Remote sensing image vehicle detection method based on super-resolution and multi-modal fusion

Through super-resolution and multimodal fusion technology, the accuracy of vehicle detection of satellite remote sensing images is improved, the problem of low resolution in severe weather and low light conditions is solved, and more accurate vehicle target detection is achieved.

CN118982800AActive Publication Date: 2024-11-19CHONGQING UNIV OF POSTS & TELECOMM

Patent Information

Application Number
CN202411128542.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-19
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

Satellite remote sensing images have low resolution in bad weather conditions, making it difficult to identify small-scale target objects, such as ground vehicles, and vehicle target detection is poor in low light and complex scenarios.

Method used

Using vehicle detection methods based on super-resolution and multimodal fusion, high-resolution images are generated through enhanced super-resolution generation adversarial networks, and feature extraction and fusion are used for multi-layer hybrid Transformer encoder and RetinaNet algorithm to generate clearer generation target characteristics.

Benefits of technology

It improves the accuracy and robustness of vehicle target detection in remote sensing images, and can effectively identify vehicle targets in severe weather and low light conditions, enhancing the performance of the detection model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982800A_ABST
    Figure CN118982800A_ABST
Patent Text Reader

Abstract

The invention discloses a remote sensing image vehicle detection method based on super-resolution and multi-modal fusion, and the method comprises the steps: carrying out the super-resolution processing of a low-resolution remote sensing image through constructing an enhanced super-resolution generative adversarial network, and generating a high-resolution image; constructing a multi-layer hybrid Transform encoder to extract features from the high-resolution image to obtain a multi-layer feature map of each modal, thereby generating a multi-scale feature map; performing feature coding and multi-modal feature fusion on the multi-scale feature map to obtain a fused feature map; constructing a classification sub-network and a regression sub-network based on a RetinaNet algorithm, and generating a detection model; inputting the fusion feature map into a detection model, and detecting the vehicle target of each anchor point and the category of the vehicle target; by introducing super-resolution generation to perform fine processing on a low-resolution remote sensing image, clearer target features are generated in combination with the advantages of different modal images, and the accuracy of remote sensing image vehicle target detection is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a remote sensing image vehicle detection method based on super-resolution and multi-modal fusion. Background Art

[0002] Remote sensing images have made significant improvements in imaging quality, scene diversity, and the number of covered targets. Satellite remote sensing images have the advantages of wide coverage and short acquisition cycle, and are widely used in a variety of application scenarios including building areas, roads, parking lots, and various target types such as vehicles, ships, and bridges. In addition, remote sensing images also play a vital role in environmental monitoring, disaster assessment, resource management, and urban planning. For example, in environmental monitoring, remote sensing images can be used to detect changes in forest cover, monitor water pollution, and assess the impact of climate change. In disaster assessment, remote sensing technology can quickly provide high-resolution images of disaster-stricken areas, helping rescuers assess the impact of disasters and develop rescue plans. In resource management, remote sensing images are used for agricultural monitoring, mineral resource exploration, and fishery resource management, thereby improving resource utilization efficiency and sustainable development capabilities. In urban planning, remote sensing technology provides reliable data support for urban expansion, traffic flow analysis, and infrastructure construction, helping planners optimize urban layout and improve urban management.

[0003] However, due to the limited spatial resolution of satellites, especially the low resolution of images obtained by high-altitude orbit satellites, it is difficult to identify small-scale target objects, such as ground vehicles, which limits the effectiveness of satellite remote sensing images in some refined applications. In addition, the color and texture features of vehicles are not rich, the types are diverse, the distribution scenes are complex and changeable, and there are many background interferences, which makes vehicle target detection difficult. Especially under low-light conditions, the color and contour information of vehicle targets in visible light images are seriously lost, the contrast between vehicles and backgrounds decreases, and the contrast is small, which makes the vehicle features extracted by the detection model insufficient, resulting in frequent missed detections. Under severe weather conditions, it is difficult for both visible light and infrared light in satellite remote sensing images to obtain effective features of vehicle targets. The imaging effect of visible light remote sensing images will be significantly reduced under conditions such as cloudy days, haze, rain and snow. In turn, visible light images provide intuitive and easy-to-understand rich color information, which is suitable for detail recognition and scene understanding; infrared images perform well under low-light conditions, have temperature sensitivity and all-weather working capabilities, and are suitable for target detection in complex environments; radar images are not affected by severe weather and can penetrate obstacles such as clouds and smoke to provide stable target information. By combining the advantages of the three types of images, more efficient and accurate target detection can be achieved in diverse and complex scenes. Therefore, developing a remote sensing image vehicle detection method based on super-resolution and multimodal fusion has important research significance and application value for improving vehicle target detection in low light and bad weather conditions. Summary of the invention

[0004] The technical problem to be solved by the present invention is that under severe weather conditions, the resolution of satellite remote sensing images is low and the detection accuracy is low. The purpose is to provide a remote sensing image vehicle detection method based on super-resolution and multimodal fusion, which introduces super-resolution generation to refine the low-resolution remote sensing images, and utilizes multimodal feature fusion technology to combine the advantages of different modal images to generate clearer target features, thereby improving the accuracy of remote sensing image vehicle target detection.

[0005] The present invention is achieved through the following technical solutions:

[0006] The present invention provides a remote sensing image vehicle detection method based on super-resolution and multimodal fusion, comprising the following specific steps:

[0007] Acquire remote sensing images and preprocess the remote sensing images;

[0008] Construct an enhanced super-resolution generative adversarial network to perform super-resolution processing on low-resolution remote sensing images and generate high-resolution images;

[0009] Construct a multi-layer hybrid Transformer encoder to extract features from high-resolution images and obtain multi-layer feature maps for each modality;

[0010] Generate multi-scale feature maps based on the multi-layer feature maps of each modality;

[0011] Perform feature encoding and multi-modal feature fusion on the multi-scale feature map to obtain a fused feature map;

[0012] Build a classification subnetwork and a regression subnetwork based on the RetinaNet algorithm to generate a detection model;

[0013] The fused feature map is input into the detection model to detect the vehicle target and its category of each anchor point.

[0014] Furthermore, the remote sensing image is preprocessed, and the specific steps include:

[0015] Acquire low-resolution remote sensing images, including visible light images, infrared images, and radar images;

[0016] De-noise and normalize remote sensing images;

[0017] The resolutions of remote sensing images acquired in multiple modes are unified to obtain preprocessed remote sensing images.

[0018] Furthermore, the enhanced super-resolution generative adversarial network is constructed to perform super-resolution processing on low-resolution remote sensing images, and the specific steps include:

[0019] Construct an enhanced super-resolution generative adversarial network, use the generator to generate high-resolution images, use the discriminator to distinguish the generated high-resolution images from the real high-resolution images, and obtain the authenticity probability of the two images;

[0020] Determine whether the authenticity probability of the two images is less than the set threshold;

[0021] If not, the generator and discriminator are trained to optimize the generated high-resolution image until the authenticity probability of the two images is less than the set threshold, and a high-resolution image is output.

[0022] Furthermore, the adversarial training of the generator and the discriminator includes the following specific steps:

[0023] Constructing a loss function, wherein the loss function includes adversarial loss, perceptual loss, and content loss;

[0024] The generator and the discriminator are trained alternately based on the loss function. During the training process:

[0025] The generator improves the quality of high-resolution images to deceive the discriminator;

[0026] The discriminator improves the ability to distinguish between generated high-resolution images and real high-resolution images.

[0027] Furthermore, the step of obtaining a multi-layer feature map of each modality includes:

[0028] Use a multi-layer hybrid Transformer encoder to extract multi-layer features of single-modal images in high-resolution images to obtain global semantic information and local detail features;

[0029] A multi-layer feature map of each modality is obtained by merging overlapping patches of global semantic information and local detail features.

[0030] Furthermore, the multi-scale feature map is generated based on the multi-layer feature map of each modality, and the specific steps include:

[0031] The highest level multi-layer feature map is used as the top feature map of the feature pyramid FPN;

[0032] The feature pyramid FPN is used to upsample and laterally connect the feature maps from top to bottom:

[0033] Get the pyramid feature map of the current level and upsample it to obtain a feature map with the same resolution as the feature map of the next level;

[0034] The upsampled feature map is convolved with the corresponding underlying feature map and then added to generate a new pyramid feature map.

[0035] A second convolution operation is performed on the generated new pyramid feature map to obtain a multi-scale feature map.

[0036] Furthermore, the step of performing feature encoding and multi-modal feature fusion on the multi-scale feature map comprises:

[0037] Construct a mixed feedforward network Mix-FFN to encode the multi-scale feature map and obtain the encoding vectors Q, K and V

[0038] Construct a cross attention network and calculate K T V gets the global vector G;

[0039] Multiply the global vector G of one branch of the cross-attention network with Q of the other branch to obtain the cross-attention feature map of cross-modal interaction;

[0040] Perform residual connection between the cross-attention feature map and the multi-scale feature map of each branch to obtain the merged feature;

[0041] Construct a residual network to reduce the dimension of the merged features and obtain a feature map that matches the input size of the decoder;

[0042] The feature maps processed by the residual network dimensionality reduction are spliced ​​through the fully connected MLP decoder to generate a fused feature map.

[0043] Furthermore, a classification subnetwork is constructed. The specific steps include:

[0044] Based on the independent convolutional layer in RetinaNet, a classification subnetwork is obtained. The classification convolutional layer is used to process the input feature map, extract features for classification, and predict whether each anchor point contains a target and belongs to the target type based on the classification features.

[0045] Furthermore, the regression sub-network is constructed. The specific steps include:

[0046] Based on RetinaNet, a regression subnetwork is obtained, and the input feature map is processed using a convolutional layer to extract regression features;

[0047] Predict the position adjustment value of the bounding box associated with each anchor point based on the regressed features.

[0048] Furthermore, before detecting the vehicle target and its category, the detection model is also optimized:

[0049] Constructing a loss function, wherein the loss function includes a focal loss function and a smooth L1 loss function, wherein the focal loss function is used to solve the imbalance of positive and negative samples, and the smooth L1 loss function is used for regression prediction;

[0050] The Adam algorithm is used for iterative training, and the gradient information is used to update the weights of the detection model, optimize the detection model parameters, and minimize the loss function until the model parameters reach the set threshold, and the optimized detection model is obtained for detection.

[0051] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0052] By introducing super-resolution generation to refine low-resolution remote sensing images, and using multi-modal feature fusion technology to combine the advantages of different modal images to generate clearer target features, the accuracy of vehicle target detection in remote sensing images is improved;

[0053] By using the generator from the low-resolution image I LR Generate high-resolution images The generated high-resolution image is then distinguished from the real high-resolution image by the discriminator. The output of the discriminator is the authenticity probability of the two images, ranging from [0,1]. Through adversarial training, the generator continuously optimizes the quality of the generated image so that the discriminator cannot distinguish between the generated image and the real image, improving the resolution and details of the image, and providing a clearer and more accurate image foundation for subsequent feature extraction and fusion;

[0054] The MiT encoder uses an overlapping patch strategy to generate multi-layer feature maps with different resolutions and numbers of channels, and captures global semantics and local details through a multi-head self-attention mechanism. Next, the Feature Pyramid Network (FPN) processes the extracted feature maps, enhances the model's ability to detect multi-scale targets through top-down paths and lateral connections, and generates a smooth series of multi-scale feature maps. Subsequently, the output of the multi-head self-attention module is fed into the Mixed Feedforward Network (Mix-FFN) for feature encoding. Mix-FFN enhances the model's perception of location information and reduces information leakage by combining convolutional layers and multi-layer perceptrons (MLP). Finally, in the multimodal feature fusion stage, the cross-attention mechanism is used to complementarily fuse feature maps from different modalities, and the merged features are processed through the residual network to generate the final fused feature map. The multimodal fusion features are effectively extracted and fused, and rich feature information is provided for subsequent vehicle target detection;

[0055] The classification subnetwork processes the fused feature map through a convolutional layer to predict whether each anchor point contains a vehicle target and its category. Specifically, the classification convolution layer is applied to the fused feature map, and the classification score is generated using the sigmoid function. The regression subnetwork focuses on predicting the bounding box adjustment value of each anchor point. The regression convolution layer processes the fused feature map and outputs the predicted bounding box coordinates. To optimize the target detection performance, RetinaNet combines the focal loss and smooth L1 loss functions. The focal loss is used to deal with the imbalance problem of positive and negative samples, and the classification accuracy is improved by adjusting the hyperparameters; the smooth L1 loss is used for regression prediction to reduce the coordinate error. Finally, the weighted sum of the classification loss and regression loss constitutes the total loss of the model, and the weight parameter is used to balance the classification and regression tasks. The complementarity of multimodal data is effectively utilized to improve the accuracy and robustness of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0057] Figure 1 A schematic diagram of a method flow in an embodiment of the present invention;

[0058] Figure 2The overall structure of the multimodal feature fusion network based on super-resolution in the embodiment of the present invention;

[0059] Figure 3 It is a cross-attention multimodal feature fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0061] As a possible implementation method, this embodiment provides a remote sensing image vehicle detection method based on super-resolution and multi-modal fusion, such as Figure 1 As shown in the figure, remote sensing images are acquired and preprocessed; an enhanced super-resolution generative adversarial network is constructed to perform super-resolution processing on low-resolution remote sensing images to generate high-resolution images; a multi-layer hybrid Transformer encoder is constructed to extract features from high-resolution images to obtain multi-layer feature maps of each modality; a multi-scale feature map is generated based on the multi-layer feature map of each modality; feature encoding and multi-modal feature fusion are performed on the multi-scale feature map to obtain a fused feature map; a classification subnetwork and a regression subnetwork are constructed based on the RetinaNet algorithm to generate a detection model; the fused feature map is input into the detection model to detect the vehicle target and its category of each anchor point. By introducing super-resolution generation to refine the low-resolution remote sensing images, and using multi-modal feature fusion technology, the advantages of different modal images are combined to generate clearer target features, thereby improving the accuracy of vehicle target detection in remote sensing images.

[0062] Specifically,

[0063] S1: Image super-resolution processing. Using the Enhanced Super-Resolution Generative Adversarial Network (ESRGAN), low-resolution remote sensing images (acquired remote sensing images include visible light images, infrared images, and radar images) are processed by super-resolution to generate high-resolution images.

[0064] The specific process includes:

[0065] First, the generator G is used to generate low-resolution remote sensing images I LR Generate high-resolution images The generated high-resolution image is then distinguished from the real high-resolution image by the discriminator D. The output of the discriminator D is the authenticity probability of the two images, which can be set in the range of [0,1];

[0066] Then, a loss function is defined, and adversarial training is performed on the generator G and the discriminator D based on the loss function. During the adversarial training, the generator G continuously optimizes the quality of the generated high-resolution images. After each optimization, the discriminator D distinguishes the generated images from the real images again until the discriminator D cannot distinguish the generated images from the real images. This process improves the resolution and details of the image, providing a clearer and more accurate image foundation for subsequent feature extraction and fusion.

[0067] Further, step S1 specifically includes:

[0068] S11: Image preprocessing. First, three low-resolution modal remote sensing images are received: visible light images, infrared images, and radar images. Each image needs to go through a preprocessing step before super-resolution processing. The preprocessing includes: denoising and normalizing the image, and then unifying the resolution of the three modal images to the same size to facilitate subsequent super-resolution processing.

[0069] S12: Super-resolution processing. Obtain pre-processed low-resolution remote sensing images of three modalities, including visible light image I RGB,LR 、Infrared image I NIR,LR and radar image I RAD,LR The resolution of these images is H LR ×W LR To facilitate the subsequent steps, I LR To represent the low-resolution image for unified processing. Use the super-resolution network ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) to perform super-resolution enhancement on the low-resolution image. First, use the generator G to generate a high-resolution image. Then use the discriminator D to distinguish the generated high-resolution image from the real high-resolution image. LR and generate images The outputs of the discriminator are D(I LR )→[0,1] and

[0070] S13: Define the loss function.

[0071] Among them, the loss functions include adversarial loss, perceptual loss, and content loss;

[0072] The adversarial loss is:

[0073]

[0074] The perceptual loss is:

[0075]

[0076] The content loss is:

[0077]

[0078] Here, φ represents the feature map extracted by a specific layer of the VGG network.

[0079] S14: Optimization process.

[0080] Optimize the discriminator D:

[0081]

[0082] Optimize generator G:

[0083]

[0084] The generator D and the discriminator G are trained alternately through optimization. The generator D continuously improves the image quality to deceive the discriminator G, and the discriminator G continuously improves its ability to distinguish between real and fake images. After training, the generator G can be directly used to transform the new low-resolution image I LR Convert to high resolution image I HR , the resolution is determined by H LR ×W LR Convert to H HR ×W HR .

[0085] S2: Multimodal feature fusion. First, features are extracted from the high-resolution image after super-resolution processing through a multi-layer mixed transformer (MiT) encoder. The MiT encoder uses an overlapping patch strategy to generate multi-layer feature maps with different resolutions and number of channels, and captures global semantics and local details through a multi-head self-attention mechanism. Next, the feature pyramid network (FPN) processes the extracted feature maps, enhances the model's detection ability for multi-scale objects through top-down paths and lateral connections, and generates a smooth series of multi-scale feature maps. Subsequently, the output of the multi-head self-attention module is fed into a mixed feedforward network (Mix-FFN) for feature encoding. Mix-FFN enhances the model's perception of position information and reduces information leakage by combining convolutional layers and multi-layer perceptrons (MLP). Finally, in the multimodal feature fusion stage, the cross-attention mechanism is used to complementarily fuse feature maps from different modalities, and the merged features are processed through a residual network to generate the final fused feature map. These steps not only effectively extract and fuse multimodal features, but also provide rich feature information for subsequent vehicle target detection.

[0086] Further, if Figure 2 and Figure 3 As shown, the above step S2 specifically includes:

[0087] S21: Feature extraction. First, the MiT encoder is used to extract the multi-scale features of the unimodal image. The MiT encoder generates a multi-level feature map through a multi-layer Transformer feature extractor. These feature maps are similar to the feature maps of the Convolutional Neural Network (CNN). The encoder can extract global semantic information and local detail features from the input image. Specifically, taking the high-resolution visible light image after super-resolution processing as an example, its resolution is H LR ×W LR , with 3 channels, is used as the input image. The MiT encoder constructs a multi-layer feature map f output by the i-th MiT encoder by merging overlapping patches. i .f i The size changes are shown below.

[0088]

[0089] Among them, i is the serial number of MiT encoder, C i is the i-th feature map f iThe number of channels. In order to ensure local continuity between patches and reduce artifact effects, a patch embedding strategy with overlap is adopted. The overlapping area between adjacent patches not only expands the perception range, but also retains the local information of each patch. Here, M, S, and P are defined to represent the patch size, the span between adjacent patches, and the padding size, respectively. Each layer of Transformer feature extraction module contains a multi-head self-attention mechanism. The encoding output of each head includes a query matrix Q, a key matrix K, and a value matrix V of dimension N×C, where N=H×W is the sequence length. The calculation process of the self-attention mechanism is as follows:

[0090]

[0091] S22: Obtain multi-scale feature maps. After the feature extraction of each modality, FPN is used to obtain multi-scale feature maps. FPN increases the contextual information of deep features through a top-down path, enhances the model's detection ability for multi-scale targets by combining features at different levels, and combines feature maps of different resolutions through lateral connections. For the extracted feature maps f1, f2, f3, f4. The resolution and number of channels of each feature map are different. As the level deepens, the resolution decreases, while the abstractness of the features increases. The highest level feature map f4 is selected as the top feature map of the pyramid, denoted as F4. For each feature map of a lower level, upsampling and lateral connection processing are performed. Upsampling is to upsample the pyramid feature map of the current level so that its resolution is the same as that of the feature map of the next level. Horizontal connection is to add the upsampled feature map to the corresponding underlying feature map after adjusting the number of channels through 1×1 convolution to generate a new pyramid feature map. The mathematical expression is:

[0092] F i =Conv 1×1 (f i )+UpSample(F i+1 ) (8)

[0093] Among them C i is the feature map from the previous layer, F i It is the output feature map of FPN, and UpSample is the upsampling operation.

[0094] After each connection, the feature map F i A 3×3 convolution operation is performed on the image to reduce the aliasing effect and ensure the smoothness and consistency of the feature map. Finally, a series of multi-scale feature maps F2, F3, and F4 are output.

[0095] S23: Feature encoding. The output of the multi-head self-attention module will be sent to the Mix FeedForword Network (Mix-FFN) for feature encoding. Mix-FFN adds a 3×3 convolutional layer directly before the MLP layer to enhance the model's perception of position information and reduce information leakage caused by zero padding. Subsequently, the GELU activation function is used to perform nonlinear operations on the features and sent to the MLP multi-layer perceptron to complete the output. The process is shown as follows:

[0096]

[0097] Among them, x in Self-attention module extracts features.

[0098] S24: Multimodal feature fusion. In the cross-attention network ( Figure 3 )middle, and They represent the multi-scale visible light feature map, infrared feature map, and radar feature map output by the feature pyramid network. The size of the feature map is determined by the stage of the encoder. First, and Construct a residual path so that the model can learn residual information and reduce the potential impact of gradient vanishing; then, use a multi-head self-attention mechanism to build two-way cross paths. The traditional self-attention score calculation is based on vector dot multiplication. The input vector is encoded to obtain Q, K, and V, and then the correlation weights used for weighting are obtained by calculating the similarity between Q and K to generate output features with attention to the input vector. However, this calculation method takes up a lot of memory. In order to reduce the amount of self-attention calculation, an efficient attention mechanism was proposed. Its main idea is to change the calculation order of the correlation matrix, that is, no longer based on QK T Instead of calculating the attention score map, K T The size of V is The global vector G of is multiplied by Q to obtain the attention score map. Based on this, for branches from different modalities, the present invention multiplies the G of one branch by the Q of another branch to construct a cross-modal interactive cross-attention feature map The process can be expressed as follows:

[0099]

[0100] Cross-attention feature map for multimodal information complementation and The merged features are concatenated with the input of each branch through the residual connection to obtain the merged features. Then, the merged features are reduced in dimension by constructing a residual network to match the input size of the decoder. The residual network mainly includes 1×1 convolution operation, 3×3 convolution operation, ReLU activation function and batch normalization. To further reduce the number of parameters, the depthwise separable convolution layer is used in the residual module to replace the conventional 3×3 convolution. After the residual network, the size of the input after concatenation is H LR ×W LR ×3C will eventually generate a merged feature of size H LR ×W LR ×C fusion feature map Afterwards, it is passed to the fully connected MLP decoder for multi-level fusion and splicing for vehicle target detection tasks.

[0101] S3: Vehicle target detection. The classification subnetwork and regression subnetwork in the RetinaNet algorithm are used. The classification subnetwork processes the fused feature map through a convolutional layer to predict whether each anchor point contains a vehicle target and its category. Specifically, the classification convolution layer is applied to the fused feature map, and the classification score is generated using the sigmoid function. The regression subnetwork focuses on predicting the bounding box adjustment value of each anchor point. The regression convolution layer processes the fused feature map and outputs the predicted bounding box coordinates. To optimize the target detection performance, RetinaNet combines the focal loss and smooth L1 loss functions. The focal loss is used to deal with the imbalance problem of positive and negative samples, and improves the classification accuracy by adjusting the hyperparameters; the smooth L1 loss is used for regression prediction to reduce the coordinate error. Finally, the weighted sum of the classification loss and regression loss constitutes the total loss of the model, and the weight parameter is used to balance the classification and regression tasks. This design effectively utilizes the complementarity of multimodal data and improves the accuracy and robustness of target detection.

[0102] Furthermore, the above step S3 specifically includes:

[0103] S31: Build a classification subnetwork. The classification subnetwork is an independent convolutional layer in RetinaNet, which is used to predict whether each anchor point contains a target (vehicle) and which category it belongs to. The classification convolutional layer is used to process the input feature map to extract features for classification. Specifically, for the fused feature map output in S2 The classification convolutional layer is,

[0104]

[0105] Among them C i is the classification score on the feature map of the i-th layer, σ is the sigmoid function, Conv cls It is a convolutional layer designed for classification, and the number of its output channels is equal to the number of anchor points multiplied by the number of categories.

[0106] S32: Construct a regression subnetwork. The regression head is used to predict the position adjustment value of the bounding box associated with each anchor point. The convolution layer is applied to process the input feature map to extract features for regression. Specifically, for the fused feature map output in S2 The regression convolution layer is:

[0107]

[0108] Among them B i is the bounding box regression output on the i-th layer feature map. Conv reg It is a convolutional layer designed for regression, and its output channel number is the number of anchor points multiplied by 4 (4 coordinate values ​​for each bounding box).

[0109] S33: Loss function for target detection. RetinaNet uses focal loss and smooth L1 loss function for optimization: focal loss is used to solve the imbalance problem of positive and negative samples, as shown in the following formula.

[0110] L cls =-α t (1-p t ) γ log(p t )(18)

[0111] where p t is the probability of predicting the positive class, and α and γ are hyperparameters for adjusting the importance of positive and negative samples.

[0112] The regression head uses Smooth L1 Loss, which is specifically used for regression prediction:

[0113] L reg =smooth L1 (b pred -b gt ) (19)

[0114] where b pred are the predicted bounding box coordinates, b gt are the true bounding box coordinates.

[0115] The final loss is the weighted sum of these two parts, and the total loss L of the model is expressed as:

[0116] L=λ cls ·L cls +λ reg ·L reg (20)

[0117] λ cls is the weight of the classification loss.reg is the weight of the regression loss. This method can not only make full use of the complementarity of multimodal data, but also improve the accuracy and robustness of detection through focal loss.

[0118] S34: Optimization and training. The Adam algorithm is used to gradually optimize the model parameters through multiple iterations of training to minimize the loss function. The Adam optimization algorithm uses gradient information to update the weights of the model, so that the detection model continuously improves its performance during the training process until the performance of the detection model reaches the expected goal. After the training is completed, the input fusion features can be used for target detection.

[0119] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A remote sensing image vehicle detection method based on super-resolution and multimodal fusion, characterized in that: The specific steps include: Acquire remote sensing images and preprocess the remote sensing images; Construct an enhanced super-resolution generative adversarial network to perform super-resolution processing on low-resolution remote sensing images and generate high-resolution images; Construct a multi-layer hybrid Transformer encoder to extract features from high-resolution images and obtain multi-layer feature maps for each modality; Generate multi-scale feature maps based on the multi-layer feature maps of each modality; Perform feature encoding and multi-modal feature fusion on the multi-scale feature map to obtain a fused feature map; Build a classification subnetwork and a regression subnetwork based on the RetinaNet algorithm to generate a detection model; The fused feature map is input into the detection model to detect the vehicle target and its category of each anchor point.

2. The remote sensing image vehicle detection method based on super-resolution and multimodal fusion according to claim 1 is characterized in that: The remote sensing image is preprocessed, and the specific steps include: Acquire low-resolution remote sensing images, including visible light images, infrared images, and radar images; De-noise and normalize remote sensing images; The resolutions of remote sensing images acquired in multiple modes are unified to obtain preprocessed remote sensing images.

3. The remote sensing image vehicle detection method based on super-resolution and multimodal fusion according to claim 2 is characterized in that: The enhanced super-resolution generative adversarial network is constructed to perform super-resolution processing on low-resolution remote sensing images, and the specific steps include: Construct an enhanced super-resolution generative adversarial network, use the generator to generate high-resolution images, use the discriminator to distinguish the generated high-resolution images from the real high-resolution images, and obtain the authenticity probability of the two images; Determine whether the authenticity probability of the two images is less than the set threshold; If not, the generator and discriminator are trained to optimize the generated high-resolution image until the authenticity probability of the two images is less than the set threshold, and a high-resolution image is output.

4. The remote sensing image vehicle detection method based on super-resolution and multimodal fusion according to claim 3 is characterized in that: The adversarial training of the generator and the discriminator includes the following specific steps: Constructing a loss function, wherein the loss function includes adversarial loss, perceptual loss, and content loss; The generator and the discriminator are trained alternately based on the loss function. During the training process: The generator improves the quality of high-resolution images to deceive the discriminator; The discriminator improves the ability to distinguish between generated high-resolution images and real high-resolution images.

5. The remote sensing image vehicle detection method based on super-resolution and multimodal fusion according to claim 1, characterized in that: The step of obtaining the multi-layer feature map of each modality includes: Use a multi-layer hybrid Transformer encoder to extract multi-layer features of single-modal images in high-resolution images to obtain global semantic information and local detail features; A multi-layer feature map of each modality is obtained by merging overlapping patches of global semantic information and local detail features.

6. The remote sensing image vehicle detection method based on super-resolution and multi-modal fusion according to claim 1 is characterized in that: The step of generating a multi-scale feature map based on the multi-layer feature map of each modality comprises: The highest level multi-layer feature map is used as the top feature map of the feature pyramid FPN; The feature pyramid FPN is used to upsample and laterally connect the feature maps from top to bottom: Get the pyramid feature map of the current level and upsample it to obtain a feature map with the same resolution as the feature map of the next level; The upsampled feature map is convolved with the corresponding underlying feature map and then added to generate a new pyramid feature map. A second convolution operation is performed on the generated new pyramid feature map to obtain a multi-scale feature map.

7. The remote sensing image vehicle detection method based on super-resolution and multimodal fusion according to claim 1 is characterized in that: The step of performing feature encoding and multi-modal feature fusion on the multi-scale feature map specifically includes: Construct a mixed feedforward network Mix-FFN to encode the multi-scale feature map and obtain the encoding vectors Q, K and V Construct a cross attention network and calculate K T V gets the global vector G; Multiply the global vector G of one branch of the cross-attention network with Q of the other branch to obtain the cross-attention feature map of cross-modal interaction; Perform residual connection between the cross-attention feature map and the multi-scale feature map of each branch to obtain the merged feature; Construct a residual network to reduce the dimension of the merged features and obtain a feature map that matches the input size of the decoder; The feature maps processed by the residual network dimensionality reduction are spliced ​​through the fully connected MLP decoder to generate a fused feature map.

8. The remote sensing image vehicle detection method based on super-resolution and multimodal fusion according to claim 1, characterized in that: Construct a classification subnetwork. The specific steps include: Based on the independent convolutional layer in RetinaNet, a classification subnetwork is obtained. The classification convolutional layer is used to process the input feature map, extract features for classification, and predict whether each anchor point contains a target and belongs to the target type based on the classification features.

9. The remote sensing image vehicle detection method based on super-resolution and multimodal fusion according to claim 1, characterized in that: Construct a regression subnetwork. The specific steps include: Based on RetinaNet, a regression subnetwork is obtained, and the input feature map is processed using a convolutional layer to extract regression features; Predict the position adjustment value of the bounding box associated with each anchor point based on the regressed features.

10. The remote sensing image vehicle detection method based on super-resolution and multi-modal fusion according to claim 1, characterized in that: Before detecting vehicle targets and their categories, the detection model is also optimized: Constructing a loss function, wherein the loss function includes a focal loss function and a smooth L1 loss function, wherein the focal loss function is used to solve the imbalance of positive and negative samples, and the smooth L1 loss function is used for regression prediction; The Adam algorithm is used for iterative training, and the gradient information is used to update the weights of the detection model, optimize the detection model parameters, and minimize the loss function until the model parameters reach the set threshold, and the optimized detection model is obtained for detection.

Citation Information

Patent Citations

  • Optical remote sensing image target detection method based on super-resolution feature aggregation

    CN115909088A

  • Super-resolution reconstruction method for remote sensing image vehicle target detection

    CN116485652A

  • Multi-modal target detection method and device based on feature enhancement and collaborative interaction

    CN117423007A

Cited By

  • Vehicle target detection method based on multi-modal dual-stage refined binding relation

    CN119625657A

  • Foggy day image vehicle detection method based on cyclic generative adversarial network and interest domain enhancement

    CN119672651A

  • Remote sensing target detection method and system for low-visibility image

    CN120747487A

  • Remote sensing target detection method

    CN121482606A

  • Target detection method for event and image fusion perception

    CN121811158A