A Vehicle Detection Method Based on Super-Resolution and Multimodal Fusion Remote Sensing Images

Abstract

This invention discloses a vehicle detection method for remote sensing images based on super-resolution and multimodal fusion. It involves constructing an enhanced super-resolution generative adversarial network (GAN) to perform super-resolution processing on low-resolution remote sensing images, generating high-resolution images; constructing a multi-layer hybrid Transformer encoder to extract features from the high-resolution images, obtaining multi-layer feature maps for each modality, thus generating multi-scale feature maps; performing feature encoding and multimodal feature fusion on the multi-scale feature maps to obtain fused feature maps; constructing classification and regression sub-networks based on the RetinaNet algorithm to generate a detection model; inputting the fused feature maps into the detection model to detect vehicle targets and their categories at each anchor point; and improving the accuracy of vehicle target detection in remote sensing images by introducing super-resolution generation to refine the processing of low-resolution remote sensing images and combining the advantages of different modalities to generate clearer target features.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and specifically to a method for vehicle detection in remote sensing images based on super-resolution and multimodal fusion. Background Technology

[0002] Remote sensing imagery has seen significant improvements in imaging quality, scene diversity, and the number of targets covered. Based on the advantages of satellite remote sensing imagery, such as wide coverage and short acquisition cycles, it is widely used in various application scenarios, including built-up areas, roads, and parking lots, and for various target types, such as vehicles, ships, and bridges. Furthermore, remote sensing imagery plays a crucial role in environmental monitoring, disaster assessment, resource management, and urban planning. For example, in environmental monitoring, remote sensing images can be used to detect changes in forest cover, monitor water pollution, and assess the impacts of climate change. In disaster assessment, remote sensing technology can quickly provide high-resolution images of affected areas, helping rescue workers assess disaster impacts and develop relief plans. In resource management, remote sensing imagery is used for agricultural monitoring, mineral resource exploration, and fisheries resource management, thereby improving resource utilization efficiency and sustainable development capabilities. In urban planning, remote sensing technology provides reliable data support for urban expansion, traffic flow analysis, and infrastructure construction, helping planners optimize urban layout and improve urban management.

[0003] However, due to the limited spatial resolution of satellites, especially the low resolution of images acquired by high-orbit satellites, it is difficult to identify small-scale targets, such as ground vehicles, which limits the effectiveness of satellite remote sensing images in certain refined applications. Furthermore, vehicles have limited color and texture features, diverse types, and complex and variable distribution scenarios, coupled with significant background interference, making vehicle target detection challenging. Particularly under low-light conditions, the color and contour information of vehicle targets in visible light images is severely lost, the contrast between the vehicle and the background decreases, and the low contrast results in insufficient vehicle features extracted by the detection model, leading to frequent missed detections. Under adverse weather conditions, both visible and infrared light from satellite remote sensing images are insufficient to obtain effective features of vehicle targets. Visible light remote sensing images show a significant decrease in imaging quality under conditions such as overcast skies, fog, rain, and snow. Conversely, visible light images provide rich and easily understandable color information, suitable for detail recognition and scene understanding; infrared images perform well under low-light conditions, have temperature sensitivity and all-weather capability, and are suitable for target detection in complex environments; radar images are unaffected by adverse weather conditions, can penetrate clouds, smoke, and other obstacles, and provide stable target information. By combining the advantages of three types of images, more efficient and accurate target detection can be achieved in diverse and complex scenes. Therefore, developing a vehicle detection method based on super-resolution and multimodal fusion remote sensing images is of significant research importance and application value for improving vehicle target detection performance under low light and adverse weather conditions. Summary of the Invention

[0004] The technical problem this invention aims to solve is that satellite remote sensing images have low resolution and low detection accuracy under adverse weather conditions. The goal is to provide a vehicle detection method based on super-resolution and multimodal fusion of remote sensing images. By introducing super-resolution generation to refine the low-resolution remote sensing images, and by utilizing multimodal feature fusion technology to combine the advantages of different modal images to generate clearer target features, the accuracy of vehicle target detection in remote sensing images is improved.

[0005] This invention is achieved through the following technical solution:

[0006] This invention provides a vehicle detection method based on super-resolution and multimodal fusion of remote sensing images, comprising the following specific steps:

[0007] Acquire remote sensing images and preprocess them;

[0008] An enhanced super-resolution generative adversarial network is constructed to perform super-resolution processing on low-resolution remote sensing images and generate high-resolution images.

[0009] A multi-layer hybrid Transformer encoder is constructed to extract features from high-resolution images, resulting in multi-layer feature maps for each modality;

[0010] Generate multi-scale feature maps based on multi-layer feature maps for each modality;

[0011] Feature encoding and multimodal feature fusion are performed on the multi-scale feature map to obtain a fused feature map;

[0012] A classification subnetwork and a regression subnetwork are constructed based on the RetinaNet algorithm to generate a detection model;

[0013] The fused feature map is input into the detection model to detect vehicle targets and their categories at each anchor point.

[0014] Furthermore, the preprocessing of the remote sensing images includes the following specific steps:

[0015] Acquire low-resolution modal remote sensing images, including visible light images, infrared images, and radar images;

[0016] The remote sensing images are subjected to noise reduction and normalization processing.

[0017] The resolution of the acquired remote sensing images of various modalities is unified to obtain the preprocessed remote sensing image.

[0018] Furthermore, the construction of the enhanced super-resolution generative adversarial network to perform super-resolution processing on low-resolution remote sensing images includes the following specific steps:

[0019] An enhanced super-resolution generative adversarial network is constructed. A generator generates high-resolution images, and a discriminator distinguishes between the generated high-resolution images and real high-resolution images to obtain the authenticity probabilities of the two images.

[0020] Determine whether the probability of two images being real is less than a set threshold;

[0021] If not, the generator and discriminator are trained to optimize the generated high-resolution image until the probability of authenticity of the two images is less than a set threshold, and then the high-resolution image is output.

[0022] Furthermore, the adversarial training of the generator and discriminator specifically includes the following steps:

[0023] Construct a loss function, which includes adversarial loss, perceptual loss, and content loss;

[0024] The generator and discriminator are trained alternately based on a loss function. During the training process:

[0025] The generator improves the quality of high-resolution images to deceive the discriminator;

[0026] The discriminator improves the ability to distinguish between generated high-resolution images and real high-resolution images.

[0027] Furthermore, the specific steps for obtaining the multi-layer feature map for each modality include:

[0028] Multi-layer hybrid Transformer encoders are used to extract multi-layer features of single-modal images in high-resolution images to obtain global semantic information and local detail features;

[0029] Multi-layer feature maps for each modality are obtained by overlapping and merging global semantic information and local detail features.

[0030] Furthermore, the specific steps for generating multi-scale feature maps based on multi-layer feature maps of each modality include:

[0031] The highest-level multi-layer feature map is used as the top-level feature map of the Feature Pyramid (FPN).

[0032] The feature map is upsampled and horizontally joined sequentially from top to bottom using the Feature Pyramid Network (FPN).

[0033] Upsample the pyramid feature map of the current level to obtain a feature map with the same resolution as the feature map of the next level.

[0034] The upsampled feature map is convolved with the corresponding low-level feature map and then added to generate a new pyramid feature map.

[0035] A second convolution operation is performed on the newly generated pyramid feature map to obtain a multi-scale feature map.

[0036] Furthermore, the specific steps for feature encoding and multimodal feature fusion of the multi-scale feature maps include:

[0037] A hybrid feedforward network (Mix-FFN) is constructed to encode features from multi-scale feature maps, yielding encoded vectors Q, K, and V.

[0038] Construct a cross-attention network and compute K T V obtains the global vector G;

[0039] Multiply the global vector G of one branch of the cross-attention network with the Q of the other branch to obtain the cross-attention feature map for cross-modal interaction;

[0040] The cross-attention feature map is residually connected with the multi-scale feature maps of their respective branches to obtain the merged feature;

[0041] A residual network is constructed to reduce the dimensionality of the merged features, resulting in a feature map of the input size to the matching decoder.

[0042] The feature maps processed by the dimensionality reduction of the residual network are stitched together using a fully connected MLP decoder to generate a fused feature map.

[0043] Furthermore, a classification subnetwork is constructed, and the specific steps include:

[0044] A classification subnetwork is obtained based on the independent convolutional layers in RetinaNet. The input feature map is processed by the classification convolutional layer to extract features for classification. Based on the classification features, it is predicted whether each anchor point contains the target and what type of target it belongs to.

[0045] Furthermore, the regression subnetwork is constructed, and the specific steps include:

[0046] A regression sub-network is obtained based on RetinaNet, and convolutional layers are used to process the input feature map to extract regression features;

[0047] Predict the position adjustment value of the bounding box associated with each anchor point based on regression features.

[0048] Furthermore, before detecting vehicle targets and their categories, the detection model is optimized.

[0049] Construct a loss function, which includes a focus loss function and a smoothed L1 loss function. The focus loss function is used to address the imbalance between positive and negative samples, and the smoothed L1 loss function is used for regression prediction.

[0050] The Adam algorithm is used for iterative training. Gradient information is used to update the weights of the detection model, optimize the detection model parameters, minimize the loss function, until the model parameters reach a set threshold, and the optimized detection model is obtained for detection.

[0051] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0052] By introducing super-resolution generation to refine low-resolution remote sensing images and utilizing multimodal feature fusion technology to combine the advantages of different modal images to generate clearer target features, the accuracy of vehicle target detection in remote sensing images is improved.

[0053] Using a generator to extract low-resolution image I LR Generate high-resolution images Then, a discriminator distinguishes the generated high-resolution image from the real high-resolution image. The output of the discriminator is the probability of the two images being real, ranging from [0, 1]. Through adversarial training, the generator continuously optimizes the quality of the generated images so that the discriminator cannot distinguish between the generated and real images, improving the resolution and detail of the images and providing a clearer and more accurate image foundation for subsequent feature extraction and fusion.

[0054] The MiT encoder uses an overlapping patching strategy to generate multi-layer feature maps with different resolutions and channel numbers, and captures global semantics and local details through a multi-head self-attention mechanism. Next, a Feature Pyramid Network (FPN) processes the extracted feature maps, enhancing the model's ability to detect multi-scale targets through top-down paths and lateral connections, generating a smooth series of multi-scale feature maps. Subsequently, the output of the multi-head self-attention module is fed into a Mix-FFN for feature encoding. The Mix-FFN enhances the model's perception of positional information and reduces information leakage by combining convolutional layers and a Multi-Layer Perceptron (MLP). Finally, in the multimodal feature fusion stage, a cross-attention mechanism is used to complementaryly fuse feature maps from different modalities, and the merged features are processed by a residual network to generate the final fused feature map. This effectively extracts and fuses multimodal features, and also provides rich feature information for subsequent vehicle target detection.

[0055] The classification sub-network processes the fused feature map through convolutional layers to predict whether each anchor point contains a vehicle target and its category. Specifically, the classification convolutional layer is applied to the fused feature map, using the sigmoid function to generate a classification score. The regression sub-network focuses on predicting the bounding box adjustment value for each anchor point; the regression convolutional layer processes the fused feature map and outputs the predicted bounding box coordinates. To optimize object detection performance, RetinaNet combines focus loss and smoothing L1 loss functions. Focus loss addresses the imbalance between positive and negative samples by adjusting hyperparameters to improve classification accuracy; smoothing L1 loss is used for regression prediction to reduce coordinate errors. Finally, the weighted sum of the classification and regression losses constitutes the model's total loss, with weight parameters used to balance the classification and regression tasks. This effectively utilizes the complementarity of multimodal data, improving the accuracy and robustness of object detection. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0057] Figure 1 This is a schematic diagram of the method flow in an embodiment of the present invention;

[0058] Figure 2This is the overall structure of the super-resolution-based multimodal feature fusion network in this embodiment of the invention;

[0059] Figure 3 This refers to the cross-attention multimodal feature fusion module in this embodiment of the invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0061] As one possible implementation, this embodiment provides a vehicle detection method based on super-resolution and multimodal fusion of remote sensing images, such as... Figure 1 As shown, remote sensing images are acquired and preprocessed. An enhanced super-resolution generative adversarial network (GAN) is constructed to perform super-resolution processing on low-resolution remote sensing images, generating high-resolution images. A multi-layer hybrid Transformer encoder is constructed to extract features from the high-resolution images, obtaining multi-layer feature maps for each modality. Multi-scale feature maps are generated based on the multi-layer feature maps for each modality. Feature encoding and multi-modal feature fusion are performed on the multi-scale feature maps to obtain fused feature maps. A classification sub-network and a regression sub-network are constructed based on the RetinaNet algorithm to generate a detection model. The fused feature maps are input into the detection model to detect vehicle targets and their categories at each anchor point. By introducing super-resolution generation to refine the processing of low-resolution remote sensing images and utilizing multi-modal feature fusion technology to combine the advantages of different modalities to generate clearer target features, the accuracy of vehicle target detection in remote sensing images is improved.

[0062] Specifically,

[0063] S1: Image super-resolution processing. An Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) is used to perform super-resolution processing on low-resolution remote sensing images (acquired remote sensing images include visible light images, infrared images, and radar images) to generate high-resolution images;

[0064] The specific process includes:

[0065] First, generator G is used to process the low-resolution remote sensing image I. LR Generate high-resolution images Then, a discriminator D distinguishes the generated high-resolution image from the real high-resolution image. The output of discriminator D is the probability of the two images being real, and its range can be set between [0,1].

[0066] Then, a loss function is defined, and the generator G and discriminator D are trained adversarially based on this loss function. During the adversarial training process, the generator G continuously optimizes the quality of the generated high-resolution images. After each optimization, the discriminator D again distinguishes between the generated images and the real images, until the discriminator D can no longer distinguish between the generated images and the real images. This process improves the resolution and detail of the images, providing a clearer and more accurate image foundation for subsequent feature extraction and fusion.

[0067] Furthermore, step S1 specifically includes:

[0068] S11: Image Preprocessing. First, three low-resolution modalities of remote sensing images are received: visible light images, infrared images, and radar images. Each image requires a preprocessing step before super-resolution processing. Preprocessing includes denoising and normalization of the images, and then unifying the resolution of the three modalities to the same size to facilitate subsequent super-resolution processing.

[0069] S12: Super-resolution processing. Acquire preprocessed low-resolution remote sensing images in three modalities, including visible light image I. RGB,LR Infrared image I NIR,LR and radar image I RAD,LR These images all have a resolution of H. LR ×W LR To facilitate subsequent steps, use I. LR To represent low-resolution images and perform unified processing, the Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) is used to perform super-resolution enhancement on low-resolution images. First, a generator G is used to generate a high-resolution image. Then, discriminator D is used to distinguish between the generated high-resolution image and the real high-resolution image. For the real image I... LR and generating images The outputs of the discriminator are D(I) LR → [0,1] and

[0070] S13: Define the loss function.

[0071] The loss functions include adversarial loss, perceptual loss, and content loss.

[0072] The losses from the confrontation are:

[0073]

[0074] The perceived loss is:

[0075]

[0076] Content loss is:

[0077]

[0078] Where φ represents the feature map extracted from a specific layer of the VGG network.

[0079] S14: Optimization process.

[0080] Optimize discriminator D:

[0081]

[0082] Optimize generator G:

[0083]

[0084] The generator D and discriminator G are trained alternately through optimization. Generator D continuously improves image quality to deceive discriminator G, while discriminator G continuously improves its ability to distinguish between real and fake images. After training, generator G can be directly used to process new low-resolution images I. LR Convert to high resolution image I HR Resolution increased by H LR ×W LR Convert to H HR ×W HR .

[0085] S2: Multimodal Feature Fusion. First, features are extracted from the high-resolution image after super-resolution processing using a multi-layered hybrid Transformer (MiT) encoder. The MiT encoder uses an overlapping patching strategy to generate multi-layered feature maps with different resolutions and channel numbers, and captures global semantics and local details through a multi-head self-attention mechanism. Next, a Feature Pyramid Network (FPN) processes the extracted feature maps, enhancing the model's ability to detect multi-scale objects through top-down paths and lateral connections, generating a smooth series of multi-scale feature maps. Subsequently, the output of the multi-head self-attention module is fed into a hybrid feedforward network (Mix-FFN) for feature encoding. Mix-FFN enhances the model's perception of location information and reduces information leakage by combining convolutional layers and a multi-layer perceptron (MLP). Finally, in the multimodal feature fusion stage, a cross-attention mechanism is used to complementaryly fuse feature maps from different modalities, and the merged features are processed by a residual network to generate the final fused feature map. These steps not only effectively extract and fuse multimodal features, but also provide rich feature information for subsequent vehicle target detection.

[0086] Furthermore, such as Figure 2 and Figure 3 As shown, step S2 specifically includes:

[0087] S21: Feature Extraction. First, a MiT encoder is used to extract multi-scale features from the single-modality image. The MiT encoder generates multi-level feature maps through a multi-layer Transformer feature extractor, similar to feature maps in a Convolutional Neural Network (CNN). This encoder is able to extract both global semantic information and local detail features from the input image. Specifically, taking a high-resolution visible light image after super-resolution processing as an example, its resolution is H... LR ×W LR With 3 channels as input, the MiT encoder constructs the multi-layer feature map f of the i-th MiT encoder output by merging overlapping patches. i f i The dimensional changes are shown below.

[0088]

[0089] Where i is the sequence number of the MiT encoder, C i It is the i-th feature map f iThe number of channels. To ensure local continuity between patches and reduce artifact effects, an overlapping patch embedding strategy is adopted. The overlapping region between adjacent patches not only expands the receptive range but also preserves the local information of each patch. Here, M, S, and P are defined as patch size, span between adjacent patches, and padding size, respectively. Each Transformer feature extraction module contains a multi-head self-attention mechanism. The encoded output of each head includes a query matrix Q of dimension N×C, a key matrix K, and a value matrix V, where N = H×W is the sequence length. The self-attention mechanism calculation process is as follows:

[0090]

[0091] S22: Obtaining Multi-Scale Feature Maps. After feature extraction for each modality, FPN is used to obtain multi-scale feature maps. FPN adds contextual information to deep features through a top-down path, enhances the model's ability to detect multi-scale targets by combining features from different levels, and combines feature maps of different resolutions through lateral connections. For the extracted feature maps f1, f2, f3, and f4, each feature map has a different resolution and number of channels. As the layer deepens, the resolution decreases while the abstraction of the features increases. The highest-level feature map f4 is selected as the top-level feature map of the pyramid, denoted as F4. For each lower-level feature map, upsampling and lateral connections are performed. Upsampling involves upsampling the current-level pyramid feature map to make its resolution the same as the feature map of the next level. Lateral connections involve adding the upsampled feature map and the corresponding lower-level feature map after adjusting the number of channels through a 1×1 convolution to generate a new pyramid feature map. Mathematically, this is represented as:

[0092] F i =Conv 1×1 (f i )+UpSample(F i+1 (8)

[0093] Where C i It is a feature map from the previous layer, F i It is the output feature map of FPN, and UpSample is the upsampling operation.

[0094] Feature map F after each connection i A 3×3 convolution operation is then performed to reduce aliasing and ensure the smoothness and consistency of the feature maps. Finally, a series of multi-scale feature maps F2, F3, and F4 are output.

[0095] S23: Feature Encoding. The output of the multi-head self-attention module is fed into a Mix FeedForward Network (Mix-FFN) for feature encoding. Mix-FFN enhances the model's perception of positional information and reduces information leakage caused by zero-padding by adding a 3×3 convolutional layer directly before the MLP layer. Subsequently, the features are non-linearly processed using the GELU activation function and then fed into the MLP multilayer perceptron to complete the output. The process is as follows:

[0096]

[0097] Where, x in Features are extracted using a self-attention module.

[0098] S24: Multimodal feature fusion. In cross-attention networks ( Figure 3 )middle, and These represent the multi-scale visible light feature map, infrared feature map, and radar feature map output by the feature pyramid network, respectively. The size of the feature map is determined by the stage of the encoder. First, and To mitigate the potential impact of gradient vanishing, residual paths are constructed to allow the model to learn residual information. Then, a multi-head self-attention mechanism is used to build pairwise cross paths. Traditional self-attention score calculation is based on vector dot product, encoding the input vector into Q, K, and V. The similarity between Q and K is then used to obtain relevance weights for weighting, generating output features with attention to the input vector. However, this method is computationally expensive. To reduce the computational cost of self-attention, an efficient attention mechanism is proposed. Its main idea is to change the order of relevance matrix calculation, i.e., no longer relying on QK. T Instead of calculating the attention score map, it is done via K. T The size of V is The global vector G is multiplied by Q to obtain the attention score map. Based on this, for branches from different modalities, this invention uses G from one branch to multiply Q from another branch, thereby constructing a cross-modal interactive cross-attention feature map. The process can be represented as follows:

[0099]

[0100] Cross-attention feature maps that complement multimodal information will be used. and The inputs from each branch, after residual connections, are concatenated to obtain merged features. Then, a residual network is constructed to reduce the dimensionality of the merged features to match the input size of the decoder. This residual network mainly includes 1×1 convolution operations, 3×3 convolution operations, ReLU activation function, and batch normalization. To further reduce the number of parameters, depthwise separable convolutional layers are used instead of regular 3×3 convolutions in the residual module. After passing through the residual network, the size of the concatenated input is H. LR ×W LR The merging feature of ×3C will ultimately generate a size of H. LR ×W LR ×C fusion feature map It is then fed into a fully connected MLP decoder for multi-level fusion and splicing, which is used for vehicle target detection tasks.

[0101] S3: Vehicle Target Detection. This uses the classification and regression sub-networks from the RetinaNet algorithm. The classification sub-network processes the fused feature map through convolutional layers to predict whether each anchor point contains a vehicle target and its category. Specifically, the classification convolutional layer is applied to the fused feature map, using the sigmoid function to generate a classification score. The regression sub-network focuses on predicting the bounding box adjustment values ​​for each anchor point. The regression convolutional layer processes the fused feature map and outputs the predicted bounding box coordinates. To optimize target detection performance, RetinaNet combines focus loss and smoothing L1 loss functions. Focus loss addresses the imbalance between positive and negative samples by adjusting hyperparameters to improve classification accuracy; smoothing L1 loss is used for regression prediction to reduce coordinate errors. Finally, the weighted sum of the classification and regression losses constitutes the model's total loss, with weight parameters used to balance the classification and regression tasks. This design effectively utilizes the complementarity of multimodal data, improving the accuracy and robustness of target detection.

[0102] Furthermore, step S3 specifically includes:

[0103] S31: Constructing the classification sub-network. In RetinaNet, the classification sub-network is a separate convolutional layer used to predict whether each anchor point contains an object (vehicle) and which class it belongs to. The classification convolutional layer processes the input feature map to extract features for classification. Specifically, for the fused feature map output in S2... The classification convolutional layer is,

[0104]

[0105] Among them, C i It is the classification score on the feature map of the i-th layer, σ is the sigmoid function, and Conv cls It is a convolutional layer designed for classification, and its output channel number is equal to the number of anchor points multiplied by the number of categories.

[0106] S32: Construct the regression sub-network. The regression head is used to predict the position adjustment value of the bounding box associated with each anchor point. Convolutional layers are applied to process the input feature map to extract features for regression. Specifically, for the fused feature map output in S2... The regressive convolutional layer is:

[0107]

[0108] Among them B i It is the bounding box regression output on the feature map of the i-th layer. reg It is a convolutional layer designed for regression, and its output channel number is the number of anchor points multiplied by 4 (4 coordinate values ​​for each bounding box).

[0109] S33: Loss function for object detection. RetinaNet uses Focal Loss and smoothed L1 loss function for optimization: Focal Loss is used to solve the problem of imbalance between positive and negative samples, as shown in the following equation.

[0110] L cls =-α t (1-p t ) γ log(p t (18)

[0111] Where p t It represents the probability of being predicted as positive, while α and γ are hyperparameters that adjust the importance of positive and negative samples.

[0112] The regression head uses Smooth L1 Loss, specifically designed for regression prediction:

[0113] L reg =smooth L1 (b pred -b gt (19)

[0114] Where b pred These are the predicted bounding box coordinates, b gt These are the actual bounding box coordinates.

[0115] The final loss is the weighted sum of these two parts, and the total loss L of the model is expressed as:

[0116] L=λ cls ·L cls +λ reg ·L reg (20)

[0117] λ cls λ represents the weights of the classification loss.reg These are the weights of the regression loss. This method not only fully utilizes the complementarity of multimodal data, but also improves the accuracy and robustness of detection through focus loss.

[0118] S34: Optimization and Training. The Adam algorithm is used for iterative training to gradually optimize the model parameters and minimize the loss function. Specifically, the Adam optimization algorithm utilizes gradient information to update the model's weights, allowing the detection model to continuously improve its performance during training until it reaches the expected performance target. After training, object detection can be performed on the input fused features.

[0119] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A vehicle detection method based on super-resolution and multimodal fusion remote sensing images, characterized in that, The specific steps include the following: Acquire remote sensing images and preprocess them. The remote sensing images include visible light images, infrared images, and radar images. An enhanced super-resolution generative adversarial network is constructed to perform super-resolution processing on low-resolution remote sensing images and generate high-resolution images. A multi-layer hybrid Transformer encoder is constructed to extract features from high-resolution images, resulting in multi-layer feature maps for each modality; Generate multi-scale feature maps based on multi-layer feature maps for each modality; The multi-scale feature map is subjected to feature encoding and multi-modal feature fusion to obtain a fused feature map. The specific steps include: A hybrid feedforward network (Mix-FFN) is constructed to encode features from multi-scale feature maps, resulting in encoded vectors. , and ; Construct a cross-attention network and compute... Obtain the global vector ; The global vector of one branch of the cross-attention network With another branch Multiply the features to obtain the cross-attention feature map for cross-modal interaction; The cross-attention feature map is residually connected with the multi-scale feature maps of their respective branches to obtain the merged feature; A residual network is constructed to reduce the dimensionality of the merged features, resulting in a feature map of the input size to the matching decoder. The feature maps processed by the dimensionality reduction of the residual network are stitched together using a fully connected MLP decoder to generate a fused feature map; A classification subnetwork and a regression subnetwork are constructed based on the RetinaNet algorithm to generate a detection model; The fused feature map is input into the detection model to detect vehicle targets and their categories at each anchor point.

2. The vehicle detection method based on super-resolution and multimodal fusion of remote sensing images according to claim 1, characterized in that, The preprocessing of remote sensing images includes the following steps: Acquire remote sensing images of low-resolution modalities; The remote sensing images are subjected to noise reduction and normalization processing. The resolution of the acquired remote sensing images of various modalities is unified to obtain the preprocessed remote sensing image.

3. The vehicle detection method based on super-resolution and multimodal fusion of remote sensing images according to claim 2, characterized in that, The construction of the enhanced super-resolution generative adversarial network to perform super-resolution processing on low-resolution remote sensing images includes the following specific steps: An enhanced super-resolution generative adversarial network is constructed. A generator is used to generate high-resolution images from low-resolution remote sensing images. A discriminator is used to distinguish between the generated high-resolution images and the real high-resolution images. The output of the discriminator is the probability of the two images being real. The generator and discriminator are trained adversarially to optimize the quality of the generated high-resolution images. After each optimization, the discriminator distinguishes between the generated and real images again until the discriminator can no longer distinguish between the generated and real images.

4. The vehicle detection method based on super-resolution and multimodal fusion of remote sensing images according to claim 3, characterized in that, The specific steps for adversarial training of the generator and discriminator include: Construct a loss function, which includes adversarial loss, perceptual loss, and content loss; The generator and discriminator are trained alternately based on a loss function. During the training process: The generator improves the quality of high-resolution images to deceive the discriminator; The discriminator improves the ability to distinguish between generated high-resolution images and real high-resolution images.

5. The vehicle detection method based on super-resolution and multimodal fusion of remote sensing images according to claim 1, characterized in that, The specific steps for obtaining the multi-layer feature map for each modality include: Multi-layer hybrid Transformer encoders are used to extract multi-layer features of single-modal images in high-resolution images to obtain global semantic information and local detail features; Multi-layer feature maps for each modality are obtained by overlapping and merging global semantic information and local detail features.

6. The vehicle detection method based on super-resolution and multimodal fusion of remote sensing images according to claim 1, characterized in that, The specific steps for generating multi-scale feature maps based on multi-layer feature maps of each modality include: The highest-level multi-layer feature map is used as the top-level feature map of the Feature Pyramid (FPN). The feature map is upsampled and horizontally connected sequentially from top to bottom using the Feature Pyramid Network (FPN). Upsample the pyramid feature map of the current level to obtain a feature map with the same resolution as the feature map of the next level. The upsampled feature map is convolved with the corresponding low-level feature map and then added to generate a new pyramid feature map. A second convolution operation is performed on the newly generated pyramid feature map to obtain a multi-scale feature map.

7. The vehicle detection method based on super-resolution and multimodal fusion of remote sensing images according to claim 1, characterized in that, The specific steps for constructing a classification subnetwork include: A classification subnetwork is obtained based on the independent convolutional layers in RetinaNet. The input feature map is processed by the classification convolutional layer to extract features for classification. Based on the classification features, it is predicted whether each anchor point contains the target and what type of target it belongs to.

8. The vehicle detection method based on super-resolution and multimodal fusion of remote sensing images according to claim 1, characterized in that, The specific steps for constructing a regression subnetwork include: A regression sub-network is obtained based on RetinaNet, and convolutional layers are used to process the input feature map to extract regression features; Predict the position adjustment value of the bounding box associated with each anchor point based on regression features.

9. The vehicle detection method based on super-resolution and multimodal fusion of remote sensing images according to claim 1, characterized in that, Before detecting vehicle targets and their categories, the detection model is also optimized. Construct a loss function, which includes a focus loss function and a smoothing loss function. The loss function, the focal loss function, is used to address the imbalance between positive and negative samples; the smoothing... Loss functions are used for regression prediction; The Adam algorithm is used for iterative training. Gradient information is used to update the weights of the detection model, optimize the detection model parameters, minimize the loss function, until the model parameters reach a set threshold, and the optimized detection model is obtained for detection.

Images