Multi-modal remote sensing image weak and small target detection method based on super-resolution

Abstract

The invention discloses a multi-mode remote sensing image weak and small target detection method based on super-resolution. The method comprises the following steps that firstly, feature fusion is carried out on input RGB and infrared remote sensing images through BFM to generate comprehensive features, and the BFM can effectively make up for the defect that weak and small target information is insufficient in a single mode; the features are then sent to a backbone network to generate multi-level features. Secondly, a backbone network in the SA-YOLO is responsible for further fusing the multi-level features in a feature pyramid network, and then classification and regression are carried out in a detection head; a branch network in SA-YOLO selects low-level and high-level features output by a backbone network, and local texture and semantic information are fused to carry out super-resolution reconstruction. According to the method provided by the invention, the detection precision of the weak and small target in the multi-mode remote sensing image is effectively improved, and the method has a wide application prospect in the aspects of military reconnaissance, civil monitoring, disaster early warning and the like.

Description

Technical Field

[0001] This invention relates to the problem of target detection in remote sensing images in the field of deep learning, and in particular to a method for detecting weak targets in multimodal remote sensing images based on super-resolution. Background Technology

[0002] Single-modal remote sensing images contain limited pixel information about small targets and are easily affected by environmental factors such as lighting and weather, resulting in low accuracy in detecting small targets. With the rapid development of imaging technology, multimodal remote sensing images collected from different scenes can supplement information about small targets, and multimodal data fusion can effectively improve the detection accuracy of small targets in remote sensing images.

[0003] In recent years, with the rapid development of deep learning technology, the performance of remote sensing target detection algorithms has made significant breakthroughs, and its application in the field of remote sensing has yielded some results. However, their performance in detecting small targets often falls short of expectations. These methods are easily affected by factors such as lighting and weather, and are mainly designed for single modalities. Now, the development of imaging technology has brought about multimodal remote sensing images, such as infrared images and synthetic aperture radar images. These remote sensing images have their own advantages and can complement each other to improve the detection accuracy of algorithms for small targets. Currently, the fusion of RGB images and infrared images has become a hot topic in the field of remote sensing.

[0004] The key to fusing RGB and infrared images lies in the design of the fusion method. Currently, researchers have proposed various image fusion methods. Early fusion methods mainly involved simple addition or concatenation. Liu et al. used a GAN architecture to preserve target information in the infrared image while maintaining texture details in the visible image. Tang et al. designed an illumination-aware network to estimate the illumination distribution of the image for better image fusion. Xu et al. proposed a novel structure that can adaptively preserve information from the source (infrared / visible) images to construct a fused image using features extracted by the network, alleviating the problem of lacking ground truth in image fusion tasks. While these image fusion methods have achieved some success in vision, they have not considered whether the fused image is suitable for subsequent target detection tasks. Summary of the Invention

[0005] The purpose of this invention is to provide a method for detecting weak targets in multimodal remote sensing images based on super-resolution. First, SA-YOLO utilizes a designed balanced multimodal feature fusion module to fuse feature information, compensating for the insufficient information on weak targets in a single modality. Second, to facilitate the separation of weak targets from the complex background of the remote sensing image, SA-YOLO introduces a super-resolution branch to reconstruct high-resolution features of the weak targets. Considering the model's complexity, this branch is only used during the training phase.

[0006] The present invention specifically adopts the following technical solution:

[0007] A method for detecting weak targets in multimodal remote sensing images based on super-resolution is proposed. The main features of this method are:

[0008] a. A balanced multimodal feature fusion module is proposed, which is specifically designed to fuse weak target features in RGB and infrared remote sensing images.

[0009] b. A multimodal remote sensing image weak target detection method, SA-YOLO, combining BFM and Super-Resolution Module (SRM), is proposed.

[0010] like Figure 1 As shown, the method mainly includes the following steps:

[0011] (1) The input RGB and infrared remote sensing images are fused at the pixel level through BFM to generate comprehensive features. These features are then fed into the backbone network to generate multi-level image features.

[0012] (2) The super-resolution branch network in SA-YOLO selects the low-level and high-level features output by the backbone network and fuses local texture and semantic information to perform super-resolution reconstruction. The output size of the super-resolution module is twice the size of the input feature.

[0013] (3) The feature pyramid network in SA-YOLO is responsible for further fusing multi-level features, and then classifying and regressing them in the detection head. Attached Figure Description

[0014] Figure 1 This is a framework diagram of the weak target detection method based on super-resolution multimodal remote sensing images of the present invention.

[0015] Figure 2 This is a structural diagram of the balanced multimodal feature fusion module of the present invention.

[0016] Figure 3 This is a diagram of the super-resolution branch network structure of the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above-described invention, and these improvements and adjustments should still fall within the scope of protection of the present invention.

[0018] The specific explanation of the Balanced Multimodal Feature Fusion (BFM) module in step (1) is as follows:

[0019] like Figure 2 As shown, the BFM module employs a symmetrical design to process the input RGB and infrared remote sensing image features. Each module consists of two branches. First, it processes the original input RGB and IR images X... RGB X IR I was obtained by sampling RGB I IR I RGB I IR First, the channel features F are extracted using the CBAM attention module. RGB F IR Its expression is:

[0020] F RGB =CBAM(I RGB ),F IR =CBAM(I IR (1)

[0021] Then, after 1×1 convolution, attention maps m representing the internal relationships of different modalities in the spatial domain are obtained. IR m RGB ,

[0022] m IR =f1(F IR ),m RGB =f2(F RGB (2) where f1 and f2 represent 1×1 convolutions of the RGB and IR modes, respectively. This represents element-wise matrix multiplication. Multiplying the channel features with the internal relational attention pixel-wise yields the internal spatial information F between different modalities. in1 F in2 :

[0023]

[0024] To integrate internal view information and spatial texture information, by modifying the original input X... RGB X IR Perform addition and feed it into a 1×1 convolution. The complete features include:

[0025] Fful1 =f3(F in1 +I RGB ),F ful2 =f4(F in2 +I IR (4)

[0026] Where f3 and f4 represent 1×1 convolutions. Finally, these features are fused as follows:

[0027] F o =CBAM(Concat(F ful1 ,F ful2 (5)

[0028] Concat(·) represents a concatenation operation along the channel axis.

[0029] The specific explanation of the SA-YOLO multimodal remote sensing image weak target detection method combining BFM and SRM in steps (2) and (3) is as follows:

[0030] SA-YOLO introduces SOCA as the encoder structure. In the encoder, the first CR module (Conv and ReLU for short) operates on low-level features. For high-level features, an upsampling operation is used to match the spatial size of the low-level features, and then a connection operation and two CR modules are used to merge the low-level and high-level features. For the decoder, low-resolution features are magnified to a high-resolution space, where the output size of the super-resolution module is twice that of the input image. The super-resolution branch guides the learning related to spatial dimensions and passes it to the main branch, thereby improving the algorithm's performance in detecting small targets. Considering the complexity of the model, this branch is only used during the training phase.

[0031] SA-YOLO utilizes the Balanced Multimodal Feature Fusion (BFM) module to fuse feature information, compensating for the lack of information on small targets in a single modality. Secondly, to facilitate the separation of small targets from the complex background of remotely sensed images, a super-resolution branch is introduced to learn their high-resolution features. Considering the model's complexity, this branch is only used during the training phase.

[0032] The loss function of the SA-YOLO network consists of two parts: the detection loss L... d And the loss of super-resolution reconstruction L SR , can be represented as:

[0033] L Al =λ1L d +λ2L SR (6)

[0034] Here, λ1 and λ2 are weighting factors set to balance the losses of the two tasks. Detection loss L d The expression is as follows:

[0035]

[0036] Among them, L loc L represents the target localization loss. obj L represents the confidence loss of the target. cls The classification loss for the target; λ loc , λ obj and λ cls It is a weighting factor used to balance the oriented bounding box localization loss, confidence loss, and object classification; α i β i and This represents the weighting factor for the first three output layers of the detection. The loss L is used to calculate the difference between the input image X and the super-resolution reconstructed image Y. SR The L1 loss function is used, and its expression is as follows:

[0037] L SR =|YX||1. (8)

[0038] This embodiment utilizes the publicly available dataset VEDAI for training and testing, achieving optimal results after 300 training rounds. Quantitative detection performance is shown in Tables 1 and 2, demonstrating the superior detection performance of this invention. Bold text indicates the best experimental results, while underlined text indicates the second-best. Since targets such as cars, pickup trucks, campervans, trucks, other vehicles, tractors, tugboats, and vans are all relatively small, they are considered as weak targets in this discussion.

[0039] Table 1. Experimental results of SA-YOLO and other algorithms on VEDAI data.

[0040]

[0041] Table 2 shows the parameter count and computational cost of SA-YOLO compared to other algorithms. Compared to the baseline methods, SA-YOLO strikes a balance between detection performance and complexity, achieving good detection performance for weak targets without significantly increasing model parameters and computational cost. YOLOFusion's results are slightly better than SA-YOLO; however, its parameter count and computational cost are considerably higher. Overall, SA-YOLO's model performance is superior to YOLOFusion.

[0042] Table 2 shows the parameter quantity and computational complexity of SA-YOLO compared to other algorithms.

[0043]

Claims

1. A method for detecting weak targets in multimodal remote sensing images based on super-resolution, characterized in that: a. A Balanced Multimodal Feature Fusion (BFM) module is proposed, which is specifically designed to fuse weak target features in RGB and infrared remote sensing images; b. A multimodal remote sensing image weak target detection method, SA-YOLO, combining BFM and Super-Resolution Module (SRM) is proposed; This method mainly includes the following steps: (1) The input RGB and infrared remote sensing images are fused at the pixel level through BFM to generate comprehensive features. These features are then fed into the backbone network to generate multi-level image features. (2) The super-resolution branch network in SA-YOLO selects the low-level and high-level features output by the backbone network and fuses local texture and semantic information to perform super-resolution reconstruction. The output size of the super-resolution module is twice the size of the input feature. (3) The feature pyramid network in SA-YOLO is responsible for further fusing multi-level features, and then classifying and regressing them in the detection head.

2. The Balanced Multimodal Feature Fusion (BFM) module as described in claim 1, characterized in that... In step (1), a BFM (Browser Function Mechanism) is designed to fuse weak target features from RGB and infrared remote sensing images. The design method is as follows: The BFM module employs a symmetrical design to process the input RGB and infrared remote sensing image features. Each module consists of two branches: first, it processes the original input RGB and IR images X... RGB X IR I was obtained by sampling RGB I IR I RGB I IR First, the channel features F are extracted using the CBAM attention module. RGB F IR Then, after 1×1 convolution, attention maps of the internal relationships of different modalities in the spatial domain are obtained. Finally, the channel features are multiplied pixel by pixel with the internal relationship attention to obtain the internal spatial information F between different modalities. in1 F in2 To integrate internal view information and spatial texture information, the original input X is processed... RGB X IR An addition operation is performed to feed the data into a 1×1 convolutional layer, followed by a channel concatenation operation to feed the complete features into the CBAM module.

3. The SA-YOLO multimodal remote sensing image weak target detection method combining BFM and SRM according to claim 1, characterized in that... In step (3), the super-resolution module (SRM) is constructed as follows: SOCA is introduced as the encoder structure. In the encoder, the first CR module (Conv and ReLU for short) is used on low-level features. For high-level features, an upsampling operation is used to match the spatial size of the low-level features, and then a connection operation and two CR modules are used to merge the low-level and high-level features. For the decoder, low-resolution features are magnified to high-resolution space, where the output size of the super-resolution module is twice that of the input image. The super-resolution branch guides the learning related to spatial dimension and passes it to the main branch, thereby improving the algorithm's performance in detecting weak targets. Considering the complexity of the model, this branch is only used during the training phase.

4. The SA-YOLO multimodal remote sensing image weak target detection method combining BFM and SRM according to claim 1, characterized in that... SA-YOLO combines the Balanced Multimodal Feature Fusion (BFM) module in step (1) and the Super-Resolution Branch Network in step (3). BFM can efficiently fuse feature information from RGB and infrared remote sensing images to compensate for the lack of information on weak targets in a single modality. The Super-Resolution Branch can promote the separation of weak targets from the complex background of the remote sensing image. The Super-Resolution Branch is discarded in the inference stage to avoid introducing additional computational overhead. SA-YOLO improves the detection accuracy of weak targets in multimodal remote sensing images through the combined action of BFM and SRM.

Images