Small target detection method based on multimodal remote sensing images

Abstract

This invention discloses a small target detection method based on multimodal remote sensing images. The method leverages convolutional neural networks and the similar, complementary, and multi-level features of targets in multimodal remote sensing images. Through an end-to-end target detection network, it improves the detection accuracy of small targets in remote sensing images while fully utilizing relevant features. This invention fully utilizes the similar and complementary features of registered multimodal remote sensing images. Based on the characteristics of small targets in remote sensing images in practical engineering—small size and weak features—this invention proposes a multimodal target detection method for such small targets. This method fuses multimodal input images at the pixel level, extracts target features layer by layer through convolutional neural networks, fuses multi-scale target features across layers, and enhances subtle features and improves overall recognition ability through a weighted convolutional attention mechanism, forming an end-to-end target detection network to achieve high-precision and high-speed detection of small targets in remote sensing images. Experiments show that this invention can effectively utilize multimodal remote sensing images to simultaneously improve the recall and precision of small target detection in remote sensing images.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and machine learning technology, specifically a method for small target detection based on multimodal remote sensing images. Background Technology

[0002] With the development of target detection in remote sensing images, small target detection has gradually become a research focus. Small targets in remote sensing images are characterized by weak features and small size. Weak features are mainly reflected in unclear outlines, indistinct texture features, and high similarity to the adjacent background; small size is manifested in occupying few pixels in the image, only a few dozen pixels. Therefore, it is difficult to extract sufficient effective features from remote sensing images, making small target detection challenging.

[0003] Multimodal image fusion technology extracts complementary information from different sensors, making the fused image more comprehensive and reliable, thereby improving the accuracy of subsequent image processing and decision-making tasks. Its core idea is to select the optimal data under interference-free conditions, extract it effectively, eliminate redundant information, enhance robustness, and improve recognition accuracy.

[0004] In recent years, deep learning has demonstrated significant advantages in object detection. Traditional methods require explicit features, lack flexibility and robustness, and are unsuitable for infrared and panchromatic images. Deep learning can learn features from massive datasets, extracting both shallow visual features and deep semantic features. When there are sufficient samples, the trained network model exhibits strong generalization ability and can handle object detection in complex environments. For targets with clear features and large size, conventional methods such as Faster R-CNN, SSD, and YOLO can achieve high accuracy. However, to effectively utilize multimodal data for accurate detection of small targets, conventional methods are no longer sufficient. Therefore, it is necessary to propose key technologies to improve the accuracy of small target detection in multimodal detection tasks, based on the characteristics of small targets. Summary of the Invention

[0005] To address the low accuracy of conventional target detection methods in detecting small targets in multimodal remote sensing images and to meet the requirements of target detection under complex conditions, this invention proposes a small target detection method based on multimodal remote sensing images.

[0006] This invention proposes a small target detection method based on multimodal remote sensing images. This method utilizes a YOLOv5 convolutional neural network, leveraging the similarity and complementarity features of targets in multimodal images, as well as the multi-level features of small targets. Through an end-to-end target detection network, it improves the accuracy of small target detection in multimodal remote sensing images. The specific process of the detection method is as follows:

[0007] Step 1: Input the multimodal remote sensing images into the multimodal fusion module, fuse the multimodal images, and obtain the fused multimodal feature map;

[0008] Step 2: Input the feature map obtained in Step 1 into the convolutional neural network. The backbone feature extraction network samples the remote sensing image to form multi-level features of the image. A weighted convolutional attention module is added to the backbone feature extraction network.

[0009] Step 3: Input the feature map obtained in Step 2 into the BSS-FPN feature pyramid for feature fusion. Perform double upsampling and downsampling operations on some layers of the multi-level features of the image, and fuse the upper and lower level features of adjacent feature extraction layers and cross-scale feature layers to extract fused features;

[0010] Step 4: Perform three-branch prediction; finally, use the non-maximum suppression algorithm (NMS) to obtain the multimodal small target detection results.

[0011] Furthermore, step 1 specifically includes:

[0012] The multimodal remote sensing image containing small targets is first input into the multimodal fusion module (in this module, the input multimodal images are processed along two identical branches; here, only one branch is used as an example). After feature extraction via the SE attention mechanism and two 1×1 convolutional layers, the input image is multiplied pixel-by-pixel with the feature map before convolution, and then added pixel-by-pixel to obtain a weighted image. This weighted image is then subjected to a 3×3 convolution operation to obtain the final feature map. The final feature map is then added to the final feature map obtained from the other branch on a channel-by-channel basis, and finally passed through the SE attention module to form the final feature map input to the network.

[0013] Furthermore, step 2 specifically includes:

[0014] A remote sensing image containing small targets is input into a convolutional neural network. The backbone feature extraction network performs four consecutive downsampling operations on the remote sensing image, each downsampling by a factor of two. This process extracts shallow to deep features of the image, corresponding to layer number p. i (i = 4, 6, 9), forming multi-level features of the image.

[0015] Furthermore, step 3 specifically includes:

[0016] The feature maps obtained in step 2 are input into the BSS-FPN feature pyramid for feature fusion. The p9 output from the backbone network is upsampled by double, and cross-scale fusion is performed on adjacent layer features and cross-scale feature layers to achieve feature fusion. First, BSS-FPN obtains input feature maps from different layers, including p4, p6, and p9. The p9 feature map is doubled in size using nearest neighbor interpolation. Next, the p6 feature map and the upsampled p9 feature map are compared at node p... 12 Feature fusion is performed at the location. Then, p 12 New feature map p generated from features 14 During the second-level feature fusion process, the p4 feature map and the upsampled p6 feature map are compared at node p. 17 The feature map p after upsampling 14 The feature maps are fused to generate a new feature map p. 17 p 17 The features are convolved to generate a new feature map p. 19 Feature map p 19 At node p 20 Located in feature map p 14 Cross-layer fusion is performed with feature map p6. Feature map p 20 After convolution at node p 23 Located in feature map p 10 Perform cross-level fusion. The fused features are then passed to p. 18 p 21 and p 24 These features, after convolutional processing, are used in three detection layers to output the target's location and category information.

[0017] The feature fusion is represented by the following formula:

[0018]

[0019] p 23 =p 10 +p 20 ↓ 2×

[0020] This method enables the network to maintain high-resolution feature details, thus ensuring accuracy in small target detection. Simultaneously, it provides a wider sensory domain to determine the detection accuracy for targets of regular size. When combining data from different scales in the backbone network, upsampling and downsampling are performed using equal feature resolution. Direct connections are established between initial input and output nodes with the same features, further enriching the feature set without increasing overhead.

[0021] Furthermore, step 4 specifically includes:

[0022] These features, after convolutional processing, are used in three detection layers to output the target's location and category information. This yields features for regular targets and small targets. Small target prediction is then performed on the feature maps of the prominent small target branches, and finally, Non-Maximum Suppression (NMS) is used to obtain the small target detection results.

[0023] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0024] This invention is applicable to small target detection tasks in multimodal remote sensing images. It solves the problem of low detection accuracy of existing target detection methods when performing multimodal small target detection tasks, and effectively improves the recall and precision of multimodal small targets. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall process of the target detection method of the present invention;

[0026] Figure 2 This is an example image of a multimodal remote sensing image containing small targets;

[0027] Figure 3 This is a schematic diagram of the multimodal fusion module in the target detection method of the present invention;

[0028] Figure 4 This is a schematic diagram of the weighted convolutional attention module in the target detection method of the present invention;

[0029] Figure 5 This is an example diagram of the BSS-FPN feature pyramid structure in the target detection method of this invention;

[0030] Figure 6 This is an example diagram showing the results of small target detection in remote sensing images using the target detection method of this invention. Detailed Implementation

[0031] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0032] This invention proposes a small target detection method based on multimodal remote sensing images. This method utilizes a YOLOv5 convolutional neural network, combining the similarity and complementarity features of targets in multimodal images with the multi-level features of small targets. Through an end-to-end target detection network, it improves the detection accuracy of small targets in multimodal remote sensing images. The network structure diagram is shown below. Figure 1 As shown, this invention can effectively utilize multimodal remote sensing images to accurately detect small targets, significantly improving the recall and precision of small target detection.

[0033] The specific process of the small target detection method based on multimodal remote sensing images is as follows:

[0034] Step 1: First, input the multimodal remote sensing image containing small targets into the multimodal fusion module (in this module, the input multimodal images are processed along two identical branches; here, only one branch is used as an example). The module structure diagram is shown below. Figure 3 As shown, the input image is processed by the SE attention mechanism and two 1×1 convolutional layers to extract features. These features are then multiplied pixel-by-pixel by the feature map before convolution, and then added pixel-by-pixel to the input image to obtain a weighted image. The weighted image is then subjected to a 3×3 convolution operation to obtain the final feature map. The final feature map is then added to the final feature map obtained from another branch on a channel-by-channel basis, and finally passed through the SE attention module to form the final feature map input to the network.

[0035] Step 2: Input the feature map obtained in Step 1 into the convolutional neural network. The backbone feature extraction network performs four consecutive downsampling operations on the remote sensing image, each downsampling by a factor of two, thereby extracting the shallow to deep features of the image. The corresponding layer number is p. i (i = 4, 6, 9), forming multi-level image features. Simultaneously, a weighted convolutional attention module is added to the backbone feature extraction network. The structure diagram of this module is shown below. Figure 4 As shown, this module extracts information from the compressed feature map of each channel, giving higher weights to feature layers with stronger semantic information;

[0036] Step 3: Input the feature maps obtained in Step 2 into the BSS-FPN feature pyramid for feature fusion. The p9 output from the backbone network is upsampled by double, and cross-scale fusion is performed on adjacent layer features and cross-scale feature layers to achieve feature fusion. First, BSS-FPN obtains input feature maps from different layers, including p4, p6, and p9. The p9 feature map is doubled in size using nearest neighbor interpolation. Next, the p6 feature map and the upsampled p9 feature map are compared at node p... 12 Feature fusion is performed at the location. Then, p 12 Feature convolution processing generates new feature maps p 14 During the second-level feature fusion process, the p4 feature map and the upsampled p6 feature map are compared at node p. 17 The feature map p after upsampling 14 The feature maps are fused to generate a new feature map p. 17 p 17 The features are convolved to generate a new feature map p. 19 Feature map p 19 At node p 20 Located in feature map p 14 Cross-layer fusion is performed with feature map p6. Feature map p 20 After convolution at node p 23 Located in feature map p 10Perform cross-level fusion. The fused features are then passed to p. 18 p 21 and p 24 These features, after convolutional processing, are used in three detection layers to output the target's location and category information.

[0037] The feature fusion is represented by the following formula:

[0038]

[0039] p 23 =p 10 +p 20 ↓ 2×

[0040] Step 4: After convolutional processing of these features, they are used in three detection layers to output the target's location and category information. This yields features for regular targets and small targets. Small target prediction is then performed on the feature maps of the prominent small target branches, and finally, Non-Maximum Suppression (NMS) is used to obtain the small target detection results.

[0041] The following is a specific example of small target detection in multimodal remote sensing images:

[0042] Figure 2 This example illustrates a multimodal remote sensing image containing small targets. As seen in the image, small targets in multimodal remote sensing images exhibit characteristics such as small pixel size, minimal spectral differences from the background, potential temporal dynamics, indistinct texture and shape, easy obscuration by complex backgrounds, and susceptibility to noise and interference. By combining data from multiple sensors, effective background modeling, and advanced image processing and data fusion methods, the detection and identification capabilities of these small targets can be enhanced.

[0043] Application Step 1: First, input the multimodal remote sensing image containing small targets into the multimodal fusion module (in this module, the input multimodal images are processed along two identical branches; here, only one branch is used as an example). After the input image passes through the SE attention mechanism and two 1×1 convolutional layers to extract features, it is multiplied pixel-by-pixel with the feature map before convolution, and then added pixel-by-pixel to obtain the weighted image. The weighted image is then subjected to a 3×3 convolution operation to obtain the final feature map. The final feature map is then added to the final feature map obtained from the other branch on the same channel, and then passed through the SE attention module to form the final feature map input to the network.

[0044] Application Step Two: Input the feature map obtained in Step 1 into the convolutional neural network. The backbone feature extraction network performs four consecutive downsampling operations on the remote sensing image, each downsampling by a factor of two, thereby extracting the shallow to deep features of the image, corresponding to layer number p. i(i = 4, 6, 9) to form multi-level image features. Simultaneously, a weighted convolutional attention module is added to the backbone feature extraction network. This module extracts information from the compressed feature map of each channel, giving higher weights to feature layers with stronger semantic information.

[0045] Application Step 3: Input the feature maps obtained in Step 2 into the BSS-FPN feature pyramid for feature fusion. The p9 output from the backbone network is upsampled by double, and cross-scale fusion is performed on adjacent layer features and cross-scale feature layers to achieve feature fusion. First, BSS-FPN obtains input feature maps from different layers, including p4, p6, and p9. The p9 feature map is doubled in size using nearest neighbor interpolation. Next, the p6 feature map and the upsampled p9 feature map are compared at node p... 12 Feature fusion is performed at the location. Then, p 12 Feature convolution processing generates new feature maps p 14 During the second-level feature fusion process, the p4 feature map and the upsampled p6 feature map are compared at node p. 17 The feature map p after upsampling 14 The feature maps are fused to generate a new feature map p. 17 p 17 The features are convolved to generate a new feature map p. 19 Feature map p 19 At node p 20 Located in feature map p 14 Cross-layer fusion is performed with feature map p6. Feature map p 20 After convolution at node p 23 Located in feature map p 10 Perform cross-level fusion. The fused features are then passed to p. 18 p 21 and p 24 These features, after convolutional processing, are used in three detection layers to output the target's location and category information.

[0046] The feature fusion is represented by the following formula:

[0047]

[0048] p 23 =p 10 +p 20 ↓ 2×

[0049] Application Step 4: After convolutional processing of these features, they are applied to the three detection layers, outputting the target's location and category information. This yields features for regular targets and small targets. Small target prediction is then performed on the feature maps of the prominent small target branches, and finally, Non-Maximum Suppression (NMS) is used to obtain the small target detection results.

[0050] Examples of specific test results are as follows: Figure 6 As shown above, the specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for small target detection based on multimodal remote sensing images, characterized in that, The specific process of the detection method is as follows: Step 1: Input the multimodal remote sensing image into the pixel-level multimodal fusion module to perform pixel-level fusion of the multimodal image and obtain the fused multimodal feature map; in Step 1, the pixel-level multimodal fusion module specifically comprises: First, the multimodal remote sensing image is input into the pixel-level multimodal fusion module. The image after feature extraction through the SE attention mechanism layer and convolutional layer is multiplied pixel by pixel with the feature map before the input convolutional layer. Then, it is added pixel by pixel with the input multimodal remote sensing image to obtain the weighted image. The weighted image is then subjected to a second convolution operation to obtain the first final feature map. The first final feature map is added to the final feature map obtained from another branch in the multimodal fusion module, and then passed through the SE attention mechanism layer to form the second final feature map; Step 2: Input the multimodal feature map obtained in Step 1 into the backbone feature extraction network of the convolutional neural network. The backbone feature extraction network samples the multimodal feature map to form multi-level features of the image. The backbone feature extraction network includes a weighted convolutional attention module. The weighted convolutional attention module extracts information by compressing each channel of the multimodal feature map and enhances the weights of feature layers with strong semantic information. Step 3: Input the feature map obtained in Step 2 into the BSS-FPN feature pyramid for feature fusion. Perform upsampling and downsampling operations on some levels of the multi-level features. Fuse the upper and lower level features of adjacent feature extraction layers and cross-scale feature layers to extract fused features. In Step 3, the BSS-FPN feature pyramid structure is as follows: The process of fusing the upper and lower level features of adjacent feature extraction layers and cross-scale feature layers to extract the fused features is as follows: The weighted convolutional attention module diagram M obtained in step 2 WCAM Feature fusion is performed on the input BSS-FPN feature pyramid, and the p9 output of the backbone network is upsampled by double. Features from adjacent levels and features at different scales are fused across scales in the feature pyramid. First, the BSS-FPN feature pyramid includes feature map p4, feature map p6, and feature map p9; The p9 feature map is doubled in size by upsampling using nearest neighbor interpolation. The p6 feature map and the upsampled p9 feature map are then compared at node p. 12 Feature fusion is performed at p 12 Feature convolution processes generate new feature maps p 14 ; At the fusion point between the p4 feature map and the upsampled p6 feature map, i.e., node p 17 p after upsampling 14 Feature maps are fused to generate a new feature map p. 17 , will p 17 The features are convolved to generate a new p. 19 Feature map; p 19 Feature map at node p 20 At p 14 Cross-layer fusion of feature maps and p6 feature maps; p 20 Feature convolution at node p 23 p 10 Feature maps are fused across layers; the fused features are then passed to p. 18 Feature map, p 21 Feature map and feature map p 24 ; p 18 Feature map, p 21 Feature map and feature map p 24 After convolution processing, the data are used in three detection layers to output the target's location and category information. The feature fusion is represented by the following formula: p 23 =p 10 +p 20 ↓ 2× ; Step 4: Use the feature maps of the three branches to predict the target; finally, use the Non-Maximum Suppression (NMS) algorithm to obtain the target detection result.

2. The method for small target detection based on multimodal remote sensing images according to claim 1, characterized in that, In step 2, the weighted convolutional attention module specifically comprises: The second final feature map is input, and firstly, an average pooled feature map F is generated through global average pooling and global max pooling. gap and max pooling feature map F gmp The average pooling feature map F is weighted by weight factors α and (1-α). gap and max pooling feature map F gmp Combined to generate weighted feature maps; Then, the weighted feature map is used to generate channel weights F through a shared multilayer perceptron. c The channel weight F c The second final feature map is applied to generate a weighted channel attention map; Then, the weighted channel attention is subjected to global max pooling and global average pooling again, and concatenated along the channel dimension, and then passed through a convolutional layer to generate spatial weights F. sw ; Finally, the spatial weight F sw Applying this to the weighted channel attention map yields the weighted convolutional attention module map M. WCAM .

Images