Remote sensing image tiny target detection method based on re-parameter and layered multi-scale adaptive feature fusion

Abstract

The invention belongs to the field of image processing, discloses a remote sensing tiny target detection method based on re-parameter and hierarchical multi-scale adaptive feature fusion, and is used for solving the problem that robustness detection is difficult to realize under the conditions of tiny target size, rare features, low signal-to-noise ratio and the like in the traditional remote sensing tiny target detection method. The model comprises a feature extraction network and a small target detection head, wherein the feature extraction network is composed of a re-parameterization-based multi-scale feature extraction module and a layered multi-scale adaptive feature fusion module; according to the method, the feature extraction network is designed by using re-parameter, gradient flow optimization and hierarchical multi-scale adaptive feature fusion strategies, so that the target characterization capability of the network is enhanced; the method can significantly improve the precision of remote sensing image small target detection, reduces the model parameter quantity, and improves the reasoning speed.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a remote sensing tiny target detection method based on heavy parameter and hierarchical multi-scale adaptive feature fusion. Background Art

[0002] In recent years, advances in earth observation technology have made it easier to obtain massive amounts of data, greatly promoting the development of small target detection in optical remote sensing images. Small target detection has important applications in military reconnaissance, ocean monitoring, intelligent monitoring, ecological monitoring, and post-disaster emergency response. However, due to the extremely small size of targets in remote sensing images (often less than 16×16 pixels or even less), feature extraction is more difficult. In addition, complex backgrounds and adverse meteorological conditions, such as lighting changes and weather interference, lead to a decrease in the signal-to-noise ratio, exacerbating the confusion between targets and backgrounds. Targets of different scales and dense arrangements further increase the difficulty of detection.

[0003] With the development of convolutional neural networks, many advanced detection methods have been applied to the field of remote sensing and have become mainstream. However, two-stage and single-stage detectors still have insufficient performance when dealing with tiny targets. Although detection effects can be improved through super-resolution, context information extraction, and multi-scale feature fusion, these strategies often increase the amount of training, rely on manual annotation, or expand model parameters, resulting in a decrease in inference speed and difficulty in balancing accuracy and speed. Therefore, designing an algorithm that can balance detection accuracy and speed has become a key requirement in the field of optical remote sensing. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a remote sensing image small target detection method based on heavy parameters and hierarchical multi-scale adaptive feature fusion, which can significantly improve the accuracy of remote sensing image small target detection, reduce the amount of model parameters and slightly improve the inference speed.

[0005] The technical solution of the present invention to solve the above technical problems is:

[0006] A remote sensing small target detection method based on heavy parameters and hierarchical multi-scale adaptive feature fusion includes the following steps:

[0007] (S1), build data set: preprocess the AI-TODv2 remote sensing data set;

[0008] (S2), construct RepHGStem module: design RepHGStem module based on 4x4 convolution, 2x2 convolution, reparameterization and residual connection technology to replace the original YOLOv8 Stem module;

[0009] (S3), construct HMSAF-Bolck module: in the C2f-Bolck module of the original YOLOv8, the HMSAF-Bolck module is designed based on the gradient flow, hierarchical multi-scale and adaptive feature fusion technology to replace the C2f-Bolck module of the original YOLOv8;

[0010] (S4), construct a small target detection head module: add a small target detection head p2, and remove the detection head p5 to replace the original YOLOv8 detection head module;

[0011] (S5), constructing the HMSAF-YOLO network: combining the RepHGStem module, the HMSAF-Bolck module and the small target detection head module to construct the HMSAF-YOLO network;

[0012] (S6) Train the HMSAF-YOLO network and evaluate it: Use the HMSAF-YOLO network structure as the small target recognition network for remote sensing images, and obtain a deep learning model for small target detection in remote sensing images through training.

[0013] Preferably, in step (S1), data preprocessing is performed on the AI-TODv2 remote sensing dataset, comprising the following steps:

[0014] (S1-1) Data preprocessing: A series of image processing operations are performed on the dataset, including color enhancement, rotation and translation, scaling and shearing, perspective, flipping, and mosaic.

[0015] Preferably, in step (S2), a RepHGStem module is designed according to 4x4 convolution, re-parameterized 2x2 convolution and residual connection technology to replace the original YOLOv8 Stem module, comprising the following steps:

[0016] (S2-1) HGStem is introduced to replace the native YOLOv8Stem module. HGStem uses a delayed downsampling strategy, that is, introducing multiple 3x3 convolution and maximum pooling layers after preliminary downsampling to gradually extract features and better preserve image information and details.

[0017] (S2-2), RepConv is designed using 4x4 convolution and reparameterization, assuming X∈R H×W×C is the input feature, then 4x4RepConv can be expressed as:

[0018] X=γ(bn(Conv 4×4 (X))+bn(Conv 2×2 (X))) (1)

[0019] Among them, bn represents batch normalization, γ represents the activation function ReLU, and then, by adopting the reparameterization technology, after the model is trained, the 2x2 convolution kernel will be integrated into the 4x4 convolution kernel to reduce the number of parameters and calculations of the model;

[0020] (S2-3) Replace the original 2x2 convolution of the HGStem module with RepConv, and then add residual connection to the two RepConvs to design the RepHGStem module, and then use the RepHGStem module to replace the original YOLOv8 Stem module.

[0021] Preferably, in step (S3), in the C2f Bolck module of the original YOLOv8, a HMSAF-Bolck module is designed according to the gradient flow and hierarchical multi-scale adaptive feature fusion technology to replace the C2f-Bolck module of the original YOLOv8, comprising the following steps:

[0022] (S3-1), HMSAF-Bolck module introduces more gradient flows and hierarchical multi-scale adaptive feature fusion mechanisms based on C2f;

[0023] (S3-2) Assume that X is the input feature. X first undergoes a 1x1 convolution and then introduces a split operation to divide the feature into X1 and X2. The split data stream is divided into two branches. X2 undergoes a 1x1 convolution and then introduces a split operation to divide the feature into X 21 and X 22 , where X 21 is directly fed to the Concat operation, while X 22 Before entering the bottleneck, it is fused with the original information retained in X1 through adaptive feature fusion to obtain X. It is then sent to the bottleneck structure for deep multi-scale feature extraction;

[0024] (S3-3), all feature maps generated by the right branch in step (S3-2) are fused and reduced in dimension through 3x3 convolution, and then the left and right branches are merged again and the feature representation is unified through 1x1 convolution.

[0025] Preferably, in step (S4), a small target detection head p2 is added, and the detection head p5 is removed to replace the original YOLOv8 detection head module.

[0026] Preferably, in step (S5), the RepHGStem module, the HMSAF-Bolck module and the small target detection head are used to replace the Stem module, the C2f-Bolck module and the detection head of the original YOLOv8 respectively, and the HMSAFYOLO network is constructed in combination.

[0027] Preferably, in step (S6), the AI-TODv2 dataset is input into HMSAF-YOLO for training and evaluation.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The invention proposes a remote sensing image small target detection method based on reparameterization and hierarchical multi-scale feature fusion, and designs a RepHGStem module. This module enhances the model's ability to capture background information and detail features by expanding the receptive field. The multi-scale feature extraction and fusion technology used significantly improves the target recognition accuracy of the model, especially when dealing with small target detection. Although this strategy slightly increases the computational cost, it is crucial to achieve efficient feature extraction and optimize network performance.

[0030] 2. The invention proposes a remote sensing image small target detection method based on reparameterization and hierarchical multi-scale feature fusion, and designs a HMSAF-Bolck module. The module provides more comprehensive feature information by enriching the gradient flow, and combines the retained original feature information with the enhanced feature information, and realizes efficient fusion in the channel and spatial dimensions through an adaptive fusion strategy. This method enables the model to fully extract and utilize information at multiple scales, thereby significantly improving the adaptability and recognition effect of small targets and complex features.

[0031] 3. The remote sensing image small target detection method based on re-parameterization and hierarchical multi-scale feature fusion of the present invention designs a small target detection head module, adds small target detection to enhance the recognition of small targets, and removes the detection head p5 for detecting large targets to reduce the model parameters and calculation amount, thereby improving the small target detection accuracy while optimizing the overall calculation efficiency and the number of model parameters.

[0032] 4. The remote sensing image small target detection method based on re-parameterization and hierarchical multi-scale feature fusion of the present invention can significantly improve the accuracy of small target detection in remote sensing images while reducing the model parameters and improving the reasoning speed by combining the RepHGStem module, the HMSAF-Bolck module and the small target detection head module to design the HMSAF-YOLO network. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The present invention is a flow chart of a remote sensing image micro-target detection method based on heavy parameters and hierarchical multi-scale adaptive feature fusion.

[0034] Figure 2 It is a schematic diagram of the RepHGStem module of the remote sensing image small target detection method based on heavy parameters and hierarchical multi-scale adaptive feature fusion of the present invention.

[0035] Figure 3 It is a schematic diagram of the HMSAF-Bolck module of the remote sensing image small target detection method based on heavy parameters and hierarchical multi-scale adaptive feature fusion of the present invention.

[0036] Figure 4 It is a schematic diagram of the HMSAF-YOLO network of the remote sensing image small target detection method based on heavy parameters and hierarchical multi-scale adaptive feature fusion of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0038] See also Figure 1 The remote sensing image small target detection method based on large kernel convolution and hierarchical multi-scale feature fusion of the present invention comprises the following steps:

[0039] (S1), build data set: preprocess the AI-TODv2 remote sensing data set;

[0040] (S2), construct RepHGStem module: design RepHGStem module based on 4x4 convolution, 2x2 convolution, reparameterization and residual connection technology to replace the original YOLOv8 Stem module;

[0041] (S3), construct HMSAF-Bolck module: in the C2f-Bolck module of the original YOLOv8, the HMSAF-Bolck module is designed based on the gradient flow, hierarchical multi-scale and adaptive feature fusion technology to replace the C2f-Bolck module of the original YOLOv8;

[0042] (S4), construct a small target detection head module: add a small target detection head p2, and remove the detection head p5 to replace the original YOLOv8 detection head module;

[0043] (S5), constructing the HMSAF-YOLO network: combining the RepHGStem module, the HMSAF-Bolck module and the small target detection head module to construct the HMSAF-YOLO network;

[0044] (S6) Train the HMSAF-YOLO network and evaluate it: Use the HMSAF-YOLO network structure as the small target recognition network for remote sensing images, and obtain a deep learning model for small target detection in remote sensing images through training.

[0045] See also Figure 1 In step (S1), the AI-TODv2 remote sensing dataset is preprocessed, including the following steps:

[0046] (S1-1) Data preprocessing: A series of image processing operations are performed on the dataset, including color enhancement, rotation and translation, scaling and shearing, perspective, flipping, and mosaic.

[0047] See also Figure 1 In step (S2), the RepHGStem module is designed based on 4x4 convolution, re-parameterized 2x2 convolution and residual connection technology to replace the original YOLOv8 Stem module, see Figure 2 , including the following steps:

[0048] (S2-1) In the existing network structure, as features are downsampled layer by layer, especially when features flow through detection heads such as shallow feature maps, semantic information is greatly reduced due to multiple compressions, limiting the network's ability to capture fine features. Therefore, the full extraction of underlying features is particularly important for the performance of subsequent networks (such as the neck layer). To this end, we introduced the HGStem module to replace the native YOLOv8 Stem module. HGStem uses a delayed downsampling strategy to introduce multiple 3x3 convolutional layers and maximum pooling layers after initial downsampling, gradually extracting features and better retaining the overall information and details of the image;

[0049] (S2-2) In order to expand the receptive field and enhance the model's ability to capture background information and detail features, the multi-scale feature extraction and fusion strategy introduced significantly improves the model's recognition of targets. We use 4x4 convolution and reparameterization to design RepConv, assuming X∈R H×W×C is the input feature, then RepConv can be expressed as:

[0050] X=γ(bn(Conv 4×4 (X))+bn(Conv 2×2 (X))) (1)

[0051] Among them, bn represents batch normalization, γ represents the activation function ReLU, and then, by adopting the reparameterization technology, after the model is trained, the 2x2 convolution kernel will be integrated into the 4x4 convolution kernel to reduce the number of parameters and calculations of the model;

[0052] (S2-3) Replace the original 2x2 convolution of the HGStem module with RepConv, and then add residual connection to the two RepConvs to design the RepHGStem module, and then use the RepHGStem module to replace the original YOLOv8Stem module.

[0053] See also Figure 1In step (S3), in the C2fbolck module of the original YOLOv8, the HMSAF-Bolck module is designed based on the gradient flow and hierarchical multi-scale adaptive feature fusion technology to replace the C2f-Bolck module of the original YOLOv8, see Figure 3 , including the following steps:

[0054] (S3-1) To enhance the network’s ability to process complex features and improve recognition accuracy, the HMSAF-Bolck module introduces more gradient flows and hierarchical multi-scale adaptive feature fusion mechanisms based on C2f;

[0055] (S3-2) Assume that X is the input feature. X first undergoes a 1x1 convolution and then introduces a split operation to divide the feature into X1 and X2. The split data stream is divided into two branches. X2 undergoes a 1x1 convolution and then introduces a split operation to divide the feature into X 21 and X 22 , where X 21 is directly fed to the Concat operation, while X 22 Then, the original information retained in X1 before entering the bottleneck is fused through adaptive feature fusion to obtain X'. The adaptive feature fusion can be expressed as:

[0056] X'=sigmoid(a)X+(1-sigmoid(a))X (2)

[0057] Among them, α is a weight factor that can be optimized by back propagation, and then normalized by the sigmoid function. It is then sent to the bottleneck structure for deep multi-scale feature extraction. By combining the retained original feature information with the enhanced feature information, and achieving efficient fusion in the channel and spatial dimensions through an adaptive fusion strategy, the module can effectively extract and utilize information at different scales, enhancing the ability of multi-scale feature extraction;

[0058] (S3-3), all feature maps generated by the right branch in step (S3-2) are fused and reduced in dimension through 3x3 convolution, and then the left and right branches are merged again and the feature representation is unified through 1x1 convolution.

[0059] See also Figure 1 In step (S4), add the small target detection head p2 and remove the detection head p5 to replace the original YOLOv8 detection head module:

[0060] (S4-1) Small target detection is added to enhance the recognition of small targets, and the detection head p5 for detecting large targets is removed to reduce the number of model parameters and the amount of calculation. While improving the accuracy of small target detection, the overall calculation efficiency and the number of model parameters are optimized.

[0061] See also Figure 1 In step (S5), the RepHGStem module, HMSAF-Bolck module and small target detection head are used to replace the original YOLOv8 Stem module, C2f-Bolck module and detection head, respectively, and the HMSAF-YOLO network is constructed. Figure 4 .

[0062] See also Figure 1 ,In step (S6), the AI-TODv2 dataset is input into HMSA-YOLO for training and evaluation.

[0063] The above is a preferred embodiment of the present invention, but the embodiment of the present invention is not limited to the above content. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A remote sensing small target detection method based on heavy parameters and hierarchical multi-scale adaptive feature fusion, characterized in that: The following steps are involved: (S1), build data set: preprocess the AI-TODv2 remote sensing data set; (S2), construct RepHGStem module: design RepHGStem module based on 4x4 convolution, 2x2 convolution, reparameterization and residual connection technology to replace the original YOLOv8 Stem module; (S3), construct HMSAF-Bolck module: in the C2f-Bolck module of the original YOLOv8, the HMSAF-Bolck module is designed based on the gradient flow, hierarchical multi-scale and adaptive feature fusion technology to replace the C2f-Bolck module of the original YOLOv8; (S4), construct a small target detection head module: add a small target detection head p2, and remove the detection head p5 to replace the original YOLOv8 detection head module; (S5), constructing the HMSAF-YOLO network: combining the RepHGStem module, the HMSAF-Bolck module and the small target detection head module to construct the HMSAF-YOLO network; (S6) Train the HMSAF-YOLO network and evaluate it: Use the HMSAF-YOLO network structure as the small target recognition network for remote sensing images, and obtain a deep learning model for small target detection in remote sensing images through training.

2. A remote sensing small target detection method based on heavy parameter and hierarchical multi-scale adaptive feature fusion according to claim 1, characterized in that: In step (S1), data preprocessing is performed on the AI-TODv2 remote sensing dataset, including the following steps: (S1-1) Data preprocessing: A series of image processing operations are performed on the dataset, including color enhancement, rotation and translation, scaling and shearing, perspective, flipping, and Mosaic data enhancement.

3. The remote sensing small target detection method based on heavy parameter and hierarchical multi-scale adaptive feature fusion according to claim 1 is characterized in that: In step (S2), a RepHGStem module is designed based on 4x4 convolution, re-parameterized 2x2 convolution and residual connection technology to replace the original YOLOv8 Stem module, including the following steps: (S2-1) HGStem is introduced to replace the native YOLOv8 Stem module. HGStem uses a delayed downsampling strategy, that is, introducing multiple 3x3 convolution and maximum pooling layers after preliminary downsampling to gradually extract features; (S2-2), RepConv is designed using 4x4 convolution and reparameterization, assuming X∈R H×W×C is the input feature, then 4x4RepConv can be expressed as: X=γ(bn(Conv 4×4 (X))+bn(Conv 2×2 (X))) (1) Among them, bn represents batch normalization, and γ represents the activation function ReLU; (S2-3) Replace the original 2x2 convolution of the HGStem module with RepConv, and then add residual connection to the two RepConvs to design the RepHGStem module, and then use the RepHGStem module to replace the original YOLOv8 Stem module.

4. The remote sensing small target detection method based on heavy parameter and hierarchical multi-scale adaptive feature fusion according to claim 1 is characterized in that: In step (S3), in the C2f Bolck module of the original YOLOv8, the HMSAF-Bolck module is designed according to the gradient flow and hierarchical multi-scale adaptive feature fusion technology to replace the C2f-Bolck module of the original YOLOv8, including the following steps: (S3-1), HMSAF-Bolck module introduces more gradient flows and hierarchical multi-scale adaptive feature fusion mechanisms based on C2f; (S3-2) Assume that X is the input feature. X first undergoes a 1x1 convolution and then introduces a split operation to divide the feature into X1 and X2. The split data stream is divided into two branches. X2 undergoes a 1x1 convolution and then introduces a split operation to divide the feature into X 21 and X 22 , where X 21 is directly fed to the Concat operation, while X 22 Before entering the bottleneck, it is fused with the original information retained in X1 through adaptive feature fusion to obtain X. It is then sent to the bottleneck structure for deep multi-scale feature extraction; (S3-3), all feature maps generated by the right branch in step (S3-2) are fused and reduced in dimension through 3x3 convolution, and then the left and right branches are merged again and the feature representation is unified through 1x1 convolution.

5. The remote sensing small target detection method based on heavy parameter and hierarchical multi-scale adaptive feature fusion according to claim 1 is characterized in that: In step (S4), a small target detection head p2 is added and the detection head p5 is removed to replace the original YOLOv8 detection head module.

6. The remote sensing small target detection method based on heavy parameter and hierarchical multi-scale adaptive feature fusion according to claim 1 is characterized in that: In step (S5), the RepHGStem module, HMSAF-Bolck module and small target detection head are used to replace the Stem module, C2f-Bolck module and detection head of the original YOLOv8 respectively, and combined to construct the HMSAF-YOLO network.

Images