An improved pyramid-based remote sensing image target detection method

Abstract

The application provides a remote sensing image target detection method based on an improved pyramid, in a constructed detection model, a feature extraction module is a two-stage feature pyramid structure based on ResNet50, and a bottom-to-top pyramid feature map fusion structure is added; two enhancement modules respectively take two feature maps of high layer size and two feature maps of low layer size as input, perform channel attention extraction and spatial attention extraction, and multiply the extracted feature maps to obtain double-scale feature maps; then two detection head modules process the double-scale feature maps, and respectively output the category and the boundary box information of the target through a classification branch and a regression branch. While the high layer rich semantic information is retained, the edge, corner, texture and other detail information of the bottom layer are more comprehensively captured, and for the remote sensing target under the condition of density and overlap, each edge of the object can be accurately identified, and the detection precision is higher.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a remote sensing image target detection method based on an improved pyramid. Background Technology

[0002] In recent years, with the increasing number of space imaging devices, the spatial resolution of remote sensing images has been continuously improved. Compared with traditional medium and low resolution images, high-resolution remote sensing images can more clearly express the characteristic distribution and spatial correlation of ground targets. The large amount of texture, color and other details contained therein provide new information channels and intelligence sources, which facilitates the refined and comprehensive management and decision-making of the region.

[0003] Pyramid networks are an image processing method that decomposes an image into multiple scales for processing. Specifically, they divide the input image into multiple scales and process each scale separately. Pyramid networks are widely used in tasks such as image classification and object detection because they can improve model performance and robustness. However, high-resolution remote sensing images are often captured at long distances, and the targets in remote sensing images are much smaller than those in natural scene images, thus placing higher demands on the accuracy of detection methods. Existing pyramid networks only retain rich semantic information in the higher levels. For remote sensing images, which often contain dense targets, they cannot accurately identify object edges, resulting in low detection accuracy. Summary of the Invention

[0004] The purpose of this invention is to solve the problem of low target detection accuracy caused by using pyramid networks to detect remote sensing images in existing technologies. Instead, it provides a remote sensing image target detection method based on an improved pyramid. This method retains rich semantic information at high levels while capturing more comprehensive low-level detail information. For dense remote sensing targets, it can highlight object features, better detect dense small targets, and improve detection accuracy.

[0005] To achieve the above objectives, the technical solution provided by this invention is:

[0006] A remote sensing image target detection method based on an improved pyramid is provided, comprising the following steps:

[0007] Step 1: Obtain the dataset of remote sensing images;

[0008] Step 2: Construct a remote sensing image target detection model based on the improved pyramid. The target detection model includes a feature extraction module, two enhancement modules, and two detection head modules.

[0009] The feature extraction module is a two-level feature pyramid structure based on ResNet50. The first-level pyramid structure is used to extract four feature maps of different sizes from the input image and obtain a fused feature map through the FPN structure. The second-level pyramid structure is used to perform feature fusion again on the obtained fused feature map from bottom to top.

[0010] The two enhancement modules take the two high-level feature maps and the two low-level feature maps from the four feature maps obtained by re-fusion as input, respectively, perform channel attention extraction and spatial attention extraction on the input feature maps, and multiply the channel attention feature maps and spatial attention feature maps to obtain the final dual-scale feature maps;

[0011] The two detection head modules are used to process the dual-scale feature maps, and output the target's category and bounding box information through classification and regression branches, respectively;

[0012] Step 3: Iteratively train the constructed object detection model using the acquired dataset. After the detection model converges, obtain the optimal network model parameters.

[0013] Step 4: Use the obtained optimal network model parameters to detect the target remote sensing image and obtain the classification and location information of the target in the image.

[0014] Furthermore, in step 1, the acquired dataset is either a publicly available dataset or a dataset obtained through self-photography.

[0015] Furthermore, step 1 also includes: dividing the acquired data into a training set and a test set, with a split ratio of 8:2 or 7:3.

[0016] Furthermore, in step 2, the feature extraction module extracts images of sizes that are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size, respectively.

[0017] Furthermore, in step 2, the two enhancement modules are the first enhancement fusion module and the second enhancement fusion module. The input of the first enhancement fusion module is two feature maps with a size of 1 / 16 and 1 / 32 of the input image size from the four feature maps output by the feature extraction module. The input of the second enhancement fusion module is two feature maps with a size of 1 / 8 and 1 / 4 of the input image size from the four feature maps output by the feature extraction module.

[0018] Furthermore, in step 2, the processing procedure of each enhancement module for the input feature map is as follows: the image size is kept consistent by upsampling or downsampling, and then a stitching operation is performed; a convolution operation is performed on the stitched feature map; channel attention extraction and spatial attention extraction are performed on the convolution feature map respectively; and the channel attention feature map and the spatial attention feature map are multiplied together.

[0019] Furthermore, in step 2, the spatial attention extraction process for each enhancement module is as follows: global max pooling and global average pooling are performed on the feature map along the channel direction to obtain two feature maps respectively. After convolution operation, element-wise operation is performed, and finally sigmoid activation operation is performed to generate the final spatial attention feature map.

[0020] Furthermore, in step 2, the channel attention extraction process for each enhancement module is as follows: global max pooling and global average pooling are performed on the feature map along the height and width directions to obtain two feature maps respectively. After convolution, element-wise operation is performed, and finally sigmoid activation is performed to generate the final channel attention feature map.

[0021] Furthermore, in step 2, the detection head module uses an SSD detection head, and the loss function includes category loss and localization loss.

[0022] The advantages of this invention are:

[0023] 1. The remote sensing image target detection method based on the improved pyramid of the present invention adopts a two-level feature pyramid structure based on ResNet50 for feature extraction, and uses two enhancement modules to process the feature maps of the high-level and low-level dimensions respectively. Compared with the traditional pyramid network, while retaining the rich semantic information of the high level, it captures more comprehensive details such as edges, corners, and textures of the low level. For remote sensing targets with dense and overlapping conditions, it can accurately identify the edges of the objects and has higher detection accuracy.

[0024] 2. This invention adds spatial attention and channel attention extraction steps after splicing two adjacent feature maps after extraction, which comprehensively captures the key information in the features and provides a more comprehensive and effective feature extraction capability for the target detection model.

[0025] 3. This invention addresses the challenge of large differences in the scale of remote sensing targets. The enhancement module outputs dual-scale feature maps for detection by the detection head. The dual-scale feature maps retain key information from the top and bottom layers of the feature extraction module, taking into account both large and small targets in the image. This solves the problem of missed detection of small targets in the multi-scale target feature representation problem to a certain extent. Attached Figure Description

[0026] The above and / or other features and advantages of the present invention will become more readily understood from the following description with reference to the accompanying drawings, in which:

[0027] Figure 1 This is a flowchart of the remote sensing image target detection method based on the improved pyramid of the present invention;

[0028] Figure 2 This is an overall structural diagram of the remote sensing image target detection model in this invention;

[0029] Figure 3(a) is a schematic diagram of the structure of the first enhanced fusion module, and Figure 3(b) is a schematic diagram of the structure of the first enhanced fusion module;

[0030] Figure 4 This is a remote sensing image of the present invention in an example;

[0031] Figure 5 This is a diagram showing the detection results of the present invention in an example. Detailed Implementation

[0032] The present invention will now be described in detail with reference to the accompanying drawings and exemplary embodiments thereof. It should be noted that the following detailed description of the present invention is for illustrative purposes only and is not intended to limit the scope of the invention.

[0033] This invention provides a remote sensing image target detection method based on an improved pyramid, which can effectively extract the feature map of the target and improve the accuracy of remote sensing image target detection.

[0034] First, the overall reference Figure 1 The remote sensing image target detection method based on the improved pyramid, as an exemplary embodiment of the present invention, includes the following steps:

[0035] Step S1: Obtain the dataset of remote sensing images;

[0036] Step S2: Construct a remote sensing image target detection model based on the improved pyramid. The target detection model includes a feature extraction module, two enhancement modules, and two detection head modules.

[0037] The feature extraction module is a two-level feature pyramid structure based on ResNet50. The first-level pyramid structure is used to extract four feature maps of different sizes from the input image and obtain a fused feature map through the FPN structure. The second-level pyramid structure is used to perform feature fusion again on the obtained fused feature map from bottom to top.

[0038] The two enhancement modules take the two high-level feature maps and the two low-level feature maps from the four feature maps obtained by re-fusion as input, respectively, perform channel attention extraction and spatial attention extraction on the input feature maps, and multiply the channel attention feature maps and spatial attention feature maps to obtain the final dual-scale feature maps;

[0039] The two detection head modules are used to process the dual-scale feature maps, and output the target's category and bounding box information through classification and regression branches, respectively;

[0040] Step S3: Iteratively train the constructed object detection model using the acquired dataset. After the detection model converges, obtain the optimal network model parameters.

[0041] Step S4: Use the obtained optimal network model parameters to detect the target remote sensing image and obtain the classification and location information of the target in the image.

[0042] In step S1, the obtained remote sensing dataset can be a public dataset, including optical remote sensing datasets such as DIOR and NWPU VHR-10; or it can be a dataset taken by oneself. If a dataset taken by oneself is used, the image size needs to be normalized first, and then the normalized image needs to be color standardized. In order to prevent overfitting, the training set images taken by oneself should be no less than 800.

[0043] Furthermore, the acquired data can be divided into a training set and a test set, with a ratio of 8:2 or 7:3. Accordingly, in step S3, the processed training set data can be input into the model for iterative training. After the detection model converges, the optimal network model parameters are obtained, and then the optimal network model parameters are used to detect the test set data.

[0044] For step S2, Figure 2 The overall structure of the remote sensing image target detection model is shown. To effectively address the gradient vanishing and model saturation problems in the network model, the feature extraction module uses ResNet-50 as the backbone to extract features from the image. A total of four feature maps are extracted, resulting in four feature maps C1, C2, C3, and C4 from bottom to top. The image sizes can be 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size, respectively, with 64, 128, 256, and 512 channels. After obtaining feature maps of different sizes, an FPN (pyramid network) structure is used to perform upsampling and downsampling operations to ensure that the parts to be fused have the same size. Through lateral connections and fusion operations, the fused feature maps P1, P2, and P3 are obtained. The P4 image sizes are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size, with 64, 128, 256, and 512 channels, respectively. Traditional FPN networks retain rich semantic information in high-level layers. However, for remote sensing images, there are often dense targets in the images. In this case, it is necessary to accurately identify the edges of the objects. A large number of edge details are retained in the low-level network. In order to retain more detail information and ensure the detection accuracy of dense small targets in remote sensing images, this invention adds a bottom-up pyramid feature map fusion structure after the above process. The four output feature maps are M1, M2, M3, and M4, which are consistent with the size and number of channels of the feature maps P1, P2, P3, and P4 in the previous step.

[0045] There are two enhancement modules following the feature extraction module, namely the first enhancement fusion module and the second enhancement fusion module. Referring to Figure 3(a), the input of the first enhancement fusion module, i.e., enhancement module 1, is two of the four feature maps output by the feature extraction module, with sizes of 1 / 16 and 1 / 32 of the input image size; referring to Figure 3(b), the input of the second enhancement fusion module, i.e., enhancement module 2, is two of the four feature maps output by the feature extraction module, with sizes of 1 / 8 and 1 / 4 of the input image size. The input feature maps of each enhancement module are first upsampled or downsampled to maintain consistent image size, and then stitched together. When unifying the size of the input feature maps for enhancement module 1, the higher-level dimensions are prioritized, meaning the two sampled feature maps are each 1 / 32 the size of the input image. Conversely, when unifying the size of the input feature maps for enhancement module 2, the lower-level dimensions are prioritized, meaning the two sampled feature maps are each 1 / 4 the size of the input image. Assuming the stitched feature map size is H×W×2C, a 3×3 convolution operation is then performed with C kernels, resulting in a feature map X1 of size H×W×C, where the feature maps of enhancement module 1 and enhancement module 2 are X1 and X2, respectively. 11 and X 12 Then, channel attention extraction and spatial attention extraction are performed on the obtained feature maps respectively. There is no fully connected step in the extraction process of channel attention and spatial attention. Fully convolution is used instead, which reduces the model parameters and is more friendly to small or lightweight devices.

[0046] This invention addresses the challenge of large differences in the scale of remote sensing targets. The enhancement module outputs dual-scale feature maps for detection by the detection head. The dual-scale feature maps retain key information from the top and bottom layers of the feature extraction module, taking into account both large and small targets in the image. This solves the problem of missed detection of small targets in the multi-scale target feature representation problem to a certain extent.

[0047] The spatial attention extraction process is as follows: Feature map X1 is subjected to global max pooling (MP) and global average pooling (AP) along the C (channel) direction, resulting in two H×W×1 feature maps. These are then subjected to 3×3 convolution operations followed by element-wise operations. Finally, a sigmoid activation operation is performed to generate the final spatial attention feature map S1. The spatial attention feature maps of enhancement module 1 and enhancement module 2 are S1 and S2, respectively. 11 and S 12 The dimensions are H×W×1.

[0048] The formula for spatial attention extraction is:

[0049] S1=σ(Conv(AP C (X1))+(Conv(MP C(X1))

[0050] In the formula, σ represents the sigmoid operation, Conv represents the convolution calculation, and AP C This indicates a global average pooling operation along the C (channel) direction, MP C This indicates that a global max pooling operation is performed along the C (channel) direction.

[0051] The channel attention extraction process is as follows: Feature map X1 is subjected to global max pooling (MP) and global average pooling (AP) along the H and W (height and width) directions, respectively, to obtain two 1×1×C feature maps. These are then subjected to 3×3 convolution operations followed by element-wise operations. Finally, a sigmoid activation operation is performed to generate the final channel attention feature map S2. The spatial attention feature maps of enhancement module 1 and enhancement module 2 are S2 and S2, respectively. 21 and S 22 The dimensions are 1×1×C. By stitching together two adjacent feature maps after extraction, spatial attention and channel attention steps are added to comprehensively capture the key information in the features, providing a more comprehensive and effective feature extraction capability for the object detection model.

[0052] The channel attention extraction formula is:

[0053] S2=σ(Conv(AP HW (X1))+(Conv(MP HW (X1))

[0054] In the formula, σ represents the sigmoid operation, Conv represents the convolution calculation, and AP HW This indicates a global average pooling operation along the H and W (height and width) directions, MP HW This indicates that global max pooling is performed along the H and W (height and width) directions.

[0055] The formula for the global max pooling (MP) operation described above is:

[0056]

[0057] In the formula, y kij Represents the relationship between the k-th feature map and R. ij Max pooling output value, x kpq Represents a rectangular region R ij The value of the element located at (p, q).

[0058] The formula for the global average pooling (AP) operation described above is as follows:

[0059]

[0060] In the formula, g kij Represents the relationship between the k-th feature map and R. ij Average pooling output value, t kpq Represents a rectangular region R ij The element value at (p, q) in the array, |R ij | indicates the number of all elements.

[0061] After performing spatial attention extraction and channel attention extraction operations, the feature map X is... 11 (or X) 12 ) respectively with spatial attention feature map S 11 (or S) 12 ) and channel attention feature map S 21 (or S) 22 Multiply by , and you get the final feature map X. 12 (or X) 22 The feature maps output by the first and second enhanced fusion modules are X and X, respectively. 21 and X 22 That is, a dual-scale feature map.

[0062] Following the enhancement module is the detection head module, which uses an SSD detection head. The loss function includes category loss and localization loss. The feature map X2 is fed into the detection head, and finally compared with the label value to calculate the loss.

[0063] The formula for category loss is:

[0064] F L (p t )=-α(1-p t ) γ log(p t )

[0065] In the formula, p t Indicates the confidence level. p is the prediction probability, y = 1 represents the foreground and y ≠ 1 represents the background, α is the weighting parameter, which can suppress the imbalance between the number of positive and negative samples, and γ is the hyperparameter, which takes values ​​in the range of [0, 5].

[0066] The formula for positioning loss is:

[0067]

[0068] In the formula, y i It is the actual value. This is the predicted value, and n is the number of sample points.

[0069] Step S3 may specifically include the following steps:

[0070] Store the training and test datasets in separate folders named "train" and "test," and move them into the "data" folder of the algorithm package. Run the "train.py" file in the terminal to start training the network. During training, the feature extraction module is first called to extract features from the input image. Then, the enhancement module is called to fuse high-level semantic information with low-level detail information. Finally, the detection head network is called to output the fused feature map X2 to the loss function to calculate the loss value between the original and the true value. This completes one forward propagation. Subsequently, the neural network adjusts the corresponding parameter values ​​in the model based on the loss value, continuously updating and iterating until the preset number of iterations is completed. At this point, the curve of the loss value versus time tends to flatten and converge. The entire model at this point is the network model with optimal parameters that has been trained.

[0071] Step S4 includes the following steps:

[0072] The remote sensing image of the target is used as the detection model trained in step S3 for forward propagation to obtain the predicted target's location information and class confidence information. Finally, the non-maximum suppression method (NMS) is used to filter the generated bounding boxes to obtain the bounding box and class confidence of each target.

[0073] This invention is based on Figure 4 The example image of the remote sensing image in the image is used as input, and the example image of the detection result is shown below. Figure 5 As shown, the effectiveness of the method proposed in this invention is verified.

[0074] As described above, this invention uses a two-level feature pyramid structure based on ResNet50 for feature extraction, and employs two enhancement modules to process the feature maps of the high-level and low-level dimensions respectively. Compared with traditional pyramid networks, it retains the rich semantic information of the high-level layers while capturing more comprehensive details such as edges, corners, and textures of the low-level layers. For remote sensing targets that are dense and overlapping, it can accurately identify the edges of the objects and has higher detection accuracy.

[0075] Finally, it should be noted that the features mentioned and / or shown in the above description of exemplary embodiments of the present invention can be combined in the same or similar manner with one or more other embodiments, combined with features in other embodiments, or substituted for corresponding features in other embodiments. These combined or substituted technical solutions should also be considered to be included within the scope of protection of the present invention.

Claims

1. A remote sensing image target detection method based on an improved pyramid, characterized in that, Includes the following steps: Step 1: Obtain the dataset of remote sensing images; Step 2: Construct a remote sensing image target detection model based on an improved pyramid. The target detection model includes a feature extraction module, two enhancement modules, and two detection head modules. The feature extraction module is a two-level feature pyramid structure based on ResNet50. The first-level pyramid structure is used to extract four feature maps of different sizes from the input image and obtain a fused feature map through the FPN structure. The second-level pyramid structure is used to perform feature fusion again on the obtained fused feature map from bottom to top, resulting in four feature maps with image sizes of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image size, respectively. The two enhancement modules are a first enhancement fusion module and a second enhancement fusion module. The input of the first enhancement fusion module is two feature maps with a size of 1 / 16 and 1 / 32 of the input image size from the four feature maps output by the feature extraction module. The input of the second enhancement fusion module is two feature maps with a size of 1 / 8 and 1 / 4 of the input image size from the four feature maps output by the feature extraction module. The two enhancement modules maintain consistent image sizes through upsampling or downsampling before stitching. Specifically, the first enhancement fusion module uses high-level features as the primary size for input feature maps, resulting in two sampled feature maps with a size of 1 / 32 of the input image. The second enhancement fusion module uses low-level features as the primary size for input feature maps, resulting in two sampled feature maps with a size of 1 / 4 of the input image. Then, a convolution operation is performed on the stitched feature maps. Following this, channel attention and spatial attention are extracted from the convolutional feature maps, and the channel attention feature maps are multiplied by the spatial attention feature maps to obtain the final dual-scale feature maps. The two detection head modules are used to process the dual-scale feature map, and output the target's category and bounding box information through classification and regression branches, respectively. Step 3: Iteratively train the constructed object detection model using the acquired dataset. After the detection model converges, obtain the optimal network model parameters. Step 4: Use the obtained optimal network model parameters to detect the target remote sensing image and obtain the classification and location information of the target in the image.

2. The remote sensing image target detection method based on the improved pyramid according to claim 1, characterized in that, In step 1, the obtained dataset is either a publicly available dataset or a dataset taken by the user.

3. The remote sensing image target detection method based on the improved pyramid according to claim 2, characterized in that, Step 1 also includes: splitting the acquired dataset into a training set and a test set, with a split ratio of 8:2 or 7:

3.

4. The remote sensing image target detection method based on the improved pyramid according to claim 1, characterized in that, In step 2, the spatial attention extraction process for each enhancement module is as follows: global max pooling and global average pooling are performed on the feature map along the channel direction to obtain two feature maps respectively. After convolution operation, element-wise operation is performed, and finally sigmoid activation operation is performed to generate the final spatial attention feature map.

5. The remote sensing image target detection method based on the improved pyramid according to claim 1, characterized in that, In step 2, the channel attention extraction process for each enhancement module is as follows: global max pooling and global average pooling are performed on the feature map along the height and width directions to obtain two feature maps respectively. After convolution, element-wise operation is performed, and finally sigmoid activation is performed to generate the final channel attention feature map.

6. The remote sensing image target detection method based on the improved pyramid according to claim 1, characterized in that, In step 2, the detection head module uses an SSD detection head, and the loss function includes category loss and localization loss.

Images