Multi-scale detection optimization method based on packet convolution attention network
By introducing a packet convolutional attention network module into the remote sensing image detection model, the problem of incomplete feature extraction caused by the difference in target scales in remote sensing images is solved, the detection accuracy and performance are significantly improved, and more accurate object detection is achieved.
Patent Information
- Application Number
- CN202411269948.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-11
AI Technical Summary
Large scale differences in the targets in remote sensing images lead to incomplete feature extraction and low detection accuracy.
Build a grouping convolutional attention network module, introduce it into an object detection model based on convolutional neural network, feature fusion is performed through feature grouping module and attention extraction module, and coordinated training data sets are optimized to improve the representation of multi-scale features. ability.
The remote sensing image detection accuracy is significantly improved and the performance of the detection model is improved. Especially when dealing with large-scale differences, the detection results are more accurate and the prediction speed is not reduced.
Smart Images

Figure CN119068355A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of remote sensing image building detection technology, and more specifically to a multi-scale detection optimization method and system based on a grouped convolutional attention network. The present invention can be used to enhance the detection accuracy in remote sensing images. Background Art
[0002] With the rapid development of high-resolution satellites, it has become important to obtain useful target detection results from remote sensing images and understand satellite images, and it has become a research hotspot in the current remote sensing field. Among them, building detection is of great significance to rural revitalization, urban planning, disaster detection, ecological environment management and military. Target detection in remote sensing images can provide basic data for many remote sensing applications. For example, building detection in remote sensing images can provide important illegal building information for illegal building monitoring in urban management. Due to the powerful feature extraction ability of convolutional neural networks, target detection algorithms based on deep convolutional neural networks are currently popular in the field of building detection. However, remote sensing images are different from general images. For example, the width of remote sensing images is relatively large, and the width of ordinary images can reach tens of kilometers. As a result, the scale of targets that can be distinguished in remote sensing images varies greatly, which brings difficulties and challenges to target detection in remote sensing images.
[0003] In 2014, Girshick et al. published a paper titled "Rich feature hierarchies for accurate object detection and semantic segmentation" (In Proceedings of the IEEE conference on computer vision and pattern recognition), which successfully applied deep learning methods to object detection. It used the traditional selective search algorithm to crop the image, then extracted features through a neural network, and finally classified it through SVM. However, since it needed to feed the cropped image into the network 2,000 times, it greatly increased the network overhead and the detection speed was slow.
[0004] In 2015, Girshick drew on the idea of SPPNet and published the paper "Fast r-cnn" (In Proceedings of the IEEE international conference on computer vision). He proposed Fast R-CNN by using mapping to improve R-CNN. First, he took a picture, selected the suggestion box using selective search, and then input the original picture into the convolutional neural network to obtain the feature map. For each suggestion box, he found the corresponding position from the feature map (just search according to the proportion), cut out the feature box (the depth remains unchanged), divided each feature box into (7x7) grids, and then stretched each matrix into a vector, which was used as the input of the subsequent fully connected layer. The output of the fully connected layer has two, calculating the classification score and bounding box regression, and finally using the non-maximum suppression method to select a few boxes from the output score matrix. This method not only improves the detection speed, but also improves the detection accuracy. However, it does not solve the end-to-end problem, and the detection speed is still relatively slow.
[0005] Shaoqing Ren, Kaiming He and others published the paper "Faster r-cnn: Towards real-time object detection with region proposal networks" (Advances in neural information processing systems), which continued to improve on the basis of Fast R-CNN. They introduced a region proposal network (RPN) based on Fast R-CNN, and generated proposal boxes for the extracted features through the RPN network. Then the generated proposal boxes and feature maps were sent to ROIpooling for cropping, and then sent to the Fast R-CNN prediction head for prediction. This method makes Fast R-CNN an end-to-end object detection network, which significantly improves the detection speed while maintaining the detection accuracy.
[0006] In the SKNet proposed by Li et al. in their paper "Selective kernel networks", a network module called SK unit was designed, which contains multiple branches with different kernel sizes. These branches are fused using the softmax attention mechanism under the guidance of information, thereby achieving adaptive adjustment of the receptive field size. It shows that grouped convolution helps improve the representation ability of multi-scale features.
[0007] Qin et al. published a paper titled "Fcanet: Frequency channel attention networks", proposing a multi-spectral channel attention framework that generalizes the channel attention mechanism from the traditional scalar representation to multiple frequency component representations in the frequency domain. This framework can capture channel feature information more comprehensively and improve the representation ability of the model. This shows that the channel attention mechanism can focus on the multi-scale features that are most useful for the task, which also demonstrates its ability to capture multi-scale features.
[0008] However, compared with images of ordinary scenes, the scales of targets that can be distinguished in remote sensing images vary greatly, so the features extracted by general convolutional neural network-based detection models are imperfect during the feature extraction stage, resulting in unsatisfactory subsequent detection accuracy. Summary of the invention
[0009] The purpose of the present invention is to propose a multi-scale detection optimization method based on a grouped convolutional attention network to address the shortcomings of the existing technology, which solves the problems of incomplete feature extraction and low detection accuracy caused by a series of reasons such as the large scale differences of targets in remote sensing images.
[0010] The implementation idea of the present invention is: first, construct a grouped convolutional attention network module, and introduce the constructed grouped convolutional attention network module into a general object detection model based on a convolutional neural network. Then, divide the object detection data set, perform prediction processing on the images in the data set, and use the training data to perform collaborative training with the object detection model and the decoder. Finally, send the verification data to the detection model for detection to obtain the detection result.
[0011] The specific steps of the present invention are as follows: Step 1: Build a grouped convolutional attention network module: Construct a grouped convolutional attention network module consisting of a feature grouping module and an attention extraction module. The structure of the feature grouping module is: 1×1 convolution → BN layer → grouped convolution block; the structure of the grouped convolution block is to set the output channel to 1 / 8 of the input channel through a normal convolution layer and set a separate grouping number for each convolution block; The structure of the attention extraction module is: dilated channel attention block → feature splicing layer → softmax layer; the structure of the dilated channel attention block is: relu function → 1×1 convolution → 3×3 convolution → ECA module; the ECA module is composed of global average pooling and one-dimensional convolution; the feature splicing layer splices the attention weights obtained from the dilated channel attention module; Then, the features obtained by the feature grouping module are concatenated and fused with the attention weights obtained in the attention extraction module. The obtained features are followed by a BN layer and a ReLU activation function. Step 2: Introduce the grouped convolutional attention network module into the general convolutional neural network-based object detection model: The constructed grouped convolutional attention network module is introduced into the backbone network of the object detection model. After placing it in three different stages of the backbone network, the obtained features are fused with the original features. Step 3: Divide the object detection dataset: Select some annotated optical remote sensing images with unlimited length and width (each batch will preprocess the images, scale the images in the batch, and then fill them into images of uniform size). If the dataset category is difficult to train, more images are required. A simple dataset can use 1000+ images. Then 80% of the images form the training set, and 20% of the images form the validation set and test set. Step 4: Perform prediction processing on the images in the dataset: First, the data in each mini-batch is horizontally flipped with a probability of 50%. Then, the length and width of the images in each mini-batch are fixed, the length is scaled to an integer multiple of 32 between 400 and 600, and the width is padded to an integer multiple of 32 between 400 and 600. Step 5: Use the training data to co-train the object detection model and decoder: The first step is to input the preprocessed data into the backbone network (resnet50 or cspnext, etc.) for feature extraction to generate feature maps C3, C4, and C5; In the second step, the feature map is sent to the object detection head (YOLO detection head or Fast R-CNN detection head, etc.) to predict the category and bounding box, and then the cross entropy loss and bounding box loss are calculated with the true label; In the third step, the feature maps C3, C4, and C5 generated in the first step are sent to the grouped convolutional attention network module for feature grouping module, attention extraction module, and feature fusion, and then the feature map of the same size as the image after preprocessing is output, and then the mean square error loss function between the preprocessed image and the feature map is calculated; The fourth step is to simultaneously pass back the loss functions of the second and third steps to update the entire network structure; Step 6: Send the verification data to the detection model for detection: In the first step, the preprocessed data is input into the backbone network for feature extraction to generate feature maps C3, C4, and C5; In the second step, feature maps C3, C4, and C5 are sent to the object detection head for category and bounding box prediction, and then processed by non-maximum suppression (NMS) to generate detection results; Step 7, get the test results: Map the detection results obtained in step 6 to the corresponding positions of the original image and mark the predicted category information.
[0012] Compared with the prior art, the present invention has the following advantages: First, the structure has strong embedding properties. It can be embedded in some mainstream object detection models, such as RtmDet and Faster R-CNN. Second, although it takes longer to train than the detection model without the grouped convolutional attention network module, it does not increase the prediction time, and is not inferior to the prediction results of deeper backbone networks. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 This is a flow chart of the multi-scale detection optimization method based on the grouped convolutional attention network;
[0014] Figure 2 It is a test result diagram of the present invention. DETAILED DESCRIPTION
[0015] The present invention will be described in detail below in conjunction with the accompanying drawings.
[0016] Reference Figure 1 , the steps of the present invention are further described.
[0017] Step 1: Build a grouped convolutional attention network module: Construct a grouped convolutional attention network module consisting of a feature grouping module and an attention extraction module. The structure of the feature grouping module is: 1×1 convolution → BN layer → grouped convolution block; the structure of the grouped convolution block is to set the output channel to 1 / 8 of the input channel through a normal convolution layer and set a separate grouping number for each convolution block; The structure of the attention extraction module is: dilated channel attention block → feature splicing layer → softmax layer; the structure of the dilated channel attention block is: relu function → 1×1 convolution → 3×3 convolution → ECA module; the ECA module is composed of global average pooling and one-dimensional convolution; the feature splicing layer splices the attention weights obtained from the dilated channel attention module; Then, the features obtained by the feature grouping module are concatenated and fused with the attention weights obtained in the attention extraction module. The obtained features are followed by a BN layer and a ReLU activation function. Step 2: Introduce the grouped convolutional attention network module into the general convolutional neural network-based object detection model: The constructed grouped convolutional attention network module is introduced into the backbone network of the object detection model. After placing it in three different stages of the backbone network, the obtained features are fused with the original features. Step 3: Divide the object detection dataset: Select some annotated optical remote sensing images with unlimited length and width (each batch will preprocess the images, scale the images in the batch, and then fill them into images of uniform size). If the dataset category is difficult to train, more images are required. A simple dataset can use 1000+ images. Then 80% of the images form the training set, and 20% of the images form the validation set and test set. Step 4: Perform prediction processing on the images in the dataset: First, the data in each mini-batch is horizontally flipped with a probability of 50%. Then, the length and width of the images in each mini-batch are fixed, the length is scaled to an integer multiple of 32 between 400 and 600, and the width is padded to an integer multiple of 32 between 400 and 600. Step 5: Use the training data to co-train the object detection model and decoder: The first step is to input the preprocessed data into the backbone network (resnet50 or cspnext, etc.) for feature extraction to generate feature maps C3, C4, and C5; In the second step, the feature map is sent to the object detection head (YOLO detection head or Fast R-CNN detection head, etc.) to predict the category and bounding box, and then the cross entropy loss and bounding box loss are calculated with the true label; In the third step, the feature maps C3, C4, and C5 generated in the first step are sent to the grouped convolutional attention network module for feature grouping module, attention extraction module, and feature fusion, and then the feature map of the same size as the image after preprocessing is output, and then the mean square error loss function between the preprocessed image and the feature map is calculated; The fourth step is to simultaneously pass back the loss functions of the second and third steps to update the entire network structure; Step 6: Send the verification data to the detection model for detection: In the first step, the preprocessed data is input into the backbone network for feature extraction to generate feature maps C3, C4, and C5; In the second step, feature maps C3, C4, and C5 are sent to the object detection head for category and bounding box prediction, and then processed by non-maximum suppression (NMS) to generate detection results; Step 7, get the test results: Map the detection results obtained in step 6 to the corresponding positions of the original image and mark the predicted category information.
[0018] The effects of the present invention are further described below in conjunction with simulation.
[0019] 1. Simulation experiment conditions: The simulation experiment conditions of the present invention are as follows: server GPU: GeForce RTX3090, video memory 24G; The software platform of the simulation experiment of the present invention: ubuntu18.04 system, python 3.10, pytorch-gpu2.0.0.
[0020] 2. Analysis of simulation content and experimental results: The simulation of the present invention is to detect remote sensing images by using several existing technologies (Faster R-CNN detection method, RtmDet method, etc.) and their introduced grouped convolutional attention network modules to obtain detection results; the data sets used in the simulation are: SSDD, DIOR-R and MS-Building.
[0021] The two existing technologies used in the simulation experiment are: The Faster R-CNN detection method is an object detection method proposed by Shaoqing Ren, Kaiming He and others in "Ren, S., He, K., Girshick, R., Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 91-99.", referred to as the Faster R-CNN detection method; The Cascade R-CNN detection method is an object detection method proposed by Cai Z et al. in "Cai Z, Vasconcelos N. Cascade r-cnn: Delving into high quality object detection[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 6154-6162." The Dynamic R-CNN detection method is an object detection algorithm proposed by Zhang et al. in "Zhang H, Chang H, Ma B, et al. Dynamic R-CNN: Towards high quality object detection via dynamic training [C] / / Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16. Springer International Publishing, 2020: 260-275." The Libra R-CNN detection method is an object detection method proposed by Pang et al. in "Pang J, Chen K, Shi J, et al. Librar-cnn: Towards balanced learning for object detection[C] / / Proceedings of theIEEE / CVF conference on computer vision and pattern recognition. 2019: 821-830." The Grid R-CNN detection method is an object detection method proposed by Zhang et al. in "Lu X, Li B, Yue Y, et al. Grid r-cnn[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 7363-7372." The YOLOX detection method is an object detection method proposed by Lu et al. in "Ge Z, Liu S, Wang F, et al. Yolox: Exceedingyolo series in 2021[J]. arXiv preprint arXiv:2107.08430, 2021." The RtmDet detection method is an object detection method proposed by Lyu C et al. in “Lyu C, Zhang W, Huang H, et al. Rtmdet:An empirical study of designing real-time object detectors[J]. arXiv preprintarXiv:2212.07784, 2022.”
[0022] The input images used in the simulation experiment of the present invention are the public SSDD dataset, DIOR-R dataset and MS-Building dataset. The SSDD dataset was released by Wuhan University in 2017. Its images were extracted from RadarSat-2, TerraSAR-X, and Sentinel-1, including 1,160 SAR ship images with an image size of 500×500×3. We use 1,000 ship images as a training set and 160 images as a validation set. The DIOR dataset was released by Northwestern Polytechnical University in 2019. Its images were extracted from Google Earth with an image size of , including 20 categories and a total of 190,288 targets. We use 11,725 remote sensing images as a training set and 11,738 images as a validation set for testing and evaluation. The MS-Building dataset was released by Jinan University in 2023. Its images come from Gaofen Satellite 1 and Gaofen Satellite 2, and contain nearly 1,200 panchromatic images, multispectral images, and fused images in total.
[0023] Simulation experiment 1 introduces the grouped convolutional attention network module into each model and compares the detection results on DIOR-R. Simulation experiment 2 introduces the grouped convolutional attention network module into each model and compares the detection results on MS-Building. In order to verify the effect of the present invention on remote sensing image detection, the coco evaluation index is used to evaluate the detection results. The calculation results are plotted in the following table.
[0024] Table 1. Comparison of simulation experimental results of the present invention on the DIOR-R dataset Method AP (%) AP50 (%) AP75(%) Faster R-CNN 39.0 64.7 39.7 Cascade R-CNN 42.3 66.2 44.4 Libra R-CNN 40.3 65.8 41.5 Dynamic R-CNN 40.5 64.8 42.0 Grid R-CNN 42.5 64.2 45.2 RtmD 52.8 75.9 56.4 YOLOX 50.6 78.2 78.2 Faster R-CNN+ Improved Module 39.2 65.0 39.7 Cascade R-CNN+ Improved Module 42.6 66.7 44.4 Libra R-CNN+ Improved Module 40.5 66.2 41.7 Dynamic R-CNN+ Improved Module 40.8 65.1 42.5 Grid R-CNN+ Improved Module 42.0 65.7 44.4 RtmDet+ Improved Module 53.6 76.6 57.2 YOLOX+ Improvement Module 51.2 78.9 54.6 Table 2. Comparison of simulation results of the present invention on the MS-Builiding dataset Method AP (%) AP50 (%) AP75(%) Faster R-CNN 39.7 76.9 36.9 Cascade R-CNN 40.3 77.9 37.8 Libra R-CNN 39.5 75.9 37.1 Dynamic R-CNN 37.3 72.7 33.2 Grid R-CNN 38.6 76.4 34.3 RtmD 41.0 77.6 39.5 YOLOX 39.8 76.6 37.7 Faster R-CNN+ Improved Module 40.0 77.9 36.2 Cascade R-CNN+ Improved Module 41.9 79.6 39.0 Libra R-CNN+ Improved Module 40.7 77.0 39.0 Dynamic R-CNN+ Improved Module 38.3 74.2 34.7 Grid R-CNN+ Improved Module 38.5 77.1 33.5 RtmDet+ Improved Module 43.5 79.9 42.1 YOLOX+ Improvement Module 39.7 78.1 37.4 Combined with Table 1 above, it is not difficult to see that the detection algorithm that introduces the grouped convolutional attention network module for training has significantly improved the detection accuracy compared to the one that has not been introduced. On the DIOR-R dataset, the best AP50 was obtained on the YOLOX model after embedding the grouped convolutional attention network module in our experiment, with AP50 reaching 78.9%. On RtmDet, mAP reached 53.6% and AP75 reached 57.2%. Combined with Table 2, the table shows the experimental results of mainstream detection models on the MS_Building dataset. We can observe that the performance of almost all models has been improved to a greater or lesser extent after embedding the grouped convolutional attention network. On RtmDet, mAP, AP50 and AP75 can reach 43.5%, 79.9% and 42.1% respectively.
[0025] Combine the following Figure 2 The effect of this simulation experiment is further explained.
[0026] Figure 2 (a)~(d) in the figure are the prediction results using the RtmDet network, and (e)~(h) are the prediction results after the improved model. Figure 2 We can see that the prediction results of images (a) to (d) in the marked area have missed detections and false detections, while the detection results of images (e) to (h) on the same image have fewer missed detections and false detections. This shows that the present invention is superior to the original detection model without the present invention in terms of accuracy, missed detections and false detections.
Claims
1. A multi-scale detection optimization method based on grouped convolutional attention network, characterized in that: The method comprises: The grouped convolutional attention network module and the object detection model are used for collaborative training to enhance the feature extraction network of the detection model and improve the detection accuracy. The specific steps of this method include the following: Step 1: Build a grouped convolutional attention network module: Construct a grouped convolutional attention network module consisting of a feature grouping module and an attention extraction module. The structure of the feature grouping module is: 1×1 convolution → BN layer → grouped convolution block; the structure of the grouped convolution block is to set the output channel to 1 / 8 of the input channel through a normal convolution layer and set a separate grouping number for each convolution block; The structure of the attention extraction module is: dilated channel attention block → feature splicing layer → softmax layer; the structure of the dilated channel attention block is: relu function → 1×1 convolution → 3×3 convolution → ECA module; the ECA module is composed of global average pooling and one-dimensional convolution; the feature splicing layer splices the attention weights obtained from the dilated channel attention module; Then, the features obtained by the feature grouping module are concatenated and fused with the attention weights obtained in the attention extraction module. The obtained features are followed by a BN layer and a ReLU activation function. Step 2: Introduce the grouped convolutional attention network module into the general convolutional neural network-based object detection model: The constructed grouped convolutional attention network module is introduced into the backbone network of the object detection model. After placing it in three different stages of the backbone network, the obtained features are fused with the original features. Step 3: Divide the object detection dataset: Select some annotated optical remote sensing images with unlimited length and width (each batch will preprocess the images, scale the images in the batch, and then fill them into images of uniform size). If the dataset category is difficult to train, more images are required. A simple dataset can use 1000+ images. Then 80% of the images form the training set, and 20% of the images form the validation set and test set. Step 4: Perform prediction processing on the images in the dataset: First, the data in each mini-batch is horizontally flipped with a probability of 50%. Then, the length and width of the images in each mini-batch are fixed, the length is scaled to an integer multiple of 32 between 400 and 600, and the width is padded to an integer multiple of 32 between 400 and 600. Step 5: Use the training data to co-train the object detection model and decoder: The first step is to input the preprocessed data into the backbone network (resnet50 or cspnext, etc.) for feature extraction to generate feature maps C3, C4, and C5; In the second step, the feature map is sent to the object detection head (YOLO detection head or Fast R-CNN detection head, etc.) to predict the category and bounding box, and then the cross entropy loss and bounding box loss are calculated with the true label; In the third step, the feature maps C3, C4, and C5 generated in the first step are sent to the grouped convolutional attention network module for feature grouping module, attention extraction module, and feature fusion, and then the feature map of the same size as the image after preprocessing is output, and then the mean square error loss function between the preprocessed image and the feature map is calculated; The fourth step is to simultaneously pass back the loss functions of the second and third steps to update the entire network structure; Step 6: Send the verification data to the detection model for detection: In the first step, the preprocessed data is input into the backbone network for feature extraction to generate feature maps C3, C4, and C5; In the second step, feature maps C3, C4, and C5 are sent to the object detection head for category and bounding box prediction, and then processed by non-maximum suppression (NMS) to generate detection results; Step 7, get the test results: Map the detection results obtained in step 6 to the corresponding positions of the original image and mark the predicted category information.
2. The multi-scale detection optimization method based on grouped attention network according to claim 1, characterized in that: The construction of the grouped convolutional attention network module and the embedding training of the grouped convolutional attention network module described in steps 1, 2 and 5, the design of the corresponding grouped convolutional attention network for the backbone network of the detection model, and the use of the constructed grouped convolutional attention network module for collaborative training with the object detection model to enhance the feature extraction capability in the backbone network and make the detection stronger.
Citation Information
Patent Citations
Image segmentation method based on lightweight recursive non-local self-attention
CN112132844A
Coronary CTA stenosis detection method based on YOLOv5-LA
CN116612087A
Medical image segmentation method based on joint encoder network GPA-TUNet
CN117197151A
Dual-polarized SAR ship detection method based on grouped mixed attention
CN117422946A
Target detection method and apparatus, storage medium, and terminal
WO2022111352A1
Cited By
Fixed-wing unmanned aerial vehicle timing sequence state prediction method and system fusing multi-scale embedding and grouping channel attention, and medium
CN121808718A
Fixed-wing unmanned aerial vehicle time series state prediction method and system fusing multi-scale embedding and grouped channel attention, and medium
CN121808718B
