A multi-scale detection optimization method based on grouped convolutional attention network

Abstract

This application discloses a multi-scale detection optimization method and system based on a grouped convolutional attention network. The implementation steps of the method are as follows: constructing a grouped convolutional attention network module; introducing the grouped convolutional attention network module into a general convolutional neural network-based object detection model; dividing the object detection data set; performing predictive processing on the images in the data set; using the training data to co-train the object detection model and the grouped convolutional attention network module; feeding the verification data into the detection model for detection; and obtaining the detection results. The present invention utilizes the constructed grouped convolutional attention network module and a general deep convolutional neural network-based object detection model for co-training to enhance the detection model's feature extraction network for multi-scale object detection, thereby improving object detection accuracy without increasing computational cost.

Description

Technical Field

[0001] This application relates to the field of building detection technology in remote sensing images, and more specifically to a multi-scale detection optimization method and system based on a grouped convolutional attention network. The present invention can be used to enhance detection accuracy in remote sensing images. Background Art

[0002] With the rapid development of high-resolution satellites, obtaining useful target detection results from remote sensing imagery and thereby understanding satellite imagery has become increasingly important and a research hotspot in the field of remote sensing. Building detection, in particular, is crucial for rural revitalization, urban planning, disaster detection, ecological and environmental management, and the military. Object detection in remote sensing imagery can provide fundamental data for many remote sensing applications. For example, building detection in remote sensing imagery can provide crucial information about illegal buildings for monitoring in urban management. Due to the powerful feature extraction capabilities of convolutional neural networks, object detection algorithms based on deep convolutional neural networks are currently popular in the field of building detection. However, remote sensing images differ from general images in that they are typically large, with typical image widths reaching tens of kilometers. This results in a wide range of discernible object scales in remote sensing images, making target detection in remote sensing imagery difficult and challenging.

[0003] In 2014, Girshick et al. published the paper "Rich feature hierarchies for accurate object detection and semantic segmentation" (In Proceedings of the IEEE conference on computer vision and pattern recognition), successfully applying deep learning methods to object detection. This approach used a traditional selective search algorithm to crop images, then extracted features using a neural network, and finally performed classification using a support vector machine (SVM). However, this approach required feeding the cropped image into the network 2,000 times, significantly increasing network overhead and resulting in slower detection speeds.

[0004] In 2015, Girshick drew on the ideas of SPPNet and published the paper "Fast R-CNN" (In Proceedings of the IEEE international conference on computer vision). Fast R-CNN, an improvement on R-CNN using a mapping approach, was proposed. First, it takes an image and uses selective search to select suggested boxes. The original image is then fed into a convolutional neural network to obtain a feature map. For each suggested box, the corresponding position is found in the feature map (using a proportional search). The feature boxes are then cropped (the depth remains unchanged). Each feature box is divided into a (7x7) grid. Each matrix is ​​then stretched into a vector, which serves as the input to the subsequent fully connected layer. The fully connected layer outputs two vectors: a classification score and bounding box regression. Finally, the non-maximum suppression method is used on the output score matrix to select a small number of boxes. This method not only improves detection speed but also accuracy. However, it does not address the end-to-end problem, and detection speed remains relatively slow.

[0005] Shaoqing Ren, Kaiming He, and others published the paper "Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks" (Advances in Neural Information Processing Systems), further improving on Fast R-CNN by introducing a Region Proposal Network (RPN). The RPN generates proposal boxes based on the extracted features. The generated proposal boxes and feature maps are then processed by ROIpooling for cropping and then fed into the Fast R-CNN prediction head for prediction. This approach makes Fast R-CNN an end-to-end object detection network, significantly improving detection speed while maintaining accuracy.

[0006] Li et al. published a paper titled "Selective Kernel Networks," which proposed the SKNet framework. They designed a network module called the SK unit, which consists of multiple branches with varying kernel sizes. These branches are fused using a softmax attention mechanism, guided by information, to achieve adaptive receptive field size adjustment. This demonstrates that grouped convolutions can improve the representation of multi-scale features.

[0007] Qin et al. published the paper "Fcanet: Frequency Channel Attention Networks," proposing a multispectral channel attention framework that generalizes the channel attention mechanism from traditional scalar representation to multiple frequency components in the frequency domain. This framework captures channel feature information more comprehensively, improving the model's representational capabilities. This demonstrates that the channel attention mechanism can focus on the multi-scale features that are most useful for a task, demonstrating its effectiveness in capturing multi-scale features.

[0008] However, compared with images of ordinary scenes, the scales of targets that can be distinguished in remote sensing images vary greatly, which makes the features extracted by general convolutional neural network-based detection models incomplete during the feature extraction stage, resulting in unsatisfactory detection accuracy. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of the existing technology and propose a multi-scale detection optimization method based on a grouped convolutional attention network, which solves the problems of incomplete feature extraction and low detection accuracy caused by a series of reasons such as the large scale differences of targets in remote sensing images.

[0010] The implementation strategy of this invention is as follows: First, a grouped convolutional attention network module is constructed and introduced into a general convolutional neural network-based object detection model. Then, the object detection dataset is divided, and prediction processing is performed on the images in the dataset. The object detection model and decoder are collaboratively trained using the training data. Finally, the verification data is fed into the detection model for detection, resulting in detection results.

[0011] The specific steps of the present invention are as follows:

[0012] Step 1: Build a grouped convolutional attention network module:

[0013] Construct a grouped convolutional attention network module consisting of a feature grouping module and an attention extraction module. The structure of the feature grouping module is as follows: 1×1 convolution → BN layer → grouped convolution block; the structure of the grouped convolution block is to set the output channel to 1 / 8 of the input channel through a normal convolution layer and set a unique grouping number for each convolution block;

[0014] The structure of the attention extraction module is as follows: dilated channel attention block → feature splicing layer → softmax layer; the structure of the dilated channel attention block is as follows: relu function → 1×1 convolution → 3×3 convolution → ECA module; the ECA module consists of global average pooling and one-dimensional convolution; the feature splicing layer splices the attention weights obtained from the dilated channel attention module;

[0015] Then, the features obtained by the feature grouping module are concatenated and fused with the attention weights obtained in the attention extraction module. The obtained features are followed by a BN layer and a ReLU activation function.

[0016] Step 2: Introduce the grouped convolutional attention network module into the general convolutional neural network-based object detection model:

[0017] The constructed grouped convolutional attention network module is introduced into the backbone network of the object detection model. After placing it at three different stages in the backbone network, the obtained features are fused with the original features.

[0018] Step 3: Divide the object detection dataset:

[0019] Select some annotated optical remote sensing images with no restrictions on image length and width (each batch will be preprocessed, the images in the batch will be scaled, and then padded to a uniform size). If the dataset is more difficult to train, more images will be required. For simple datasets, 1000+ images can be used. Then 80% of the images will form the training set, and 20% of the images will form the validation set and test set;

[0020] Step 4: Perform prediction processing on the images in the dataset:

[0021] First, the data in each mini-batch is horizontally flipped with a probability of 50%. Then, the length and width of the images in each mini-batch are fixed, and the length is scaled to an integer multiple of 32 between 400 and 600 pixels. The width is then padded to an integer multiple of 32 between 400 and 600 pixels.

[0022] Step 5: Use the training data to co-train the object detection model and decoder:

[0023] The first step is to input the preprocessed data into the backbone network (resnet50 or cspnext, etc.) for feature extraction to generate feature maps C3, C4, and C5;

[0024] In the second step, the feature map is fed into the object detection head (YOLO detection head or Fast R-CNN detection head, etc.) to predict the category and bounding box, and then the cross entropy loss and bounding box loss are calculated with the true label.

[0025] In the third step, the feature maps C3, C4, and C5 generated in the first step are fed into the grouped convolutional attention network module for feature grouping, attention extraction, and feature fusion. Then, a feature map of the same size as the preprocessed image is output, and the mean square error loss function between the preprocessed image and the feature map is calculated.

[0026] The fourth step is to pass the loss functions of the second and third steps back to update the entire network structure at the same time;

[0027] Step 6: Send the verification data to the detection model for detection:

[0028] In the first step, the preprocessed data is input into the backbone network for feature extraction to generate feature maps C3, C4, and C5;

[0029] In the second step, the feature maps C3, C4, and C5 are fed into the object detection head for category and bounding box prediction, and then processed by non-maximum suppression (NMS) to generate the detection results.

[0030] Step 7: Get the test results:

[0031] Map the detection results obtained in step 6 to the corresponding positions of the original image and mark the predicted category information.

[0032] Compared with the existing technology, the present invention has the following advantages:

[0033] First, the structure has strong embedding properties. It can be embedded in some mainstream object detection models, such as RtmDet and Faster R-CNN.

[0034] Second, although it takes longer to train than the detection model without the grouped convolutional attention network module, it does not increase the prediction time, and the prediction results are not inferior to those of deeper backbone networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flowchart of the multi-scale detection optimization method based on the grouped convolutional attention network;

[0036] Figure 2 It is a test result diagram of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be described in detail below with reference to the accompanying drawings.

[0038] Reference Figure 1 , the steps of the present invention are further elaborated.

[0039] Step 1: Build a grouped convolutional attention network module:

[0040] Construct a grouped convolutional attention network module consisting of a feature grouping module and an attention extraction module. The structure of the feature grouping module is as follows: 1×1 convolution → BN layer → grouped convolution block; the structure of the grouped convolution block is to set the output channel to 1 / 8 of the input channel through a normal convolution layer and set a unique grouping number for each convolution block;

[0041] The structure of the attention extraction module is as follows: dilated channel attention block → feature splicing layer → softmax layer; the structure of the dilated channel attention block is as follows: relu function → 1×1 convolution → 3×3 convolution → ECA module; the ECA module consists of global average pooling and one-dimensional convolution; the feature splicing layer splices the attention weights obtained from the dilated channel attention module;

[0042] Then, the features obtained by the feature grouping module are concatenated and fused with the attention weights obtained in the attention extraction module. The obtained features are followed by a BN layer and a ReLU activation function.

[0043] Step 2: Introduce the grouped convolutional attention network module into the general convolutional neural network-based object detection model:

[0044] The constructed grouped convolutional attention network module is introduced into the backbone network of the object detection model. After placing it at three different stages in the backbone network, the obtained features are fused with the original features.

[0045] Step 3: Divide the object detection dataset:

[0046] Select some annotated optical remote sensing images with no restrictions on image length and width (each batch will be preprocessed, the images in the batch will be scaled, and then padded to a uniform size). If the dataset is more difficult to train, more images will be required. For simple datasets, 1000+ images can be used. Then 80% of the images will form the training set, and 20% of the images will form the validation set and test set;

[0047] Step 4: Perform prediction processing on the images in the dataset:

[0048] First, the data in each mini-batch is horizontally flipped with a probability of 50%. Then, the length and width of the images in each mini-batch are fixed, and the length is scaled to an integer multiple of 32 between 400 and 600 pixels. The width is then padded to an integer multiple of 32 between 400 and 600 pixels.

[0049] Step 5: Use the training data to co-train the object detection model and decoder:

[0050] The first step is to input the preprocessed data into the backbone network (resnet50 or cspnext, etc.) for feature extraction to generate feature maps C3, C4, and C5;

[0051] In the second step, the feature map is fed into the object detection head (YOLO detection head or Fast R-CNN detection head, etc.) to predict the category and bounding box, and then the cross entropy loss and bounding box loss are calculated with the true label.

[0052] In the third step, the feature maps C3, C4, and C5 generated in the first step are fed into the grouped convolutional attention network module for feature grouping, attention extraction, and feature fusion. Then, a feature map of the same size as the preprocessed image is output, and the mean square error loss function between the preprocessed image and the feature map is calculated.

[0053] The fourth step is to pass the loss functions of the second and third steps back to update the entire network structure at the same time;

[0054] Step 6: Send the verification data to the detection model for detection:

[0055] In the first step, the preprocessed data is input into the backbone network for feature extraction to generate feature maps C3, C4, and C5;

[0056] In the second step, the feature maps C3, C4, and C5 are fed into the object detection head for category and bounding box prediction, and then processed by non-maximum suppression (NMS) to generate the detection results.

[0057] Step 7: Get the test results:

[0058] Map the detection results obtained in step 6 to the corresponding positions of the original image and mark the predicted category information.

[0059] The effects of the present invention are further described below in conjunction with simulation.

[0060] 1. Simulation experiment conditions:

[0061] The simulation experiment conditions of the present invention are as follows: server GPU: GeForce RTX3090, video memory 24G;

[0062] The software platform for the simulation experiment of the present invention is: Ubuntu 18.04 system, Python 3.10, and PyTorch-GPU 2.0.0.

[0063] 2. Analysis of simulation content and experimental results:

[0064] The simulation of the present invention is to detect remote sensing images using several existing technologies (Faster R-CNN detection method, RtmDet method, etc.) and their introduction of grouped convolutional attention network modules to obtain detection results; the datasets used in the simulation are: SSDD, DIOR-R and MS-Building.

[0065] The two existing technologies used in the simulation experiments are:

[0066] The Faster R-CNN detection method is an object detection method proposed by Shaoqing Ren, Kaiming He, and others in "Ren, S., He, K., Girshick, R., Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural informationprocessing systems, 28, 91-99.", referred to as the Faster R-CNN detection method;

[0067] The Cascade R-CNN detection method is an object detection method proposed by Cai Z et al. in "Cai Z, Vasconcelos N. Cascade r-cnn: Delving into high quality object detection[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 6154-6162."

[0068] Dynamic R-CNN is an object detection algorithm proposed by Zhang et al. in “Zhang H, Chang H, Ma B, et al. Dynamic R-CNN: Towards high quality object detection via dynamic training[C] / / Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16. Springer International Publishing, 2020: 260-275.”

[0069] The Libra R-CNN detection method is an object detection method proposed by Pang et al. in "Pang J, Chen K, Shi J, et al. Librar-cnn: Towards balanced learning for object detection[C] / / Proceedings of theIEEE / CVF conference on computer vision and pattern recognition. 2019: 821-830."

[0070] The Grid R-CNN detection method is an object detection method proposed by Zhang et al. in "Lu X, Li B, Yue Y, et al. Grid r-cnn[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 7363-7372."

[0071] The YOLOX detection method is an object detection method proposed by Lu et al. in "Ge Z, Liu S, Wang F, et al. Yolox: Exceedingyolo series in 2021[J]. arXiv preprint arXiv:2107.08430, 2021."

[0072] The RtmDet detection method is an object detection method proposed by Lyu C et al. in "Lyu C, Zhang W, Huang H, et al. Rtmdet:An empirical study of designing real-time object detectors[J]. arXiv preprintarXiv:2212.07784, 2022."

[0073] The input images used in the simulation experiment of the present invention are the publicly available SSDD dataset, DIOR-R dataset, and MS-Building dataset. The SSDD dataset was released by Wuhan University in 2017. Its images were extracted from RadarSat-2, TerraSAR-X, and Sentinel-1. It contains 1,160 SAR ship images with an image size of 500×500×3. We use 1,000 ship images as a training set and 160 images as a validation set. The DIOR dataset was released by Northwestern Polytechnical University in 2019. Its images were extracted from Google Earth with an image size of , including 20 categories and a total of 190,288 targets. We use 11,725 ​​remote sensing images as a training set and 11,738 images as a validation set for testing and evaluation. The MS-Building dataset was released by Jinan University in 2023. Its images come from Gaofen Satellite 1 and Gaofen Satellite 2, and contain a total of nearly 1,200 panchromatic images, multispectral images, and fused images.

[0074] Simulation Experiment 1 introduces the Grouped Convolutional Attention Network module into various models and compares the detection results on DIOR-R. Simulation Experiment 2 introduces the Grouped Convolutional Attention Network module into various models and compares the detection results on MS-Building. To verify the effectiveness of this invention for remote sensing image detection, the detection results were evaluated using the Cocoa evaluation metric. The calculation results are plotted in the table below.

[0075] Table 1. Comparison of simulation results of the present invention on the DIOR-R dataset

[0076] Method AP (%) AP50 (%) AP75 (%) Faster R-CNN 39.0 64.7 39.7 Cascade R-CNN 42.3 66.2 44.4 Libra R-CNN 40.3 65.8 41.5 Dynamic R-CNN 40.5 64.8 42.0 Grid R-CNN 42.5 64.2 45.2 RtmDet 52.8 75.9 56.4 YOLOX 50.6 78.2 78.2 Faster R-CNN+ improved module 39.2 65.0 39.7 Cascade R-CNN+ improved module 42.6 66.7 44.4 Libra R-CNN+ improved module 40.5 66.2 41.7 Dynamic R-CNN+ improved module 40.8 65.1 42.5 Grid R-CNN+ improved module 42.0 65.7 44.4 RtmDet+ improved module 53.6 76.6 57.2 YOLOX+ Improvement Module 51.2 78.9 54.6

[0077] Table 2. Comparison of simulation results of the present invention on the MS-Builiding dataset

[0078] Method AP (%) AP50 (%) AP75 (%) Faster R-CNN 39.7 76.9 36.9 Cascade R-CNN 40.3 77.9 37.8 Libra R-CNN 39.5 75.9 37.1 Dynamic R-CNN 37.3 72.7 33.2 Grid R-CNN 38.6 76.4 34.3 RtmDet 41.0 77.6 39.5 YOLOX 39.8 76.6 37.7 Faster R-CNN+ improved module 40.0 77.9 36.2 Cascade R-CNN+ improved module 41.9 79.6 39.0 Libra R-CNN+ improved module 40.7 77.0 39.0 Dynamic R-CNN+ improved module 38.3 74.2 34.7 Grid R-CNN+ improved module 38.5 77.1 33.5 RtmDet+ improved module 43.5 79.9 42.1 YOLOX+ Improvement Module 39.7 78.1 37.4

[0079] Combined with Table 1 above, it is not difficult to see that the detection algorithms that incorporate the Grouped Convolutional Attention Network module for training have significantly improved detection accuracy compared to those without it. On the DIOR-R dataset, the YOLOX model embedded with the Grouped Convolutional Attention Network module in our experiments achieved the best AP50, reaching 78.9%. On RtmDet, mAP reached 53.6%, and AP75 reached 57.2%. Combined with Table 2, the table shows the experimental results of mainstream detection models on the MS_Building dataset. We can observe that almost all models have achieved performance improvements of varying degrees after embedding the Grouped Convolutional Attention Network. On RtmDet, mAP, AP50, and AP75 can reach 43.5%, 79.9%, and 42.1%, respectively.

[0080] The following combination Figure 2 The effects of this simulation experiment are further explained.

[0081] Figure 2 (a)~(d) in the figure are the prediction results using the RtmDet network, and (e)~(h) are the prediction results after the improved model. Figure 2 We can see that the prediction results for images (a)-(d) in the annotated areas have some omissions and false positives, while the detection results for the same images (e)-(h) show fewer omissions and false positives. This demonstrates that our invention outperforms the original detection model without our invention in terms of accuracy, omissions, and false positives.

Claims

1. A multi-scale detection optimization method based on grouped convolutional attention network, characterized in that: The method comprises: The grouped convolutional attention network module is used to perform collaborative training with the object detection model to enhance the feature extraction network of the detection model and improve the detection accuracy. The specific steps of this method include the following: Step 1: Build a grouped convolutional attention network module: Construct a grouped convolutional attention network module consisting of a feature grouping module and an attention extraction module. The structure of the feature grouping module is as follows: 1×1 convolution → BN layer → grouped convolution block. The structure of the grouped convolution block is to set the output channel to 1 / 8 of the input channel through a normal convolution layer and set a unique grouping number for each convolution block. The structure of the attention extraction module is as follows: dilated channel attention block → feature splicing layer → softmax layer; the structure of the dilated channel attention block is as follows: relu function → 1×1 convolution → 3×3 convolution → ECA module; the ECA module consists of global average pooling and one-dimensional convolution; the feature splicing layer splices the attention weights obtained from the dilated channel attention module; Then, the features obtained by the feature grouping module are concatenated and fused with the attention weights obtained in the attention extraction module. The obtained features are followed by a BN layer and a ReLU activation function. Step 2: Introduce the grouped convolutional attention network module into the general convolutional neural network-based object detection model: The constructed grouped convolutional attention network module is introduced into the backbone network of the object detection model. After placing it at three different stages in the backbone network, the obtained features are fused with the original features. Step 3: Divide the object detection dataset: Select some annotated optical remote sensing images with no restrictions on image length and width (each batch will be preprocessed, the images in the batch will be scaled, and then padded to a uniform size). If the dataset is more difficult to train, more images will be required. For simple datasets, 1000+ images can be used. Then 80% of the images will form the training set, and 20% of the images will form the validation set and test set; Step 4: Perform prediction processing on the images in the dataset: First, the data in each mini-batch is horizontally flipped with a probability of 50%. Then, the length and width of the images in each mini-batch are fixed, and the length is scaled to an integer multiple of 32 between 400 and 600 pixels. The width is then padded to an integer multiple of 32 between 400 and 600 pixels. Step 5: Use the training data to co-train the object detection model and decoder: The first step is to input the preprocessed data into the backbone network (resnet50 or cspnext, etc.) for feature extraction to generate feature maps C3, C4, and C5; In the second step, the feature map is fed into the object detection head (YOLO detection head or Fast R-CNN detection head, etc.) to predict the category and bounding box, and then the cross entropy loss and bounding box loss are calculated with the true label. In the third step, the feature maps C3, C4, and C5 generated in the first step are fed into the grouped convolutional attention network module for feature grouping, attention extraction, and feature fusion. Then, a feature map of the same size as the preprocessed image is output, and the mean square error loss function between the preprocessed image and the feature map is calculated. The fourth step is to pass the loss functions of the second and third steps back to update the entire network structure at the same time; Step 6: Send the verification data to the detection model for detection: In the first step, the preprocessed data is input into the backbone network for feature extraction to generate feature maps C3, C4, and C5; In the second step, the feature maps C3, C4, and C5 are fed into the object detection head for category and bounding box prediction, and then processed by non-maximum suppression (NMS) to generate the detection results. Step 7: Get the test results: Map the detection results obtained in step 6 to the corresponding positions of the original image and mark the predicted category information.

2. The multi-scale detection optimization method based on grouped convolutional attention network according to claim 1, characterized in that: The construction of the grouped convolutional attention network module and the embedding training of the grouped convolutional attention network module described in steps 1, 2 and 5, the design of the corresponding grouped convolutional attention network for the backbone network of the detection model, and the use of the constructed grouped convolutional attention network module for collaborative training with the object detection model to enhance the feature extraction capability in the backbone network and make the detection stronger.

Images