A hybrid attention-based small target detection method for anchor-free remote sensing images

Abstract

The present invention provides a method for detecting small targets in remote sensing images without anchor frames based on hybrid attention, including: constructing a network structure for detecting small targets in remote sensing images without anchor frames based on hybrid attention; using transfer learning to train the network structure to obtain a trained network structure; inputting a remote sensing data set into the trained network structure to obtain a target detection result of the remote sensing image: during the network structure training process, using ResNet50 to extract features of each layer of the input image, and introducing a U-shaped network structure for strengthening features, and after associating high-level and low-level features, adding an attention mechanism, and finally using a loss function to train the network structure to obtain a trained network model. The present invention provides the following beneficial effects: improving the detection capability, speed, robustness and accuracy of small targets in high-resolution remote sensing images.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and in particular to a method for detecting small targets in remote sensing images without anchor frames based on hybrid attention. Background Art

[0002] Small target detection in remote sensing images is one of the research hotspots in the field of remote sensing. The development of high spatial resolution (HSR) remote sensing image sensors has accelerated the acquisition of various remote sensing images such as aerial and satellite images with sufficiently detailed spatial structure information. These remote sensing images can facilitate a wide range of military and civilian applications, such as ocean monitoring, urban area detection, cargo transportation, port management, etc. Different from acquiring natural images on the ground from the horizontal direction, obtaining high spatial resolution remote sensing images requires viewing from a top-down perspective, which is an image acquisition method that is easily affected by weather and lighting. In addition, the small and variable scale characteristics of many types of geospatial objects and the lack of manually annotated training samples make the detection task more challenging. Many studies related to small target detection in remote sensing images have been carried out.

[0003] The main steps of remote sensing image target detection based on traditional machine learning methods are: 1) Use a sliding window to traverse the image to obtain the region of interest; 2) Count the underlying features in the image, such as the Histogram of Oriented Gradient (HOG) feature; 3) Train the classifier with the extracted features, such as the commonly used Support Vector Machine (SVM) method; 4) Determine whether the region of interest contains the target. Cheng et al. used HOG features and sliding windows to extract features to achieve target recognition in remote sensing images. Aytekin et al. detected airport targets based on image texture features. This type of algorithm has two main defects: 1) The region selection strategy based on the sliding window is not targeted, the window is redundant, and the time complexity is high; 2) The manual features designed based on background knowledge are not very robust to diverse environmental changes.

[0004] There are two mainstream routes for target detection algorithms based on deep learning: one is the two-stage detection algorithm represented by Fast R-CNN (Fast-regions with Convolutional Neural Network), Faster R-CNN (Faster-regions with Convolutional Neural Network) and R-FCN (Region-based Fully Convolutional Networks), and the other is the single-stage detection algorithm represented by the YOLO (You Only Look Once) series and SSD. In the two-stage detection algorithm, the first stage extracts candidate regions from the input image, and the second stage obtains the prediction results based on the mapped feature map of the candidate regions. The single-stage detection algorithm directly obtains the prediction box through the anchor box and generates the prediction results at the same time. The speed of the single-stage detection algorithm is much faster than the two-stage detection algorithm, but the accuracy is slightly inferior to the latter. With the rapid development of deep learning technology, scholars have begun to use deep learning methods to solve the problem of target detection in remote sensing images. Summary of the invention

[0005] In view of the fact that small targets in existing high-resolution remote sensing images occupy fewer image pixels, the information features they contain are not obvious, and there are often missed detections and false detections during detection, which seriously affects the target detection effect and the problem of insufficient high-quality small target training data, the present invention provides a method for detecting small targets in remote sensing images without anchor frames based on hybrid attention, which is used to improve the detection capability, speed, robustness and accuracy of small targets in high-resolution remote sensing images. The method for detecting small targets in remote sensing images without anchor frames specifically includes:

[0006] S101: Constructing a network structure for small target detection in remote sensing images without anchor boxes based on hybrid attention;

[0007] S102: training the network structure using transfer learning to obtain a trained network structure;

[0008] S103: Inputting the remote sensing data set into the trained network structure to obtain the target detection result of the remote sensing image.

[0009] Furthermore, the process of obtaining the trained network structure is as follows:

[0010] Firstly, in the process of training the network structure by transfer learning, ResNet50 is used to extract features of each layer of the input image. Then, a U-shaped network structure with enhanced features is introduced. The U-shaped network structure is used to associate the feature layers extracted by the multi-scale backbone network. After the association, an attention mechanism is added to adaptively adjust the feature maps between different channels. Finally, the adaptively adjusted feature layer is input into the detection layer, and the network structure is trained by combining the loss function to obtain a trained network structure.

[0011] Furthermore, the U-shaped network structure transforms the previous direct downsampling into a step-by-step upward fusion upsampling to obtain a feature map that gathers more small target information.

[0012] Furthermore, the attention mechanism is added after the association, and the process of adaptively adjusting the features between different channels includes channel attention adjustment and spatial attention adjustment.

[0013] Furthermore, the process of adjusting the channel attention is as follows: using average pooling and maximum pooling to process the associated feature map respectively, and then performing dimension upscaling and point addition operations on the processed results to obtain weight information on the feature map channel, and weighting this weight information with the original feature map to obtain a channel-adjusted feature map.

[0014] Furthermore, the process of spatial attention adjustment is: using dimensionality reduction to obtain weight information of each layer in the network structure, weighting this weight information with the feature map adjusted by the channel, and obtaining the final feature map adjusted by the attention mechanism.

[0015] Furthermore, before using the loss function, it is necessary to optimize it using the stochastic gradient descent method.

[0016] The beneficial effects provided by the present invention are: improving the small target detection capability, speed, robustness and accuracy of high-resolution remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of a network structure in an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of channel attention adjustment in an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of spatial attention adjustment in an embodiment of the present invention;

[0020] Figure 4 Schematic diagram of overall adjustment of the attention mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0022] A method for detecting small targets in remote sensing images without anchor boxes based on hybrid attention, including the following:

[0023] S101: Constructing a network structure for small target detection in remote sensing images without anchor boxes based on hybrid attention;

[0024] S102: training the network structure using transfer learning to obtain a trained network structure;

[0025] Please refer to Figure 1 , Figure 1 The improved network structure schematic diagram of the present invention; during the network structure training process, ResNet50 is used to extract features of each layer of the input image, and a U-shaped network structure for strengthening features is introduced. The U-shaped network structure can associate feature layers extracted by a multi-scale backbone network to improve the feature representation capability of small targets. The U-shaped network structure combines low-level features with high-level features, effectively combines high-level semantic information with shallow position information and texture information, and can improve the detection accuracy of small targets and improve the missed detection rate of the network; after the high-level and low-level features are associated, an attention mechanism is added, and the obtained features are adaptively adjusted to reduce the interference of background error information. The structure adaptively adjusts the features between different channels, and can effectively improve the false detection rate of the network.

[0026] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the overall adjustment of the attention mechanism. In order to achieve adaptive adjustment of the fusion feature layer, the channel attention of the feature map is first adjusted, such as Figure 2 As shown in , average pooling and maximum pooling are used to process the associated feature map, and then the processed results are dimensionally upgraded and point-added to obtain the weight information on the feature map channel. This weight information is weighted with the original feature map to obtain the channel-adjusted feature map; then spatial attention is used for adjustment, as shown in Figure 3 As shown, the weight information of each layer in the network structure is obtained by dimensionality reduction, and the weight information is weighted with the channel-adjusted feature map obtained by the channel attention mechanism to obtain the final feature map adjusted by the attention mechanism.

[0027] The output feature layer (that is, Figure 1 The final feature layer shown in ) is input to the detection layer, and the detection layer has 3 branches, which are used to obtain the specific location information of the target. The network structure is trained using the loss function to obtain the trained network structure.

[0028] S103: Inputting the remote sensing data set into the trained network structure to obtain the target detection result of the remote sensing image.

[0029] In order to verify the performance of the algorithm of the present invention, the present invention was tested on both indoor and outdoor image datasets. Four algorithms were used for comparative experiments, namely YOLOv3, YOLOv4, RetinaNet, CenterNet and SSD. All experiments of the present invention were carried out on a 64-bit Windows 10 computer equipped with a Core i9-10940X CPU and a 3090 with 24G video memory. All experimental results were obtained under the same IoU threshold of 0.5 and detection confidence threshold of 0.5.

[0030] In order to reliably evaluate and verify the proposed method, the present invention uses the NWPU VHR-10 dataset proposed by Professor Han Junwei's team in 2014, which includes not only optical remote sensing images but also color infrared images. It includes 715 color images and 85 color infrared images. The color images are all collected from Google Earth Proc with a spatial resolution of 0.5 meters to 2 meters. The pan-infrared images are obtained from the Vaihingen dataset with a spatial resolution of 0.08 meters. The team divides the entire dataset into two sets, positive and negative. The positive set contains 650 images, including 10 target categories in total, and the negative set contains 150 images, each of which does not contain a target category. Because the negative set is mainly used for weakly supervised learning tasks and semi-supervised learning tasks, the negative set is not used in this experiment. The entire positive set is divided into a training set (accounting for 80%) and a test set (accounting for 20%). The specific data of the dataset are shown in Table 1.

[0031] Table 1 NWPU VHR-10 dataset

[0032]

[0033] During the training process, only the labeled positive sample images in the NWPU VHR-10 dataset are used, and the 650 positive sample images are divided into a training set of 550 and a test set of 100.

[0034] Multi-strategy training consists of two stages: pre-training and training. In the pre-training stage, the convolutional layer of ResNet50 is first pre-trained using the ImageNet dataset, and then the convolutional layer of ResNet50 in the ResNet50 model is frozen. The other network parts are trained using the VOC2007+VOC2012 datasets, and then the entire network is fine-tuned. In the training stage, the network is first initialized using the pre-trained model, the convolutional layer parameters of ResNet50 are frozen, the other parameters are trained, and then the entire network is fine-tuned to obtain the target model.

[0035] Data enhancement adopts random cropping, random rotation and scaling operations, and the stochastic gradient descent method is used to optimize the total loss function. The batch size is set to 8, and the initial learning rate of the training set training phase is set to 0.001, and it decays by 50% every 10,000 iterations thereafter.

[0036] The invention sets the IoU value of the predicted box and the real box as long as it is greater than 0.5, then the detection is considered correct. At the same time, the algorithm proposed in the invention shows better performance on the NWPU VHR-10 dataset than other detection algorithms. The quantitative results of different methods on the NWPU VHR-10 dataset, including the AP value and mAP value of 10 categories, are shown in Table 2. The mAP value of Ours (disclosed in the invention) algorithm is 15.2%, 10.4%, 7.8%, 5.7% and 13.5% higher than that of SSD512, YOLOv3, YOLOv4, RetinaNet and CenterNet, respectively, and the performance is significantly improved on vehicles and ships. The results show that compared with the existing popular general target detection algorithms, the algorithm proposed in the invention has better performance advantages in the detection of small targets in remote sensing images. By comparing the FPS of different methods, it can be seen that after adding the U-shaped structure and attention mechanism, the algorithm of the invention can improve the detection accuracy on the basis of ensuring real-time detection, and performs well in the task of small target detection in remote sensing images.

[0037] Table 2 Algorithm performance comparison

[0038]

[0039]

[0040] The U-shaped network structure provides more structural spatial information of the detected target, so that stronger semantic information and more detailed information can be obtained. Experimental results show that the algorithm proposed in the present invention has excellent performance in small target detection in remote sensing images. The present invention also demonstrates that making full use of deep and shallow feature information helps to obtain a better target detector.

[0041] This paper studies the task of quickly detecting small targets in optical remote sensing images by improving the network structure model. In order to improve the detection accuracy and speed up the detection, the number of layers of the network structure is modified, a feature fusion module is added, and an evaluation experiment is conducted on the validation set. The experiment proves that the present invention solves the problem of low accuracy in small target detection. By analyzing the detection results of small and medium-sized targets such as ships and vehicles, the following conclusions are obtained:

[0042] 1) The network structure proposed in the present invention is more reasonable, and the training process is easier to converge, and good experimental results can be obtained with the support of transfer learning.

[0043] 2) Compared with the SSD algorithm, YOLO series algorithms, and RetinaNet algorithm, the algorithm proposed in the present invention has significantly improved the overall accuracy of the original CenterNet algorithm, especially in detecting small targets. This shows that high-level features and low-level features obtain better semantic information and position information after feature extraction through the U-shaped network proposed in the present invention, and some useless information is suppressed by adding the attention mechanism.

[0044] The anchor-free remote sensing image small target detection algorithm based on hybrid attention has certain value for efficient application in fast target detection of remote sensing images, especially in the detection of small targets in remote sensing images. It provides certain reference value for the field of remote sensing image processing as well as traffic management, military target detection and other related work. In the next work, we will continue to optimize the network results and improve its calculation speed.

[0045] The beneficial effects provided by the present invention are: improving the small target detection capability, speed, robustness and accuracy of high-resolution remote sensing images.

[0046] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for small target detection in remote sensing images without anchor boxes based on hybrid attention, characterized by: The anchor-free remote sensing image small target detection method comprises the following steps: S101: Constructing a network structure for small target detection in remote sensing images without anchor boxes based on hybrid attention; S102: training the network structure using transfer learning to obtain a trained network structure; The process of obtaining the trained network structure is as follows: First, in the process of training the network structure by using transfer learning, ResNet50 is used to extract features of each layer of the input image, and then a U-shaped network structure with enhanced features is introduced. The U-shaped network structure is used to associate the feature layers extracted by the multi-scale backbone network, and after the association, an attention mechanism is added to adaptively adjust the feature maps between different channels. Finally, the adaptively adjusted feature layer is input into the detection layer, and the network structure is trained by using a loss function to obtain a trained network structure; The U-shaped network structure transforms the previous direct downsampling into a step-by-step upward fusion upsampling to obtain a feature map that gathers more small target information; The attention mechanism is added after the association, and the process of adaptively adjusting the features between different channels includes channel attention adjustment and spatial attention adjustment; S103: Inputting the remote sensing data set into the trained network structure to obtain the target detection result of the remote sensing image.

2. The method for detecting small targets in remote sensing images without anchor frames based on hybrid attention as claimed in claim 1, characterized in that: The process of channel attention adjustment is: using average pooling and maximum pooling to process the associated feature map respectively, and then performing dimension increase and point addition operations on the processed results to obtain weight information on the feature map channel, and weighting this weight information with the original feature map to obtain the channel-adjusted feature map.

3. The method for detecting small targets in remote sensing images without anchor frames based on hybrid attention as claimed in claim 1, characterized in that: The process of spatial attention adjustment is: using dimensionality reduction to obtain the weight information of each layer in the network structure, weighting this weight information with the feature map adjusted by the channel, and obtaining the final feature map adjusted by the attention mechanism.

4. The method for detecting small targets in remote sensing images without anchor frames based on hybrid attention as claimed in claim 1, characterized in that: Before using the loss function, it needs to be optimized using the stochastic gradient descent method.

Images