Remote sensing image small target detection method based on context and attention

Abstract

The invention discloses a remote sensing image small target detection method based on context and attention. The method comprises the following steps of: firstly, fully extracting feature information of a small target through a designed backbone network EResNet50; the EResNet50 combines switchable convolution, context and attention, and can adaptively adjust the size of a receptive field. Secondly, the extracted features are sent to a designed feature fusion network AM-FPN for feature fusion; the AM-FPN establishes the semantic and spatial dependency relationship between each sub-region of the current feature layer and the adjacent feature layers through a self-attention mechanism, so that the problems of small target feature dislocation and information loss can be effectively relieved. And finally, more effective features obtained through AM-FPN are used for subsequent classification and regression tasks. The method provided by the invention effectively improves the small target detection precision, and has wide application prospects in the aspects of smart cities, marine shipping, national defense security and the like.

Description

Technical Field

[0001] The present invention relates to the problem of remote sensing target detection in the field of deep learning, and in particular to a remote sensing image small target detection method based on context and attention. Background Art

[0002] As an important computer vision task, remote sensing target detection is widely used in smart cities, marine shipping, national defense security, ecological protection and other fields. Thanks to the rapid development of deep learning technology, the powerful feature expression ability of convolutional neural networks has achieved remarkable results in the field of computer vision, and many advanced target detection algorithms have been extended to remote sensing target detection. Compared with general target detection, remote sensing target detection has problems such as close target arrangement, arbitrary direction, and small targets. Previous methods focused on solving the problem of arbitrary target direction, but lacked research on the problem of small targets, which is extremely important for the performance optimization of remote sensing target detection algorithms.

[0003] As a branch of target detection, remote sensing target detection is usually regarded as a rotation target positioning problem by mainstream methods. Among them, angle-based detectors dominate, and they use predicted angles to construct rotated rectangular boxes. For example, SCRDet, R3Det, ​​Mask OBB, and Oriented R-CNN have achieved good detection performance. However, these methods may not be able to accurately locate due to the periodic boundedness of angles and the limitations of bounding box definitions. CSL and DCL use circular smooth labeling technology to transform the angle regression problem into a classification problem, achieving more stable results. The above are all anchor-based methods. In recent years, anchor-free methods have attracted much attention. FCOS, Oriented RepPoints, etc. are all relatively representative anchor-free methods. Compared with anchor-based methods, anchor-free methods do not require fine-tuning of anchor parameters. Moreover, the aspect ratio of objects in remote sensing images varies greatly, making it difficult to adjust anchor parameters. Therefore, anchor-free methods have high flexibility and low computational complexity. Compared with anchor-based methods, anchor-free methods have more flexible feature expressions.

[0004] In response to the problem of small targets, researchers at home and abroad have proposed a series of solutions. They can be mainly divided into data enhancement methods, feature fusion methods, super-resolution methods, context-based methods, attention-based methods, etc. Data enhancement methods aim to solve the problem of imbalanced number of small targets in the data set, which affects the performance of small target detection. Its main idea is to increase the number of small target instances. Feature fusion methods are to alleviate the information imbalance problem between deep features and shallow features to obtain better small target feature representation. Super-resolution methods are to enrich the information of small targets, and they mainly use generative adversarial networks to generate high-quality feature representations that are conducive to small target detection. Context-based methods use the relationship between targets and the environment to promote the recognition of small targets. Attention-based methods use attention mechanisms to better detect small targets. In complex remote sensing scenarios, achieving superior detection performance for small targets cannot rely solely on one of the above methods.

[0005] The existing remote sensing target detection methods generally have low detection performance for small targets. Small targets have few pixels themselves, and current methods rely on standard deep neural backbone networks (ResNet, Darknet53, etc.) to extract features and use feature pyramid networks (FPN) as feature fusion networks for feature fusion. It is undeniable that these standard settings have achieved certain results in the field of target detection. However, these standard settings have the following problems in detecting small targets: first, the backbone network does not extract sufficient feature information of small targets; second, when using FPN for feature fusion, small targets have information loss and feature misalignment problems. Summary of the invention

[0006] The purpose of the present invention is to provide a method for detecting small targets in remote sensing images based on context and attention. First, the feature information of small targets is fully extracted through the designed backbone network EResNet50. EResNet50 combines switchable convolution, context and attention, and can adaptively adjust the size of the receptive field. Secondly, the extracted features are sent to the designed feature fusion network AM-FPN for feature fusion. AM-FPN uses a self-attention mechanism to establish semantic and spatial dependencies between adjacent feature layers, which can effectively alleviate the problem of small target feature misalignment. Moreover, AM-FPN captures the semantic and spatial dependencies between the sub-regions of the current layer features through the self-attention mechanism to retain and expand channel information, which can effectively alleviate the problem of small target information loss. Finally, the more effective features obtained by AM-FPN are used for subsequent classification and regression tasks. Due to the introduction of attention and context, the feature information of small targets is preserved as effectively as possible, the identification and regression capabilities of small target features are improved, and thus the detection accuracy of small targets is improved.

[0007] The present invention specifically adopts the following technical solutions:

[0008] A small target detection method for remote sensing images based on context and attention is proposed. The main features of this method are:

[0009] a. Propose a backbone network EResNet50 based on switchable convolution and SE attention;

[0010] b. Propose a feature fusion network AM-FPN based on self-attention mechanism and context;

[0011] c. Propose a remote sensing image small target detection method that combines the backbone network EResNet50, the feature fusion network AM-FPN and the anchor-free detection head;

[0012] like Figure 1 As shown, the method mainly includes the following steps:

[0013] (1) The backbone network EResNet50 is used to extract the feature information of small targets and obtain features F1, F2, F3 and F4.

[0014] (2) The extracted features are fused using the feature fusion network AM-FPN, where features F1, F2, and F3 first pass through the context attention module CxAM and then are horizontally connected through 1×1 convolution; then they are upsampled 2 times from top to bottom with the previous level features that have passed through the context content attention module CnAM, and added and fused to obtain more effective features P1, P2, and P3. Feature F4 is directly output to obtain P4, and 2 times downsampled to obtain feature P5.

[0015] (3) The features obtained by the backbone network EResNet50 and the feature fusion network AM-FPN are sent to the shared detection head for classification and regression. The shared detection head is anchor-free and uses the detection head of the baseline method. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a framework diagram of the small target detection method in remote sensing images based on context and attention of the present invention.

[0017] Figure 2 This is the residual block structure diagram of the backbone network EResNet50 of the present invention.

[0018] Figure 3 This is a top-down and lateral connection structure diagram of the feature fusion network AM-FPN of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical personnel familiar with the relevant field of technology may make some non-essential improvements and adjustments to the present invention based on the above invention content for specific implementation, which should still fall within the scope of protection of the present invention.

[0020] The specific description of the backbone network EResNet50 based on switchable convolution and SE attention in step (1) is as follows:

[0021] like Figure 2 As shown in the figure, the residual module of the backbone network EResNet50 is mainly composed of three parts: 1×1 convolution, switchable convolution module SACM and SE attention. All 3×3 convolutions in the residual blocks of ResNet50 (i.e. Conv2_x, Conv3_x, Conv4_x and Conv5_x) are replaced with switchable convolution modules SACM, and SE attention is added after them, which enables the backbone network to focus on features that are useful for the current task in complex remote sensing scenarios while suppressing useless features. The switchable convolution module SACM is mainly composed of two global context modules (Pre-Global Context and Post-Global Context) and switchable convolution SAC. In order not to increase the number of parameters and computation of the network, the two global context modules are lightweight. The first context module (Pre-Global Context) first performs global average pooling on the input features to obtain global feature information, and then adds it to the input features through 1×1 convolution; the second context module (Post-GlobalContext) first performs global average pooling on the features output by the switchable convolution SAC to obtain global feature information, and then adds it to the features output by the switchable convolution SAC through 1×1 convolution. These two context modules are designed to retain the feature information of small targets as much as possible, which is beneficial to the subsequent processing of the feature information of small targets. The essence of switchable convolution SAC is to combine dilated convolutions with different dilation coefficients by switching the function S(x), which can not only adjust the receptive field, but also does not lose resolution. Moreover, switchable convolution SAC does not require the introduction of additional hyperparameters and does not increase the amount of additional calculations.

[0022] The specific description of the feature fusion network AM-FPN based on the self-attention mechanism and context in step (2) is as follows:

[0023] like Figure 1 and Figure 3 As shown, the feature F extracted by the backbone network i (i=1,2,3) is divided into two branches, one upward and deep feature P i+1Input the contextual content attention module CnAM to generate the feature D with contextual content attention i Then, for feature D i Perform 2 times upsampling to obtain feature D i U The other one directly generates the self-attention feature U through the context attention module CxAM i Then, after 1×1 convolution, feature U is obtained i S Finally, the features of the two branches are added to obtain the output feature P i The output feature P of the feature fusion network AM-FPN i The expression is:

[0024]

[0025] Among them, the function f x () represents the context attention module CxAM, function f n () represents the contextual content attention module CnAM, function f dc () represents a 1×1 convolution compression channel operation, function f up () indicates a 2x upsampling operation, function f down () indicates a 2x downsampling operation.

[0026] This embodiment uses the public data set DOTA for training and testing, and trains for 40 epochs to achieve the optimal effect. The quantitative detection effect is shown in Table 1. The detection performance of the present invention exceeds the current optimal network. The abbreviations of the categories are defined as: PL-Plane, BD-Baseball Diamond, BR-Bridge, GTF-Ground Track Field, SV-SmallVehicle, LV-Large Vehicle, SH-Ship, TC-Tennis Court, BC-Basketball Court, ST-Storage Tank, SBF-Soccer-ball Field, RA-Roundabout, HA-Harbor, SP-Swimming Pool, and HC-Helicopter.

[0027]

[0028]

Claims

1. A method for detecting small targets in remote sensing images based on context and attention, characterized in that: a. Propose a backbone network EResNet50 based on switchable convolution and SE attention; b. Propose a feature fusion network AM-FPN based on self-attention mechanism and context; c. Propose a remote sensing image small target detection method that combines the backbone network EResNet50, the feature fusion network AM-FPN and the anchor-free detection head; The method mainly includes the following steps: (1) Extract the feature information of small targets through the backbone network EResNet50 to obtain features F1, F2, F3 and F4; (2) The extracted features are fused using the feature fusion network AM-FPN, where features F1, F2, and F3 first pass through the context attention module CxAM and then are horizontally connected through 1×1 convolution; then they are upsampled by a factor of 2 with the previous level features of the context content attention module CnAM from top to bottom, and added and fused to obtain more effective features P1, P2, and P3; feature F4 is directly output to obtain P4, and then downsampled by a factor of 2 to obtain feature P5; (3) The features obtained by the backbone network EResNet50 and the feature fusion network AM-FPN are sent to the shared detection head for classification and regression; the shared detection head is the anchor-free detection head of the baseline method.

2. The switchable convolution and SE attention based backbone network EResNet50 as claimed in claim 1, characterized in that In step (1), the residual blocks (i.e., Conv2_x, Conv3_x, Conv4_x, and Conv5_x) in the backbone network EResNet50 are designed as follows: All 3×3 convolutions in the ResNet50 residual blocks (i.e., Conv2_x, Conv3_x, Conv4_x, and Conv5_x) are replaced with switchable convolution modules SACM, and SE attention is added afterwards, which enables the backbone network to focus on features that are useful for the current task in complex remote sensing scenarios while suppressing useless features; the switchable convolution module SACM is mainly composed of two global context modules (Pre-Global Context and Post-Global Context) and switchable convolution SAC; in order not to increase the number of parameters and computations of the network, the two global context modules are lightweight; the previous context module (Pre-Global Context) first performs global average pooling on the input features to obtain global feature information, and then adds it to the input features through 1×1 convolution; The post-global context module first performs global average pooling on the features output by the switchable convolution SAC to obtain global feature information, and then adds the features output by the switchable convolution SAC through a 1×1 convolution. These two context modules are designed to retain the feature information of small targets as much as possible, which is beneficial to the subsequent processing of the feature information of small targets; the essence of switchable convolution SAC is to combine dilated convolutions with different dilation coefficients by switching the function S(x), which can not only adjust the receptive field but also does not lose resolution; and switchable convolution SAC does not require the introduction of additional hyperparameters and does not increase the amount of additional calculations.

3. The feature fusion network AM-FPN based on self-attention mechanism and context according to claim 1 is characterized in that In step (2), the feature fusion network AM-FPN is constructed as follows: Using the contextual content attention module CnAM to connect adjacent layer features from top to bottom and establish semantic and spatial dependency relationships between adjacent feature layers can effectively alleviate the problem of small target feature misalignment; Use the contextual attention module CxAM for lateral connection to capture the semantic and spatial dependencies between the sub-regions of the current layer features to retain and expand channel information, which can effectively alleviate the problem of small target information loss; Feature F extracted by backbone network EResNet50 i (i=1,2,3) is divided into two branches, one upward and deep feature P i+1 Input the contextual content attention module CnAM to generate the feature D with contextual content attention i Then, for feature D i Perform 2 times upsampling to obtain feature D i U ; The other one directly generates the self-attention feature U through the context attention module CxAM i Then, after 1×1 convolution, feature U is obtained i S ; Finally, the features of the two branches are added to obtain the output feature P i ; Output feature P of feature fusion network AM-FPN i The expression is: Among them, the function f x () represents the context attention module CxAM, function f n () represents the contextual content attention module CnAM, function f dc () represents a 1×1 convolution compression channel operation, function f up () indicates a 2x upsampling operation, function f down () indicates a 2x downsampling operation.

4. The remote sensing image small target detection method combining the backbone network EResNet50, the feature fusion network AM-FPN and the anchor-free detection head according to claim 1 is characterized in that In step (3), the backbone network EResNet50 in step (1) and the feature fusion network AM-FPN in step (2) are combined to fully mine the small target information; the backbone network EResNet50 fully extracts the feature information of the small target, while the feature fusion network AM-FPN allows the small target to obtain more semantic information from the deep feature layer and retains the features of the small target itself as much as possible; in addition, the anchor-free detection head used in step (3) uses an adaptive point set to capture the inherent geometric structure of the small target, which can characterize the features of the small target in a more fine-grained manner; These methods have effectively improved the detection accuracy of small targets.

Images