A remote sensing small target detection method, system, device and medium based on fusion cascade attention mechanism

Abstract

A remote sensing small target detection method, system, device and medium based on a fused cascade attention mechanism, the method comprising: inputting an image into a feature extraction network to obtain a multi-scale fused feature map, placing the map into a rotatable region candidate network, defining a new rotation target detection method and a custom loss function, selecting a candidate region map containing the target, performing classification prediction and bounding box regression on the map, and obtaining a detection result; the system, device and medium are used to implement a remote sensing small target detection method based on a fused cascade attention mechanism; the present invention provides an end-to-end target detection network, combines a multi-scale attention module (MAM) with a fused cascade attention module (FCAM), and optimizes and improves the model's loss function based on a new rotation box representation method; the invention has the characteristics of reducing the influence of background noise on detection results, enhancing small target feature information, improving the imbalance between positive and negative samples, improving the average detection accuracy of the model, and improving the performance of optical remote sensing target detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a remote sensing small target detection method, system, device and medium based on a fused cascade attention mechanism. Background Art

[0002] Optical remote sensing technology uses optical remote sensing sensors from satellites, aircraft, drones, and other devices to conduct large-scale ground observations. This technology enables efficient observation of a wide range of ground targets and resources. In recent years, with the continuous advancement of remote sensing imaging technology, the resolution and quality of remote sensing images have also continued to rise. A large amount of high-quality annotated optical remote sensing image data has begun to emerge, providing strong data support for object detection in remote sensing images and fine-grained recognition techniques specific to target subcategories.

[0003] Remote sensing target detection technology is a fundamental task in remote sensing image processing. Its goal is to locate targets of interest in remote sensing images, determine their spatial location within the image, and perform basic classification and identification. With the advancement of remote sensing technology, fine-grained recognition techniques, widely used in natural scenes, have begun to achieve breakthroughs in remote sensing. Compared with coarse-grained recognition techniques, fine-grained recognition techniques can significantly expand the application scope of remote sensing technology.

[0004] Existing remote sensing target detection methods include:

[0005] (1) Target detection methods in optical remote sensing images

[0006] Remote sensing target detection is a branch of general target detection technology. Due to its unique high-altitude bird's-eye view, target detection technology for remote sensing images needs to overcome more challenges. The typical two-stage target detector based on Faster RCNN proceeds as follows: first, feature extraction is performed through a convolutional network, candidate regions are generated through a region proposal network (RPN), and then the candidate regions are classified and regressed based on the features. However, when applied to optical remote sensing images, this type of method cannot adapt to the various rotating targets in remote sensing images, requiring additional methods to encode the orientation information of the anchor frame. At the same time, remote sensing images contain a large number of densely packed small targets, making accurate detection of these targets a challenge.

[0007] (2) Rotating target detection

[0008] In order to adapt to remote sensing targets in various directions, RRPN introduced rotated anchor frames into FasterRCNN. This algorithm was originally applied to text detection, but it also has good results in the field of remote sensing target detection. It improves the expression of anchor frames and adds a rotation angle θ to the original anchor frame parameters (x, y, w, h). Among them, x, y are the coordinates of the center point of the anchor frame, w, h are the width and height of the anchor frame, and θ is the angle between the anchor frame and the positive direction of the x-axis. Since this method presets parameters for 6 rotation angles, the total number of anchor frame parameters is increased by 6 times, which greatly increases the amount of calculation. In addition, this method sets too many anchor frames, which are too redundant, resulting in a large waste of resources.

[0009] To address the problem of redundant anchor frames in RRPN, a self-learning feature extraction module is designed. Through the rotation-sensitive RoIAlign, the directional rotation-invariant feature region is mapped to the horizontal feature region and fed into the final classifier. This method learns the rotation angle through the network and does not require additional anchor frames in multiple directions, thus solving the problem of redundant anchor frames. However, the network itself has a large number of parameters and also requires a lot of computation.

[0010] (3) Feature alignment method for rotating targets

[0011] The RefineDet network regresses the bounding box in a coarse-to-fine manner, making the positioning of features and candidate boxes more accurate. That is, the coarse-grained anchor box information is first obtained through the RPN network, and then a more accurate bounding box is obtained through the regression branch. However, the anchor box obtained by RefineDet's two regressions is not accurate enough, and the features used for regression are all features that are not aligned with the region of interest. The first step of the AlignDet method is to regress the initial anchor box to obtain the learned anchor box (learnedanchor). At this time, the anchor box is already closer to the real candidate box; in the second step, AlignDet uses variable convolution to add the offset given at this position to the convolution kernel, map the convolution kernel to the aligned position, and use the aligned features to regress the final anchor box position. The object features extracted by this method are not accurate enough and cannot reflect the specific detailed features of the object.

[0012] The invention, entitled "A Remote Sensing Small Target Detection Method Based on Dual Attention of FPN and PAN Networks", with publication number "CN114821341A", first pools the top-level feature maps in the FPN network to obtain channel vectors, then performs matrix operations on the channel vectors to obtain a channel attention matrix, then normalizes it to obtain a channel weight matrix, and multiplies the weights by the feature map to obtain a feature map with channel weights, which is then fused with low-level features. In the PAN network, the bottom-level feature maps are first channel-compressed to obtain spatial vectors, then perform matrix operations on the spatial vectors to obtain a spatial attention matrix, then normalize it to obtain a spatial weight matrix, and multiply the weights by the feature map to obtain a feature map with spatial weights, which is then fused with high-level features and finally sent to the detection head to generate detection results. This method uses the attention mechanism of the dual structure of FPN and PAN, and although the accuracy is improved, the complex structure makes this method have the disadvantages of large computational complexity and bloated model structure. Summary of the Invention

[0013] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a remote sensing small target detection method, system, equipment and medium based on a fused cascade attention mechanism. By providing an end-to-end target detection network, the multi-scale attention module MAM is combined with the fused cascade attention module FCAM. On the basis of the new rotating frame representation method, the loss function of the model is optimized and improved, with less background noise affecting the detection results, enhancing the feature information of small targets, improving the detection accuracy of various small targets, improving the imbalance problem of positive and negative samples, improving the average detection accuracy of the model, and improving the characteristics of optical remote sensing target detection performance.

[0014] A remote sensing small target detection method based on the fusion cascade attention mechanism includes the following steps:

[0015] Step 1: Input the image into the feature extraction network to obtain a multi-scale fusion feature map of the channel attention mechanism and multi-scale fusion;

[0016] Step 2: The channel attention mechanism and the multi-scale fusion feature map obtained in step 1 are put into the rotatable region candidate network. A new detection method for rotated objects and a custom loss function are defined for the candidate network. The candidate region map containing the object is selected through the region candidate network.

[0017] Step 3: Perform classification prediction and bounding box regression on the candidate region map obtained in step 2 to obtain the remote sensing small target detection result based on the fusion cascade attention mechanism.

[0018] The feature extraction network in step 1 is based on Resnet-50 as the backbone network. After optimization, a multi-scale attention module (MAM) and a fused cascade attention module (FCAM) are obtained. The specific process of obtaining a multi-scale fusion feature map of channel attention mechanism and multi-scale fusion in step 1 is as follows:

[0019] Step 1.1: Input the image to the backbone network Resnet-50 of the feature extraction network and perform a 6-layer convolution operation to extract the feature maps after the 3rd and 4th convolution operations;

[0020] Step 1.2: The features after the 3rd and 4th convolution operations extracted according to step 1.1 Figure 1 The data are sent to the fused cascade attention module FCAM to obtain a feature map with fused multi-scale features;

[0021] Step 1.3: The feature map with fused multi-scale features obtained in step 1.2 is added to the feature map obtained after the four-layer convolution operation of the backbone network through element-by-element addition to obtain a multi-scale fused feature map.

[0022] Step 1.4: Output the feature map with fused multi-scale features in step 1.2, the multi-scale fused feature map in step 1.3, and the feature map after the 6-layer convolution operation of the backbone network, and output the above feature maps in parallel to obtain the multi-scale fused feature map.

[0023] The construction process of the fused cascade attention module FCAM in step 1.2 is specifically as follows:

[0024] Step 1.2.1: Perform a convolution operation on the 4th layer of the backbone network through a deconvolution operation with holes to output the feature map;

[0025] Step 1.2.2: Input the feature map output by the backbone network layer 3 convolution operation into the multi-scale attention module MAM1 to obtain the feature Figure 2 , the features are obtained through step 1.2.1 Figure 1 and features Figure 2 , and fuse them by element-by-element addition to obtain the fused feature map, which is sent to the multi-scale attention module MAM2 to obtain a multi-scale fused feature map that combines the attention mechanism and multi-scale fusion.

[0026] The multi-scale attention module MAM1 or MAM2 combines the channel attention mechanism with the improved Inception structure and introduces a spatial attention module with jump links in parallel. The specific construction process of the multi-scale attention module MAM1 or MAM2 in step 1.2.2 is as follows:

[0027] Step 1.2.2.1: Input the feature map and apply 1×1 convolution to the feature map to compress the input feature dimension, thus obtaining a feature map with reduced output channels.

[0028] Step 1.2.2.2: Apply three convolution operations (1×1 convolution, 3×1 convolution combined with 1×3 convolution, and 5×1 convolution combined with 1×5 convolution) to the feature map output in step 1.2.2.1 to obtain features of different scales.

[0029] Step 1.2.2.3: For the three features of different scales output in step 1.2.2.2, connect them with 3×3 dilated convolutions to output features with 2 to 10 times the receptive field;

[0030] Step 1.2.2.4: Concatenate the three features of different scales output in step 1.2.2.3 by concat, and adjust the dimensions again by 1×1 convolution to obtain multi-scale fusion features;

[0031] Step 1.2.2.5: Perform channel-wise addition on the multi-scale fusion features output from step 1.2.2.4 and the channel attention feature map output from the channel attention mechanism branch, and output the feature map of the intermediate layer.

[0032] Step 1.2.2.6: Perform spatial attention on the feature map of the intermediate layer in step 1.2.2.5, and finally output a multi-scale attention feature map.

[0033] The channel attention mechanism branch construction process in step 1.2.2.5 is as follows:

[0034] Step 1.2.2.5.1: Input the original feature map and compress it into a 1-dimensional vector feature map through a global average pooling;

[0035] Step 1.2.2.5.2: Pass the feature map obtained in step 1.2.2.5.1 through two fully connected layers and then perform a ReLU activation function operation to learn the important weights of the channels and obtain the activated feature map.

[0036] Step 1.2.2.5.3: Based on the activation feature map obtained in step 1.2.2.5.2, map the weights to the real number domain through the Sigmoid function Get activation features of different layers;

[0037] Step 1.2.2.5.4: The activation features of different layers obtained in step 1.2.2.5.3 are multiplied element by element and applied to F i , performs feature selection in the channel domain, and its calculation formula is as follows. Finally, the channel attention feature map is output, and its calculation formula is as follows:

[0038] F c =σ(MLP(AvgPool(F))),

[0039] Where σ is the Sigmoid function, F is the input feature map, MLP is a multi-layer perceptron consisting of two fully connected layers and ReLU activation function, and AvgPool is the global average pooling operation.

[0040] The design process of the rotatable region candidate network in step 2 is as follows:

[0041] Step 2.1: Based on the multi-scale fusion feature map that has passed through the feature extraction network, connect 1×1 convolution to change the number of channels of the multi-scale fusion feature map from 256 to 64, that is, H×W×256->H×W×64, to obtain a feature map with reduced channels;

[0042] Step 2.2: Based on the feature map with reduced channel number obtained in step 2.1, a sliding window is used to traverse each pixel. At each pixel, a candidate region frame is generated according to the five preset horizontal anchor frames with the pixel as the center. The candidate region frame is then screened to obtain the region frame.

[0043] Five ratios of horizontal anchor boxes are set in the features of all levels at each spatial position, namely {3:1, 2:1, 1:1, 1:2, 1:3}. For each anchor box at a different position, six regression parameters (x, w, h, Δα, Δβ) are output as a directed anchor box.

[0044] The method for obtaining the regional frame diagram in step 2.2 is as follows:

[0045] The representation method of the area frame diagram is the representation method of the center point plus the offset. The coordinate representation formula of this method is:

[0046]

[0047] Where x and y are the coordinates of the center point of the anchor box, w and h are the width and height of the circumscribed horizontal rectangular anchor box of the directed anchor box, respectively. Δα and Δβ represent the top and right vertices of the directed anchor box, and the offset from the top and right midpoints of the circumscribed horizontal rectangular anchor box, respectively. The value can be positive or negative. (x1, y1), (x2, y2), (x3, y3) and (x4, y4) are the coordinates of the four vertices of the directed bounding box.

[0048] The custom loss function in step 2 is:

[0049]

[0050] Where i is the number of anchor boxes in each training batch, N is the maximum number of anchor boxes, and p is the maximum number of anchor boxes.i is the probability that the i-th anchor box belongs to the foreground, is the true candidate of the i-th anchor box, is the difference between the i-th anchor box and the true candidate box, that is, the offset, F reg is the SmoothL1 loss, which is defined as:

[0051]

[0052] A remote sensing small target detection system based on the fusion cascade attention mechanism, including:

[0053] Feature extraction module: extracts multi-scale features from the input image and fuses the feature maps of different scales to generate fused features;

[0054] Region Candidate Module: Based on the fusion features generated by the feature extraction module, it finds the region frame where the target is located and sends it to the classification prediction module;

[0055] Classification prediction module: classifies and predicts the region frame sent from the region candidate module, obtains and outputs the prediction results of remote sensing small target detection based on the fusion cascade attention mechanism.

[0056] A remote sensing small target detection device based on a cascaded attention mechanism, comprising:

[0057] Memory for storing computer programs;

[0058] A processor is used to implement the remote sensing small target detection method based on the fusion cascade attention mechanism when executing the computer program.

[0059] A computer-readable storage medium comprising:

[0060] The computer-readable storage medium stores a computer program, which, when executed by a processor, can detect a small remote sensing target based on a fused cascade attention mechanism.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. To address the complex and diverse backgrounds in optical remote sensing images, this paper designs a multi-scale attention module (MAM) to filter out useless features. It also designs a fused cascade attention module (FCAM) to achieve better small target detection capabilities. To address the extreme imbalance in dataset categories, this paper optimizes and improves the model's loss function based on a new rotating box representation method, thereby improving the model's average detection performance and enhancing optical remote sensing target detection performance.

[0063] 2. This paper combines the ideas of the Inception network and two attention mechanisms in the channel domain and spatial domain to design a multi-scale attention module (MAM). The multi-scale attention module (MAM) can combine the channel attention mechanism with the improved Inception structure, and then introduces a spatial attention module with skip links in parallel, which plays the role of simultaneously retaining important channel information and spatial information and expanding the receptive field.

[0064] 3. To address the problem of rotating targets, the present invention improves the region proposal network to adapt to targets with extreme aspect ratios and designs a rotatable region proposal network. That is, each layer of the feature pyramid is connected to a 3×3 convolution and two parallel 1×1 convolutions to learn directed anchor boxes. In order to adapt to targets with more extreme aspect ratios, five horizontal anchor boxes with different ratios are set in the features of all levels at each spatial position, namely {3:1, 2:1, 1:1, 1:2, 1:3}. For each anchor box at a different position, six regression parameters (x, w, h, Δα, Δβ) are output to represent a directed anchor box. The loss function of the rotatable region proposal network is defined as Extreme aspect ratio targets can be achieved.

[0065] 4. In the convolutional neural network, the present invention can extract the spatial information and channel information of each position in the image through the interconnection of a series of convolutions, nonlinear activation functions, and rotatable region candidate feature networks, and realizes the extraction of multi-scale features with strong discriminability through a mechanism.

[0066] 5. The Inception structure in the present invention reduces the number of parameters while ensuring model quality by widening the network, thereby improving the ability to extract high-dimensional features. It obtains features of different scales by performing multi-scale convolution and pooling operations, and finally fuses and superimposes these features, that is, by clustering sparse matrices into denser sub-matrices to improve computing performance.

[0067] 6. This invention imitates the way the human eye observes objects through the attention mechanism, learns the characteristic of the human eye paying extra attention to key areas, and designs a method that can strengthen local spatial areas. Through a self-learning network, it learns the weight information of each position, thereby suppressing useless background features and strengthening the features of objects such as sparse small targets, making the entire feature map more discriminative.

[0068] 7. The present invention provides an end-to-end target detection network, combines the multi-scale attention module (MAM) with the fused cascade attention module (FCAM), and optimizes and improves the model's loss function based on a new rotation box representation method. Therefore, it can achieve more accurate detection of various targets in complex backgrounds. Experiments have also verified the robustness and effectiveness of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 Flow chart of the method of the present invention.

[0070] Figure 2 This is the feature extraction network structure diagram of the present invention.

[0071] Figure 3 This is the structural diagram of the fused cascade attention module FCAM of the present invention.

[0072] Figure 4 This is the structure diagram of the multi-scale attention module MAM of the present invention.

[0073] Figure 5 This is the rotatable region candidate network diagram of the present invention.

[0074] Figure 6 This is the visualization result diagram on the DOTA dataset of the present invention.

[0075] Figure 7 This is a comparison chart of the performance of the present invention and the current methods in the same field. DETAILED DESCRIPTION

[0076] The working principle of the present invention will be described in detail below with reference to the accompanying drawings.

[0077] See also Figure 1 , a remote sensing small target detection method based on the fusion cascade attention mechanism, including the following steps:

[0078] Step 1: Input the image into the feature extraction network to obtain a multi-scale fusion feature map of attention mechanism and multi-scale fusion;

[0079] Step 2: The multi-scale fusion feature map obtained by the attention mechanism and multi-scale fusion in step 1 is put into the rotatable region candidate network. A new detection method for rotated objects and a custom loss function are defined for the candidate network. The candidate region map containing the object is selected through the network.

[0080] Step 3: Perform classification prediction and bounding box regression on the candidate region map obtained in step 2 to obtain the remote sensing small target detection result based on the fused cascade attention mechanism; the attention mechanism imitates the way the human eye observes objects, learns the characteristics of the human eye paying extra attention to key areas, and designs a method that can enhance local spatial areas. Through a self-learning network, the weight information of each position is learned, thereby suppressing useless background features and enhancing the features of sparse small targets and other objects, making the entire feature map more discriminative.

[0081] The feature extraction network in step 1 is based on Resnet-50 as the backbone network. After optimization, a multi-scale attention module MAM and a fused cascade attention module FCAM are obtained; a multi-scale attention module MAM is designed to filter useless features; a fused cascade attention module FCAM is designed to obtain better small target detection capabilities.

[0082] See also Figure 2 The specific process of step 1 to obtain the multi-scale fusion feature map of attention mechanism and multi-scale fusion is as follows:

[0083] Step 1.1: Input the image to the backbone network Resnet-50 of the feature extraction network and perform a 6-layer convolution operation to extract the feature maps after the 3rd and 4th convolution operations;

[0084] Step 1.2: The features after the 3rd and 4th convolution operations extracted according to step 1.1 Figure 1 The data are sent to the fused cascade attention module FCAM to obtain a feature map with fused multi-scale features;

[0085] Step 1.3: Perform channel addition on the feature map with fused multi-scale features obtained in step 1.2 and the feature map obtained after the four-layer convolution operation of the backbone network to obtain a multi-scale fused feature map.

[0086] Step 1.4: Output the feature map with fused multi-scale features in step 1.2, the multi-scale fused feature map in step 1.3, and the feature map after the backbone network 6-layer convolution operation, and output the above feature maps in parallel to obtain a multi-scale fused feature map; the feature extraction network in step 1 is based on Resnet-50 as the backbone network, and after optimization, a multi-scale attention module MAM and a fused cascade attention module FCAM are obtained.

[0087] See also Figure 3 The construction process of the fused cascade attention module FCAM in step 1.2 is specifically as follows:

[0088] Step 1.2.1: Through a deconvolution operation with holes, the 4th layer of the backbone network is convolutionally operated, and the output feature map, that is, the C4 layer feature map, is doubled upsampled to output the feature map. Figure 1 ;

[0089] Step 1.2.2: Input the feature map output by the backbone network layer 3 convolution operation into the multi-scale attention module MAM1 to obtain the feature Figure 2 , the features are obtained through step 1.2.1 Figure 1 and features Figure 2 , the fusion is performed by element-by-element addition to obtain the fused feature map, which is sent to the multi-scale attention module MAM2 to obtain a multi-scale fused feature map that combines the attention mechanism with multi-scale fusion; it can enhance the semantic information of the fused features and increase the number of features of small-scale targets;

[0090] A fused cascade attention module (FCAM) is added to the C3 and C4 layers of the backbone network. The serial design of such modules can improve the multi-scale feature extraction capability, thereby improving the problem of many small objects and dense distribution.

[0091] See also Figure 4 The specific construction process of the multi-scale attention modules MAM1 and MAM2 in step 1.2.2 is as follows:

[0092] Step 1.2.2.1: Input the feature map and use 1×1 convolution to compress the dimension of the input features, which can reduce the amount of computation and obtain a feature map with a reduced number of output channels;

[0093] Step 1.2.2.2: Apply three convolution operations (1×1 convolution, 3×1 convolution combined with 1×3 convolution, and 5×1 convolution combined with 1×5 convolution) to the feature map output in step 1.2.2.1 to obtain features of different scales.

[0094] Step 1.2.2.3: Connect the three features of different scales output in step 1.2.2.2 with 3×3 dilated convolutions to output features with 2 to 10 times the receptive field. This can further improve the network's receptive field and deep feature extraction capabilities while limiting the increase in computational complexity.

[0095] Step 1.2.2.4: Concatenate the three features of different scales output in step 1.2.2.3 by concat, and adjust the dimensions again by 1×1 convolution to obtain multi-scale fusion features;

[0096] Step 1.2.2.5: Perform channel-wise addition on the multi-scale fusion features output from step 1.2.2.4 and the channel attention feature map output from the channel attention mechanism branch, and output the feature map of the intermediate layer.

[0097] Step 1.2.2.6: Perform spatial attention on the feature map of the intermediate layer in step 1.2.2.5, and finally output a multi-scale attention feature map;

[0098] The multi-scale attention module MAM1 or MAM2 combines the channel attention mechanism with the improved Inception structure and introduces a spatial attention module with skip links in parallel. Combining the ideas of the Inception network and the two attention mechanisms in the channel domain and spatial domain, the multi-scale attention modules MAM, namely MAM1 and MAM2, are designed. The multi-scale attention module MAM can combine the channel attention mechanism with the improved Inception structure, and then introduces a spatial attention module with skip links in parallel, which plays the role of simultaneously retaining important channel information and spatial information and expanding the receptive field. The Inception structure reduces the number of parameters while ensuring model quality by widening the network, thereby improving the ability to extract high-dimensional features. It obtains features of different scales by performing multi-scale convolution and pooling operations, and finally fuses and superimposes these features, that is, by clustering sparse matrices into denser sub-matrices to improve computing performance.

[0099] The channel attention mechanism branch construction process in step 1.2.2.5 is as follows:

[0100] Step 1.2.2.5.1: Input the original feature map and compress it into a 1-dimensional vector feature map through a global average pooling;

[0101] Step 1.2.2.5.2: Pass the feature map obtained in step 1.2.2.5.1 through two fully connected layers and then perform a ReLU activation function operation to learn the important weights of the channels and obtain the activated feature map.

[0102] Step 1.2.2.5.3: Based on the activation feature map obtained in step 1.2.2.5.2, map the weights to the real number domain through the Sigmoid function Get activation features of different layers;

[0103] Step 1.2.2.5.4: The activation features of different layers obtained in step 1.2.2.5.3 are multiplied element by element and applied to F i , performs feature selection in the channel domain, and its calculation formula is as follows. Finally, the channel attention feature map is output, and its calculation formula is as follows:

[0104] F c =σ(MLP(AvgPool(F))),

[0105] Where σ is the Sigmoid function, F is the input feature map, MLP is a multi-layer perceptron consisting of two fully connected layers and ReLU activation function, and AvgPool is the global average pooling operation.

[0106] See also Figure 5 , the design process of the rotatable region candidate network is:

[0107] Step 2.1: Based on the multi-scale fusion feature map that has passed through the feature extraction network, connect 1×1 convolution to change the number of channels of the multi-scale fusion feature map from 256 to 64, that is, H×W×256→H×W×64, to obtain a feature map with reduced channels;

[0108] Step 2.2: Based on the feature map with reduced channel number obtained in step 2.1, a sliding window method is used to traverse each pixel point. At each pixel point, with the pixel point as the center, a candidate region frame map is generated according to the preset five ratios of horizontal anchor frames. After training and fitting, the region frame map is obtained, which is the decoding process in the figure. Taking a point (x, y) as an example, five ratios of horizontal anchor frames are set in the features of all levels at each spatial position, namely {3:1, 2:1, 1:1, 1:2, 1:3}. For each anchor frame at a different position, six regression parameters (x, w, h, Δα, Δβ) are output to represent a directed anchor frame, which can achieve the goal of extreme aspect ratios.

[0109] The method for expressing the area frame diagram in step 2.2 is the method for expressing the center point plus the offset. The coordinate expression formula of this method is as follows:

[0110]

[0111] Among them, x, y are the coordinates of the center point of the anchor box, w, h are the width and height of the circumscribed horizontal rectangular anchor box of the directed anchor box, Δα and Δβ represent the top and right vertices of the directed anchor box, and the offset from the top and right midpoints of the circumscribed horizontal rectangular anchor box, respectively. The value can be positive or negative; (x1, y1), (x2, y2), (x3, y3) and (x4, y4) are the coordinates of the four vertices of the directed bounding box.

[0112] The custom loss function in step 2 is:

[0113]

[0114] Where i is the number of anchor boxes in each training batch, N is the maximum number of anchor boxes, and p is the maximum number of anchor boxes. i is the probability that the i-th anchor box belongs to the foreground, is the true candidate of the i-th anchor box, is the difference between the i-th anchor box and the true candidate box, that is, the offset, F reg It is the Smooth L1 loss, which is defined as:

[0115]

[0116] In order to verify the effectiveness and feasibility of the rotated multi-scale attention network proposed in this paper, experiments were conducted on the DOTA dataset, UCAS-AOD dataset and HRSC2016 dataset.

[0117] On the DOTA dataset, we first validate the small target detection capability of our method for the coarse-grained target detection problem. Then, we conduct ship detection experiments on the HRSC2016 dataset, and then extend the method to fine-grained recognition tasks for preliminary verification. Finally, we conduct generalization experiments on the UCAS-AOD dataset to verify the robustness of our method.

[0118] In order to evaluate the performance of remote sensing image target detection models, the rotational multi-scale attention network RMAN is compared with some excellent methods currently designed for remote sensing target detection. On the DOTA dataset, CAD-Net, SCR-Det, DRN and R3-Det are compared.

[0119] The test results are shown in the table:

[0120]

[0121] Among them, the 15 types of targets are: airplane (PL), baseball field (BD), bridge (BR), track and field (GTF), small vehicle (SV), large vehicle (LV), ship (SH), tennis court (TC), basketball court (BC), oil tank (ST), football field (SBF), intersection (RA), port (HA), swimming pool (SP) and helicopter (HC);

[0122] As can be seen from the above table, the average precision of the multi-scale attention network RMAN of the present invention on the DOTA dataset reached 73.34%, which is 1.38% higher than the average precision of R3Det. In each test result, the multi-scale attention network RMAN achieved the best performance in the SV, LV, TC, ST and HA categories; in the small vehicle class, the precision of the multi-scale attention network RMAN reached 74.21%, which is 0.73% higher than DRN; in the oil tank class, the AP of the multi-scale attention network RMAN reached 88.53%, which is 1.67% higher than the precision of SCRDet.

[0123] See also Figure 6Figure (a) is for densely distributed vehicle category targets. The present invention accurately selects the vehicle category targets in the picture, among which the yellow frame selects a small truck and the green frame selects a large truck, and makes a correct judgment on their categories; Figure (b) is for densely distributed ship category targets, among which the green frame selects the ship category target and the blue frame selects the port category target. It can be seen that the present invention accurately finds the target and makes a correct judgment; Figure (c) is for targets of different categories and scales. The yellow frame selects the small train target, the light green frame selects the ship category target, and the dark green frame selects the tennis court target; it can be seen that the present invention accurately and completely selects the target and makes a correct classification; Figure (d) The dark green frame selects the tennis court target and the yellow frame selects the small truck target. It can be seen that the present invention has a relatively accurate detection effect on various targets under complex backgrounds.

[0124] See also Figure 7 ,Different from the DOTA dataset, the objects in the HRSC2016 dataset are mostly ship targets with a large aspect ratio and different orientations;,when performing binary classification detection on the HRSC2016 dataset, the,multi-scale attention network RMAN achieves an accuracy of 93.3% on the HRSC216 dataset,,which is 0.6% higher than DRN.

[0125] In order to verify the generalization ability, the present invention uses the multi-scale attention network RMAN to perform cross-dataset verification on the UCAS-AOD dataset. The results are shown in Table 2:

[0126] algorithm DOTA (mAP%) UCAS-AOD (mAP%) DRN 70.70 85.73 R3Det 71.69 83.16 RMAN(Ours) 73.34 87.24

[0127] Table 2: Generalization verification results on the UCAS-AOD dataset

[0128] The model proposed in this embodiment is trained on the DOTA dataset, but achieves good results on the UCAS-AOD dataset, improving the overall detection accuracy;

[0129] In addition, an ablation experiment was conducted on the DOTA dataset to verify the overall impact of the modules and optimization methods proposed in this section on the algorithm. The experimental results are shown in Table 3:

[0130]

[0131] Table 3: Ablation experiment results on the DOTA dataset

[0132] As shown in Table 3, the improved loss function improves mAP by 2.27%. Furthermore, the multi-scale attention module (MAM) improves mAP by 2.83%. The combination of the MAM and the fused cascaded attention module (FCAM) combines deep and shallow features in a top-down manner, resulting in an overall 6.30% improvement in detection accuracy compared to the method using only the improved loss function. Overall, the combination of the improved loss function, the MAM, and the FCAM improves mAP by 11.02% over the baseline.

[0133] A remote sensing small target detection system based on the fusion cascade attention mechanism, including:

[0134] Feature extraction module: extracts multi-scale features from the input image and fuses the feature maps of different scales to generate fused features;

[0135] Region Candidate Module: Based on the fusion features generated by the feature extraction module, it finds the region frame where the target is located and sends it to the classification prediction module;

[0136] Classification prediction module: classifies and predicts the region frame sent from the region candidate module, obtains the prediction result and outputs it.

[0137] A remote sensing small target detection device based on a cascaded attention mechanism, comprising:

[0138] Memory for storing computer programs;

[0139] A processor is used to implement the remote sensing small target detection method based on the fusion cascade attention mechanism when executing the computer program.

[0140] A computer-readable storage medium stores a computer program, which, when executed by a processor, can detect a small remote sensing target based on a fused cascade attention mechanism.

[0141] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The processor is the control center of the remote sensing small target detection device based on the fusion cascade attention mechanism, and utilizes various interfaces and lines to connect the various parts of the remote sensing small target detection device based on the fusion cascade attention mechanism.

[0142] When the processor executes the computer program, the steps of the above-mentioned remote sensing small target detection method based on the fused cascade attention mechanism are implemented, for example: the input image enters the feature extraction network to obtain a multi-scale fusion feature map of the channel attention mechanism and multi-scale fusion; the multi-scale fusion feature map of the channel attention mechanism and multi-scale fusion obtained in step 1 is placed in a rotatable region candidate network, a new detection method for the rotated target and a custom loss function are defined for the candidate network, and a candidate region map containing the target is selected from the region candidate network; the remote sensing small target detection result based on the fused cascade attention mechanism is achieved.

[0143] Alternatively, when the processor executes the computer program, it realizes the functions of each module in the above system, for example: feature extraction module: performs multi-scale feature extraction on the input image, and fuses the feature images of different scales to generate fusion features; region candidate module: based on the fusion features generated by the feature extraction module, finds the region frame where the target is located, and sends it to the classification prediction module; classification prediction module: performs classification prediction on the region frame sent to the region candidate module, obtains and outputs the remote sensing small target detection result prediction result based on the fusion cascade attention mechanism; the output obtains the remote sensing small target detection result based on the fusion cascade attention mechanism.

[0144] Exemplarily, the computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete preset functions, and the instruction segments are used to describe the execution process of the computer program in the device for remote sensing small target detection based on the fusion cascade attention mechanism. For example, the computer program can be divided into a feature extraction module, a region candidate module, and a classification prediction module. The specific functions of each module are as follows: Feature extraction module: performs multi-scale feature extraction on the input image and fuses the feature maps of different scales to generate fusion features; Region candidate module: Based on the fusion features generated by the feature extraction module, finds the region frame where the target is located and sends it to the classification prediction module; Classification prediction module: performs classification prediction on the region frame sent to the region candidate module, obtains and outputs the prediction results of remote sensing small target detection based on the fusion cascade attention mechanism; The output obtains the result of the remote sensing small target detection system based on the fusion cascade attention mechanism.

[0145] The remote sensing small target detection device based on the fused cascade attention mechanism can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The remote sensing small target detection device based on the fused cascade attention mechanism may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the above is an example of a remote sensing small target detection device based on the fused cascade attention mechanism and does not constitute a limitation on a remote sensing small target detection device based on the fused cascade attention mechanism. It may include more components than the above, or a combination of certain components, or different components. For example, the remote sensing small target detection device based on the fused cascade attention mechanism may also include input and output devices, network access devices, buses, etc.

[0146] The memory can be used to store the computer program and / or module, and the processor realizes the various functions of the remote sensing small target detection device based on the fused cascade attention mechanism by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory.

[0147] The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function or an image playback function); and the data storage area may store data generated based on the use of the mobile phone (such as audio data, a phone book, etc.). Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0148] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method for remote sensing small target detection based on a fusion cascade attention mechanism.

[0149] If the module / unit integrated in the remote sensing small target detection system based on the fusion cascade attention mechanism is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0150] The present invention implements all or part of the process of the aforementioned method for detecting small remote sensing targets based on a fused cascaded attention mechanism, and can also be accomplished by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the aforementioned method for detecting small remote sensing targets based on a fused cascaded attention mechanism. The computer program includes computer program code, which can be in source code form, object code form, executable file, or a preset intermediate form.

[0151] The computer-readable storage medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0152] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable storage media do not include electrical carrier signals and telecommunication signals.

[0153] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented using dedicated logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated hardware.

[0154] Those skilled in the art will appreciate that the above-described devices and methods can be implemented using computer-executable instructions and / or contained in processor control code, for example, such code being provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays and programmable logic devices, or by software executed by various types of processors, or by a combination of the above-described hardware circuits and software, such as firmware.

Claims

1. A remote sensing small target detection method based on the fusion cascade attention mechanism, characterized in that: The following steps are involved: Step 1: Input the image into the feature extraction network to obtain a multi-scale fusion feature map of the channel attention mechanism and multi-scale fusion; Step 2: The channel attention mechanism and the multi-scale fusion feature map obtained in step 1 are put into the rotatable region candidate network. A new detection method for rotated objects and a custom loss function are defined for the candidate network. The candidate region map containing the object is selected through the region candidate network. The design process of the rotatable region candidate network in step 2 is as follows: Step 2.1: Based on the multi-scale fusion feature map that has passed through the feature extraction network, connect 1×1 convolution to change the number of channels of the multi-scale fusion feature map from 256 to 64, that is, H×W×256→H×W×64, to obtain a feature map with reduced channels; Step 2.2: Based on the feature map with reduced channel number obtained in step 2.1, a sliding window is used to traverse each pixel. At each pixel, a candidate region frame is generated according to the five preset horizontal anchor frames with the pixel as the center. The candidate region frame is then screened to obtain the region frame. Five horizontal anchor boxes with different ratios are set in the features of all levels at each spatial position, namely {3:1, 2:1, 1:1, 1:2, 1:3}. For each anchor box at a different position, six regression parameters (x, w, h, Δα, Δβ) are output to represent a directed anchor box. The method for obtaining the regional frame diagram in step 2.2 is as follows: The representation method of the area frame diagram is the representation method of the center point plus the offset. The coordinate representation formula of this method is: Where x, y are the coordinates of the center point of the anchor box, w, h are the width and height of the circumscribed horizontal rectangular anchor box of the directed anchor box, Δα and Δβ represent the top and right vertices of the directed anchor box, and the offset from the top and right midpoints of the circumscribed horizontal rectangular anchor box, respectively. The value can be positive or negative; (x1, y1), (x2, y2), (x3, y3) and (x4, y4) are the coordinates of the four vertices of the directed bounding box; Step 3: Perform classification prediction and bounding box regression on the candidate region map obtained in step 2 to obtain the remote sensing small target detection result based on the fusion cascade attention mechanism.

2. A remote sensing small target detection method based on fusion cascade attention mechanism according to claim 1, characterized in that: The feature extraction network in step 1 is based on Resnet-50 as the backbone network. After optimization, a multi-scale attention module (MAM) and a fused cascade attention module (FCAM) are obtained. The specific process of obtaining a multi-scale fusion feature map of channel attention mechanism and multi-scale fusion in step 1 is as follows: Step 1.1: Input the image to the backbone network Resnet-50 of the feature extraction network and perform a 6-layer convolution operation to extract the feature maps after the 3rd and 4th convolution operations; Step 1.2: The feature maps after the 3rd and 4th convolution operations extracted in step 1.1 are fed into the fused cascade attention module FCAM to obtain a feature map with fused multi-scale features; Step 1.3: The feature map with fused multi-scale features obtained in step 1.2 is added to the feature map obtained after the four-layer convolution operation of the backbone network through element-by-element addition to obtain a multi-scale fused feature map. Step 1.4: Output the feature map with fused multi-scale features in step 1.2, the multi-scale fused feature map in step 1.3, and the feature map after the 6-layer convolution operation of the backbone network, and output the above feature maps in parallel to obtain the multi-scale fused feature map.

3. The remote sensing small target detection method based on the fusion cascade attention mechanism according to claim 2 is characterized in that: The construction process of the fused cascade attention module FCAM in step 1.2 is specifically as follows: Step 1.2.1: Perform a convolution operation on the 4th layer of the backbone network through a deconvolution operation with holes to output the feature map; Step 1.2.2: Input the feature map output by the convolution operation of the third layer of the backbone network into the multi-scale attention module MAM1 to obtain feature map 2. The feature map 1 and feature map 2 obtained in step 1.2.1 are fused by element-by-element addition to obtain the fused feature map, which is then sent to the multi-scale attention module MAM2 to obtain a multi-scale fused feature map that combines the attention mechanism and multi-scale fusion.

4. A remote sensing small target detection method based on fusion cascade attention mechanism according to claim 3, characterized in that: The multi-scale attention module MAM1 or MAM2 combines the channel attention mechanism with the improved Inception structure and introduces a spatial attention module with jump links in parallel. The specific construction process of the multi-scale attention module MAM1 or MAM2 in step 1.2.2 is as follows: Step 1.2.2.1: Input the feature map and apply 1×1 convolution to the feature map to compress the dimension of the input features, thereby obtaining a feature map with reduced output channels. Step 1.2.2.2: Apply three convolution operations (1×1 convolution, 3×1 convolution combined with 1×3 convolution, and 5×1 convolution combined with 1×5 convolution) to the feature map output in step 1.2.2.1 to obtain features of different scales. Step 1.2.2.3: For the three features of different scales output in step 1.2.2.2, connect them with 3×3 dilated convolutions to output features with 2 to 10 times the receptive field; Step 1.2.2.4: Concatenate the three features of different scales output in step 1.2.2.3 by concat, and adjust the dimension again by 1×1 convolution to obtain multi-scale fusion features; Step 1.2.2.5: Perform channel-wise addition on the multi-scale fusion features output from step 1.2.2.4 and the channel attention feature map output from the channel attention mechanism branch, and output the feature map of the intermediate layer. Step 1.2.2.6: Perform spatial attention on the feature map of the intermediate layer in step 1.2.2.5, and finally output a multi-scale attention feature map.

5. A remote sensing small target detection method based on fusion cascade attention mechanism according to claim 4, characterized in that: The channel attention mechanism branch construction process in step 1.2.2.5 is as follows: Step 1.2.2.5.1: Input the original feature map and compress it into a 1-dimensional vector feature map through a global average pooling; Step 1.2.2.5.2: Pass the feature map obtained in step 1.2.2.5.1 through two fully connected layers and then perform a ReLU activation function operation to learn the important weights of the channels and obtain the activated feature map. Step 1.2.2.5.3: Based on the activation feature map obtained in step 1.2.2.5.2, map the weights to the real number domain through the Sigmoid function Get activation features of different layers; Step 1.2.2.5.4: The activation features of different layers obtained in step 1.2.2.5.3 are multiplied element by element and applied to F i , perform feature selection in the channel domain, the calculation formula is as follows, and finally output the channel attention feature map, the calculation formula is as follows: F c =σ(MLP(AvgPool(F))), Among them, σ is the Sigmoid function, F is the input feature map, MLP is a multi-layer perceptron consisting of two fully connected layers and ReLU activation function, and AvgPool is the global average pooling operation.

6. The remote sensing small target detection method based on the fusion cascade attention mechanism according to claim 1 is characterized in that: The custom loss function in step 2 is: Where i is the number of anchor boxes in each training batch, N is the maximum number of anchor boxes, and p is the maximum number of anchor boxes. i is the probability that the i-th anchor box belongs to the foreground, is the true candidate of the i-th anchor box, is the difference between the i-th anchor box and the true candidate box, that is, the offset, F reg It is the Smooth L1 loss, which is defined as:

7. A remote sensing small target detection system based on the method of claim 1 and fused cascade attention mechanism, characterized in that: include: Feature extraction module: extracts multi-scale features from the input image and fuses the feature maps of different scales to generate fused features; Region Candidate Module: Based on the fusion features generated by the feature extraction module, it finds the region frame where the target is located and sends it to the classification prediction module; Classification prediction module: classifies and predicts the region frame sent from the region candidate module, obtains and outputs the prediction results of remote sensing small target detection based on the fusion cascade attention mechanism.

8. A remote sensing small target detection device based on a fusion cascade attention mechanism, characterized in that: include: memory for storing computer programs; A processor is used to implement the remote sensing small target detection method based on the fusion cascade attention mechanism of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that include: The computer-readable storage medium stores a computer program, which, when executed by a processor, can detect a small remote sensing target based on a fused cascade attention mechanism according to claims 1-6.

Images