Remote sensing image target detection method based on fused convolutional attention mechanism

Abstract

The present invention discloses a remote sensing image target detection method based on a fused convolutional attention mechanism, which solves the problems of low accuracy and slow convergence speed in small target detection. The implementation includes: collecting and processing remote sensing image data; building a feature extraction backbone network and a fused convolution Transformer encoding-decoding architecture; forming a fused convolutional attention mechanism target detection network model; and training and testing the target detection network model. The present invention adopts a pyramid-structured downsampling feature extraction backbone network to output feature matrices of the same size for input images of different sizes; builds a convolution module with deep convolution and point-by-point convolution to enhance the ability to extract information from local features of remote sensing images; replaces some attention heads with convolution modules to reduce the large number of parameters in matrix operations and reduce training time. It is used in fields such as aviation aircraft, remote sensing satellites, smart transportation, and smart agriculture that have high requirements for real-time and accuracy in remote sensing image target detection.

Description

Technical Field

[0001] This invention belongs to the technical field of remote sensing image target detection, primarily involving target detection in optical remote sensing images. Specifically, it provides a remote sensing image target detection method based on a fused convolutional attention mechanism. It has applications in areas such as real-time detection of ground targets by aircraft. Background Art

[0002] Remote sensing technology uses non-contact methods to acquire characteristic information about distant targets. It obtains this information by recording and analyzing the electromagnetic wave characteristics of the target, using specialized equipment and systems. Over the past few decades, remote sensing technology has been widely applied in a variety of fields, including agricultural development, geological analysis, ocean monitoring, military reconnaissance, and environmental protection.

[0003] Target detection has become a significant research hotspot in fields such as remote sensing imagery object recognition and computer vision. Target detection can identify specific targets in an image and determine their type and location. This technology plays a vital role in intelligent transportation, smart cities, public safety, military warfare, and other fields. Therefore, research on target detection in remote sensing image data is of great significance in fields such as oceanography, military affairs, and agriculture, for example, by reducing costs, improving efficiency, and promoting scientific and technological progress in these fields. With the rapid development of high-resolution satellites, the number of high-resolution remote sensing images has increased dramatically. Therefore, target detection in remote sensing images based on big data is a pressing need in the current field of high-resolution remote sensing image detection.

[0004] The development of object detection technology can be traced back to the 1990s. At the time, object detection methods were primarily based on manual feature extraction and classifier training, such as support vector machines (SVMs) and AdaBoost. However, these methods struggled to adapt to complex scene changes. With the widespread adoption of deep learning technology, object detection has made significant progress. Deep learning is an effective machine learning method with a strong ability to learn complex data representations. Convolutional neural networks (CNNs) are one of the most commonly used models for deep learning object detection. They can learn the complex underlying feature representations of each object in an image.

[0005] Early object detection methods such as the R-CNN series, Fast R-CNN, and Faster R-CNN fully utilized the learning capabilities of CNNs, significantly improving the accuracy of object detection. However, these methods still suffer from high computational complexity and slow inference speed.

[0006] To address these issues, the development of remote sensing image object detection technology has shifted towards single-stage object detection methods, such as YOLO, SSD, and RetinaNet. These methods can perform object detection in a single stage, reducing computational complexity and improving detection accuracy and inference speed. Furthermore, they have achieved excellent performance in various object detection benchmarks.

[0007] Transformer-based object detection methods are a new direction in the field of object detection that has become popular in recent years. The main idea behind this type of method is to apply the encoder-decoder architecture originally proposed for natural language processing (NLP) tasks to object detection tasks.

[0008] Compared with convolution, the visual Transformer breaks through the limitation of traditional convolutional neural network target detection models that cannot be calculated in parallel; the number of times the Transformer calculates the correlation between two target positions does not increase with distance; the self-attention mechanism can produce a more interpretable model. The encoder module calculates the attention matrix based on the feature map. Each value on the matrix actually constructs the predicted box coordinates, which can directly predict the target box.

[0009] At the heart of the Transformer approach is the self-attention mechanism, which enables the model to focus on different regions of the input image and dynamically adjust the importance of each region. Compared to traditional convolutional neural network (CNN)-based methods, Transformer-based object detection methods are more flexible and can handle complex scenes with multiple objects.

[0010] One of the first works in this field is DETR, which proposed an end-to-end object detection framework based on Transformer. DETR uses a set of queues to predict object locations and categories, and uses an encoder-decoder architecture to process images and output predictions. The self-attention mechanism allows DETR to handle instances of different scales and shapes and perform object detection in a single stage, making it more effective than traditional two-stage methods.

[0011] Although the current Transformer remote sensing image end-to-end target detection framework based on DETR can achieve good detection results in the field of remote sensing image target detection, there are still some difficulties, such as the difficulty in converging the attention mechanism, which leads to long training time, and the inability of the attention mechanism to effectively obtain local information, which leads to low accuracy of the model in detecting small targets.

[0012] In summary, although DETR proposes a simplified framework for remote sensing image target detection and improves the overall detection performance, problems such as low small target detection performance and slow model convergence speed have not been solved. Summary of the Invention

[0013] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a remote sensing image target detection method based on a fused convolutional attention mechanism with stronger ability to acquire local image features and faster model convergence speed.

[0014] The present invention is a remote sensing image target detection method based on a fused convolutional attention mechanism, characterized by comprising the following steps:

[0015] Step 1: Collect and process remote sensing image data: Obtain public remote sensing images from public websites and divide them into training data sets, validation data sets, and test data sets to form a remote sensing image dataset. The remote sensing image dataset contains a total of 15 types of targets, namely: airplanes, ships, storage tanks, baseball fields, tennis courts, basketball courts, playgrounds, ports, bridges, large vehicles, small vehicles, helicopters, roundabouts, football fields, and swimming pools; generate a txt file with the coordinates and category information of all targets in the original image data of the remote sensing image dataset, and input it into the constructed feature extraction backbone network together with the original image data;

[0016] Step 2: Build a feature extraction backbone network: The constructed feature extraction backbone network is composed of four convolution groups connected in sequence. The first convolution group is composed of a residual connection unit through a convolution layer, a Norm layer, an activation function layer, and a maximum pooling layer. The second, third, and fourth convolution groups are respectively composed of different numbers of residual connection units connected in sequence. The structure of each residual connection unit is a convolution layer, a GN layer, and an activation function layer stacked in sequence. The input original image data is downsampled by the constructed feature extraction backbone network and then outputs a remote sensing image feature matrix.

[0017] Step 3, build a fused convolution Transformer encoder: the built Transformer encoder includes a fused convolution multi-head attention module composed of a convolution module and an attention module connected in parallel. The encoder includes a fused convolution multi-head attention module, a residual connection and layer normalization module, a forward propagation module, a residual connection and layer normalization module from the input end, wherein the convolution module in the fused convolution multi-head attention module includes a first convolution layer, a first activation function layer, a second convolution layer, a BN layer, a second activation function layer, and a third convolution layer connected in sequence, and the attention module includes an LN layer, a self-attention layer, and a feedforward network layer connected in sequence; wherein the ratio of the convolution module to the attention module is 4:4, and the matrix size after the output of the convolution module is the same as that of the attention module, and the output matrix of the fused convolution Transformer encoder module is formed by concat cascading;

[0018] Step 4: Build a Transformer decoder with a hybrid attention mechanism: The decoder uses a self-attention mechanism to process redundant information on the input target query matrix. The cross-attention mechanism models the relationship between the feature matrix output by the encoder and the target query matrix. The forward propagation module performs feature transformation on the image features and the prediction box.

[0019] Step 5: Construct a fused convolutional attention mechanism target detection network model: Build a fused convolutional attention mechanism target detection network model, which is composed of a feature extraction backbone network, a fused convolutional Transformer encoder, and a hybrid attention mechanism Transformer decoder, referred to as the network model.

[0020] Step 6: Train the network model: Use the training data set to train the network model to obtain a trained fusion convolutional attention mechanism target detection network model;

[0021] Step 7, test the network model: Use the trained fusion convolutional attention mechanism target detection network model to detect the test dataset, that is, input the test set into the trained network model to obtain the detection results of each type of target in the remote sensing image dataset, including the average precision AP and the average precision mean mAP of all types of targets.

[0022] The present invention solves the technical problems of slow convergence of training models and low accuracy of detection models for small targets in an end-to-end remote sensing image target detection framework.

[0023] Compared with the prior art, the advantages of the present invention are as follows:

[0024] Improved detection accuracy of the model for small targets: The present invention designs a convolution module in the encoder, which consists of point-by-point convolution, depth-wise convolution, activation function, and normalization layer. The module obtains local information of the image without changing the size of the encoder input matrix; the encoder formed by parallel connection of the convolution module and the attention module has good feature extraction capabilities for both global and local features of the image; the present invention improves the detection accuracy of small targets while ensuring the detection accuracy of the target detection network model fused with the convolutional attention mechanism for large targets.

[0025] Reduced model training time: In the prior art, an encoder based on the attention mechanism calculates the quadratic power of the image pixel count when encoding an image, resulting in high computational complexity and a large number of parameters. The convolution module in the encoder designed in the present invention uses point-by-point convolution and depth-wise convolution, which have the advantages of a small number of parameters, reducing the computational complexity of the model, accelerating the convergence of the model, and reducing the model training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of the implementation process of the present invention;

[0027] Figure 2 This is a diagram of the backbone network structure for extracting image features in the present invention;

[0028] Figure 3 This is a structural diagram of the Transformer encoder with fused convolution constructed by the present invention;

[0029] Figure 4 is a structural diagram of a convolutional module constructed in the encoder of the present invention;

[0030] Figure 5 It is a flow chart of the encoder-decoder of the present invention;

[0031] Figure 6 is an experimental result diagram of the present invention, wherein Figure 6 (a) is a picture of the remote sensing image detection results of small vehicles and large vehicles in the test set of the DOTA dataset. Figure 6 (b) is a picture of the detection results of remote sensing images containing small vehicles and circular loop targets in the test set of the DOTA dataset. DETAILED DESCRIPTION

[0032] Example 1

[0033] In the prior art, Transformer-based object detection methods can be divided into two categories: one that uses a Transformer-based backbone network to extract image features, and the other that uses a Transformer-based set prediction method. DETR, the first end-to-end object detection framework based on the Transformer-based set prediction method, was proposed by the Facebook team in 2020. The image is passed through the backbone network to extract features, which are then fed into the encoder along with a positional encoding. The encoder output matrix and the target sequence are input into the decoder. The decoder output is fed into the prediction head, where a feed-forward neural network predicts the object category and bounding box. DETR uses a set of queues to predict object location and category, and an encoder-decoder architecture to process the image and output predictions. This framework is simple and straightforward, directly predicting detection boxes for the image sequence, addressing the drawback of traditional object detection that requires non-maximum suppression. However, DETR's use of an attention mechanism focuses more on global image features, resulting in low detection accuracy for small objects. Furthermore, the attention mechanism requires a larger number of parameters, making it more difficult to converge than traditional convolutional object detection networks. This paper addresses these issues through research and experiments, proposing a method for object detection in remote sensing images based on a fused convolutional attention mechanism.

[0034] The present invention is a remote sensing image target detection method based on the fusion convolutional attention mechanism. Figure 1, Figure 1 This is a flowchart of the implementation process of the present invention; it includes the following steps:

[0035] Step 1, collect and process remote sensing image data: obtain public remote sensing images from public websites, divide the images into training data set, verification data set, and test data set in a ratio of 4:2:3, and form a remote sensing image dataset as a whole. The images in the remote sensing image dataset are called original image data; the remote sensing image dataset contains a total of fifteen categories of targets, namely: airplanes, ships, storage tanks, baseball fields, tennis courts, basketball courts, playgrounds, ports, bridges, large vehicles, small vehicles, helicopters, roundabouts, football fields, and swimming pools. The present invention generates a txt file with the coordinates and category information of all targets in each original image data in the remote sensing image dataset, and inputs it into the constructed feature extraction backbone network together with the original image data. The dataset of the present invention is annotated using an oblique frame method, and the pixel size distribution of each image is different, which contains objects of different scales, directions and shapes.

[0036] Step 2, building a feature extraction backbone network: The constructed feature extraction backbone network is composed of four convolution groups connected in sequence. The first convolution group sequentially forms a residual connection unit through a convolution layer, a Norm layer, an activation function layer, and a maximum pooling layer; wherein the second, third, and fourth convolution groups sequentially downsample the feature map output by the previous convolution group; the input original image data is subjected to a downsampling operation of the constructed feature extraction backbone network, and then a remote sensing image feature matrix is ​​output. The design of the backbone network needs to meet the requirements of multi-scale images. In order to make the backbone network output feature matrices of the same size after extracting features from images of different sizes, the present invention adds a partial downsampling operation to the feature extraction backbone network.

[0037] Step 3: Build a Transformer encoder with fused convolution: Figure 3The constructed Transformer encoder includes a fused convolutional multi-head attention module composed of a convolutional module and an attention module connected in parallel. The encoder includes a fused convolutional multi-head attention module, a residual connection and layer normalization module, a forward propagation module, a residual connection and layer normalization module from the input end. The convolution module in the fused convolutional multi-head attention module includes a first point-by-point convolution layer, a first activation function layer, a depth convolution layer, a BN layer, a second activation function layer, and a second point-by-point convolution layer connected in sequence. The attention module includes an LN layer, a self-attention layer, and a feedforward network layer connected in sequence. The ratio of the convolution module to the attention module is 4:4, and the matrix size after the output of the convolution module is the same as that of the attention module. After concat cascading, an output matrix with the same size as the input matrix of the fused convolutional Transformer encoder is formed. In this example, the improved multi-head attention module uses eight heads. The number of attention heads selected by the multi-head attention module can be selected based on model training time and detection accuracy. Eight heads achieve a balance between the number of model parameters and detection accuracy. The equal number of convolution modules and attention modules ensures that the encoder does not favor global features or local features when extracting image features. The convolution module does not change the size of the feature map by using point-by-point convolution, and does not change the number of feature map channels by using depth-wise convolution. This ensures that the matrix output by the convolution module is equal to the matrix output by the attention module, and can be directly added.

[0038] Step 4: Build a Transformer decoder with a hybrid attention mechanism. The decoder uses a self-attention mechanism to process redundant information in the input target query matrix. A cross-attention mechanism models the relationship between the encoder's feature matrix and the target query matrix. The forward propagation module transforms the image features and predicted boxes. The decoder consists of six decoder units connected in sequence. After the matrix passes through the forward propagation module, each decoder unit outputs a set of predictions for the category and location of each object in the image. The encoder's pixel-based attention coefficient matrix is ​​converted into an attention coefficient matrix based on the relationships between objects in the image.

[0039] Step 5, form a fused convolutional attention mechanism target detection network model: establish a fused convolutional attention mechanism target detection network model composed of a feature extraction backbone network, a fused convolutional Transformer encoder, and a hybrid attention mechanism Transformer decoder, referred to as the network model.

[0040] Step 6: Train the fused convolutional attention mechanism target detection network model: Use the training dataset to train the fused convolutional attention mechanism target detection network model, which is composed of a feature extraction backbone network, a fused convolutional attention mechanism encoder, and a hybrid attention mechanism decoder connected in sequence, to obtain a trained fused convolutional attention mechanism target detection network model. In the present invention, since the number of parameters of the attention module is relatively large, when setting the training parameters, the number of images in each training batch should be minimized while ensuring the convergence speed of the model.

[0041] Step 7, test the fused convolutional attention mechanism target detection network model: Use the trained fused convolutional attention mechanism target detection network model to detect the test dataset, that is, input the test set into the trained fused convolutional attention mechanism target detection network model to obtain the detection results of each type of target in the remote sensing image dataset, including the average precision AP and the average precision mean mAP of all types of targets.

[0042] The technical idea of ​​the present invention is: by fusing the convolution layer with the encoder, the model's extraction of local image features is increased, thereby improving the model's detection accuracy for small targets; using convolution modules with small parameters to replace some attention modules with large parameters, reducing model parameters and speeding up model training.

[0043] To address the shortcomings of existing end-to-end remote sensing image target detection frameworks, this paper introduces a convolutional module to replace some of the self-attention modules in the multi-head attention module of the Transformer encoder. This paper specifically designs a remote sensing image target detection method that integrates the convolutional attention mechanism. Compared with the existing mainstream Transformer target detection end-to-end framework, this paper enhances the model's ability to extract local image information, improves the accuracy of small targets, reduces the number of model parameters, and accelerates model convergence. This method can be applied to fields such as real-time detection of ground targets by aircraft.

[0044] Example 2

[0045] The remote sensing image target detection method based on the fusion convolutional attention mechanism is the same as in Example 1. The feature extraction backbone network built in step 2 of the present invention is shown in FIG. Figure 2 , Figure 2It is the backbone network structure diagram for extracting image features in the present invention. The constructed feature extraction backbone network is composed of four convolution groups connected in sequence; in this example, the first convolution group is composed of a convolution layer with a convolution kernel size of 6×6, a number of convolution kernels of 32, and a step size of 1, a GroupNorm layer, a ReLU activation function layer, and a maximum pooling layer with a window size of 3×3 and a step size of 2; the second convolution group is composed of three identical residual modules 1 connected in sequence, and each residual module 1 is composed of a convolution layer with a convolution kernel size of 1×1, a number of convolution kernels of 128, and a step size of 1, a convolution layer with a convolution kernel size of 3×3, a number of convolution kernels of 128, and a step size of 1, and a convolution layer with a convolution kernel size of 2×2, a number of convolution kernels of 128, and a step size of 1. The third convolution group is composed of four identical residual modules 2 connected in sequence, and each residual module 2 is composed of a convolution layer with a convolution kernel size of 1×1, a number of convolution kernels of 128, and a stride of 1, a convolution layer with a convolution kernel size of 3×3, a number of convolution kernels of 128, and a stride of 1, and a convolution layer with a convolution kernel size of 1×1, a number of convolution kernels of 512, and a stride of 1. The fourth convolution group is composed of nine identical residual modules 3 connected in sequence, and each residual module 3 is composed of a convolution layer with a convolution kernel size of 1×1, a number of convolution kernels of 128, and a stride of 2, a convolution layer with a convolution kernel size of 3×3, a number of convolution kernels of 256, and a stride of 1, and a convolution layer with a convolution kernel size of 1×1, a number of convolution kernels of 512, and a stride of 1. The network structure parameters given in this example are a set of parameters that have good performance for remote sensing image target detection tasks. Depending on the specific task, the parameters of the feature extraction backbone network can be adjusted.

[0046] Since remote sensing images have the characteristics of a bird's-eye view, high resolution, uneven target scale, multi-directional target rotation, and complex image background, the design of the backbone network needs to meet the requirements of multi-scale images and downsample images of different input sizes so that the feature matrix dimensions of the feature extraction backbone network output are the same.

[0047] Example 3

[0048] The remote sensing image target detection method based on the fused convolution attention mechanism is the same as in Example 1-2. The fused convolution Transformer encoder built in step 3 is Figure 3 , Figure 3It is a fused convolution Transformer encoder constructed by the present invention. The fused convolution Transformer encoder constructed by the present invention is composed of six encoder units connected in sequence. The output sequence of the feature extraction backbone network is added with position coding to generate a position coding feature sequence as the input of the entire encoder; it is input into the first encoder unit. The structure of each encoder unit in the present invention is exactly the same, and is composed of a fused convolution multi-head attention module, a first residual connection and layer normalization module, a forward propagation module, a second residual connection and layer normalization module connected in sequence. Among them, the first residual connection and layer normalization module adds and short-circuits the input matrix of the encoder unit to which it belongs and the output matrix of the multi-head attention module, and then normalizes the short-circuited matrix; the forward propagation module is composed of a linear layer, a relu activation function layer, and a dropout layer connected in sequence. The second residual connection and layer normalization module adds and short-circuits the output matrix of the first residual connection and layer normalization module and the output matrix of the forward propagation module, and then normalizes the short-circuited matrix. For the first, second, third, fourth, and fifth encoder units, the output matrix of the current encoder unit serves as the input matrix of the next encoder module unit; in particular, the output matrix of the sixth encoder unit serves as the input matrix of the six decoder units in the decoder.

[0049] The convolution module designed in this paper consists of depthwise convolution, pointwise convolution, activation function, and normalization layer. This convolution module ensures that the output matrix is ​​the same size as the input matrix while extracting local image features. Similarly, the output matrix of the attention module is the same size as the input matrix. Only when the input matrix size is unchanged can the convolution module and the attention module be added in parallel.

[0050] Example 4

[0051] The remote sensing image target detection method based on the fusion convolutional attention mechanism is the same as in Example 1-3, see Figure 4 , Figure 4 It is a structural diagram of the convolution module constructed in the encoder of the present invention. The fused convolution multi-head attention module in the Transformer encoder of the present invention is composed of four self-attention units and four convolution units in parallel; wherein, the four self-attention units have the same structure. Each self-attention unit first multiplies the input matrix with three matrices with different parameters, that is, performs three different linear transformations on the input matrix to obtain three matrices Q, K, and V of the same size but different parameters. Then, the softmax function is used to calculate the three matrices Q, K, and V to obtain the attention parameter matrix. The formula is as follows:

[0052]

[0053] The four convolution units have the same structure. Each convolution unit consists of a point-by-point convolution layer with a convolution kernel size of 1×1, 128 convolution kernels, and a stride of 1, a GLU activation function layer, a depthwise convolution layer with a convolution kernel size of 3×3, 256 convolution kernels, and a stride of 1, a BN normalization layer, a Swish activation function layer, and a point-by-point convolution layer with a convolution kernel size of 1×1, 256 convolution kernels, and a stride of 1.

[0054] This invention extracts image features through the parallel use of convolution and attention modules, enabling the acquisition of both global and local image information. This improves the model's detection accuracy for small targets while maintaining the trained model's accuracy for large targets. Furthermore, the attention mechanism requires a quadratic calculation of the image feature dimensions, resulting in a large number of parameters. The introduction of a local convolution module reduces the number of model parameters, accelerating convergence and reducing training time.

[0055] Example 5

[0056] The remote sensing image target detection method based on the fusion convolutional attention mechanism is the same as that in Example 1-4. Figure 5 , Figure 5 It is a flow chart of the encoder-decoder of the present invention; the Transformer decoder of the hybrid attention mechanism in step 4 of the present invention is composed of six decoder units connected in sequence; each decoder unit has the same structure, and is composed of a multi-head self-attention module, a first residual connection and a layer normalization module, a multi-head cross attention module, a second residual connection and a layer normalization module, a forward propagation module, a third residual connection and a layer normalization module connected in sequence, that is, they are connected in sequence in the order of input to output.

[0057] In this example, the input target query matrix size is 100×256, the encoder output matrix size is 850×256, the mask matrix size is 25×34, and the output matrix size is 100×256. The multi-head self-attention module of the present invention removes redundant information from the input target query sequence. After residual connection and layer normalization, it performs multi-head cross-attention calculation with the feature matrix output by the encoder, shifting the attention of the image region to the attention of the objects in the image. This simplifies the process of generating target detection boxes for the image in the target detection task.

[0058] Example 6

[0059] The remote sensing image target detection method based on the fused convolutional attention mechanism is the same as that in Examples 1-5. The training network model in step 6 of the present invention is to train the fused convolutional attention mechanism target detection network using the remote sensing image training dataset, specifically:

[0060] 6.1 Hyperparameter settings: Set the initial learning rate to R, the learning rate adjustment to steps, the weight decay parameter to a, the batch size to B, and the number of training rounds to E;

[0061] 6.2 Training method: Use the stochastic gradient descent algorithm to update the weights and biases of the entire network model. The weights and biases are updated once every B training images are input, for a total of Stop updating and the training ends;

[0062] 6.3 Obtain the final trained network model: When the iteration stops, the trained fusion convolutional attention mechanism target detection network model is obtained.

[0063] In this example, the initial learning rate was set to 0.001, the learning rate adjustment was set to steps, the weight decay parameter was 0.0001, the batch size was 4, and the number of training rounds was 100. The weights and biases of the entire network model were updated using the stochastic gradient descent algorithm, with the weights and biases updated every time four training images were input. The update was stopped after 40,000 iterations to obtain the final trained network model. The above parameters are the set of parameters that achieved good training results in this example, and the present invention can be adjusted for different object detection tasks.

[0064] A more detailed example is given below to further illustrate the present invention.

[0065] Example 7

[0066] The remote sensing image target detection method based on the fusion convolutional attention mechanism is the same as that in Example 1-6.

[0067] This paper proposes a remote sensing image target detection method based on the fusion convolutional attention mechanism, see Figure 1 , Figure 1 This is a flowchart of the implementation process of the present invention; it includes the following steps:

[0068] Step 1: Collect and process remote sensing image data: Obtain publicly available remote sensing images from public websites and divide them into a training dataset, a validation dataset, and a test dataset in a ratio of 3:1:2. The images in this dataset are called raw image data. This dataset contains fifteen target categories: airplanes, ships, storage tanks, baseball fields, tennis courts, basketball courts, playgrounds, ports, bridges, large vehicles, small vehicles, helicopters, roundabouts, soccer fields, and swimming pools. For each raw image in the dataset, generate a txt file containing the coordinates and category information for all targets. This file, along with the raw image data, is then fed into the constructed feature extraction backbone network.

[0069] Step 2, build a feature extraction backbone network: The constructed feature extraction backbone network is composed of four convolution groups connected in sequence. The first convolution group is composed of convolution layer, Norm layer, activation function layer, and maximum pooling layer connected in sequence; the second, third, and fourth convolution groups are respectively composed of different numbers of residual connection units connected in sequence. The structure of each residual connection unit is a convolution layer, a GN layer, and an activation function layer stacked in sequence; the backbone network uses a feature pyramid module with different convolution modules stacked by residual connections to generate the same feature matrix after downsampling input images of different sizes; the feature matrix output by the backbone network is sent to the encoder after dimensionality reduction processing and position encoding of the same dimension is added.

[0070] See also Figure 2 The feature extraction backbone network constructed by the present invention is composed of four convolution groups connected in sequence; the first convolution group is composed of a convolution layer, a GroupNorm layer, an activation function layer, and a maximum pooling layer in sequence; the second convolution group is composed of three identical residual modules 1 connected in sequence, and each residual module 1 is composed of three different convolution layers, a normalization layer, and an activation function layer connected in sequence; the third convolution group is composed of four identical residual modules 2 connected in sequence, and each residual module 2 is composed of three different convolution layers, a normalization layer, and an activation function layer connected in sequence; the fourth convolution group is composed of nine identical residual modules 3 connected in sequence, and each residual module 3 is composed of three different convolution layers, a normalization layer, and an activation function layer connected in sequence.

[0071] The first convolution group consists of a convolution layer with a kernel size of 7×7, 64 kernels, and a stride of 2, a GroupNorm layer, a ReLU activation function layer, and a maximum pooling layer with a window size of 3×3 and a stride of 2. The second convolution group consists of three identical residual modules 1 connected in sequence. Each residual module 1 consists of a convolution layer with a kernel size of 1×1, 128 kernels, and a stride of 1, a convolution layer with a kernel size of 3×3, 128 kernels, and a stride of 1, and a convolution layer with a kernel size of 1×1, 256 kernels, and a stride of 1. The third convolution group consists of four identical residual modules 2 connected in sequence. Each residual module 2 consists of a convolution layer with a convolution kernel size of 1×1, 128 convolution kernels, and a stride of 1, a convolution layer with a convolution kernel size of 3×3, 128 convolution kernels, and a stride of 1, and a convolution layer with a convolution kernel size of 1×1, 512 convolution kernels, and a stride of 1. The fourth convolution group consists of nine identical residual modules 3 connected in sequence. Each residual module 3 consists of a convolution layer with a convolution kernel size of 1×1, 256 convolution kernels, and a stride of 1, a convolution layer with a convolution kernel size of 3×3, 256 convolution kernels, and a stride of 1, and a convolution layer with a convolution kernel size of 1×1, 1024 convolution kernels, and a stride of 1.

[0072] The feature extraction backbone network designed in the present invention downsamples the input image by stacking residual unit modules, so that input images of different sizes output feature matrices of the same size after passing through the backbone network, solving the difficulty of multi-scale detection of remote sensing image targets.

[0073] Step 3: Build a Transformer encoder with fused convolution: Figure 3The Transformer encoder constructed by the present invention is composed of six encoder units connected in sequence. Each encoder unit includes a fused convolutional multi-head attention module composed of a convolutional module and an attention module connected in parallel. The encoder includes a fused convolutional multi-head attention module, a first residual connection and layer normalization module, a forward propagation module, a second residual connection and layer normalization module from the input end. Among them, the first residual connection and layer normalization module adds and short-circuits the input matrix of the encoder unit to which it belongs and the output matrix of the multi-head attention module, and then normalizes the short-circuited matrix. The forward propagation module is composed of a linear layer, a relu activation function layer, and a dropout layer connected in sequence; the second residual connection and layer normalization module adds and short-circuits the output matrix of the first residual connection and layer normalization module and the output matrix of the forward propagation module, and then normalizes the short-circuited matrix; for the first, second, third, fourth, and fifth encoder modules, the output matrix of the current encoder module serves as the input matrix of the next encoder module; in particular, the output matrix of the sixth encoder module serves as the input matrix of the six decoder units in the decoder module.

[0074] The fused convolution multi-head attention module in the fused convolution Transformer encoder is composed of four self-attention units and four convolution units in parallel. Among them, the four self-attention units have the same structure. Each self-attention unit includes a sequentially connected LN layer, a self-attention layer, and a feedforward network layer. First, by multiplying the input matrix with three matrices with different parameters, that is, performing three different linear transformations on the input matrix, three matrices Q, K, and V of the same size but different parameters are obtained. Among them, the Q matrix is ​​the query matrix for linear transformation of the image feature matrix, the K matrix is ​​the key matrix for linear transformation of the image feature matrix, and the V matrix is ​​the value matrix for linear transformation of the image feature matrix. Then, the softmax function is used to calculate the attention parameter matrix of the three matrices Q, K, and V. The formula is as follows:

[0075]

[0076] The four convolutional units have the same structure, see Figure 4 Each convolutional unit consists of a sequentially connected point-by-point convolution layer, an activation function layer, a depth-wise convolution layer, a batch normalization layer, an activation function layer, and a point-by-point convolution layer. The ratio of the convolution module to the attention module is 4:4. The output matrices of the convolution module and the attention module are the same size. After concatenation, they form an output matrix of the same size as the input matrix of the Transformer encoder module of the fused convolution.

[0077] Step 4: Build a Transformer decoder module with a hybrid attention mechanism: the decoder processes redundant information on the input target query matrix through the self-attention mechanism, models the relationship between the feature matrix output by the encoder and the target query matrix through the cross-attention mechanism, and performs feature transformation on the image features and prediction boxes through the forward propagation module.

[0078] See also Figure 5 The Transformer decoder module of the hybrid attention mechanism constructed by the present invention is composed of six decoder units connected in sequence; each decoder unit has the same structure, and is composed of a multi-head self-attention module, a first residual connection and layer normalization module, a multi-head cross-attention module, a second residual connection and layer normalization module, a forward propagation module, a third residual connection and layer normalization module connected in sequence. First, the target query matrix is ​​input into the multi-head self-attention module for redundant information processing. The processed target query matrix and the matrix output by the encoder are input into the multi-head cross-attention module together for cross-attention calculation, and the attention matrix of the image area is converted into the attention matrix of the image target. Each decoder unit outputs a prediction matrix for the image through the forward propagation module, and the forward propagation module predicts the target.

[0079] Step 5, form a fused convolutional attention mechanism target detection network model: establish a fused convolutional attention mechanism target detection network model consisting of a feature extraction backbone network, a fused convolutional Transformer encoder module, and a hybrid attention mechanism Transformer decoder module, referred to as the network model.

[0080] Step 6: Train the fused convolutional attention mechanism target detection network model: Use the training dataset to train the fused convolutional attention mechanism target detection network model, which consists of a feature extraction backbone network, a fused convolutional attention mechanism encoder, and a hybrid attention mechanism decoder. This results in a trained fused convolutional attention mechanism target detection network model. This includes the following steps:

[0081] 6.1 Hyperparameter settings: Set the initial learning rate to R, the learning rate adjustment to steps, the weight decay parameter to a, the batch size to B, and the number of training rounds to E;

[0082] 6.2 Training method: Use the stochastic gradient descent algorithm to update the weights and biases of the entire network model. The weights and biases are updated once every B training images are input, for a total of Stop updating and the training ends;

[0083] 6.3 Obtain the final trained network model: When the iteration stops, the trained fusion convolutional attention mechanism target detection network model is obtained.

[0084] In this example, the initial learning rate is set to 0.0025, the learning rate adjustment is set to steps, the weight decay parameter is 0.0001, the batch size is 8, and the training rounds are 100. The stochastic gradient descent algorithm is used to update the weights and biases of the entire network model. The weights and biases are updated every time 8 training images are input. The update is stopped after 20,000 iterations to obtain the final trained network model.

[0085] The present invention accelerates the model convergence speed and reduces the time consumption of model training.

[0086] Step 7, test the fused convolutional attention mechanism target detection network model: Use the trained fused convolutional attention mechanism target detection network model to detect the test dataset, that is, input the test set into the trained fused convolutional attention mechanism target detection network model to obtain the detection results of each type of target in the remote sensing image dataset, including the average precision AP and the average precision mean mAP of all types of targets.

[0087] The present invention uses a pyramid-structured feature extraction backbone network with downsampling operations to output feature matrices of the same size for input images of different sizes, thus resolving the difficulty of multi-scale detection of remote sensing image targets. A convolution module comprising deep convolution and point-by-point convolution is constructed to enhance the model's ability to extract information from local features of remote sensing images. Replacing some of the attention heads in the encoder with the constructed convolution module reduces the large number of parameters caused by the encoder composed entirely of the attention mechanism, improves the model's convergence speed, and reduces the model's training time. This approach is applicable to fields such as aviation, remote sensing satellites, smart transportation, and smart agriculture, where real-time and accuracy performance of remote sensing image target detection are highly demanding.

[0088] The technical effects of the present invention are further explained below through experiments and their result data.

[0089] Example 8

[0090] The remote sensing image target detection method based on the fusion convolutional attention mechanism is the same as that in Example 1-7.

[0091] Experimental Conditions: All experiments were conducted on the same platform, equipped with an Intel 8358P CPU, an NVIDIA GeForce RTX 3090 GPU, and 24GB of video memory. The operating system used was Ubuntu 18.04LTS, the deep learning framework used was Pytorch 1.7.1, the GPU computing platform used was CUDA 11.0, and the GPU acceleration library used was cuDNN 8.0.5.

[0092] Experimental content: The remote sensing image target detection method based on the fusion convolutional attention mechanism of the present invention is used to detect targets on the public remote sensing dataset DOTA. The trained model is obtained using the training method of the present invention. The target category and position accuracy of 937 images in the test set of the DOTA dataset are tested. Two detection result images are randomly selected as shown in the figure. Figure 6 As shown, Figure 6 (a) is a detection result image containing small vehicles and large vehicles detected by the present invention.

[0093] Experimental results and analysis: see Figure 6 (a), from Figure 6 (a) It can be seen that all small and large vehicle targets in the inspected remote sensing images are detected, and the confidence of most detection frames is high. The rotating frames can accurately detect targets in rotating directions and targets rotating in multiple directions, and no targets are missed. The present invention has good detection performance for both large and small targets in remote sensing datasets.

[0094] Example 9

[0095] The remote sensing image target detection method based on the fused convolutional attention mechanism is the same as that in Examples 1-7, and the experimental conditions and experimental content are the same as those in Example 8.

[0096] Experimental results and analysis: see Figure 6 (b) Figure 6 (b) is a picture of the detection result of the present invention containing small vehicles and circular road targets. Specifically, the present invention is aimed at detecting the target category and position accuracy of remote sensing images. Figure 6 (b) It can be seen that all small vehicles and circular loop targets in the inspected remote sensing images are detected, and the confidence levels of most detection frames are high. The figure shows that the present invention can accurately detect small vehicles and circular loops, two targets with huge scale differences, without any missed targets. This verifies the ability of the feature extraction backbone network designed by the present invention to extract multi-scale features of images through downsampling operations. The present invention has good detection performance for both large and small targets in remote sensing datasets.

[0097] Example 10

[0098] The remote sensing image target detection method based on the fused convolutional attention mechanism is the same as that in Examples 1-7, and the experimental conditions are the same as those in Example 8.

[0099] Experimental content: The remote sensing image target detection method based on the fusion convolutional attention mechanism of the present invention, the remote sensing image target detection method based on the attention mechanism DETR and the remote sensing image target detection method based on the convolution are compared on the DOTA dataset. The results of the comparative experiment are shown in Table 1.

[0100] Table 1 Comparative experimental results on the DOTA remote sensing dataset

[0101]

[0102]

[0103] Experimental results and analysis: As can be seen from Table 1, compared with the convolution-based remote sensing image target detection method, the present invention has improved the detection accuracy of most targets, including targets with large scale differences, regardless of the targets of the site, vehicle, or road section. Compared with the remote sensing image target detection method based on the attention mechanism, the present invention has similar detection accuracy for large targets and better detection accuracy for small targets.

[0104] In summary, this invention is a remote sensing image target detection method based on a fused convolutional attention mechanism. It solves the technical problems of existing end-to-end remote sensing image target detection technologies, such as low accuracy for detecting small targets in remote sensing images and slow convergence during model training. The implementation includes: collecting and processing remote sensing image data; building a feature extraction backbone network; building a fused convolutional Transformer encoder; building a hybrid attention Transformer decoder; constructing a fused convolutional attention target detection network model; training the fused convolutional attention target detection network model; and testing the fused convolutional attention target detection network model. The present invention adopts a pyramid-structured downsampling feature extraction backbone network, outputting feature matrices of the same size for input images of different sizes, thus solving the difficulty of multi-scale detection of remote sensing image targets. A convolution module consisting of deep convolution, point-by-point convolution, activation function, and normalization layer is constructed to enhance the model's ability to extract information from local features of remote sensing images. Some attention heads in the multi-head attention module are replaced with the constructed convolution module, reducing the large number of parameters required by the attention mechanism to perform square matrix dot product operations on image pixels, thereby improving the model's convergence speed and reducing the model's training time. The invention can be used in fields such as aviation, remote sensing satellites, smart transportation, and smart agriculture, which have relatively high requirements for real-time and accuracy in remote sensing image target detection.

Claims

1. A remote sensing image target detection method based on a fused convolutional attention mechanism, characterized in that: The following steps are included: Step 1: Collect and process remote sensing image data: Obtain public remote sensing images from public websites and divide them into training dataset, validation dataset, and test dataset to form a remote sensing image dataset. The remote sensing image dataset contains 15 categories of targets: airplanes, ships, storage tanks, baseball fields, tennis courts, basketball courts, playgrounds, ports, bridges, large vehicles, small vehicles, helicopters, roundabouts, football fields, and swimming pools. Generate a txt file containing the coordinates and category information of all targets in the original image data of the remote sensing image dataset and input it into the constructed feature extraction backbone network together with the original image data. Step 2: Build a feature extraction backbone network: The built feature extraction backbone network consists of four convolution groups connected in sequence. The first convolution group is composed of a convolution layer, a Norm layer, an activation function layer, and a maximum pooling layer. The second, third, and fourth convolution groups are respectively composed of different numbers of residual connection units connected in sequence. The structure of each residual connection unit is a convolution layer, a GN layer, and an activation function layer stacked in sequence. The input original image data is downsampled by the built feature extraction backbone network and then outputs a remote sensing image feature matrix. Step 3, build a fused convolution Transformer encoder: the built Transformer encoder includes a fused convolution multi-head attention module composed of a convolution module and an attention module connected in parallel. The encoder includes a fused convolution multi-head attention module, a residual connection and layer normalization module, a forward propagation module, a residual connection and layer normalization module from the input end, wherein the convolution module in the fused convolution multi-head attention module includes a first convolution layer, a first activation function layer, a second convolution layer, a BN layer, a second activation function layer, and a third convolution layer connected in sequence, and the attention module includes an LN layer, a self-attention layer, and a feedforward network layer connected in sequence; wherein the ratio of the convolution module to the attention module is 4:4, and the matrix size after the output of the convolution module is the same as that of the attention module, and the output matrix of the fused convolution Transformer encoder is formed by concat cascading; Step 4: Build a Transformer decoder with a hybrid attention mechanism: The decoder uses a self-attention mechanism to process redundant information on the input target query matrix. The cross-attention mechanism models the relationship between the feature matrix output by the encoder and the target query matrix. The forward propagation module performs feature transformation on the image features and the prediction box. Step 5: Construct a fused convolutional attention mechanism target detection network model: Build a fused convolutional attention mechanism target detection network model, which is composed of a feature extraction backbone network, a fused convolutional Transformer encoder, and a hybrid attention mechanism Transformer decoder, referred to as the network model. Step 6: Train the fused convolutional attention mechanism target detection network model: Use the training dataset to train the fused convolutional attention mechanism target detection network model, which is composed of the feature extraction backbone network, the fused convolutional attention mechanism encoder, and the hybrid attention mechanism decoder connected in sequence, to obtain the trained fused convolutional attention mechanism target detection network model; Step 7, test the fused convolutional attention mechanism target detection network model: Use the trained fused convolutional attention mechanism target detection network model to detect the test dataset, that is, input the test set into the trained fused convolutional attention mechanism target detection network model to obtain the detection results of each type of target in the remote sensing image dataset, including the average precision AP and the average precision mean mAP of all types of targets.

2. The remote sensing image target detection method based on the fused convolutional attention mechanism according to claim 1 is characterized in that: The feature extraction backbone network built in step 2 is composed of four convolution groups connected in sequence; the first convolution group is composed of a convolution layer, a GroupNorm layer, an activation function layer, and a maximum pooling layer in sequence; the second convolution group is composed of three identical residual modules 1 connected in sequence, and each residual module 1 is composed of three different convolution layers connected in sequence; the third convolution group is composed of four identical residual modules 2 connected in sequence, and each residual module 2 is composed of three different convolution layers connected in sequence; the fourth convolution group is composed of nine identical residual modules 3 connected in sequence, and each residual module 3 is composed of three different convolution layers connected in sequence.

3. The remote sensing image target detection method based on the fused convolutional attention mechanism according to claim 1 is characterized in that: The fused convolution Transformer encoder built in step 3 is composed of six identical encoder units connected in sequence; the structure of each encoder unit is exactly the same, consisting of a fused convolution multi-head attention module, the first residual connection and layer normalization module, a forward propagation module, the second residual connection and layer normalization module connected in sequence; the output sequence of the feature extraction backbone network is added with the position encoding to generate a position encoding feature sequence, which is used as the input of the entire encoder and input to the first encoder unit; in the encoder unit, the first residual connection and layer normalization module combine the input matrix of the encoder unit with the output matrix of the multi-head attention module The matrix is ​​added and short-circuited, and then the short-circuited matrix is ​​normalized; the forward propagation module is composed of a linear layer, a relu activation function layer, and a dropout layer connected in sequence; the second residual connection and layer normalization module adds and short-circuits the output matrix of the first residual connection and layer normalization module with the output matrix of the forward propagation module, and then normalizes the short-circuited matrix; for the first, second, third, fourth, and fifth encoder units, the output matrix of the current encoder unit is used as the input matrix of the next encoder unit; in particular, the output matrix of the sixth encoder unit is used as the input matrix of the six decoder units in the decoder.

4. The fused convolutional Transformer encoder according to claim 1 or 3, wherein: The fused convolution multi-head attention module in the fused convolution Transformer encoder consists of four self-attention units and four convolution units in parallel. The four self-attention units have the same structure. Each self-attention unit first multiplies the input matrix by three matrices with different parameters, that is, performs three different linear transformations on the input matrix to obtain three matrices Q, K, and V of the same size but different parameters. Then, the softmax function is used to calculate the attention parameter matrix on the three matrices Q, K, and V. The formula is as follows: The four convolution units have the same structure. Each convolution unit is composed of the first convolution layer, the first activation function layer, the second convolution layer, the Norm layer, the second activation function layer, and the third convolution layer connected in sequence.

5. The remote sensing image target detection method based on the fused convolutional attention mechanism according to claim 1 is characterized in that: The Transformer decoder with the hybrid attention mechanism in step 4 is composed of six decoder units connected in sequence; each decoder unit has the same structure, which is composed of a multi-head self-attention module, a first residual connection and a layer normalization module, a multi-head cross-attention module, a second residual connection and a layer normalization module, a forward propagation module, and a third residual connection and a layer normalization module connected in sequence.

6. The remote sensing image target detection method based on the fused convolutional attention mechanism according to claim 1 is characterized in that the training network model in step 6 is to train the fused convolutional attention mechanism target detection network using the remote sensing image training dataset, specifically: 6.1 Hyperparameter settings: Set the initial learning rate to R, the learning rate adjustment to steps, the weight decay parameter to a, the batch size to B, and the number of training rounds to E; 6.2 Training method: Use the stochastic gradient descent algorithm to update the weights and biases of the entire network model. The weights and biases are updated once every B training images are input, for a total of Stop updating and the training ends; 6.3 Obtain the final trained network model: When the iteration stops, the trained fusion convolutional attention mechanism target detection network model is obtained.

Images