Transform remote sensing target detection method based on dynamic scale self-attention
By introducing a dynamic scale self-attention module into the Transformer model, the problem of high computational complexity of DETR model is solved, flexible detection of objects of different sizes is achieved, and the accuracy of remote sensing object detection is improved.
Patent Information
- Application Number
- CN202411323147.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2024-12-20
AI Technical Summary
The self-attention mechanism of DETR model in object detection leads to long training time and high computational complexity, making it difficult to effectively deal with the problems of different object sizes and imaging interference.
Using the Transformer framework of dynamic scale self-attention, features are extracted through convolutional neural networks, and scale selection modules and self-attention modules are introduced into the Transformer model to dynamically select the receptive field, reduce computing redundancy, and improve detection accuracy.
Without increasing the computational complexity, the accuracy of the object detection task is improved, and the flexibility to deal with objects of different sizes is reduced, which reduces the complexity of self-attention calculation.
Smart Images
Figure CN119169273A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of computer vision and image processing, and specifically is a Transformer remote sensing target detection method based on dynamic scale self-attention. Background Art
[0002] Object detection is a core task in the field of computer vision. Its main goal is to accurately find all objects of interest from an image and determine their categories and locations. Remote sensing object detection is widely used in many fields such as urban planning, land use, and environmental monitoring. Since various objects have different appearances, shapes, and postures, and are interfered by factors such as illumination and occlusion during imaging, object detection has always been the most challenging problem in the field of computer vision.
[0003] The DETR model applied the Transformer method to target detection for the first time, achieving true non-maximum suppression (NMS) post-processing and eliminating the need for anchor box generation. It also has the advantages of being easy to implement and having a flexible architecture. However, the self-attention mechanism of the DETR model will bring about problems of long training time and high computational complexity. Summary of the invention
[0004] To solve the above technical problems, the present invention provides a Transformer remote sensing target detection method based on dynamic scale self-attention. Features are extracted through a convolutional neural network, and target detection is performed through a DETR framework with dynamic scale self-attention. In this process, different receptive fields are dynamically selected according to the size of the object in the input image without increasing the computational complexity, focusing on key features and reducing computational redundancy, which can effectively improve the accuracy of target detection tasks on the data set.
[0005] The present invention provides a Transformer remote sensing target detection method based on dynamic scale self-attention, comprising the following steps:
[0006] S1. Obtain a remote sensing image target detection training data set and perform a preprocessing operation on the data set;
[0007] S2, feature extraction through Resnet50 convolutional neural network;
[0008] S3, input the extracted features into the encoder of the Transformer model with a dynamic scale self-attention module to perform key feature attention;
[0009] S4, input the target query and encoder output into the decoder of the Transformer model for attention operation;
[0010] S5. Input the result after the Transformer model is executed into the prediction head to obtain the classification result and bounding box.
[0011] Furthermore, in S1, the preprocessing operation on the training data set includes the following steps:
[0012] S1.1, firstly, randomly flip the image horizontally;
[0013] S1.2. Randomly select an image from the given transformation list for enhancement: randomly scale the image to a size not exceeding 1333 pixels, or randomly scale the image, randomly crop it, enhance the local features of the image, and randomly scale it again to a size not exceeding 1333 pixels;
[0014] Specifically, randomly select one from the given transformation list for application. One is to directly randomly scale the image to a size of 480, 512, 544, 576, 608, 640, 672, 704, 736, 768 or 800, with the maximum size not exceeding 1333. The other is to randomly scale the image to a size of 400, 500 or 600, and then randomly crop the image to 384 or 600 to enhance the local features of the image, and finally randomly scale the image to a size of 480, 512, 544, 576, 608, 640, 672, 704, 736, 768 or 800, with the maximum size not exceeding 1333, to ensure that the final image meets the input requirements;
[0015] S1.3. Normalize the enhanced image.
[0016] Furthermore, in step S2, feature extraction is performed through the Resnet50 convolutional neural network. The Resnet50 network initially has a single 7×7 convolutional layer, followed by four groups of large convolutional blocks, each of which consists of 3, 4, 6, and 3 small convolutional blocks, each of which consists of three convolutional layers, and finally a fully connected layer, for a total of 50 layers.
[0017] Furthermore, in step S3, the extracted features are input into the encoder of the Transformer model with a dynamic scale self-attention module, which mainly includes two parts. The first part obtains the expansion scale value according to the query Q (query) through the scale selection module, and selects K (key) and V (value) at the corresponding scale; the second part performs self-attention calculation through the self-attention module and splices the outputs at the four expansion scales together.
[0018] In the first part, the scale selection module is introduced, and the expansion scale learning process is as follows:
[0019] D=DSS(Q)
[0020] Where Q is the input query, DSS(Q) is a dynamic scale selection module, and D is a learnable dilation scale. Q(query) at position (i,j) will be dilated to select the key K(key) and value V(value) at the following coordinate set (i',j') in a window of size 3×3:
[0021]
[0022] Among them, d is the selected expansion scale to control the sparsity, and p and q are parameters for adjusting the coordinate position.
[0023] The specific steps for this part are as follows:
[0024] S3.1, input Q (query) into the scale selection module, and sequentially pass through the adaptive average pooling layer, linear layer, ReLU activation function, linear layer and softmax function to learn 4 integer expansion scales between [1, 4];
[0025] S3.2, the input features are evenly divided into 4 parts by channel. According to the expansion scale, each Q (query) in each part selects 3×3 K (key) and V (value) at the corresponding position for operation.
[0026] In the second part of self-attention calculation, the traditional self-attention calculation method is adopted. The specific self-attention calculation formula is:
[0027]
[0028] 1≤i≤W,1≤j≤H
[0029] Among them, x ij represents the output feature of the self-attention module, Q ij is Q(query) at position (i,j), K d is K(key) selected when the expansion scale is d, V d is the V(value) selected when the expansion scale is d, W is the width of the input feature, and H is the height of the input feature. ij With K d The similarity of V is used to obtain the attention weight, and the weight is normalized by softmax. d The final output feature is obtained by weighted summation. The outputs of the four expansion scales are concatenated together to form the output of the self-attention module.
[0030] In step S4, the learnable target query is input into the decoder for self-attention calculation, the output of self-attention is used as the Q (query) input of the cross attention in the decoder, the output of the encoder is used as the V (value) input of the cross attention, and the output of the encoder is added with the position code as the K (key) input of the cross attention. After the cross attention calculation, the output is output through forward propagation.
[0031] In step S5, the result after the decoder of the Transformer model is executed is input into the prediction head, and the predicted result and the ground truth bounding box are binary matched by the Hungarian algorithm, and finally the classification prediction result and the bounding box prediction result are obtained.
[0032] The beneficial effects achieved by the present invention are:
[0033] (1) The present invention uses a dynamic scale self-attention module to enable the network to dynamically select different receptive field sizes and more flexibly respond to objects of different sizes in the input image.
[0034] (2) By designing a Transformer with dynamic self-attention, the present invention reduces the complexity of self-attention calculation and improves the accuracy of remote sensing target detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a flowchart of a Transformer remote sensing target detection method based on dynamic scale self-attention in an embodiment of the present invention.
[0036] Figure 2 This is the overall network architecture diagram of the Transformer remote sensing target detection method based on dynamic scale self-attention in this embodiment.
[0037] Figure 3 Schematic diagram of the structure of the dynamic scale self-attention module in an embodiment of the present invention. DETAILED DESCRIPTION
[0038] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.
[0039] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.
[0040] A Transformer remote sensing target detection method based on dynamic scale self-attention includes the following steps:
[0041] In step S1, the preprocessing operation on the training data set includes the following steps:
[0042] S1.1, firstly, perform a random horizontal flip operation on the image;
[0043] S1.2, randomly select one from the given list of transformations to apply. One is to directly randomly scale the image to a size of 480, 512, 544, 576, 608, 640, 672, 704, 736, 768 or 800, with the maximum size not exceeding 1333 pixels. The other is to randomly scale the image to a size of 400, 500 or 600, then randomly crop the image to 384 or 600 to enhance the local features of the image, and finally randomly scale the image to a size of 480, 512, 544, 576, 608, 640, 672, 704, 736, 768 or 800, with the maximum size not exceeding 1333 pixels to ensure that the final image meets the input requirements;
[0044] S1.3, normalize the enhanced image.
[0045] In step S2, feature extraction is performed using the Resnet50 convolutional neural network. The Resnet50 network starts with a single 7×7 convolutional layer, followed by four groups of large convolutional blocks, each of which consists of 3, 4, 6, and 3 small convolutional blocks. Each small convolutional block consists of three convolutions, and finally a fully connected layer. There are 50 layers in total.
[0046] like Figure 2 As shown, step S2 corresponds to the backbone of the convolutional neural network in the figure. The output features are passed to the encoder of the Transformer model and then passed through the decoder of the Transformer model to finally obtain the prediction result. The specific description of the encoder and decoder is explained in detail in the following steps.
[0047] In step S3, the extracted features are input into the encoder of the Transformer model with a dynamic scale self-attention module. The dynamic scale self-attention module mainly consists of two parts. The first part obtains the expansion scale value according to the query Q (query) through the scale selection module, and selects K (key) and V (value) at the corresponding scale; the second part performs self-attention calculation through the self-attention module and splices the four outputs together.
[0048] like Figure 3 As shown in the figure, the input is passed through the linear layer to obtain Q(query), K(key) and V(value), and Q(query) is learned into a set of expansion scales [1,2,2,3] through the scale selection module. In the dynamic scale attention, K(key) and V(value) are selected according to the expansion scale. This is the content of the first part. The learning process of the expansion scale in the scale selection module is as follows:
[0049] D=DSS(Q)
[0050] Where Q is the input query, DSS(Q) is a dynamic scale selection module, and D is a learnable dilation scale. Q(query) at position (i,j) will be dilated to select the key K(key) and value V(value) at the following coordinate set (i',j') in a window of size 3×3:
[0051]
[0052] Where d is the selected expansion scale to control the sparsity. The specific steps of this part are as follows:
[0053] S3.1, input Q (query) into the scale selection module, and sequentially pass through the adaptive average pooling layer, linear layer, ReLU activation function, linear layer and softmax function to learn 4 integer expansion scales between [1, 4];
[0054] S3.2, the input features are evenly divided into 4 parts by channel. According to the expansion scale, each Q (query) in each part selects 3×3 K (key) and V (value) at the corresponding position for operation.
[0055] Then, if Figure 3 As shown in the figure, in the dynamic scale attention, the Q (query) at each position is self-attentioned with the corresponding K (key) and V (value) with four expansion scales of 1, 2, 2, and 4, and finally the outputs under the four expansion scales are spliced together. When calculating the second part of the self-attention, the traditional self-attention calculation method is used. The specific self-attention calculation formula is:
[0056]
[0057] 1≤i≤W,1≤j≤H
[0058] Among them, Q ij is Q(query) at position (i,j), K d is K(key) selected when the expansion scale is d, V d is the V(value) selected when the expansion scale is d, W is the width of the input feature, and H is the height of the input feature. ij With K d The similarity of V is used to obtain the attention weight, and the weight is normalized by softmax. d The final output feature is obtained by weighted summation. The output of the four parts is concatenated together to obtain the output of the self-attention module.
[0059] In step S4, the learnable target query is input into the decoder for self-attention calculation, the output of self-attention is used as the Q (query) input of cross attention, the output of the encoder is used as the V (value) input of cross attention, and the output of the encoder is added with position encoding as the K (key) input of cross attention. After the cross attention calculation, the output is output through forward propagation.
[0060] In step S5, the result after the Transformer is executed is input into the prediction head, and the predicted result and the ground truth bounding box are binary matched by the Hungarian algorithm, and finally the classification prediction result and the bounding box prediction result are obtained.
[0061] The algorithm of the present invention and the baseline algorithm of the algorithm of the present invention are pre-trained on the dataset COCO2017, and then fine-tuned on the remote sensing dataset NWPUVHR-10. The results are shown in Table 1. The common indicators of the COCO dataset are used. AP represents the average precision, and AP represents the average precision. 50 and AP 75 Respectively represent the accuracy when IOU is 50 and IOU is 75, AP S 、AP M and AP L Respectively represent the detection accuracy of small targets, medium targets and large targets. According to the experimental results, it can be seen that the algorithm of the present invention has a 0.7% improvement in the evaluation index AP. 50 It has a 1.1% improvement on the original algorithm and is also better than the basic algorithm in the other four evaluation indicators, effectively improving the accuracy of target detection tasks.
[0062] Table 1
[0063]
[0064] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.
Claims
1. A Transformer remote sensing target detection method based on dynamic scale self-attention, comprising the following steps: S1. Obtain a remote sensing image target detection training data set and perform a preprocessing operation on the data set; S2, feature extraction through Resnet50 convolutional neural network; S3, the extracted features are input into the encoder of the Transformer model with dynamic scale self-attention to perform key feature attention; S4, input the target query and encoder output into the decoder of the Transformer model for attention operation; S5. Input the result after the Transformer model is executed into the prediction head to obtain the classification result and bounding box.
2. According to claim 1, a Transformer remote sensing target detection method based on dynamic scale self-attention, It is characterized in that In S1, the preprocessing operation of the training data set includes the following steps: S1.1, firstly, perform a random horizontal flip operation on the image; S1.
2. Randomly select an image from the given transformation list for enhancement: Randomly scale the image to a size not exceeding 1333 pixels, or randomly scale the image, randomly crop it, enhance the local features of the image, and then randomly scale it again to a size not exceeding 1333 pixels; S1.
3. Normalize the enhanced image.
3. According to claim 1, a Transformer remote sensing target detection method based on dynamic scale self-attention, It is characterized in that In S2, the Resnet50 network includes a single 7×7 convolutional layer and four groups of large convolutional blocks. The four groups of large convolutional blocks include 3, 4, 6, and 3 small convolutional blocks respectively. Each small convolutional block consists of three convolutional layers and finally a fully connected layer.
4. According to claim 1, a Transformer remote sensing target detection method based on dynamic scale self-attention, It is characterized in that In S3, the extracted features are input into the encoder of the Transformer model with dynamic scale self-attention. The dynamic scale self-attention consists of two parts: the first part obtains the expansion scale value according to the query Q through the scale selection module, and selects K and V at the corresponding scale; the second part performs self-attention calculation through the self-attention module and splices the outputs together.
5. According to claim 4, a Transformer remote sensing target detection method based on dynamic scale self-attention, Features: The learning process of the dilation scale in the scale selection module in S3 is as follows: D=DSS(Q) Where Q is the input query, DSS(Q) is a dynamic scale selection module, and D is a learnable dilation scale. Q at position (i, j) will select the key K and value V located at the following coordinate set (i', j') in a window of size 3×3 through the dilation scale; Among them, d is the selected expansion scale to control the sparsity; p and q are parameters for adjusting the coordinate position; The specific steps are as follows: S3.1, input Q into the scale selection module, and learn 4 integer expansion scales between [1,4] through the adaptive average pooling layer, linear layer, ReLU activation function, linear layer and softmax function in sequence; S3.
2. The input features are evenly divided into 4 parts according to the channel. According to the expansion scale, each Q in each part selects 3×3 K and V at the corresponding position for operation.
6. According to claim 4, a Transformer remote sensing target detection method based on dynamic scale self-attention, Features: The self-attention module in S3 uses the traditional self-attention calculation method. The self-attention calculation formula is: 1≤i≤W,1≤j≤H Among them, x ij represents the output feature of the self-attention module, Q ij is Q, K at position (i, j) d K, V are selected when the expansion scale is d d is the V selected when the expansion scale is d, W is the width of the input feature, and H is the height of the input feature; by calculating Q ij With K d The similarity of V is used to obtain the attention weight, and the weight is normalized by softmax. d The weighted sum is performed to obtain the final output features; the outputs at the four expansion scales are concatenated together to obtain the output of the self-attention module.
7. The method for remote sensing target detection based on dynamic scale self-attention of Transformer according to claim 1, It is characterized in that S4 is specifically: The learnable target query is input into the decoder for self-attention calculation, the output of self-attention is used as the Q input of the cross attention in the decoder, the output of the encoder is used as the V input of the cross attention, the output of the encoder plus the position encoding is used as the K input of the cross attention, and the cross attention calculation is performed and then output through forward propagation.
8. The Transformer remote sensing target detection method based on dynamic scale self-attention according to claim 1, It is characterized in that In S5, the result after the decoder of the Transformer model is executed is input into the prediction head, and the predicted result and the ground truth bounding box are binary matched through the Hungarian algorithm, and finally the classification prediction result and the bounding box prediction result are obtained.