DETR-based aerial image-oriented target detection method

By introducing sparse attention module, scale-aware feature pyramid module and optimization query module into the DETR object detection network, the problems of low detection accuracy and poor generalization ability of small targets in aerial images are solved, and more efficient multi-scale targets are achieved Test.

CN118968350APending Publication Date: 2024-11-15UNIV OF ELECTRONICS SCI & TECH OF CHINA

Patent Information

Application Number
CN202410964200.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing aerial image object detection algorithms have low detection accuracy and poor generalization capabilities in high-resolution aerial images, especially for multi-category detection tasks.

Method used

The target detection network based on DETR is improved, and the encoder calculation amount is reduced by introducing a sparse attention module, and the feature pyramid module based on scale perception is designed to enhance multi-scale target perception capabilities, and the optimization query module is designed on the decoder side to improve the target expression capabilities. .

Benefits of technology

It effectively improves the detection accuracy and detection efficiency of small targets, significantly improves the detection effect of multi-scale targets, and optimizes the training convergence speed and detection accuracy of the network.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968350A_ABST
    Figure CN118968350A_ABST
Patent Text Reader

Abstract

The invention provides an improved target detection method for a high-definition aerial image, which particularly focuses on improving the detection precision of a small target. The method is constructed based on an end-to-end attention target detection network DETR, and three key technology modules, namely a sparse attention module, a scale perception-based feature pyramid module and an optimization query module, are introduced. In a sparse attention module, the time complexity of a DETR encoder is remarkably reduced by adopting a preset offset point and a weight factor to calculate a characteristic value, so that the network is allowed to effectively process and introduce shallow layer characteristics, the detail information of a small target is favorably reserved, and the detection precision is improved. The feature pyramid module based on scale perception further enhances the perception capability of the network to different scale targets. In order to improve the target expression ability, an optimization query module is introduced into a DETR decoder, and the module accelerates the network training process and improves the target expression ability in the decoding process by providing initial position information for query.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to image processing and pattern recognition technology, and in particular to a target detection technology for aerial images based on deep learning and an end-to-end attention target detection network DETR. Background Art

[0002] Aerial images refer to ground image data taken from the air by aircraft such as airplanes, helicopters and drones. They contain geographic information such as terrain, buildings, roads and water bodies. With the rapid development of my country's aerospace technology and imaging technology, the resolution of aerial images is also constantly improving. High-resolution aerial images have become an important means of obtaining large-scale and high-precision geographic information. As one of the key links in aerial image processing, target detection for aerial images has the main task of realizing the automatic positioning and identification of target objects in aerial images, thereby providing strong support for military operations, security monitoring and environmental protection.

[0003] Object detection based on deep learning usually refers to the use of neural networks to detect specific targets from input images and output the location, category, and confidence information of the target. Its main processes generally include feature extraction, candidate box generation, bounding box regression, prediction classification, and NMS. With the development of deep learning, the main algorithms in object detection can be divided into two categories, namely CNN-based object detection algorithms and Transformer-based object detection algorithms. In the CNN-based target detection algorithm, convolution is the core operation. The algorithm first uses a pre-trained CNN backbone network, such as VGG, ResNet or MobileNet, to extract features from the input image. These networks can capture the hierarchical features of the image. Next, the feature map is used to generate candidate regions, which can be anchor boxes or candidate boxes based on specific algorithms. Then, the algorithm classifies these candidate regions to determine whether they contain the target object, and performs bounding box regression at the same time to predict the exact location of the target object. During the classification and regression process, the algorithm assigns a confidence score to each candidate region, indicating the probability that it contains the target object. Finally, non-maximum suppression (NMS) is applied to these confidence-sorted candidate regions to remove redundancy and ultimately output accurate target detection results.

[0004] In the Transformer-based object detection algorithm, attention is the most core operation. The first step of the algorithm is the same as the CNN-based detection model. The pre-trained CNN backbone network is used to extract features from the input image. These features are then fed into the encoder part of the Transformer. Next, DETR uses the encoder to process these features through the self-attention mechanism to capture global context information and obtain the encoded features. Then, DETR introduces a fixed number of learnable queries, which interact with the output features of the encoder in the decoder. Through the self-attention and cross-attention mechanisms of the decoder, each query can aggregate information from the entire image to predict the category and bounding box of the target. DETR does not rely on traditional anchor boxes or candidate region generation steps, but directly generates target predictions in the decoder. Finally, DETR uses a set prediction loss function to train the model. The loss function is optimized by the Hungarian algorithm to achieve the best match between the prediction results and the true annotations. In this way, DETR can output a set of accurate target detection results, including categories and bounding boxes.

[0005] At present, the two mainstream target detection methods can complete some aerial target detection tasks, but we believe that they still have the following problems:

[0006] 1) Low detection accuracy for small targets: When the imaging altitude is high, some small targets such as vehicles and ships will not have too many pixels in the aerial image, thus appearing as small targets. How to detect objects with less information is a major problem in the current target detection field, and it is also a common problem of the two networks.

[0007] 2) Poor generalization: Due to the large variation in the scale of targets in aerial images, there are not many detection algorithms for general scenes, and most algorithms are designed to detect a single aerial target. Due to the large scale differences between different types of targets, algorithms for multi-category detection need to have strong generalization capabilities, which is why there are not many methods for general detection of aerial images. Summary of the invention

[0008] The technical problem to be solved by the present invention is that, in view of the challenge of small target detection in high-definition aerial images, a one-stage target detection network DETR, which runs faster than a two-stage target detection network, is used as a base network, and the model is improved according to the small characteristics of the targets to be detected in aerial images. An improved solution for the target detection network based on DETR is proposed to improve the detection accuracy and efficiency and obtain better detection results.

[0009] The technical solution adopted by the present invention to solve the above technical problems is a target detection method for aerial images based on DETR, comprising the steps of:

[0010] In order to reduce the amount of computation of the encoder and introduce shallow features, the present invention introduces a sparse attention module. The sparse attention module reduces the computational complexity of the DETR encoder when processing shallow features by sparse self-attention mechanism, so that it can introduce shallow features and retain more small target detail information, thereby improving the detection accuracy of small targets. In addition, in order to make full use of shallow feature information, the present invention designs a feature pyramid module based on scale perception. This module makes the information interaction between shallow features and deep features more sufficient by introducing a two-way information flow mechanism, enhances the network's perception of targets of different scales, and thus improves the detection effect of multi-scale targets. Finally, in order to improve the DETR decoder's ability to express targets, the present invention designs a query optimization module on the decoder side. This module accelerates the training convergence of the network by providing initial position information for the query, reduces the training time, and improves the network's ability to express targets in the decoding process, further improving the detection accuracy and efficiency.

[0011] The beneficial effect of the present invention is that it effectively improves the detection accuracy of small targets. By introducing a sparse attention module, the computational complexity of the encoder is reduced to introduce shallow features. Secondly, a feature pyramid module based on scale perception is designed by making full use of shallow features. Finally, in order to improve the expression ability of the DETR decoder for the target, a query optimization module is designed on the decoder side. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the overall network structure of the present invention.

[0013] Figure 2 Schematic diagram of the sparse attention module structure.

[0014] Figure 3 Schematic diagram of the scale-aware feature pyramid module structure.

[0015] Figure 4 Schematic diagram of the optimized query module structure.

[0016] Figure 5 This is a comparison chart of experimental results, showing the detection effect before and after improvement. DETAILED DESCRIPTION

[0017] The present invention selects a fast-running one-stage target detection network DETR as the base network, and improves the model according to the small characteristics of the target to be detected in the aerial image to obtain better detection effect. Figure 1 As shown:

[0018] Step 1: Design and build a DETR basic network model to basically detect the target in the input image; the input of the DETR basic model is the image to be detected, and the output is the detected target location box and category information;

[0019] The DETR basic model includes: input module, backbone network backbone, encoder Encoder, decoder Decoder and output module;

[0020] Step 2: Design a sparse-attention module to calculate the self-attention by selecting reference points to reduce the amount of calculation of the encoder;

[0021] The sparse attention module first calculates the vector x to be paid attention to. k Perform full connection operation to obtain L relative to the reference point p k The offset Δp mkl And the weight A of each offset point mkl , then extract the feature value of the corresponding position from the image feature x according to the reference point and the offset, and then use the weight A mkl The L eigenvalues ​​are weighted and summed to obtain the output of a single attention head. Finally, the outputs of all attention heads are merged through the fully connected layer to obtain the final output. k is the ordinal variable of the image feature x vector, l is the ordinal variable of the offset point of the reference point, and m is the ordinal number of the attention head.

[0022] Step 3: Design the scale-aware pyramid module S-FPN to add a flow channel from bottom to top for the features, so that the small target information in the shallow features can be fully utilized.

[0023] The scale-aware pyramid module includes multi-level input features, feature sampling units, scale perception and feature fusion units.

[0024] like Figure 2 As shown in the figure, the input multi-level features include four multi-scale features C2, C3, C4, and C5 output by the ResNet-50 module, which are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image in size, respectively.

[0025] The feature sampling unit is used to unify the size of the features to be fused. For shallow features, downsampling is used to reduce the size; for deep features, upsampling is used to increase the size; for features in the same layer, no size adjustment is performed and the original size can be maintained.

[0026] The scale-aware unit scale_weight is used to balance the importance of features at different levels. It consists of a pooling layer avg_pool, a fully connected layer fc, and activation functions relu and sigmoid. For the input feature map x, it first performs average pooling on each dimension of x, reduces the scale of x to 1×1, and then uses a fully connected function to reduce the dimension of x to 1 dimension. Then, the relu function is used to activate x, and finally the sigmoid function is used to obtain the eigenvalues ​​of the feature map x.

[0027] The feature fusion module is used to perform weighted fusion on the features after balancing the importance. The specific operation is to add the elements in the same position in the features.

[0028] Step 4: Design and optimize the prompt-query query module to improve the detection accuracy by adding the center point expression of the target in the cross-attention.

[0029] The optimized query module mainly acts on the input end of the cross-attention module, and enhances the feature extraction capability of the decoder by increasing the expression of the target center point. The optimized query module consists of two branches: the first branch converts the position encoding of the query query into a two-dimensional coordinate representing the center point of the object through a multi-layer perceptron, and then uses the σ function to activate the generated coordinates, and finally converts the above coordinates into a vector with the same dimension as the query through the position encoding; the second branch performs nonlinear projection of the query through a feedforward network and converts it into a scaling factor of the first branch of the coordinates; the final output consists of the product of the two branches.

[0030] The following is a detailed description of each detail involved in the embodiment.

[0031] (1) Designing a sparse self-attention module

[0032] The sparse self-attention module structure proposed in this invention is as follows Figure 2 As shown. The input of the entire sparse attention module consists of three parts: image feature x, vector x to be calculated for attention k and the reference point p k , where x k Indicates the corresponding reference point p in the image feature x k For each attention head m, it first performs k Perform a full connection operation to obtain L relative to the reference point p k The offset Δp mkl And the weight A of each offset point mkl , then extract the feature value of the corresponding position from the image feature x according to the reference point and the offset, and then use the weight A mklThe L eigenvalues ​​are weighted and summed to obtain the output of a single attention head. Finally, the outputs of the M attention heads are merged through a fully connected layer to obtain the final output. As shown in the figure, L is 4 and M is 2.

[0033] The above process can be expressed by formula (1):

[0034]

[0035] Where M is the number of attention heads, T represents transpose; represents the learnable weight projection matrix of the attention head, is the real number domain, C is the number of channels of x, and the intermediate quantity C v =C / M; L is the number of preset offset points; Δp mkl Indicates that each offset point is relative to the reference point p k The offset of A mkl Represents the weight factor of each offset point;

[0036] A mkl The calculation is shown in formula (2):

[0037]

[0038] In the formula is a learnable weight projection matrix.

[0039] Combining equations (1) and (2), we can calculate that the time complexity of the sparse attention-based encoder is O(HWL), which is the same as the square time complexity of DETR O(H 2 W 2 ) comparison, it can be found that the sparse attention module proposed in this invention can effectively reduce the time complexity of the encoder, and the time complexity is reduced from O(n 2 ) is reduced to O(n), which also provides a basis for introducing shallow features into the DETR target detector.

[0040] (2) Design a scale-aware feature pyramid module

[0041] The scale-aware feature pyramid module structure proposed in this invention is as follows: Figure 3As shown in the figure. When calculating the features of a certain layer, S-FPN no longer only considers the features deeper than the current feature level, but also adds shallow features to the generation of new features. In this way, information can flow from top to bottom and from bottom to top between feature pyramids, so that the information of shallow features can also be added to deep features, making full use of the richer target position information in shallow features; when fusing features, unlike FPN which directly adds features of different scales, S-FPN aligns features of different scales in size and dimension, obtains the weights of features of different scales through the scale perception module, and then weights and superimposes these features to output the final features of the current layer.

[0042] The expression of S-FPN is in the form of formula (3):

[0043]

[0044] In the formula represents the i-th layer feature map of the input; Conv represents 1×1 convolution, which is used to adjust the dimension of the feature; Sample represents the sampling operation, which is used to unify the scale of the feature. When the variable k=-1, it means downsampling, when k=1, it means upsampling, and when k=0, it keeps the original scale; f L (x) represents the scale perception module, and its principle is shown in formula (4):

[0045] f L (x)=σ(relu(fc(avg_pool(x)))) (4)

[0046] In the formula represents the input feature map, avg_pool represents average pooling, fc represents full connection, relu and σ are both activation functions. The overall process of the scale-aware module is as follows: for the input feature map x, it first performs average pooling on each dimension of x to reduce the scale of x to 1×1, then uses the full connection function to reduce the dimension of x to 1 dimension, then uses the relu function to activate x, and finally uses the σ function to obtain the eigenvalue of the feature map x. The larger the eigenvalue, the greater the importance of the current feature map.

[0047] (3) Design and optimize query module

[0048] The optimized query module structure proposed by the present invention is as follows: Figure 4As shown. The optimized query module mainly acts on the input end of the Cross-Attention module, and enhances the feature extraction capability of the decoder by increasing the expression of the target center point. The optimized query module consists of two branches: the first branch converts the positional encoding 1 of the query into a two-dimensional coordinate representing the center point of the object through a multi-layer perceptron, and then uses the σ function to activate the generated coordinates, and finally converts the above coordinates into a vector positional encoding 2 with the same dimension as the query through positional encoding; the second branch nonlinearly projects the query through the feedforward network FFN and converts it into a scaling factor of the first branch of the coordinates; the final output consists of the product of the two branches. The above process is shown in formula (5):

[0049] Prompt-query(q)=pos2(σ(MLP(pos1(q))))*ffn(q) (5)

[0050] In the formula represents query; pos1 represents the original decoder's position encoding of the query; pos2 represents the position encoding of the center point; σ is the activation function; MLP represents a multi-layer perceptron; ffn represents a feedforward network.

[0051] In order to decouple the semantic information of the target from the position information, the cross attention is no longer the addition of the feature encoder feature map and the positional encoding at the input, but the concatenation In this way, the network can directly distinguish the features from the position encoding during training, so that it no longer needs to spend a lot of time decoupling the input. Finally, the cross-attention module calculation method for optimizing the query module is introduced as shown in formula (6):

[0052]

[0053] In the formula represents query; C is the number of channels, that is, the dimension of the feature; Represents the query′ output by the self-attention module; represents the concatenation operation; the prompt-query function is shown in formula (5); is the feature of the encoder output; and are all learnable matrices.

[0054] In order to prove the effectiveness of the improved DETR-based target detection network algorithm proposed in the present invention, the present invention tested the above model on the public aerial target detection dataset DOTA, and used the area under the PR curve mAP as the model evaluation indicator. The dataset contains 18 different types of targets. The model is trained using a general strategy and mAP is calculated. The test results of the method in the present invention exceed the detection results of some current cutting-edge methods.

[0055] Table 1 shows the comparison between Ours and other cutting-edge methods on the DOTA dataset. The experimental comparison evaluation index is the area under the PR curve mAP:

[0056]

[0057]

[0058] Table 1

[0059] The first three columns of the table are one-stage target detection networks based on CNN, the fourth to sixth columns are target detection networks based on Transformer, and the seventh column Ours is the method proposed by the present invention. It can be found from the table that, compared with FCOS and RepPoints in the CNN detection network, the method proposed by the present invention leads the FCOS network by 7.1% and the RepPoints network by 7.9% in average accuracy, which is much higher than these two methods, indicating the superiority of the present invention in realizing target detection using the Transformer architecture; compared with the target detection network based on the Transformer architecture, the method proposed by the present invention is higher than these networks in terms of average AP and AP of most categories, especially in the categories with more small targets such as ships, small vehicles and large vehicles, the method proposed by the present invention is obviously ahead, indicating that the optimization of DETR for small targets of the present invention is effective; compared with YOLO v8, although the method proposed by the present invention has a large gap in running speed, the detection accuracy of the present invention is higher.

[0060] Figure 5The detection results of some pictures are shown to show the improved effect. The Input column represents the input original picture, the GT column is the abbreviation of Ground Truth, which represents the real annotation corresponding to the original picture, the DETR column represents the detection result using DETR, and the Ours column represents the detection effect after using the improved method of the present invention. Each row represents the detection results of different test pictures. In samples 1, 2, and 3, the unimproved DETR model missed detection, and in sample 4, the unimproved DETR had a wrong detection. Sample 1 is an empty scene with sparse targets. The DETR model missed the small target of the small car, while the improved method of the present invention successfully detected the small target; in sample 2, the DETR model missed the large target of the baseball field, which was successfully detected by the improved method of the present invention; sample 3 is a scene with dense small targets. It can be seen that the DETR model missed most of the targets. Although the improved method of the present invention missed detection, the detection effect is greatly improved; in sample 4, the DETR model mistakenly detected the house as a large vehicle, and the improved method of the present invention did not have this problem.

[0061] The main features of the present invention are: 1) In order to reduce the computational complexity of the encoder in the DETR model, the present invention designs a sparse attention module, which calculates self-attention by selecting reference points instead of incorporating all feature points into the calculation, thereby reducing the amount of computation of the encoder. 2) In order to make full use of shallow features, the present invention designs a feature pyramid module based on scale perception. Unlike the traditional feature pyramid structure, the pyramid adds a flow channel from bottom to top for the features, and can assign weights according to the scales of different features when generating new features, so that the small target information in the shallow features can be fully utilized. 3) In order to improve the DETR decoder's ability to express the target, the present invention designs a query optimization module on the decoder side, which increases the DETR training speed and improves the detection accuracy by increasing the expression of the center point of the target in the cross-attention.

Claims

1. A target detection method for aerial images based on DETR, characterized in that: Includes steps: Input step: receiving input aerial images to be subjected to target detection; Backbone network processing steps: perform target detection on the input aerial image based on the end-to-end attention target detection network DETR, and output the preliminary target image to the encoder; Encoder processing steps: The encoder that introduces the sparse attention module calculates the eigenvalues ​​of the input target image through the preset offset points and weight factors, and outputs the eigenvalues ​​to the feature pyramid module; Feature pyramid module processing steps : The feature pyramid module performs multi-scale fusion of input feature values ​​based on scale perception and outputs the fused features to the decoder; Decoder processing steps: Introduce the optimized query module decoder to output the detected target position box and category information based on the input fusion features.

2. The method according to claim 1, characterized in that: The specific processing method of the encoder that introduces the sparse attention module for the input target image is: Perform a full connection operation on the vectors in the target image that need to calculate attention, obtain the offset relative to the reference point and the weight of each offset point; extract the feature value of the corresponding position from the target image according to the reference point and the offset; use the weight to perform weighted summation on the feature values ​​to obtain the output of a single attention head; The outputs of all attention heads are merged through the fully connected layer to obtain the feature values ​​of the final output of the encoder.

3. The method according to claim 1, characterized in that: The specific processing method of the feature pyramid module processing step is as follows: when the feature pyramid module processes a layer of features in a multi-level feature value, it fuses the current layer features and the features deeper than the current feature layer and the features shallower than the current feature layer, thereby utilizing the target position information in the shallow feature; when fusing, it uses the scale perception unit to balance the importance of features at different levels to obtain the weighted fusion feature; Among them, the scale-aware unit reduces the scale of the input feature value to 1×1 through the pooling layer, and then reduces the dimension of the feature value to 1 dimension through the fully connected layer. After that, the 1-dimensional feature value is nonlinearly transformed through the activation function, and finally the feature value after nonlinear transformation is input into the weight acquisition function to obtain the weights of different scale features.

4. The method according to claim 1, characterized in that: In the decoder processing step, the fused features are used to obtain the target position box and category information of the predicted target through the self-attention mechanism and the cross-attention mechanism. The query module is optimized to output the query vector q input of the cross-attention mechanism, and the feature extraction capability of the decoder is enhanced by increasing the expression of the target center point. The specific processing method of optimizing the query module is: The position encoding of the query is converted into two-dimensional coordinates representing the center point of the object through a multi-layer perceptron; the generated coordinates are activated using an activation function; the activated coordinates are converted into a vector with the same dimension as the query through position encoding; the query is nonlinearly projected through a feedforward network and converted into a scaling factor for the coordinates; the outputs of the two branches are multiplied to obtain the final optimized query output.

5. The method according to claim 4, characterized in that: At the key vector k input end of the cross attention, the fusion feature and the position encoding are fused in a splicing manner and input as the key vector k.

Citation Information

Patent Citations

  • Aerial photography target detection method based on attention mechanism and adaptive feature fusion

    CN116524379A

  • Yolov5 target detection method based on cross-stage routing attention module and residual information fusion module

    CN116721398A

  • DETR improved model-based sparse attention target detection method

    CN117152416A

  • Aero-engine blade defect detection method based on multi-scale DETR

    CN117173449A

  • Three-dimensional point cloud target detection method based on Transform sparse coding and decoding

    CN117727022A

Cited By

  • Leaf vegetable disease detection regulation and control method and system based on deep learning

    CN119863061A