Unmanned aerial vehicle target detection system and method based on improved RT-DETR

Abstract

The invention discloses an improved RT-DETR-based unmanned aerial vehicle target detection system and method. The method comprises the steps of constructing an improved RT-DETR-based unmanned aerial vehicle target detection network, wherein the improved RT-DETR-based unmanned aerial vehicle target detection network comprises an improved backbone network structure, an improved encoder structure and a decoder structure; shallow feature extraction, processing and self-attention calculation are carried out through a feature preprocessing module, a flexible fusion module and an attention-based internal scale feature interaction module; feature fusion and coding output are carried out through a multi-scale perception feature pyramid network SAFPN and a multi-feature fusion network MSFNet; the decoder structure outputs target position information and category information to the coding features; and training an unmanned aerial vehicle target detection network of the improved RT-DETR to obtain a detection model, and evaluating and selecting an optimization model to perform target detection on an unmanned aerial vehicle image. According to the method, the sensitivity to small targets, dense targets and shielded targets is enhanced, and the accuracy and real-time performance of target detection of the unmanned aerial vehicle are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of UAV target detection, and in particular to a UAV target detection system and method based on improved RT-DETR. Background Technique

[0002] In recent years, unmanned aerial vehicle (UAV) technology has been widely used in military, industrial, agricultural, urban management, disaster relief and other fields. With the advantages of high efficiency, flexibility and low cost, UAVs have become an indispensable tool. Compared with traditional fixed cameras, UAV surveillance has higher mobility and a larger surveillance range; compared with remote sensing satellites, UAVs have lower costs, are easier to operate, and are not restricted by weather and time, and can obtain high-quality low-altitude images. However, UAV target detection faces many challenges. First, the changes in target scale and perspective make the presentation effects of the same type of targets in images vary greatly, which easily leads to false detections and missed detections. Second, during aerial photography, UAVs may collect small and dense targets in scenes such as crowds, traffic flows, and crops due to the long shooting distance, which poses high requirements for the accurate recognition and positioning capabilities of algorithms. In addition, complex backgrounds and target occlusions, such as trees, buildings, and wires, may partially or completely occlude the target object, resulting in missing target information. Finally, limited by volume and weight, the computing power of UAVs is limited, and it faces great challenges to achieve real-time detection tasks in complex scenarios. Existing UAV target detection schemes are mainly divided into two categories: UAV target detection based on traditional algorithms and UAV target detection based on deep learning. The target detection algorithms of traditional computer vision require complex manual designs, such as feature extraction and template design, and have limited generalization ability, usually only being able to adapt to specific simple scenarios. The UAV target detection algorithms based on deep learning enhance the generalization ability, enabling UAVs to handle various complex scenarios. However, existing algorithms need to rely on preprocessing of cropping or postprocessing of non-maximum suppression (NMS), with high computational complexity and poor performance in UAV target detection tasks. Therefore, improving the performance and efficiency of UAV target detection, especially in complex scenarios, has become an urgent problem to be solved.

[0003] The patent application of the System Engineering Research Institute of the Academy of Military Sciences of the Chinese People's Liberation Army, "A Drone Small Target Detection Method Based on Anchor-Free Algorithm" (Patent Application No.: 202310503724.7). The first step: Use a drone device to collect small target images. The second step: Extract target features of different sizes from the small target images based on the Backbone network. The FCOS model of the Backbone network uses the VovNet network. The third step: According to the target features, use a feature pyramid to obtain multi-scale features, and perform a bidirectional dense connection operation on the multi-scale features to obtain a fused feature map. The fourth step: Input the fused feature map into the IoU-Head network to detect targets of different scales. The main drawback of this algorithm is that the VovNet network used lacks the ability to globally model features and cannot solve the occlusion problem in the drone target detection task.

[0004] The patent application of Hangzhou Dianzi University, "A Drone Aerial Small Target Detection Method Based on Improved RT-DETR Network" (Patent Application No.: 202410988027.X). The first step: Obtain a public dataset and preprocess it. The second step: Build a small target detection model with the RT-DETR network as the basic network. The RT-DETR network includes a backbone network and an efficient hybrid encoder. Replace the conventional convolution in the backbone network Resnet-r18 with a partial convolution reparameterization module. Replace the AIFI module in the efficient hybrid encoder with a deformable attention module. Introduce a p2 detection head into the CCFM module in the efficient hybrid encoder to obtain the CCFM-p2 module. The third step: Configure the training environment and use the preprocessed dataset as input to train the small target detection model. The fourth step: Apply the trained small target detection model and use the target image to be detected as input to output the detection result. The main drawback of this algorithm is that directly introducing the P2 small target layer increases the computational complexity of the network and affects the inference efficiency of the network. And this algorithm cannot solve the problem of scale perspective transformation in the drone target detection task.

[0005] The first step of the patent application of State Grid Shandong Electric Power Company Electric Power Research Institute, "Method and System for Identifying Electric Power Components in UAV Inspection Images Based on Faster R-CNN" (Patent Application No.: 201610906708.2), is to pre-train the ZFnet model and extract the feature map of the UAV inspection image; the second step is to train the initialized RPN region proposal network model to obtain the region extraction network; the third step is to use the position feature, deep feature, and feature map of the target to train the initialized Faster R-CNN detection network to obtain the electric power component detection model; the fourth step is to use the electric power component detection model to perform actual electric power component identification and detection. The main disadvantage of this algorithm is that the Faster R-CNN network model is a two-stage detection model, which generates a large number of candidate boxes during the detection process, has a slow inference speed, and a large number of network parameters, and is not suitable for scenarios such as UAV target detection that have requirements for network parameters and real-time performance. And this model cannot solve the problems of dense targets and occlusion in UAV target detection tasks.

[0006] The first step of the patent application of Qiqihar University, "UAV Image Small Target Detection Algorithm Based on Improved YOLOv7" (Patent Application No.: 202410258507.0), is to add an attention mechanism between the Backbone backbone network and the Neck module of YOLOv7; the second step is to replace the activation function in the original CBS structure of YOLOv7 with the ACON activation function; the third step is to add a new NWD small target detection metric to the loss function; the fourth step is to modify the upsampling operation of YOLOv7 and use the CARAFE lightweight general upsampling operator to complete the improvement of YOLOv7; the fifth step is to use a public dataset to verify the performance of the improved YOLOv7 model in step four and evaluate the effectiveness of the improved model. The main disadvantage of this algorithm is that the YOLOv7 algorithm relies on the NMS process to obtain the final output, and dealing with the dense target problem in the UAV target detection task will significantly increase the inference delay. And this algorithm cannot solve the problems of target scale and perspective transformation and target occlusion.

[0007] As can be seen from the above, the UAV target detection task mainly solves problems such as target scale and perspective transformation, small target dense targets, image occlusion, etc., and has certain requirements for the lightweight of the algorithm. Among the above four methods, the first method lacks the global modeling ability of features and cannot solve the target occlusion problem. The second and third methods have a high computational complexity and the models are not lightweight enough. Moreover, the second method cannot solve the problems of target scale and perspective transformation, and the third method cannot solve the dense target detection problem. The fourth method cannot solve the problems of target scale and perspective transformation and target occlusion.

[0008] While maintaining high accuracy on public datasets, the RT-DETR algorithm has a small number of parameters and a fast detection speed, showing good detection performance. However, when applied to the UAV target detection task, it cannot solve problems such as variable target scales, complex backgrounds, and small and dense targets in UAV target detection, resulting in limited detection accuracy. Summary of the Invention

[0009] To solve the problems of dense target scale and perspective transformation, target occlusion, and limited detection accuracy in the UAV target detection task in the existing technology, the purpose of the present invention is to propose a UAV target detection system and method based on improved RT-DETR, which reduces the number of parameters of the model, can learn multi-scale features well, improves the global context learning ability of the model, enhances the sensitivity to small targets, dense targets, and occluded targets, and improves the accuracy and real-time performance of UAV target detection, obtaining better detection results.

[0010] The present invention is realized through the following technical solutions.

[0011] In one aspect of the present invention, a UAV target detection method based on improved RT-DETR is provided, including:

[0012] Construct a UAV target detection network based on improved RT-DETR, including an improved backbone network structure, an improved encoder structure, and a decoder structure;

[0013] The improved backbone network structure includes a feature preprocessing module, 4 flexible fusion modules, and an attention-based inner-scale feature interaction module, which respectively perform shallow feature extraction, shallow feature processing, and self-attention calculation on multiple scales;

[0014] The improved encoder structure includes a multi-scale perception feature pyramid network SAFPN and a multi-feature fusion network MSFNet, which respectively perform feature fusion and encode and output the fused encoded features;

[0015] The decoder structure adopts the original RT-DETR structure to output the target position information and category information for the fused encoded features output by the improved encoder structure;

[0016] Use the VisDrone training set to train the constructed UAV target detection network based on improved RT-DETR to obtain a UAV target detection model based on improved RT-DETR;

[0017] Use the VisDrone validation set to evaluate the UAV target detection model based on improved RT-DETR, and adjust the model training parameters according to the training log and evaluation metrics to obtain an optimized model;

[0018] Select the optimization model with the best performance to perform object detection on the images collected from the UAV perspective, and obtain the object detection result image.

[0019] Preferably, construct a UAV object detection network based on the improved RT-DETR, including:

[0020] Construct an improved backbone network structure for multi-scale feature extraction;

[0021] Construct an improved encoder structure for feature fusion and encoding to output the fused encoded features;

[0022] Through the original RT-DETR decoder structure, output the object position information and category information for the fused encoded features output by the improved encoder structure.

[0023] Preferably, construct an improved backbone network structure for multi-scale feature extraction, including:

[0024] Adopt 4 flexible fusion modules and an attention-based intra-scale feature interaction module to gradually process the shallow feature layer X0 output by the feature preprocessing module to obtain the small object feature layer P2, the medium object feature layer P3, the large object feature layer P4, and the feature layer F5 of global context information in sequence.

[0025] Preferably, the processing process of the 4 flexible fusion modules and the attention-based intra-scale feature interaction module is as follows:

[0026] The shallow feature X0 output by the feature preprocessing module is input into the first flexible fusion module to obtain the output feature X3 of the first flexible fusion module. The output feature X3 is copied twice, one as the small object P2 feature layer, and the other as the input feature of the second flexible fusion module;

[0027] And so on. After the previous output feature is input into the fourth flexible fusion module and undergoes the same calculation process, the output feature X6 is obtained. The output feature X6 is input into the attention-based intra-scale feature interaction module and undergoes self-attention calculation between features to obtain the feature layer F5 containing global context information.

[0028] Preferably, construct an improved encoder structure for feature fusion and encoding to output the fused encoded features, including:

[0029] Adopt a multi-feature fusion network MSFNet to guide a multi-scale perception feature pyramid network SAFPN to fuse the small object feature layer P2, the medium object feature layer P3, the large object feature layer P4, and the feature layer F5 of global context information to obtain the fused encoded features;

[0030] Use the multi - feature fusion network MSFNet to extract guiding features containing global information for subsequent guiding the feature pyramid network SAFPN with multi - scale perception to fuse feature layers of multiple scales.

[0031] Preferably, the structure of the multi - feature fusion network MSFNet module includes three 1×1 convolutions, a channel fusion module, a 3D convolution, a LeakyReLU activation function module, a 3D max - pooling module, and a dimension compression module.

[0032] Preferably, the structure of the feature pyramid network SAFPN with multi - scale perception includes five fusion modules and six convolutions.

[0033] Preferably, the processing process of the feature pyramid network SAFPN with multi - scale perception is as follows:

[0034] The multi - feature fusion network MSFNet outputs multi - scale fusion feature F final and the backbone network extracts the small - object feature layer P2, the medium - object feature layer P3, the large - object feature layer P4, and the feature layer F5 of global context information as the input of the feature pyramid network SAFPN with multi - scale perception;

[0035] The feature layer F5 of global context information is upsampled to the resolution of the large - object feature layer P4 by bilinear interpolation, and the upsampled feature layer F5 of global context information and the large - object feature layer P4 are weighted - fused to generate the first intermediate feature P4';

[0036] The first intermediate feature P4' is upsampled to the resolution of the medium - object feature layer P3 by bilinear interpolation and weighted - fused with the medium - object feature layer P3 to generate the second intermediate feature P3'; the small - object feature layer P2 is downsampled to the resolution of the second intermediate feature P3' and weighted - fused with the second intermediate feature P3' and the medium - object feature layer P3 to generate the small - scale encoded feature P3'';

[0037] After passing through a convolution, the small - object fusion encoded feature F S is obtained, and the small - object fusion encoded feature F S is copied twice, one as the output, and the other is downsampled to the resolution of the first intermediate feature P4' and weighted - fused with the first intermediate feature P4' and the large - object feature layer P4 to generate the medium - scale encoded feature P4'';

[0038] The medium - scale encoded feature P4'' passes through a convolution to obtain the medium - object fusion encoded feature F M is obtained, and the medium - object fusion encoded feature F M is copied twice, one as the output, and the other is downsampled to the resolution of the feature layer F5 of global context information and weighted - fused with the feature layer F5 of global context information and the multi - scale fusion feature F finalWeighted fusion generates large-scale encoded feature P5”;

[0039] The large-scale encoded feature P5” undergoes a convolution to obtain the large-object fusion encoded feature F L , and the multi-scale perception feature pyramid network SAFPN outputs the small-object fusion encoded feature F S , the medium-object fusion encoded feature F M and the large-object fusion encoded feature F L Three fused encoded features.

[0040] Preferably, the constructed drone target detection network based on the improved RT-DETR is trained using the VisDrone training set, including:

[0041] Convert the annotation information in the VisDrone dataset into the YOLO format convenient for training; the annotation information of each target after conversion includes the category of the target and the position of the target in the picture;

[0042] Use the VisDrone training set to train the improved drone target detection network based on the improved RT-DETR;

[0043] Record the training log, and the log content is the loss value and AP metric for each iteration during training, to obtain the drone target detection model based on the improved RT-DETR.

[0044] Another aspect of the present invention provides a drone target detection system based on the improved RT-DETR for the said method, including:

[0045] The improved backbone network structure, including:

[0046] The feature preprocessing module is used for shallow feature extraction of multiple scales;

[0047] 4 flexible fusion modules are used for processing shallow features;

[0048] The attention-based inner-scale feature interaction module is used for self-attention calculation;

[0049] The improved encoder structure, including:

[0050] The multi-scale perception feature pyramid network SAFPN is used for feature fusion;

[0051] The multi-feature fusion network MSFNet is used for extracting guiding features containing global information to guide the multi-scale perception feature pyramid network SAFPN to fuse feature layers of multiple scales;

[0052] Decoder structure, adopting the original RT-DETR structure, outputs target position information and category information for the fused encoded features output by the improved encoder structure.

[0053] Due to the above technical solutions adopted by the present invention, it has the following beneficial effects:

[0054] 1. By adopting the above technical solutions, in order to improve the network's feature extraction ability for targets of different scales and address the scale transformation problem in UAV target detection, a multi-branch structure composed of convolutions of different sizes is adopted in the FlexiFusionBlock of the backbone network. Compared with the single-branch convolution structure, convolutions of different sizes can obtain different receptive fields, and the multi-branch structure fuses the features of multiple receptive fields, effectively enhancing the network's feature extraction ability for targets of different scales; in order to reduce the computational amount and the number of parameters of the model, the present invention adopts partial convolution in the FlexiFusionBlock of the backbone network, and the partial convolution only performs convolution operations on the first quarter of the channels in the convolution, greatly reducing the memory access amount and the computational amount.

[0055] 2. In order to further improve the network's detection ability for small targets and dense targets, learnable fusion weights are designed in the multi-scale perception feature pyramid network SAFPN of the improved encoder structure. These weights can be continuously optimized during the training process, and finally the optimal fusion weights of each feature layer are obtained. Compared with the method of adding feature layers in RT-DETR, it significantly enhances the network's feature fusion ability and improves the network's detection ability for small and dense targets.

[0056] 3. In order to further improve the network's detection ability for occluded targets, in the multi-feature fusion network MSFNet of the improved encoder structure, 3D convolution is used to fuse the P3, P4, and F5 feature layers, effectively enhancing the network's global feature understanding ability of features. The process of MSFNet guiding SAFPN for feature fusion integrates the features of targets of different scales with global features, significantly improving the network's recognition ability for occluded targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not constitute an improper limitation to the present invention. In the drawings:

[0058] Figure 1 is the flowchart of the specific implementation manner provided by the present invention;

[0059] Figure 2 is the structural diagram of a UAV target detection algorithm based on the improved RT-DETR provided by the present invention;

[0060] Figure 3Structural diagram of the multi-receptive field fusion backbone network provided by the present invention;

[0061] Figure 4 Structural diagrams of the multi-feature fusion network and the multi-scale perception feature pyramid network in the multi-scale perception fusion encoder designed for the present invention

[0062] Figure 5 Comparison of the detection results between the RT-DETR algorithm and the UAV target detection algorithm based on the improved RT-DETR of the present invention. Detailed implementation manners

[0063] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. Here, the illustrative embodiments of the present invention and the description are used to explain the present invention, but not to limit the present invention.

[0064] Refer to Figure 1 , the present invention provides a UAV target detection method based on improved RT-DETR, and the specific implementation includes the following steps:

[0065] Step 101, construct a UAV target detection network based on improved RT-DETR, including an improved backbone network structure, an improved encoder structure and a decoder structure.

[0066] Step S11, extract multi-scale features through the improved backbone network.

[0067] Refer to Figure 2 , the improved backbone network uses a feature preprocessing module, 4 flexible fusion modules and an attention-based internal scale feature interaction module to replace the ResNet50 backbone network in RT-DETR.

[0068] Among them, the feature preprocessing module uses operations of convolution-batch normalization-activation function (Conv-BatchNorm-Sliu) to extract shallow features.

[0069] The feature preprocessing module needs to extract shallow features from the input image. The extraction process is shown in formula (1),

[0070] X0 = Silu(BatchNorm(Conv 3×3 (X))) (1)

[0071] In the formula, X represents the input image, and X0 represents the output preprocessed shallow features.

[0072] Among them, the improved backbone network uses 4 flexible fusion modules and an attention-based intra-scale feature interaction module to gradually process the shallow feature layer X0 output by the feature preprocessing module, successively obtaining the small object feature layer P2, the medium object feature layer P3, the large object feature layer P4, and the feature layer F5 containing global context information.

[0073] As Figure 3 shown, the structures in the improved backbone network are the same, and they are responsible for successively extracting deep features. The attention-based intra-scale feature interaction module has the same structure as that in RT-DETR. This module is responsible for performing self-attention calculations on the features gradually extracted by the 4 flexible fusion modules to obtain the feature layer F5 containing global context information. The processing processes of the 4 flexible fusion modules and the attention-based intra-scale feature interaction module are as follows:

[0074] (a) The shallow feature X0 output by the feature preprocessing module is first input into the first flexible fusion module to obtain the small object feature layer P2.

[0075] After X0 is input into the first flexible fusion module, it first extracts feature information through a PConv with a convolution kernel size of 3 and a stride of 2 to obtain the feature X1, as shown in formula (2). Subsequently, the feature X1 is copied into four parts, and three of them respectively pass through PConv with convolution kernel sizes of 1×1, 3×3, and 5×5 to obtain three groups of features with different receptive field sizes. In addition to the three branches with different receptive fields, there is also a branch for residual connection responsible for retaining the original information and enhancing the expression ability of the deep network. Subsequently, the feature information of the four branches is summed to obtain the multi-receptive field feature X2, as shown in formula (3).

[0076] X1 = PConv(X0, 3) (2)

[0077] X2 = X1 + PConv(X1, 1) + PConv(X1, 3) + PConv(X1, 5) (3)

[0078] The feature X2 integrates channel information through a 1×1 convolution, fuses the features with different receptive fields, and finally improves the regularization effect of the features and introduces non-linearity through the BN layer and the activation function layer to obtain the output feature X3 of the module, as shown in formula (4).

[0079] X3 = Silu(BatchNorm(PConv(X2, 1))) (4)

[0080] The first feature fusion module completes the extraction of the shallow features and obtains the output feature X3. The feature X3 is copied into two parts, one part is used as the small object P2 feature layer, and the other part is used as the input feature of the second flexible fusion module.

[0081] (b) The feature X3 is input into the second flexible fusion module and undergoes the same calculation process as formula (2-4) to obtain the output feature X4 of the second flexible fusion module. Similarly, the feature X4 is copied twice, one as the output of the medium target detection P3, and the other as the input feature of the third flexible fusion module.

[0082] (c) After the output feature X4 is input into the third feature fusion module, it undergoes the calculation process of formula (2-4) to obtain the output feature X5. Similarly, X5 is copied twice, one as the output of the large target feature layer P4, and the other as the input of the fourth flexible fusion module.

[0083] (d) After the output feature X5 is input into the fourth flexible fusion module, it undergoes the same calculation process to obtain the output feature X6. The output feature X6 is input into the attention-based inner-scale feature interaction module and undergoes self-attention calculation between features to obtain the feature layer F5 containing global context information.

[0084] After the shallow feature X0 is successively extracted by 4 groups of flexible fusion modules and the attention-based inner-scale feature interaction module, the small target feature layer P2, the medium target feature layer P3, the large target feature layer P4, and the feature layer F5 containing global context information are successively output. So far, the improved backbone network is constructed.

[0085] S12. Construct an improved encoder structure to fuse and encode features and output the fused encoded features

[0086] As Figure 2 and Figure 4 shown, the improved encoder structure includes a multi-scale perception feature pyramid network SAFPN and a multi-feature fusion network MSFNet. The multi-feature fusion network MSFNet is responsible for guiding the multi-scale perception feature pyramid network SAFPN to fuse the feature layers of multiple scales of the small target feature layer P2, the medium target feature layer P3, the large target feature layer P4, and the feature layer F5 containing global context information, and obtains the fused small target encoded fusion feature F S , the medium target fusion encoded feature F M and the large target fusion encoded feature F L .

[0087] Among them, the improved encoder structure uses the multi-feature fusion network MSFNet to extract the guiding feature containing global information for subsequent guiding SAFPN to fuse the feature layers of multiple scales.

[0088] The structure of the multi-feature fusion network MSFNet module is as Figure 4It contains three 1×1 convolutions, a channel fusion module, a 3D convolution, a LeakyReLU activation function module, a 3D max pooling module, and a dimension compression module. The specific processing process is as follows;

[0089] (a) First, use 1×1 convolutions to adjust the number of channels of the three feature maps P3, P4, and F5 to 256, and adjust the size of the feature maps of each feature layer. To retain the feature information in the original feature layer, use the upsampling method to make the feature sizes of the P4 and F5 feature layers consistent with that of the P3 feature layer, obtaining feature maps S3, S4, and S5, S i ∈R B×256×H×W , where B represents the number of samples processed in one forward propagation process, 256 represents the number of input channels, H represents the height of the convolutional kernel, and W represents the width of the convolutional kernel.

[0090] (b) Use the Unsqueeze function to increase the dimension of the feature maps, obtaining three upsampled feature tensors S3′, S4′, and S5′. As shown in formula (5), concatenate the three features in the depth dimension to obtain the concatenated tensor T∈R B×256×3×H×W , as shown in formula (6). In the formula, dim represents the dimension number, and 3 represents the dimension of the tensor.

[0091]

[0092] (c) Use 3D Convolution to fuse the features of multiple feature layers to obtain a tensor, as shown in formula (7), where K represents the 3D convolutional kernel, c, d, h, w represent the number of channels, depth, height, and width dimensions of the 3D convolution, and d′, h′, w′ represent the depth, height, and width dimensions.

[0093]

[0094] (d) Subsequently, F passes through the activation function and the 3D max pooling layer MaxPool3D. In the 3D max pooling layer, the pooling window is (3, 1, 1). After pooling, the feature tensor F pool ∈R B×C×1×H×W is obtained, as shown in formula (8). Finally, use the Squeeze function to compress the number of channels to obtain the fused multi-scale fusion feature F final ∈R B×C×H×W , as shown in formula (9).

[0095]

[0096] After passing through the multi-feature fusion network, the multi-scale fusion feature F that fuses the location and global context information of the features is obtained final, which is used to guide the feature fusion process in SAFPN and significantly enhance the network's sensitivity to occluded targets.

[0097] Among them, the improved encoder structure uses the Scale-Aware Feature Pyramid Network (SAFPN) with multi-scale perception to fuse multi-scale features to produce the fused encoded features.

[0098] The structure of the Scale-Aware Feature Pyramid Network (SAFPN) with multi-scale perception is as Figure 4 shown. The SAFPN structure mainly includes five fusion modules and six convolutions. The processing process of SAFPN is as follows:

[0099] The multi-scale fused feature F of the Multi-Scale Fusion Network (MSFNet) final and the feature maps P2, P3, P4, and F5 of different scales extracted by the backbone network are used as the inputs of SAFPN. F5 is upsampled to the resolution of P4 by bilinear interpolation. The upsampled F5 and P4 are weighted and fused to generate the first intermediate feature P4'. The weighted fusion process is shown in Equation (10), where w F5 and w P4 are the learnable parameters corresponding to the F5 and P4 feature layers respectively, and ∈ is a very small decimal to prevent the denominator from being zero.

[0100]

[0101] P4' is upsampled to the P3 resolution by bilinear interpolation and weighted and fused with P3 to generate the second intermediate feature P3'. P2 is downsampled to the P3' resolution by a 3×3 convolution with a stride of 2 and weighted and fused with P3' and P3 to generate the small-scale encoded feature P3”. After passing through a convolution, the fused small-target fusion encoded feature F S , F S is copied twice, one as the output, and the other is downsampled to the P4' resolution and weighted and fused with P4' and P4 to generate the medium-scale encoded feature P4”. After passing through a convolution, the fused medium-target fusion encoded F M , F M is copied twice, one as the output, and the other is downsampled to the F5 resolution and weighted and fused with F5 and F final to generate the large-scale encoded feature P5”. After passing through a convolution, the fused large-target fusion encoded feature F L is obtained. SAFPN outputs three fused encoded features: the fused small-target fusion encoded feature F S , the fused medium-target fusion encoded feature F M , and the fused large-target fusion encoded feature F L .

[0102] After passing through the multi-feature fusion network MSFNet and the feature pyramid network SAFPN with multi-scale perception, the encoded features after multi-scale feature fusion are obtained, and the improved encoder structure is completed.

[0103] Step S13, construct a decoder structure to output the target position information and class information.

[0104] As Figure 2 shown, the encoder structure adopts the original structure in RT-DETR, and the fused encoded feature F S , F M and F L after being input into the encoder and performing cross-attention operations with 300 groups of queries preset in the encoder to obtain the features of the target. After processing the target features, the final detection results will be obtained. The final results include the class information (the class number and confidence score of the target) and the position information (the position coordinates of the target). Thus, the UAV target detection network based on the improved RT-DETR is obtained.

[0105] Step S102, prepare the VisDrone dataset, and use the VisDrone training set to train the constructed UAV target detection network based on the improved RT-DETR to obtain the UAV target detection model based on the improved RT-DETR.

[0106] Specifically, it includes the following steps:

[0107] Step S21, prepare the VisDrone dataset.

[0108] The VisDrone dataset is a large-scale UAV vision dataset, including 6471 training images and 548 validation images. The format of the VisDrone dataset cannot be directly used by RT-DETR, and there are a large number of images without compliant annotations and labels outside the specified categories. This method converts the annotation information in the dataset into the YOLO format convenient for training. The annotation information of each target after conversion contains five numbers [label, X1, Y1, X2, Y2], where X1 is an integer representing the class number of the target, and the number range from 0 to 9 represents ten categories: pedestrian, person, bicycle, car, van, truck, tricycle, sunshade tricycle, bus, motorcycle; X1, Y1 represent the upper left coordinates of the target box, and X2, Y2 represent the lower right coordinates of the target box. These five numbers represent the class of the target and the position of the target in the image.

[0109] Step S22, use the VisDrone training set to train the improved UAV target detection network based on the improved RT-DETR.

[0110] (a) Resize the resolution of the input images in the training set to 1500×1500. The specific method is to adjust the longer side of the input image length and width to 1500, and adjust the other side according to the ratio of the longer side adjustment. The area less than 1500 is filled with black pixels. The image after resolution adjustment is used as the input for model training.

[0111] (b) Determine all training parameters, load the pre-trained weights corresponding to the backbone network, select Adam as the optimizer, adjust the learning rate to 0.001, set the weight decay to 0.05, and set the Batch Size to 4.

[0112] (c) Use Focal Loss as the class loss function and GIoU, a hybrid loss function, as the bounding box loss function.

[0113] Train the improved UAV target detection network based on improved RT-DETR using the VisDrone training set.

[0114] Step S23: Record the training log, and the log content is the loss value and AP metric for each iteration during training.

[0115] Step S24: Complete the training to obtain the UAV target detection model based on improved RT-DETR.

[0116] Step S103: Evaluate the UAV target detection model based on improved RT-DETR using the VisDrone validation set, adjust the model training parameters according to the training log and evaluation metrics, and obtain the optimized model;

[0117] Step S104: Select the UAV target detection model with the best performance based on improved RT-DETR to perform target detection on the images collected from the UAV perspective, and obtain the target detection result image.

[0118] To better illustrate the effectiveness of the present invention, experiments are conducted on the VisDrone dataset and compared with the RT-DETR method.

[0119] The VisDrone dataset is a large-scale UAV vision dataset created by the Machine Learning and Data Mining Laboratory of Tianjin University (AISKYEYE team). It is constructed by collecting data through multi-model UAVs in diverse scenarios in 14 cities in China (covering megacities such as Beijing and Shanghai and small and medium-sized counties and towns), including transportation hubs, commercial streets, suburban roads, etc. VisDrone aims to promote the development of visual understanding tasks on UAV platforms.

[0120] Calculate AP 50：95 ,AP 50, AP 75 , AP S , AP M and AP L These precision metrics are used to evaluate the model. As shown in Table 1, the proposed UAV target detection algorithm based on improved RT-DETR has a lower number of parameters and is significantly superior to the RT-DETR algorithm in terms of detection accuracy.

[0121] Table 1 Comparison of experimental results

[0122] Method <![CDATA[AP 50：95 > <![CDATA[AP 50 > <![CDATA[AP 75 > <![CDATA[AP S > <![CDATA[AP M > <![CDATA[AP L > Number of parameters RT-DETR 35.5 56.7 36.8 27.6 46.2 44.6 19.9 The method of the present invention 37.8 60.5 39.1 29.6 49.5 50.7 17.4

[0123] As can be seen from Table 1, while the number of parameters is reduced by 12.6%, AP 50：95 increases by 2.3%. There is a significant improvement in the detection effect for targets of different scales. The detection accuracy of small targets, AP S increases by 2%, the accuracy of medium targets, AP M increases by 3.3%, and the accuracy of large targets, AP L increases by 6.1%.

[0124] The comparison of the detection results of the RT-DETR algorithm and the proposed UAV target detection algorithm based on improved RT-DETR on UAV images is as Figure 5 shown. As can be seen from the figure, for the target categories of "person" and "motorcycle" with dense and small target scales in the on-road vehicle flow, the RT-DETR algorithm has serious missed detection situations, while the proposed method has significantly fewer missed detections in the same area. In addition, the detection structure of the proposed method for occluded targets in the picture is also significantly better than that of RT-DETR.

[0125] Table 1 and Figure 5 fully illustrate that the proposed method significantly improves the detection accuracy for small targets, dense targets, and occluded targets in the UAV target detection scenario. To a certain extent, the proposed method solves the problems of small targets, dense targets, and target occlusion in UAV target detection.

[0126] Referring to Figure 2 shown, the embodiment of the present invention also provides a UAV target detection system based on improved RT-DETR, including:

[0127] An improved backbone network structure, including:

[0128] A feature preprocessing module for extracting shallow features at multiple scales;

[0129] 4 flexible fusion modules for processing shallow features;

[0130] An attention-based intra-scale feature interaction module for performing self-attention calculation;

[0131] An improved encoder structure, comprising:

[0132] A multi-scale perception feature pyramid network SAFPN for feature fusion;

[0133] A multi-feature fusion network MSFNet for extracting guiding features containing global information to guide the multi-scale perception feature pyramid network SAFPN to fuse feature layers of multiple scales;

[0134] A decoder structure, adopting the original RT-DETR structure, to output target position information and class information for the fused encoded features output by the improved encoder structure.

[0135] The present invention is not limited to the above embodiments. Based on the technical solutions disclosed in the present invention, those skilled in the art can make some substitutions and deformations to some of the technical features without creative labor according to the disclosed technical content, and these substitutions and deformations are all within the protection scope of the present invention.

Claims

1. A UAV target detection method based on improved RT-DETR, characterized in that: include: Construct a UAV target detection network based on the improved RT-DETR, including an improved backbone network structure, an improved encoder structure, and a decoder structure; The improved backbone network structure includes a feature preprocessing module, four flexible fusion modules, and an attention-based intra-scale feature interaction module, which respectively perform shallow feature extraction, shallow feature processing, and self-attention calculation on multiple scales; The improved encoder structure includes a multi-scale perceptual feature pyramid network SAFPN and a multi-feature fusion network MSFNet, which respectively perform feature fusion and encode and output the fused encoded features; The decoder structure adopts the original RT-DETR structure, and outputs the target position information and category information of the fused coding features output by the improved encoder structure; The constructed drone target detection network based on improved RT-DETR is trained using the VisDrone training set to obtain a drone target detection model based on improved RT-DETR. The UAV target detection model based on the improved RT-DETR was evaluated using the VisDrone validation set. The model training parameters were adjusted according to the training logs and evaluation indicators to obtain the optimized model. The optimization model with the best performance is selected to perform target detection on the images collected from the drone's perspective to obtain the target detection result image.

2. The unmanned aerial vehicle target detection method based on improved RT-DETR according to claim 1, characterized in that: Construct a drone target detection network based on improved RT-DETR, including: Construct an improved backbone network structure to extract features at multiple scales; Construct an improved encoder structure to fuse features and encode and output the fused encoded features; The improved encoder structure is output through the original RT-DETR decoder structure to output the fused encoding features and output the target position information and category information.

3. The unmanned aerial vehicle target detection method based on improved RT-DETR according to claim 2, characterized in that: Construct an improved backbone network structure to extract multiple scale features, including: Four groups of flexible fusion modules and an attention-based intra-scale feature interaction module are used to gradually process the shallow feature layer X0 output by the feature preprocessing module to obtain the small target feature layer P2, the medium target feature layer P3, the large target feature layer P4, and the feature layer F5 of the global context information.

4. The unmanned aerial vehicle target detection method based on improved RT-DETR according to claim 3 is characterized in that: The processing of the four groups of flexible fusion modules and the attention-based intra-scale feature interaction module is as follows: The shallow feature X0 output by the feature preprocessing module is input into the first flexible fusion module to obtain the output feature X3 of the first flexible fusion module. The output feature X3 is copied into two copies, one as the small target P2 feature layer, and the other as the input feature of the second fusion flexible module; By analogy, the previous output feature is input into the fourth flexible fusion module and undergoes the same calculation process to obtain the output feature X6. The output feature X6 is input into the attention-based intra-scale feature interaction module and the self-attention calculation between the features is performed to obtain the feature layer F5 containing the global context information.

5. The unmanned aerial vehicle target detection method based on improved RT-DETR according to claim 2, characterized in that: Construct an improved encoder structure to fuse features and encode and output the fused encoding features, including: The multi-feature fusion network MSFNet is used to guide the multi-scale perception feature pyramid network SAFPN to fuse the small target feature layer P2, the medium target feature layer P3, the large target feature layer P4 and the feature layer F5 of the global context information to obtain the fused encoding features; The multi-feature fusion network MSFNet is used to extract guiding features containing global information, which are then used to guide the multi-scale-aware feature pyramid network SAFPN to fuse feature layers of multiple scales.

6. The unmanned aerial vehicle target detection method based on improved RT-DETR according to claim 1, characterized in that: The structure of the multi-feature fusion network MSFNet module includes three 1×1 convolutions, a channel fusion module, a 3D convolution, a LeakyReLU activation function module, a 3D maximum pooling module, and a dimension compression module.

7. The unmanned aerial vehicle target detection method based on improved RT-DETR according to claim 1, characterized in that: The multi-scale-aware feature pyramid network SAFPN structure consists of five fusion modules and 6 convolutions.

8. The unmanned aerial vehicle target detection method based on improved RT-DETR according to claim 1, characterized in that: The processing process of the multi-scale perceived feature pyramid network SAFPN is as follows: Multi-feature fusion network MSFNet outputs multi-scale fusion features F final The backbone network extracts the small target feature layer P2, the medium target feature layer P3, the large target feature layer P4 and the feature layer F5 of the global context information as the input of the multi-scale perceptual feature pyramid network SAFPN; The feature layer F5 of the global context information is upsampled to the resolution of the large object feature layer P4 by bilinear interpolation, and the upsampled feature layer F5 of the global context information and the large object feature layer P4 are weighted fused to generate the first intermediate feature P4'; The first intermediate feature P4' is upsampled to the resolution of the medium target feature layer P3 by bilinear interpolation, and weighted fused with the medium target feature layer P3 to generate the second intermediate feature P3'; The small target feature layer P2 is downsampled to the resolution of the second intermediate feature P3', and weighted fused with the second intermediate feature P3' and the medium target feature layer P3 to generate the small-scale coding feature P3"; After a convolution, the small target fusion coding feature F is obtained S , small target fusion encoding feature F S Make two copies, one as output, and the other downsampled to the resolution of the first intermediate feature P4', and weighted fused with the first intermediate feature P4' and the large target feature layer P4 to generate the medium-scale coding feature P4"; The medium-scale coding feature P4" is convolved to obtain the medium-target fusion coding feature F M , the target fusion encoding feature F M Two copies are made, one as output, and the other is downsampled to the feature layer F5 resolution of the global context information, and combined with the feature layer F5 of the global context information and the multi-scale fusion feature F final Weighted fusion generates large-scale coding features P5"; The large-scale coding feature P5" is convolved to obtain the large target fusion coding feature F L , the multi-scale perceived feature pyramid network SAFPN outputs small target fusion encoding features F S , the target fusion encoding feature F M And large target fusion encoding feature F L Three fused encoding features.

9. The unmanned aerial vehicle target detection method based on improved RT-DETR according to claim 1, characterized in that: Use the VisDrone training set to train the drone target detection network based on the improved RT-DETR, including: Convert the annotation information in the VisDrone dataset into the YOLO format that is convenient for training; the annotation information of each target after conversion includes the target category and the position of the target in the image; Use the VisDrone training set to train an improved drone target detection network based on improved RT-DETR; Record the training log, which contains the loss value and AP index of each iteration during training, and obtain the drone target detection model based on the improved RT-DETR.

10. A drone target detection system based on improved RT-DETR according to any one of claims 1 to 9, characterized in that: include: Improved backbone network structure, including: Feature preprocessing module, used for shallow feature extraction at multiple scales; 4 flexible fusion modules for shallow feature processing; Attention-based intra-scale feature interaction module for self-attention calculation; Improved encoder structure, including: Multi-scale-aware feature pyramid network SAFPN, used for feature fusion; Multi-feature fusion network MSFNet is used to extract guiding features containing global information and guide the multi-scale perceptual feature pyramid network SAFPN to fuse feature layers of multiple scales; The decoder structure adopts the original RT-DETR structure and outputs the target position information and category information of the fused encoding features output by the improved encoder structure.

Images