An RT-DETR-based unmanned aerial vehicle aerial image detection method and system

Abstract

The application discloses a kind of RT-DETR-based unmanned aerial vehicle aeronautical image detection method and system.For the problem that small target in unmanned aerial vehicle aerial photography is easily disturbed by complex background and has large scale variation, the application first carries out data enhancement preprocessing on high-resolution aerial photograph image;Second, a feature extraction network is constructed by fusing CSP and convolution attention mechanism to enhance the discriminability of small target features;In the feature fusion stage, introduce P2 high-resolution detection head, lightweight adaptive feature fusion module and partial convolution CSP structure to realize efficient interaction of multi-scale features;In addition, a position relationship detection head is designed to explicitly model the target space relationship, and an improved generalized center distance loss function incorporating aspect ratio constraints is used to accelerate model convergence.Finally, the model is deployed on an edge visualization system based on Streamlit.The application effectively reduces the missed detection and false detection in dense occlusion scenarios, achieving an efficient balance between detection accuracy and real-time performance.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) target detection, and to a method and system for UAV aerial image detection based on RT-DETR. Background Technology

[0002] In recent years, drones have been widely used in urban security, traffic monitoring, disaster assessment, and infrastructure inspection due to their small size, high maneuverability, and flexible deployment. Typical drone aerial imagery datasets (such as VisDrone) often feature wide coverage, complex scenes, and targets primarily consisting of small-scale instances such as pedestrians and vehicles, exhibiting characteristics such as dense target distribution, dramatic scale variations, and significant background interference. These factors pose significant challenges to the accuracy and robustness of target detection in drone aerial images. With the rapid development of deep learning technology, convolutional neural network-based target detection methods have made significant progress in complex backgrounds, multi-scale scenarios, and low-resolution scenes, promoting the widespread application of computer vision technology in drone image analysis. Representative target detection frameworks, such as the YOLO series, Faster R-CNN, and RT-DETR, have demonstrated strong adaptability in drone scenarios, providing effective technical support for drone aerial target detection. Existing deep learning target detection methods can be mainly divided into single-stage and two-stage methods. Single-stage methods, due to their compact structure and fast inference speed, are widely used in drone platforms with limited computing resources. To address the small target characteristics of UAVs, Li et al. reduced the number of model parameters to adapt to edge devices by sharing parameters; Zhang et al. proposed UAV-DETR, which utilizes multi-scale feature fusion and semantic alignment mechanisms to improve detection performance while ensuring real-time performance; they designed an efficient high-frequency hybrid encoder for small targets and a cross-space-frequency domain hybrid (CSFH) feature extraction network, achieving an efficient balance between accuracy and parameter reduction. Two-stage methods, through candidate region generation and refined regression, still have advantages in small target detection tasks. For example, Faster R-CNN and Cascade R-CNN combined with feature pyramid structures show strong robustness under complex background conditions. In addition, research on UAV platforms also focuses on enhancing scale adaptability through deformable convolutions and reducing model complexity by combining pruning and knowledge distillation strategies to meet the needs of embedded deployment.

[0003] While existing methods have made some progress in UAV target detection, small target detection in aerial images still faces numerous challenges. On the one hand, limited by imaging height and shooting angle, small targets typically occupy only a few pixels in the image, and their texture and edge information are easily weakened or even lost during multiple downsampling processes, resulting in insufficient feature representation. On the other hand, UAV aerial scenes often feature complex backgrounds containing a large amount of high-frequency texture information such as roads and buildings, making small target features easily submerged by background noise, increasing the difficulty of target differentiation. Furthermore, aerial images often show dense target distribution and widespread occlusion, while existing methods have limited ability to model the relative spatial relationships between targets, leading to missed or false detections in crowded scenes. In addition, traditional IoU-based regression losses struggle to provide effective gradients when the predicted bounding box and the ground truth bounding box do not overlap, easily causing unstable convergence in the early stages of training and affecting target localization accuracy. More importantly, while introducing high-resolution features to improve small target detection performance, model computational complexity and inference overhead often increase significantly. Finding a balance between detection accuracy and computational efficiency remains a pressing issue. Summary of the Invention

[0004] This invention aims to address the problems of low detection speed, difficulty in recognizing degraded features, and complex computational resources in existing UAV aerial image detection algorithms. It proposes a real-time small target detection method based on RT-DETR. This method improves the network architecture, making it easier to deploy on edge devices for real-time detection. A convolutional feature interaction module combining a CSP structure and a convolutional attention mechanism is designed to enhance the discriminative power and localization robustness of small target features with lower computational overhead. A multi-scale feature interaction structure for small target detection is constructed, and by introducing a P2 detection head and combining it with lightweight feature fusion modules (PCRSP and LAFM), the advantages of high-resolution features in representing small-scale targets are fully utilized, while reducing overall computational complexity. A detection head based on positional relationship encoding is introduced to explicitly model the relative spatial relationships between targets during the decoding stage, improving the ability to model target relationships in complex scenes. The generalized center distance loss function is improved by introducing an aspect ratio constraint term to accelerate model convergence and improve localization accuracy.

[0005] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0006] The UAV aerial image detection method based on RT-DETR includes the following steps:

[0007] S1. Collect various scene images captured by the drone, and mount the mapping camera and high-definition camera on the drone to collect images;

[0008] S2. Use deep learning technology to remove blurry, incomplete or duplicate images, retain diverse data under different lighting conditions, angles and scenes, crop large images into small pieces, and finally enhance the images.

[0009] S3. Input the target image into the CCFB extraction network of the aerial target detection model, perform layer-by-layer downsampling and feature enhancement on the input image, fully mine multi-scale high-frequency information, and generate P2 to P5 multi-scale feature maps in sequence.

[0010] S4. A fusion network using an aerial small target detection model is input into an efficient small target high-frequency hybrid encoder. High-level features are modeled through an attention-based internal feature interaction module (AIFI) to obtain semantically enhanced feature representations, which are then fused layer by layer with high-resolution features in MSFIS to fully utilize the complementary information between multi-scale features.

[0011] S5. Output the feature map after cross-scale interaction to the positional relationship detection head to obtain the output result;

[0012] S6. Train the UAV aerial photography small target detection network and test it;

[0013] S7. Detect target images based on a trained UAV aerial small target detection network and obtain detection results;

[0014] Optionally, S3 specifically involves modifying the backbone of the original RT-DETR to reduce computational complexity while enhancing the feature space modeling capability. This method borrows from the C2f module of YOLOv8 to construct a feature extraction network CCFB that combines the CSP structure with the convolutional attention mechanism, thereby enhancing the discrimination capability of small target features.

[0015] Optionally, S4 specifically addresses the challenge of balancing feature representation capability and computational efficiency in efficient high-frequency hybrid encoders for small target detection tasks. It introduces a P2 high-resolution detection head during the feature fusion stage and designs a lightweight adaptive feature fusion module and a partially convolutional CSP structure to achieve efficient interaction of multi-scale features, effectively improving the representation capability of small-scale targets while reducing computational complexity.

[0016] Optional, S5 specifically includes:

[0017] S51. In order to address the shortcomings of existing detection methods for UAV aerial images in modeling the relative spatial relationships between targets during the decoding stage, a positional relationship detection head is introduced to explicitly model the spatial relationships between targets in complex scenes, thereby enhancing the target relationship modeling capability during the decoding stage.

[0018] S52. To improve the target's positioning accuracy and accelerate model convergence, the generalized center distance loss function is improved by introducing an aspect ratio constraint term to speed up model convergence and enhance positioning accuracy.

[0019] Optionally, S7 specifically involves: using the Streamlit web application development framework to build an object detection visualization system, deploying the trained RT-DETR object detection model on an edge computing device, wherein the object detection model includes multiple model configurations with different accuracies, and supports free selection of model selection and object detection confidence threshold. Attached Figure Description

[0020] Figure 1 This is an overall flowchart of the present invention.

[0021] Figure 2 This is a model architecture for UAV aerial image detection based on RT-DETR.

[0022] Figure 3 This is the overall architecture of CCFB.

[0023] Figure 4 This describes the structure of the Lightweight Adaptive Feature Fusion Module (LFAM).

[0024] Figure 5 This is a cross-stage partial network module (PCRSP) structure with partially convolutional and reparameterizable structures.

[0025] Figure 6 It is a PConv_Rep structure.

[0026] Figure 7 This is the structure of a Position Head.

[0027] Figure 8 The figure shows the experimental results compared with mainstream detectors.

[0028] Figure 9 This is a visualization of the results based on the improved RTDETR model.

[0029] Figure 10 This document presents the interface diagram for a target detection visualization system built using the Streamlit web application development framework. Detailed Implementation This invention discloses a deep learning-based method for detecting aerial images of unmanned aerial vehicles (UAVs), such as... Figure 1 As shown, it includes the following steps:

[0030] S1. Collect various scene images captured by the drone, and mount the mapping camera and high-definition camera on the drone to collect images;

[0031] S2. Use deep learning technology to remove blurry, incomplete or duplicate images, retain diverse data under different lighting conditions, angles and scenes, crop large images into small pieces, and finally enhance the images.

[0032] S3. Input the target image into the extraction network of the aerial target detection model, perform layer-by-layer downsampling and feature enhancement on the input image, fully mine multi-scale high-frequency information, and generate multi-scale feature maps P2 to P5 in sequence.

[0033] S4. A fusion network using an aerial small target detection model is input into an efficient small target high-frequency hybrid encoder. High-level features are modeled through an attention-based internal feature interaction module (AIFI) to obtain semantically enhanced feature representations, which are then fused layer by layer with high-resolution features in MSFIS to fully utilize the complementary information between multi-scale features.

[0034] S5. Output the feature map after cross-scale interaction to the positional relationship detection head to obtain the output result;

[0035] S6. Train the UAV aerial photography small target detection network and test it;

[0036] S7. Detect target images based on a trained UAV aerial small target detection network and obtain detection results;

[0037] Furthermore, in S1, image acquisition involves a drone equipped with a high-resolution multispectral camera and a high-definition video camera. The flight altitude is set at 10-20 meters, the flight speed is 5 m / s, and the coverage area of ​​farmland is ≥20 hectares. The acquisition environment includes different lighting conditions (light intensity ≤30,000 Lux), multiple angles (pitch angle 30°-90°), and different occlusion conditions to ensure data diversity.

[0038] Furthermore, S2 specifically involves using a pre-trained ResNet model to filter out blurry (image sharpness score < 0.8) or duplicate (SSIM similarity > 95%) images;

[0039] The original high-resolution image (3840×2160) was cropped into 640×640 standard sub-images using the sliding window technique, with a window step size of 320 pixels. Each image generated 53 sub-images (including 5% of targetless samples for negative sample training). Data augmentation strategies included: Mosaic blending (4-image stitching), random rotation (±30°), tone shift (ΔHue=±0.1), and Gaussian noise (σ=0.01).

[0040] In this invention, the feature extraction backbone network specifically comprises the following stages: During the feature extraction phase, the network alternately stacks convolutional layers and CCFB modules to perform layer-by-layer downsampling and feature enhancement on the input image, fully mining multi-scale high-frequency information and sequentially generating multi-scale feature maps from P2 to P5. Specifically, layer P2 retains high spatial resolution to characterize the fine-grained structural features of small targets, while layers P3–P5 progressively enhance high-level semantic expression capabilities.

[0041] In this invention, CCFB specifically refers to a feature interaction module based on CSP structure and convolutional attention mechanism, which is proposed by deeply integrating CFBLock with CSP structure. This module, based on local convolution operations, introduces a direction-sensitive convolutional attention structure to achieve efficient modeling of spatial relationships in feature maps, making it suitable for real-time object detection tasks under resource constraints.

[0042] In this invention example, CFBLock specifically consists of three parts: a channel alignment layer, a convolutional attention submodule, and a feedforward network. Firstly, through... Convolution adjusts the number of input feature channels to ensure consistency in feature dimensions. Subsequently, a convolutional attention mechanism is introduced to model spatial interactions of the features. This attention submodule uses learnable convolutional kernel parameters to aggregate features along both the horizontal and vertical directions, effectively capturing the spatial dependencies of the target in different directions. To enhance expressive power, features are grouped in a multi-head structure, and adaptive allocation of spatial weights is achieved through normalization and Softmax operations. To better align with the semantic modeling approach of the Transformer, CFBLock extends learnable vectors to learnable kernels. This transforms the similarity calculation between pixels and learnable vectors into the similarity calculation between pixel blocks with learnable kernels. Convolutional operations with learnable kernels retain more local spatial information to some extent. The operation of convolutional attention can be summarized as follows:

[0043] in, , , Let C represent the input image, H represent the learnable query, and W represent the key. C, H, and W represent the channels, height, and width of the feature map, respectively. N represents the number of learnable parameters, and K represents the kernel size of the learnable parameters. Indicates grouped double normalization, in Apply softmax for n-dimensionality and grouped L2 Norm for n-dimensionality. This indicates a convolution operation.

[0044] After completing the convolutional attention modeling, the module further introduces a feedforward network to perform nonlinear transformations on the features. This feedforward network consists of expanded point convolutions, depthwise separable 3×3 convolutions, and squeezed point convolutions in sequence, combined with the GELU activation function to enhance nonlinear expressive power. Simultaneously, a random deactivation operation is introduced in the intermediate layers to alleviate overfitting and improve the model's generalization performance. Compared to the feedforward network used in traditional Transformers, this MLP structure effectively expands the feature receptive field and improves local context modeling capabilities while maintaining lower computational overhead.

[0045] In the invention examples, the feature fusion stage specifically addresses the challenge of balancing feature representation capability and computational efficiency in efficient high-frequency hybrid encoders for small target detection tasks. This paper proposes a Multi-Scale Feature Interaction Structure (MSFIS) to effectively reduce computational overhead while maintaining detection performance. This structure introduces a high-resolution detection branch and combines lightweight feature fusion with partial convolutional modeling strategies to fully extract fine-grained information about small-scale targets in UAV aerial images.

[0046] The multi-scale feature interaction structure is specifically divided into two parts: a lightweight adaptive feature fusion module (LAFM) and a re-parameterized cross-stage partial network with partial convolution (PCRSP).

[0047] The upsampled or downsampled feature map is input into the LAFM module, which consists of two parallel branches, each corresponding to a feature map from the previous stage. and the feature layer aligned with its spatial scale This structure aims to establish an efficient information exchange mechanism between features from different sources. In LAFM, the features of the two branches are first summed element-wise, and global contextual information is extracted through global average pooling. Subsequently, the features are input into a lightweight mapping network composed of a multilayer perceptron (MLP), and normalized modulation weights are generated through Softmax. This reflects the relative importance of different branch features along the channel dimension. Finally, LAFM completes feature fusion through channel-wise weighted summation, and its output is expressed as:

[0048] in This represents channel-wise element-wise multiplication. This modulation fusion strategy can adaptively adjust the role of different branches in the fusion process based on the global response of the input features. Compared to the Concat feature fusion method, LAFM maintains a constant channel dimension after fusion, avoiding redundant computation caused by channel expansion, and significantly reducing the computational cost of subsequent convolution operations in high-resolution feature layers.

[0049] The core idea of ​​PCRSP is to partition the input features along the channel dimension using partial convolution (PConv), performing 3×3 convolutions on some channels to model local spatial relationships, while maintaining the identity mapping for the remaining channels. This effectively reduces redundant computation while preserving the original feature information. To enhance the feature representation capability during training and maintain efficiency during inference, this paper replaces the convolutional branch with RepConv, forming the PConv_Rep module. RepConv employs a multi-branch structure during training to improve non-linear expressive power, and can be equivalently folded into a single convolutional kernel during inference, strengthening the local feature modeling capability without introducing additional inference latency. This feature is particularly advantageous for targets with blurred boundaries and small scales in UAV aerial photography scenarios.

[0050] Building upon PConv_Rep, a PCRep Bottleneck is further constructed as the basic feature unit. The main branch consists of two levels of PConv_Rep for lightweight spatial feature modeling; the side branch uses 1×1 convolutions to match the output channel dimension and forms residual connections with the main branch to stabilize gradient propagation and mitigate feature degradation. Further incorporating the One-Shot Aggregation concept, PCRSP adopts a two-branch structure of CSP: one branch enhances features by stacking multiple PCRep Bottlenecks, while the other branch maintains lightweight direct connections to preserve original information. After concatenating the two branches along the channel dimension, feature fusion is achieved through 1×1 convolutions, as shown below:

[0051] in Indicates a bottleneck stacking branch. This indicates a bypass branch. Since spatial modeling only applies to a portion of the channels, PCRSP can significantly reduce computational complexity while maintaining the network's feature representation capabilities through residual structures and the CSP mechanism, thus achieving an effective balance between computational efficiency and detection performance in UAV aerial target detection tasks.

[0052] This paper introduces a Position Head, which explicitly models the geometric relationships between targets based on a position relationship encoding mechanism. Inspired by the iterative bounding box refinement concept in Deformable DETR, this design achieves joint optimization of target position and category by updating the query embedding and its corresponding reference bounding box layer by layer in a multi-layer decoder. The Position Head consists of multiple identical decoder layers stacked sequentially, with each layer further refining the detection results based on the prediction results of the previous layer. In the... During layer decoding, the relative positional relationship between queries is first constructed based on the bounding boxes predicted by the current layer and the previous layer. Each bounding box is represented by its center coordinates and width and height parameters. By encoding the center offset, scale ratio, overlap, and relative direction between targets, relative positional relationship features are formed. The representation method is processed through scale normalization, which effectively improves the stability of relationship modeling between targets at different scales.

[0053] Based on this, all bounding boxes corresponding to the queries are paired to construct a relative positional relationship tensor, which is then encoded using a lightweight multilayer perceptron composed of Linear–GELU–Linear to generate a positional relationship weight matrix. This weight matrix is ​​introduced as an attention bias term into the multi-head self-attention calculation process, enabling attention allocation to explicitly incorporate the relative spatial layout information between targets while considering feature similarity.

[0054] Subsequently, the updated query embedding interacts with the multi-scale features output by the encoder through a multi-head deformable cross-attention mechanism, adaptively sampling key feature points related to the target location under the guidance of the reference bounding box. After nonlinear transformation by the feedforward network, the query representation of the next layer is obtained and further used for bounding box regression and class prediction. Compared with the standard self-attention mechanism that relies solely on absolute position encoding and feature similarity, the proposed positional relationship detection head can directly introduce the geometric priors between targets into the decoding process, enabling the model to have stronger target relationship modeling capabilities in complex scenes. This design is particularly suitable for small target detection tasks with dense targets and significant scale variations in UAV aerial images.

[0055] To improve the stability and accuracy of small target localization in aerial photography scenarios, this paper introduces a Generalized Center Distance Loss (GCD Loss) function to measure the geometric consistency between the predicted bounding box and the ground truth annotation. Unlike traditional IoU-based regression loss, GCD Loss models the target's center position and scale relationship, providing continuous and effective gradient signals even when the predicted and ground truth boxes do not overlap. This helps alleviate the optimization instability caused by insufficient matching in the early stages of training.

[0056] Specifically, the center coordinate difference between the predicted and ground truth bounding boxes is calculated and normalized using the width and height of both boxes, thereby enhancing sensitivity to center offset of small targets while maintaining scale invariance. Simultaneously, a width and height scale difference term is introduced to reduce localization errors caused by scale mismatch by constraining the relative changes between the predicted and ground truth scales. The center distance term and scale difference term are then fused based on bidirectional normalization to form a symmetrical geometric distance metric, enabling the loss function to simultaneously characterize both positional offset and scale consistency.

[0057] Building upon this foundation, to further enhance the convergence and stability of the regression process, this paper introduces an aspect ratio constraint term in the geometric distance modeling to characterize the shape difference between the predicted and ground truth bounding boxes. This constraint effectively suppresses regression oscillations caused by aspect ratio imbalances, especially in small object detection scenarios, helping to accelerate model convergence and improve localization accuracy. To enhance numerical stability and gradient representation capabilities, an exponential mapping is used to convert the comprehensive geometric distance into a similarity metric, defined as:

[0058] in, This represents the generalized geometric distance that integrates center offset, scale difference, and shape constraints. In summary, GCD Loss, while maintaining global geometric modeling capabilities, exhibits stronger robustness to targets with small size and drastic scale changes in aerial photography scenarios, thus accelerating network convergence and positioning accuracy.

[0059] Compared with RT-DETR, the proposed method improves mAP50 by 5.5% and mAP50:95 by 3.7% on the VisDrone2019 dataset with approximately 6.1M fewer parameters than the baseline. Compared with other mainstream algorithms, it achieves a higher balance between accuracy and efficiency.

[0060] The present invention also provides an aerial image detection system for unmanned aerial vehicles based on RT-DETR, the system comprising: Image acquisition module: used to collect aerial images of various scenes using the mapping camera and high-definition camera mounted on the drone; Data preprocessing module: used to clean the acquired images, remove blurry, incomplete or duplicate images, and perform image cropping and data enhancement; Multi-scale feature extraction module: Includes CCFB feature extraction network that combines CSP structure and convolutional attention mechanism, used to downsample and enhance the features of the input image layer by layer to generate multi-scale feature maps; Feature fusion and detection module: It has a built-in multi-scale feature interaction structure (MSFIS) and positional relationship detection head, which are used to perform cross-scale fusion of multi-scale features and explicitly model the spatial relationship between targets, and finally output the target detection results and positioning information.

Claims

1. A method for detecting aerial images of unmanned aerial vehicles based on RT-DETR, characterized in that, The images captured by the drone are processed, and an improved deep learning model is used for target detection. The steps are as follows: S1. Collect various scene images captured by the drone, and mount the mapping camera and high-definition camera on the drone to collect images; S2. Use deep learning technology to remove blurry, incomplete or duplicate images, retain diverse data under different lighting conditions, angles and scenes, crop large images into small pieces, and finally enhance the images. S3. Input the target image into the extraction network of the aerial target detection model, perform layer-by-layer downsampling and feature enhancement on the input image, fully mine multi-scale high-frequency information, and generate multi-scale feature maps P2 to P5 in sequence. S4. A fusion network using an aerial small target detection model is input into an efficient small target high-frequency hybrid encoder. High-level features are modeled through an attention-based internal feature interaction module (AIFI) to obtain semantically enhanced feature representations, which are then fused layer by layer with high-resolution features in MSFIS to fully utilize the complementary information between multi-scale features. S5. Output the feature map after cross-scale interaction to the positional relationship detection head to obtain the output result; S6. Train the UAV aerial photography small target detection network and test it; S7. Detect targets based on the trained UAV aerial small target detection network and obtain the detection results.

2. The UAV aerial image detection method based on RT-DETR according to claim 1, characterized in that: In step S1, images are acquired at a preset height using the surveying camera and the high-definition camera.

3. The UAV aerial image detection method based on RT-DETR according to claim 1, characterized in that: In step S2, when processing the acquired data images, the original high-resolution image is cropped into a standard sub-image of a preset size, and the sub-images that retain the complete image content are selected as training samples.

4. The UAV aerial image detection method based on RT-DETR according to claim 1, characterized in that: In step S3, the feature extraction network is constructed from a CSP structure and a convolutional attention mechanism.

5. The UAV aerial image detection method based on RT-DETR according to claim 1, characterized in that: In step S4, a high-resolution detection head is introduced in the feature fusion stage, and a lightweight adaptive feature fusion module is used to interact with a partial convolutional CSP structure for multi-scale features.

6. The UAV aerial image detection method based on RT-DETR according to claim 1, characterized in that: In step S5, the positional relationship detection head is used to explicitly model the spatial relationship between targets in complex scenes.

7. The UAV aerial image detection method based on RT-DETR according to claim 1, characterized in that: In step S6, during the training process, an improved generalized center distance loss function is used, which introduces an aspect ratio constraint term.

8. A UAV aerial image detection system based on RT-DETR, characterized in that, include: The system comprises a drone acquisition terminal, a wireless transmission link, and a handheld edge computing terminal. The drone acquisition terminal is responsible for capturing aerial data during low-altitude flight and transmitting it back in real time via the wireless transmission link. The handheld edge computing terminal has a built-in RT-DETR target detection model trained by any one of the methods described in claims 1-7, supporting dynamic switching of multi-precision models and custom adjustment of confidence thresholds. The system is compatible with input sources such as images, videos, and real-time camera streams. The edge computing terminal performs real-time target detection and annotation, and finally outputs the detection results.

Images