Lightweight real-time unmanned aerial vehicle target detection system and method

Abstract

The invention discloses a lightweight real-time unmanned aerial vehicle target detection system and method. The method comprises the steps that collaborative design of a multi-receptive field feature interaction backbone network, a channel perception fusion module and a multi-branch feature re-extraction module is adopted; the backbone network adopts a branch structure of global context modeling and local multi-scale feature extraction; the fusion module adaptively integrates cross-resolution features through a dynamic attention mechanism; and the re-extraction module enhances feature discrimination based on channel expansion and residual interaction. According to the method, multi-scale target robust detection of the unmanned aerial vehicle in a large-span flight height scene under strict resource limitation is realized, the complex environment adaptability is remarkably improved, and the end-to-end real-time processing capability is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and unmanned aerial vehicle (UAV) application technology, and in particular to a lightweight and real-time UAV target detection system and method. Background Art

[0002] With the widespread application of drones in security inspections, disaster monitoring, agricultural plant protection, and other fields, real-time target detection technology has become the key to improving drones' autonomous operation capabilities. Current deep learning-based drone target detection solutions mainly face the following technical bottlenecks:

[0003] 1. Imbalance between lightweight and detection accuracy

[0004] Although the current mainstream lightweight real-time models (such as the YOLO series and its variants) perform outstandingly in terms of real-time performance and lightweightness, they often sacrifice detection accuracy. In addition, they are mainly designed for natural images from a ground perspective. When directly migrated to drone aerial photography scenes, the detection performance will significantly degrade due to the drastic changes in target scale caused by the drone's flight altitude.

[0005] 2. Insufficient adaptability to multi-scale targets

[0006] Drones' flight altitudes vary dynamically (from low to high altitude), causing the scale of the same target in an image to vary by more than tenfold. Existing models, which often rely on fixed-scale convolution kernels or static feature fusion mechanisms, struggle to effectively detect targets ranging from tiny ones (such as distant pedestrians) to large ones (such as vehicles). This is especially true in high-altitude imagery, where the rate of missed detection of small targets increases significantly.

[0007] 3. Lack of robustness in complex environments

[0008] Drone photography often involves interference from sudden changes in lighting, motion blur, and background clutter. Traditional convolution operations have limited ability to model global context and are prone to false detection in occluded scenes. While improved self-attention-based models can enhance global dependencies, their quadratic computational complexity exceeds the real-time processing capabilities of the airborne platform and is sensitive to motion blur.

[0009] To alleviate the above problems, the research community has proposed a variety of solutions:

[0010] Model compression techniques (such as pruning and quantization): reduce computational overhead but with significant loss of accuracy;

[0011] Dynamic receptive field mechanism (such as deformable convolution): improves scale adaptability but introduces additional computational burden;

[0012] Lightweight attention module: balances global modeling and computational efficiency, but is not robust enough to complex noise.

[0013] However, the core contradiction remains unresolved: how to simultaneously achieve high-precision detection, multi-scale adaptability, and environmental robustness under strictly limited onboard computing resources has become a technical pain point that restricts the intelligence of drones. Summary of the Invention

[0014] In view of the shortcomings of the existing technology, the present invention provides a lightweight and real-time drone target detection system and method.

[0015] In order to achieve the above object of the invention, the technical solution adopted by the present invention is as follows:

[0016] A lightweight and real-time drone target detection system, including:

[0017] An onboard image acquisition module, used to obtain real-time images of the scene to be detected;

[0018] The processing unit, deployed on the drone edge computing platform, includes:

[0019] Multi-receptive field feature interaction backbone network (MRFFIBackbone), used to extract multi-scale features of images;

[0020] Channel-aware fusion module (CAFM) dynamically weights and fuses cross-level features output by the backbone network;

[0021] Multi-branch feature re-extraction module (MFEM), which performs secondary refinement of fusion features;

[0022] Detection head, which outputs target detection results based on refined features;

[0023] The communication module is used to transmit the detection results to the ground control terminal.

[0024] Furthermore, the MRFFI Backbone is composed of multiple levels of MRFFI Blocks, each of which includes:

[0025] Channel segmentation unit: divides the input features into global branch, local branch and residual branch according to the preset ratio;

[0026] Global branch: connects the Mamba module and the Haar wavelet transform layer in sequence to capture long-range dependencies and enhance details;

[0027] Local branch: contains multiple sets of parallel depth-wise separable convolutional layers with kernels k=3, 5, and 7 to extract multi-scale local features;

[0028] Residual branch: transfers original features through identity mapping;

[0029] Feature splicing unit: fuses the output features of the three branches.

[0030] Furthermore, the Mamba module implements sequence modeling based on the structured state space model (SS2D) to obtain long-range dependencies.

[0031] Haar wavelet transform is performed after the output of the Mamba module to enhance the detail features through high-frequency components.

[0032] Furthermore, the CAFM module includes:

[0033] Channel alignment layer: unifies the input multi-level features (P3 and P4 layers) to the same channel dimension;

[0034] Dynamic attention generation unit: Generates cross-level weights through global average pooling (GAP), multi-layer perceptron (MLP) and softmax function;

[0035] Weighted fusion layer: performs weighted summation of input features according to the weights.

[0036] Furthermore, the MFEM module includes:

[0037] Channel expansion layer: expands the input channel to n times (n ≥ 3) through 1×1 convolution;

[0038] Multi-branch processing unit:

[0039] The first branch directly connects the residual output;

[0040] The remaining branches are sequentially connected to the inverted bottleneck structure and feature superimposed with the preceding branches;

[0041] Channel compression layer: compresses the multi-branch concatenation features to the target dimension.

[0042] Furthermore, the inverted bottleneck structure comprises:

[0043] Depthwise separable convolutional layers;

[0044] Linear activation function (ReLU6) and channel scaling layer.

[0045] Furthermore, the processing unit satisfies:

[0046] Parameter count ≤ 4.3M, computational complexity ≤ 9.1GFlops;

[0047] When deployed on the drone side, the real-time frame rate is ≥563FPS (input resolution 640×640).

[0048] The present invention also discloses a target detection method based on the above system, comprising:

[0049] S1: collects real-time images through the onboard camera;

[0050] S2: Executed by the processing unit:

[0051] a) Stem module extracts initial features;

[0052] b), MRFFIBackbone generates multi-scale features;

[0053] c), CAFM integrates multi-scale features;

[0054] d) MFEM re-extracts fusion features;

[0055] e) The detection head outputs the target position and category;

[0056] S3: Send the detection results back to the ground station through the communication module.

[0057] The present invention also discloses a computer-readable storage medium storing program instructions, characterized in that the instructions implement the above-mentioned target detection method when executed by an onboard processor of a drone.

[0058] The present invention also discloses an unmanned aerial vehicle (UAV), comprising an airframe, a flight control system, an image acquisition device, and a target detection system; a processing unit of the target detection system is integrated into an onboard embedded platform (such as the NVIDIA Jetson series) of the UAV.

[0059] Compared with the prior art, the advantages of the present invention are:

[0060] 1. Collaborative breakthrough in lightweight architecture and high-precision detection

[0061] Through the three-branch channel grouping strategy of the multi-receptive field feature interaction backbone network (MRFFIBackbone), computing resources are dynamically allocated to global dependency modeling, local feature extraction and basic information transmission, significantly reducing redundant calculations; combined with the cross-channel dynamic compression mechanism of the MFEM module, while ensuring the multi-scale feature expression capability, the model parameter quantity and computational complexity are simultaneously compressed to the range that the embedded platform can bear, completely resolving the contradiction of "light weight means inaccuracy" of traditional models, and realizing real-time high-precision target detection in airborne environments.

[0062] 2. Improved adaptability to large-scale changes

[0063] To address the problem of drastic changes in target scale caused by drones flying at an altitude of 20-200 meters, the innovatively designed MRFFI module covers the complete information chain from pixel-level details to semantic-level context through a differentiated feature processing mechanism of global branches (spatial long-distance dependency capture + wavelet detail enhancement) and local branches (multi-core depthwise separable convolution); the CAFM module further adaptively fuses shallow high-resolution features with deep strong semantic features through dynamic cross-layer attention weighting, enabling the model to maintain stable detection capabilities for small targets at high altitude, large targets at low altitude, and targets with gradually changing sizes, significantly reducing the risk of missed detection.

[0064] 3. Strong robustness in complex interference environments

[0065] To address typical drone interference such as sudden changes in illumination, motion blur, and background noise, the wavelet transform enhancement branch of the MRFFI module effectively separates high-frequency details from low-frequency structural features, suppressing feature degradation caused by uneven illumination. The dynamic noise suppression mechanism of the CAFM module weakens the response in interfering areas through weight redistribution. The multi-branch residual interaction structure of the MFEM module enhances feature reuse and compensation for occluded targets. These three components work together to form a "global-local-detail" triple anti-interference defense line, ensuring the model's detection stability in extreme scenarios.

[0066] 4. Engineering-friendly end-to-end deployment

[0067] The entire architecture utilizes a hardware-aware design: MRFFI's channel grouping and MFEM's inverted bottleneck structure are compatible with TensorRT operator acceleration; CAFM's dynamic attention mechanism avoids high-dimensional matrix operations; and the overall computational flow (Stem → MRFFI → CAFM → MFEM → Decoder & Head) is linearly optimized to maximize memory access efficiency. This enables millisecond-level inference on airborne platforms like Jetson without complex compression, providing a plug-and-play deployment solution for critical applications such as dynamic target tracking and real-time obstacle avoidance in drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is an overall flow chart of the drone target detection model according to an embodiment of the present invention;

[0069] Figure 2 This is a schematic diagram of the structure of the MRFFI Block in an embodiment of the present invention;

[0070] Figure 3 2 is a schematic diagram of the structure of a channel-aware fusion module (CAFM) according to an embodiment of the present invention;

[0071] Figure 4 1 is a schematic diagram of the structure of a multi-branch feature extraction module (MFEM) according to an embodiment of the present invention;

[0072] Figure 5 This is a feature flow chart of the drone target detection model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0074] like Figure 1 As shown in the figure, the overall flow chart of the UAV target detection model proposed in the present invention is as follows. The network architecture designed by the present invention first completes the initial feature extraction of the input image through the stem module, and then uses MRFFIBackBone to realize the hierarchical extraction of multi-scale features. The P3 and P4 layer features are selected and multi-scale feature fusion and multi-branch feature extraction are performed through the channel perception fusion module CAFM. The secondary feature refinement is completed by the module MFEM, and finally the target detection task is completed through the Decoder and Head modules.

[0075] like Figure 2 As shown in the figure, MRFFI Block is the basic component unit of the model backbone network. In the MRFFI Block, its MRFFI core unit integrates a global, local, and residual three-branch architecture to better realize the interaction of multi-scale features and effectively reduce the computational complexity through the preset ratio segmentation strategy of the channel dimension. Specifically, by dividing the input features along the channel dimension into a global perception branch (using the Mamba module enhanced by long-distance wavelet transform to capture contextual dependencies), a multi-scale local perception branch (based on multi-kernel deep separable convolution to extract different receptive field features) and a residual mapping branch (eliminating high-dimensional redundancy), while ensuring multi-scale feature extraction, the computational efficiency is significantly improved.

[0076] like Figure 3 As shown in the figure, the CAFM module is used for feature fusion. The module first transforms the different input features into the same dimension. After the splicing operation, it generates dynamic attention weights through global average pooling (GAP), multi-layer perceptron (MLP) and SoftMax function in sequence. Guided by this weight, it performs weighted fusion of features at different layers, improves the ability to integrate multi-scale information, and enhances the detection performance of small targets and complex backgrounds.

[0077] like Figure 4As shown in the figure, the MFEM module is used to perform secondary refinement of cross-layer fusion features. The input features are first subjected to a 1*1*n convolution operation, expanding the number of channels to n times the original number of channels. The expanded features are then divided into n branches. The first branch directly uses a residual structure for feature transfer, while the remaining branches are first processed through an inverted bottleneck structure and then subjected to a residual connection operation. Finally, the n branch features that have undergone the above processing are spliced ​​together to complete the fusion and integration of multi-branch features. This module improves multi-scale expression and fusion capabilities through the interaction of multi-receptive field features and reduces redundancy through dynamic cross-channel fusion.

[0078] 1. The lightweight real-time drone target detection system of this embodiment is deployed on the NVIDIA Jetson AGX Orin platform and includes the following components:

[0079] Image input: The onboard camera (SONY IMX577) captures 1920×1080 resolution video stream and scales it to 640×640 input

[0080] like Figure 5 As shown, the processing flow is as follows:

[0081] ①The input image first passes through the stem layer to extract preliminary features.

[0082] ② Input the initially extracted features into the backbone network

[0083] MRFFIBackbone backbone network: adopts a four-stage backbone, each stage is an MRFFIBlock.

[0084] Detailed processing flow of the MRFFI module:

[0085] Input feature segmentation (channel dimension grouping, divided into three branches)

[0086] Global branches: 25%, extracting long-distance dependencies through Mamba (SSM (S6))

[0087] Details are enhanced through Haar wavelet transform.

[0088] Local branch: accounts for 25%, used to extract multi-scale local features. The local branches are divided into three groups by channel, and convolution kernels with k=3, 5, and 7 are used to capture multi-scale receptive fields.

[0089] Residual branch: accounts for 50%, directly reduces redundancy through identity mapping.

[0090] ③ After the backbone extracts multi-scale features, the neck part performs fusion and re-extraction. This part is composed of the CAFM module and the MFEM module, which perform multiple feature fusion and feature re-extraction respectively.

[0091] CAFM module:

[0092] Input features: backbone network P3 and P4 level features.

[0093] Weight generation: Generate cross-layer attention weights through MLP, and then perform weighted fusion after Softmax normalization.

[0094] Output features: The fused image after dynamically adjusting the weights of each layer.

[0095] MFEM module:

[0096] Input feature conv Expanded to 288 dimensions, split into three 96-dimensional branches;

[0097] Branch 1 is directly transferred, and branches 2-3 extract multi-scale details through a cascaded inverted bottleneck structure;

[0098] Residual connections between branches (channel = channel + channels [i-1]) enhance feature reuse;

[0099] Output via out conv Compress to 96 dimensions and connect to the subsequent detection head.

[0100] ④The features processed by the neck part are finally passed to the decoder and head for target detection.

[0101] 2. The implementation details of the core module are as follows:

[0102] (1) Multi-receptive field feature interaction backbone network (MRFFIBackbone)

[0103] Network structure: 4 processing stages (Stage 1-Stage 4), each stage contains 1 MRFFI Block MRFFIBlock processing flow:

[0104] Step 1: Channel segmentation:

[0105] Input feature map X∈R H×W×C Split by channel dimension into:

[0106] Global branch: X g ∈R H×W×0.25C ;

[0107] Local branch: X l ∈RH×W×0.25C ;

[0108] Residual branch: X r ∈R H×W×0.5C ;

[0109] Step 2 branch processing:

[0110] Global branch:

[0111] Y g =Mamba(SS2D)(X g )

[0112]

[0113] Extract LH / HL high frequency components

[0114] Local branch:

[0115] Y l1 =DepthwiseConv3×3(X l[0:0.33C] )

[0116] Y l2 =DepthwiseConv5×5(X l[0.33C:0.66C] )

[0117] Y l3 =DepthwiseConv7×7(X l[0.66C:C] )

[0118]

[0119] Residual branch: Step 3: Feature fusion:

[0120]

[0121] Output feature map:

[0122] Stage 3 output: 80×80 resolution feature map (1 / 8 of the input 640×640) Stage 4 output: 40×40 resolution feature map (1 / 16 of the input 640×640) (2) Channel Aware Fusion Module (CAFM)

[0123] enter:

[0124] 1 / 8 resolution feature map (80×80×96);

[0125] 1 / 16 resolution feature map (40×40×192);

[0126] Processing flow:

[0127] Step 1 Channel alignment:

[0128] P3'=Conv1×1(UpSample(1 / 16 feature map)), upsample to 80×80; P4'=Conv1×1(1 / 8 feature map), keep 80×80;

[0129] Step 2 Dynamic weight generation:

[0130] P cat =Concat(P3′,P4′)

[0131] z=GlobalAvgPool(P cat )

[0132]

[0133] FC1:192→24, FC2:24→2

[0134] Step 3: Weighted fusion:

[0135] P out =w1×P3′+w2×P4′

[0136] (3) Multi-branch feature extraction module (MFEM)

[0137] Input: CAFM output feature F in ∈R 80×80×96 ;

[0138] Processing flow:

[0139] Step 1 Channel expansion:

[0140] F exp =Conv1×1(F in ), output 288 channels;

[0141] Step 2: Three-branch processing:

[0142] Branch 1: b1 = F exp[:,:,:96] , the first 96 channels;

[0143] Branch 2:

[0144] T2=DepthwiseConv3×3(F exp[:,:,96:192] )

[0145] B2=InvertedBottleneck(T2,expansion=6)

[0146] Branch 3:

[0147] T3=DepthwiseConv5×5(F exp[:,:,192:288] )

[0148] B3=InvertedBottleneck(T3,expansion=6)

[0149] Step 3: Residual enhancement:

[0150] B′2=B2+B 1[:,:,:96] , superimpose the 96 channels of branch 1;

[0151] B′3=B3+B′2[:,:,:96], superimposing the 96 channels of branch 2;

[0152] Step 4 Output compression:

[0153] F out =Conv1×1(Concat(B1,B′2,B′3))#output 96 channels;

[0154] 3. Deployment and optimization of key parameters

[0155] (1) TensorRT quantization configuration:

[0156] Precision mode setting: Enable FP16 half-precision inference mode by setting the BuilderFlag.FP16 flag

[0157] Video memory resource allocation: The upper limit of the workspace memory pool is set to 2GB (2,147,483,648 bytes)

[0158] Layer fusion optimization: Automatically fuse Conv-BN-ReLU calculation sequences to reduce the number of kernel calls Dynamic shape configuration: Support input resolution dynamic range from 512×512 to 1024×1024

[0159] (2) Real-time guarantee:

[0160] Single frame processing time ≤ 1.8ms (640×640 input);

[0161] Memory optimization: pre-allocate four 640×640 input buffers;

[0162] Asynchronous pipeline:

[0163] Image acquisition thread: responsible for video stream capture;

[0164] Preprocessing thread: performs scaling / normalization operations;

[0165] Detection thread: runs neural network inference;

[0166] The three-stage pipeline is executed in parallel, with a frame delay of ≤0.5ms.

[0167] 4. Variant Embodiments

[0168] (1) Channel ratio adjustment range:

[0169] Global branch: 25% ≤ G ≤ 35%;

[0170] Local branching: 35%≤L≤45%;

[0171] Residual branch: 25% ≤ R ≤ 35%;

[0172] Constraint: G+L+R=100%.

[0173] (2) MFEM branch expansion:

[0174] The relationship between the number of branches n and the channel expansion multiple;

[0175] When the number of branches n=2: the input feature channels are expanded to twice the original number of channels;

[0176] When the number of branches n = 3 (typical configuration): expansion to 3 times;

[0177] When the number of branches n=4: expand to 4 times.

[0178] (3) Inverted bottleneck structure expansion ratio adjustment rules

[0179] When n=2: the inverted bottleneck expansion ratio is set to 4 to 6 (typical value 5);

[0180] When n=3: the inverted bottleneck expansion ratio is fixed at 6;

[0181] When n=4: the inverted bottleneck expansion ratio increases to 8;

[0182] (4) Adaptive adjustment of residual connection mechanism

[0183] Each additional branch:

[0184] Added a branch to perform depthwise separable convolution (kernel size increases: 3×3 → 5×5 → 7×7);

[0185] The residual superposition range is extended to all the preceding branches (all preceding channels are superimposed).

[0186] The following experiment compares the model designed by the present invention with the existing SOTA target detection model.

[0187] Hardware platform: NVIDIA 4090s

[0188] Evaluation indicators: GFLOps: measures the computational complexity of the model. A larger value indicates a higher computational workload.

[0189] Params: measures the number of model parameters;

[0190] Ap: Average precision, which comprehensively measures the detection accuracy under different Lou (intersection-over-union ratio of the detection box and the true box) values;

[0191] Ap50: detection accuracy when the IoU threshold is 0.5;

[0192] FPS: The number of frames processed per second. The larger the FPS value, the better the real-time performance of the model.

[0193] Table 1 Experimental results on the general dataset visdrone

[0194]

[0195]

[0196] Table 2 Results on the general dataset UAVDT

[0197]

[0198] Tables 1 and 2 show that, given the same level of Params (parameters) and GFLOps (computational complexity), the proposed model achieves the best detection accuracy (as measured by Ap and Ap50). Furthermore, its real-time performance is among the best of all models.

[0199] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A lightweight and real-time drone target detection system, characterized by: include: An onboard image acquisition module, used to obtain real-time images of the scene to be detected; The processing unit, deployed on the drone edge computing platform, includes: Multi-receptive field feature interaction backbone network MRFFIBackbone, used to extract multi-scale features of images; The channel-aware fusion module (CAFM) dynamically weights and fuses the cross-level features output by the backbone network. Multi-branch feature re-extraction module MFEM, which performs secondary refinement extraction on the fusion features; Detection head, which outputs target detection results based on refined features; The communication module is used to transmit the detection results to the ground control terminal.

2. The system according to claim 1, wherein: The MRFFI Backbone is composed of multiple levels of MRFFI Blocks, which include: Channel segmentation unit: divides the input features into global branch, local branch and residual branch according to the preset ratio; Global branch: connects the Mamba module and the Haar wavelet transform layer in sequence to capture long-range dependencies and enhance details; Local branch: contains multiple sets of parallel depth-wise separable convolutional layers with kernels k=3, 5, and 7 to extract multi-scale local features; Residual branch: transfers original features through identity mapping; Feature splicing unit: fuses the output features of the three branches.

3. The system according to claim 2, characterized in that: The Mamba module implements sequence modeling based on the structured state space model SS2D to obtain long-distance dependencies; Haar wavelet transform is performed after the output of the Mamba module to enhance the detail features through high-frequency components.

4. The system according to claim 1, wherein: The CAFM module includes: Channel alignment layer: unifies the multi-level features of the input to the same channel dimension; Dynamic attention generation unit: Generates cross-level weights through global average pooling (GAP), multi-layer perceptron (MLP), and softmax function; Weighted fusion layer: performs weighted summation of input features according to the weights.

5. The system according to claim 1, wherein: The MFEM module includes: Channel expansion layer: expand the input channel to n times through 1×1 convolution, n ≥ 3; Multi-branch processing unit: The first branch directly connects the residual output; The remaining branches are sequentially connected to the inverted bottleneck structure and feature superimposed with the preceding branches; Channel compression layer: compresses the multi-branch concatenation features to the target dimension.

6. The system according to claim 5, characterized in that The inverted bottleneck structure comprises: Depthwise separable convolutional layers; Linear activation function ReLU6 and channel scaling layer.

7. The system according to claim 1, wherein: The processing unit satisfies: Parameter count ≤ 4.3M, computational complexity ≤ 9.1GFlops; When deployed on the drone side, the real-time frame rate is ≥563FPS.

8. A target detection method based on the system according to any one of claims 1 to 7, characterized in that: include: S1: collects real-time images through the onboard camera; S2: Executed by the processing unit: a) Stem module extracts initial features; b), MRFFIBackbone generates multi-scale features; c), CAFM integrates multi-scale features; d) MFEM re-extracts fusion features; e) The detection head outputs the target position and category; S3: Send the detection results back to the ground station through the communication module.

9. A computer-readable storage medium storing program instructions, characterized in that: When the instructions are executed by the drone's onboard processor, the method of claim 8 is implemented.

10. An unmanned aerial vehicle, comprising an airframe, a flight control system, and an image acquisition device, characterized in that: Also includes: The target detection system according to any one of claims 1 to 7; The processing unit is integrated into an onboard embedded platform.

Images