Unmanned aerial vehicle urban inspection lightweight target detection method based on multi-scale attention fusion network

Abstract

The invention provides a lightweight unmanned aerial vehicle inspection target detection method, and aims to solve the problems that small targets are difficult to detect, serious in shielding, similar in background, limited in calculation power and the like in urban aerial photography, and the precision and speed are improved by the following steps: firstly, preprocessing data and balancing samples; then, a lightweight convolution-MLP hybrid backbone network is adopted, a high-resolution P2 branch is reserved to enhance small target perception, and a context enhancement module is embedded to aggregate long-range and short-range contexts; a learnable channel normalization fusion module is introduced into the feature fusion neck to realize adaptive multi-scale fusion, and space-channel collaborative attention is added in front of each detection layer to suppress noise and highlight a key region; and finally, constructing a P2-P5 multi-scale decoupling detection head, and supporting quantitative deployment in combination with weight preheating, a cosine learning rate and EMA strategy training. The method significantly improves the robustness of small target and abnormal target detection in a complex city scene, and is suitable for municipal inspection, security monitoring, emergency command and other applications.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, deep learning, and intelligent unmanned systems, and belongs to the real-time detection method of multi-scale small targets in urban inspection and multi-source scene perception by unmanned aerial vehicles. Specifically, it covers edge intelligent reasoning, lightweight model design, attention enhancement, and reproducible deployment techniques, and has important practical application value. Background Technology

[0002] With the rapid growth in demand for refined urban governance, intelligent transportation, public safety, emergency management, and municipal facility maintenance, drones, with their flexible flight path planning and rapid response capabilities in temporary areas, are gradually becoming an important tool for urban spatial information collection and dynamic monitoring. Compared to fixed cameras, drones offer advantages such as high mobility, fewer blind spots, and lower scheduling costs, providing supplementary perception in scenarios such as road congestion, construction zones, emergencies, post-rain drainage, and river pollution. However, aerial images are characterized by a large target scale range, a mix of overhead and oblique views, drastic changes in lighting and weather, and numerous obstructions and dynamic interferences, making the detection of small targets difficult, such as missing manhole covers, litter, illegal parking signs, and densely populated pedestrian areas.

[0003] For current mainstream detection frameworks, such as YOLOv8, YOLO11, and RT-DETR, there are several problems in the urban inspection scenario using drones: (1) shallow features do not directly participate in detection, resulting in insufficient recall of very small targets; (2) multi-scale feature fusion uses static fusion, does not distinguish the contribution of the source, and easily amplifies redundancy or noise; (3) conventional attention mechanisms have limited attention granularity, insufficient spatial and channel interaction, and the use of nondeterministic operators affects reproduction; (4) directly adding complex modules will cause gradient oscillation and overfitting, resulting in poor training stability; (5) it is difficult to balance accuracy and latency under the condition of limited computing power on edge platforms. Although existing methods have attempted to expand the receptive field by adding shallower detection heads, introducing channel or spatial attention, and using multi-dilated convolution, there is still a lack of an overall architectural solution that simultaneously achieves: lightweight, multi-scale learning-based fusion, deterministic attention, small target-specific reinforcement, phased stable training and incremental adaptation, and energy consumption and latency adaptation.

[0004] To address the aforementioned challenges, this invention proposes a systematically improved lightweight multi-scale attention fusion UAV inspection target detection method, which solves problems such as insufficient small target detection, inefficient cross-layer fusion, non-reproducible attention, unstable training, and high energy consumption. Summary of the Invention

[0005] In view of this, this invention proposes a multi-scale target detection method for UAV urban patrol based on a lightweight backbone, a lightweight context enhancement model, a learnable feature fusion module, and a deterministic spatial-channel attention module, and extends the P2 shallow detection branch. This method significantly improves the detection accuracy and robustness of small targets and rare event categories while maintaining real-time performance through structural collaboration and phased training strategies. It enhances the detection performance of small targets and rare event categories in complex urban scenarios, while maintaining real-time performance and reproducibility.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] A lightweight target detection method for UAV urban patrol based on a multi-scale attention fusion network includes:

[0008] Step 1) Data acquisition and preprocessing.

[0009] Step 2) The input image is processed through a staged FastBlock cascade lightweight backbone network to extract multi-scale backbone features.

[0010] Step 3) Insert the multi-branch features extracted from the backbone network into the lightweight context enhancement module to improve the joint representation of local and contextual features;

[0011] Step 4) Replace static fusion with a learnable channel normalization fusion module in the top-down and bottom-up neck layers to achieve cross-layer feature adaptive fusion;

[0012] Step 5) Add a spatial-channel collaborative attention module before prediction at each detection scale. This module uses mean and maximum deterministic pooling to ensure the reproducibility of training and inference.

[0013] Step 6) Use a multi-scale detector head for classification and regression, and combine it with a training strategy of phased freezing, differentiated learning rate and multi-scale loss weighting to complete model training and deployment.

[0014] Furthermore, step 1) specifically includes the following steps:

[0015] Step 11) Collect and integrate aerial or oblique view images of multiple urban inspection scenarios such as roads, blocks, construction, and waterways, and unify the category system, such as pedestrians, vehicles, garbage, manhole covers, obstructions, equipment, and floating objects, and divide the training / validation / test sets in an 8:1:1 ratio; adopt stratified sampling and category reweighting strategies to improve the coverage of long-tail classes and small target samples.

[0016] Step 12) Data Preprocessing: Perform small-scale enhancements such as size normalization, color dithering, motion blur simulation, contrast enhancement, and mosaic or copy-paste enhancement; unify the category system and annotation format. To alleviate class imbalance, class weights w can be introduced to reduce classification and confidence loss. c

[0017]

[0018] Where N is the total number of samples, K is the number of categories, and n c Let r be the number of samples in category c. The oversampling ratio can be calculated as r. c =min(r max ,μ / p c ) setting, where p c Let μ be the class prior frequency and μ be the target equilibrium constant.

[0019] Furthermore, step 2) specifically includes the following steps:

[0020] Step 21) Lightweight backbone construction: A FastBlock cascaded convolutional-MLP hybrid structure is adopted, combining a hierarchical depth mixing strategy with small-scale P3 branch widening to improve the representation density at small and medium scales while controlling parameters and computing power. Given the input... Divided into two branches according to the channel: X = [X dw ||X mlp ], the ratio r∈(0,1), the default is r≈0.5. After the first depthwise convolution branch, formula (1):

[0021] U dw =BN(φ(DWConv) k×k (X dw )),k∈{3,5} (1)

[0022] The second channel MLP branch, formula (2):

[0023] U mlp =W2δ(W1X) mlp ),W1:C mlp →eC mlp ,W2:eC mlp →C mlp (2)

[0024] Where e is the channel expansion factor, δ is the nonlinearity (ReLU or GeLU), and W1 and W2 are implemented by 1×1 convolution.

[0025] Finally, residual fusion is performed, as shown in formula (3):

[0026] Z = W c [U dw ||U mlp],Y=X+βZ,β∈(0,1] (3)

[0027] Through W c 1×1 convolution renormalizes the channels and is trained with residual scaling β-stabilization.

[0028] Step 22) Multi-scale feature generation: By alternating between hierarchical downsampling and FastBlock stacking, feature maps P2, P3, P4, and P5 are extracted to provide a foundation for subsequent fusion and detection; among them, P2 retains more spatial resolution to serve small target detection.

[0029] Furthermore, step 3) specifically includes the following steps:

[0030] Step 31) Insert a lightweight context enhancement module after each branch output. Each module contains a multi-dilation rate convolution branch and a 1×1 convergence, and the enhancement strength is controlled by the residual scaling factor.

[0031] Step 32) Context enhancement mechanism: Convolutions with different dilation rates cover the nearest and far-field contexts. After fusion, they are compressed back to the original channel dimension by 1×1 convolution and then added to the input residual to achieve fine-grained supplementation of edges, contours and textures.

[0032] Let the input features be FEM contains 3 dilated convolution branches, as shown in formula (4):

[0033]

[0034] After aggregation, the residue is compressed by 1×1 and then scaled as shown in formula (5):

[0035] G = W 1×1 [B0||B1||B2], Y=F+α FEM G,α FEM ∈(0,1] (5)

[0036] Furthermore, step 4) specifically includes the following steps:

[0037] Step 41) Learnable fusion strategy: In the neck of the feature pyramid from top to bottom and from bottom to top, a learnable channel normalization fusion module is used to replace static fusion; this module assigns learnable weights to features from different sources and performs numerically stable normalization to suppress redundancy and noise amplification.

[0038] Step 42) Fusion Morphology Preservation: This module's output maintains the original expected channel morphology, facilitating seamless integration with subsequent convolutions and the detection head. For two identical feature paths... Let the channel weight parameter be... Using Softplus normalization, as shown in formula (6):

[0039] The fused output is then given by formula (7):

[0040] Y = Concat(λ) a ⊙X a ,λ b ⊙X b (7)

[0041] Where ⊙ represents pixel-by-channel multiplication.

[0042] Furthermore, step 5) specifically includes the following steps:

[0043] Step 51) Deterministic Attention: Spatial-channel collaborative attention modules are added before the P2, P3, P4, and P5 prediction heads, respectively. This module uses global average pooling and global max pooling to aggregate global information, and uses 1×1 convolution to generate a simplified self-attention structure of Query(Q), Key(K), and Value(V) to achieve cross-channel and spatial context interaction; GAP and GMP are used instead of nondeterministic adaptive max pooling to ensure reproducibility.

[0044] Step 52) Gated Residual: The attention-weighted features are fused with the original features using a gated residual method. The strength is enhanced by adjusting the coefficients, balancing fine-grained enhancement and training stability.

[0045] set up As in formula (8), first calculate the global statistics of the channel:

[0046]

[0047] Will After cascading and linear mapping, the channel weights are obtained through Softmax, as shown in formula (9):

[0048]

[0049] Q, K, and V are generated using 1×1 convolution, as shown in formula (10):

[0050] Q = Conv 1×1 (F), K = Conv 1×1 (F), V=Conv 1×1 (F) (10) Position-wise similarity based on the dot product within the channel, as shown in formula (11):

[0051]

[0052] The weights and gated residuals are combined, as shown in formula (12):

[0053]

[0054] The above GAP / GMP, 1×1 convolution, element-wise multiplication, and Softmax are deterministic implementations, meeting the requirements for reproducible experiments and industrial deployment.

[0055] Furthermore, step 6) specifically includes the following steps:

[0056] Step 61) Multi-scale detection: Classify and regress bounding boxes for the four scales P2 to P5 respectively. P2 focuses on small-scale targets, such as pedestrians, manhole covers, and garbage.

[0057] Step 62) Phased training:

[0058] a. Cold start phase: Freeze the latter half of the backbone and prioritize training the neck and detection head; set a lower loss weight for the P2 branch to reduce early gradient interference;

[0059] b. Deployment phase: Unfreeze the entire network, gradually increase the P2 weights, and strengthen the learning of small objectives and dense scenarios;

[0060] c. Fine-tuning stage: Reduce the learning rate and attention gating strength to suppress overfitting and improve generalization;

[0061] Step 63) Loss weighting and optimization: Multi-scale loss weighting, cosine annealing and preheating strategies are adopted, combined with EMA and mixed precision training to improve convergence speed and stability.

[0062] The total loss across multiple scales can be expressed as formula (13):

[0063]

[0064] Where λ s For the scale weights, the P2 weights are preheated and scheduled, as shown in formula (14):

[0065]

[0066] Learning rate cosine annealing formula (15):

[0067]

[0068] EMA parameter updates are as shown in formula (16):

[0069]

[0070] Attention gating strength linear annealing formula (17):

[0071] Attached Figure Description

[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0073] Figure 1 This is a flowchart of the method;

[0074] Figure 2 This is a diagram illustrating the overall network framework of this method;

[0075] Figure 3 This is a diagram of the backbone network module structure proposed in this method;

[0076] Figure 4 This is a structural diagram of the lightweight context enhancement module proposed in this method;

[0077] Figure 5 This is a structural diagram of the learnable channel normalization fusion module proposed in this method;

[0078] Figure 6 This is a structural diagram of the deterministic spatial-channel collaborative attention module proposed in this method. Detailed Implementation

[0079] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0080] This invention proposes a lightweight target detection method for UAV urban patrol based on a multi-scale attention fusion network. The method flow is as follows: Figure 1 As shown, the overall network framework is as follows: Figure 2 As shown, the main steps include:

[0081] Step 1) Construction and partitioning of the urban inspection dataset;

[0082] Step 2) Construct a lightweight backbone network;

[0083] Step 3) Build a lightweight context enhancement module;

[0084] Step 4) Construct a learnable channel normalization and fusion module;

[0085] Step 5) Construct a deterministic space-channel collaborative attention module;

[0086] Step 6) Training and Testing

[0087] Furthermore, step 6) specifically includes the following steps:

[0088] Step 61) Configure the corresponding environment and set the training parameters to carry out the training task;

[0089] Step 62) The training environment configured is as follows: Operating system Windows 11 / Ubuntu 22.04, CUDA 11.8, Python 3.10, PyTorch 2.3; Hardware devices include CPU: Intel(R) Xeon(R) CPU E5-2680 v4@2.40GHz, memory 32GB; GPU: NVIDIA RTX 3090, memory 24GB.

[0090] Step 63) Training parameters: Input size 640×640, batch size 16, base learning rate 0.01 (cosine annealing + warm-up), training epochs 500; P2 branch loss weight gradually increased from 0.3 to 0.7; use EMA and mixed precision to accelerate and stabilize training.

[0091] Step 64) During the testing phase, the trained weights are loaded into the detection network, evaluated and deployed on the urban road network dataset, and the detection boxes, categories and confidence scores are output. Event geolocation and archiving are performed in conjunction with UAV pose information.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A lightweight target detection method for UAV urban patrol based on a multi-scale attention fusion network. Its characteristics are: Includes the following steps: Step 1: Obtain urban inspection data from publicly available data from urban management departments and self-collected drone aerial videos, perform frame extraction and cleaning, unify the category system, and divide it into training set, validation set and test set in an 8:1:1 ratio; perform appropriate oversampling and hard case mining for rare events to alleviate long-tail distribution and improve the visibility of small targets and abnormal targets. Step 2: Construct a lightweight FastBlock cascaded convolutional-MLP hybrid backbone network and extract multi-scale features: A multi-stage hierarchical design is adopted to obtain P2, P3, P4, and P5 feature maps. Channel widening and layer depth are introduced in the small-scale stage to improve the representation density of small and medium scales within the range of onboard computing power. Each stage is composed of alternating stacks of FastBlocks, taking into account both local space and pixel-by-pixel channel modeling, and controlling parameters and computational overhead to adapt to real-time deployment on embedded platforms. Step 3: Insert a lightweight context enhancement module at the output position of the backbone network: After the output of each branch, set a context enhancement module consisting of a multi-dilation rate convolution branch and a 1×1 convergence to aggregate nearest and far-field information, and stabilize the training by residual scaling to enhance fine-grained representations such as edges, textures, and contours to suppress background interference. Step 4: Replace static fusion with a learnable channel normalization fusion module in the neck of the feature pyramid from top to bottom and from bottom to top: assign learnable channel weights to features from different sources and perform numerical stability normalization, keep the output channel shape unchanged, realize adaptive selective fusion of cross-layer information, and reduce redundant information amplification and conflict. Step 5: Add a spatial-channel collaborative attention module before prediction at each detection scale: global average pooling and global max pooling are used to aggregate global channel information, and 1×1 convolution is used to generate Query, Key and Value to form a simplified self-attention structure. Contextual interaction and weighting are performed in the channel and spatial dimensions to suppress background noise and enhance target discrimination. Among them, GAP and GMP, as well as operators such as 1×1 convolution and Softmax, are all implemented deterministically to ensure reproducibility across devices and multiple rounds of training. Step 6, construct and train / deploy a multi-scale detection head: perform classification and bounding box regression for the four scales P2 to P5. During the training phase, strategies such as phased freezing and differential learning rate, P2 branch loss weight warm-up, multi-scale loss weighting, cosine annealing and warm-up, exponential moving average (EMA) and mixed precision are adopted to achieve stable optimization. During the inference phase, the trained weights are loaded into the detection network, evaluated and deployed on the urban road network dataset, and the detection boxes, categories and confidence scores are output. Event geolocation and archiving are performed in conjunction with UAV pose information to meet the real-time detection requirements of UAV onboard terminals.

2. A lightweight target detection method for UAV urban patrol based on a multi-scale attention fusion network as described in claim 1, characterized in that: In step 2, the lightweight backbone network achieves efficient representation under limited computing power through staged multi-scale and convolutional-MLP hybrid stacking: Let the output strides of stages s∈{2,3,4,5} be 4, 8, 16, and 32, respectively, corresponding to channels C s With the depth d of the base block s In the small-scale stage, channel widening C′3 = round(κ·C3) is used, where κ∈[1.1,1.5] is rounded to balance integer channels and cache alignment; depth elastic adjustment is achieved through d′ s =round(τ s ·d s ) or d′ s =d s +Δd s (τ s ∈[0.75,1.50],Δd s Before deployment, the memory access bandwidth and inference latency of the target platform are searched according to the budget of [-2,+2]). The core FastBlock splits the input into two in the channel dimension: the local branch uses DWConv(k×k)(k∈{3,5}) plus normalization plus nonlinear capture of spatial correlation, and the channel branch uses 1×1 convolution to realize dimensionality increase or decrease (W1,W2) and pixel-by-pixel channel rearrangement. The two are fused and then fused again by 1×1 convolution and added to the input with the residual scaling factor β (β∈(0,1], preferably 0.5~0.9). The first unit of each stage uses a stride=2 convolution / depth-separable convolution to perform downsampling, and then repeats the stacking of d. ′ s The FastBlock output features {P2, P3, P4, P5} are used, where high-resolution P2 is retained to improve the detection rate of small, elongated, and low-contrast targets; the entire network uses only ReLU activation and BN to avoid introducing normalization that is incompatible with quantization; the design maintains It achieves linear complexity while reducing the quadratic space overhead of global attention and facilitating low-bit representation.

3. A lightweight target detection method for UAV urban patrol based on a multi-scale attention fusion network as described in claim 1, characterized in that: The lightweight context enhancement module in step 3 is inserted after the backbone output to capture nearest and far-field context with parallel multi-dilation rate convolutional branches: applying a 3×3 convolution with dilation rate d∈{1,2,3} to the input X produces features {Z}. d }, spliced ​​together in the channel dimension to form Z cat Then, it is compressed to the original number of channels C by 1×1 convolution, and optional normalization and activation are performed to obtain U; ​​the final output Y = X + α FEM ·U, where α FEM ∈(0,1] can be set to learnable or annealed along the training steps to reduce early gradient oscillations; this structure expands the receptive field through multi-scale dilated convolutions while avoiding the introduction of additional fully connected attention overhead, and its parameter count approximately increases. During the initialization phase, the convolution weights of each branch are distributed using the He / Kaiming normal distribution, and the BN folding is maintained uniformly during the quantization process to ensure stable enhancement of fine edges, texture breaks, and blurred contours in complex aerial photography backgrounds.

4. A lightweight target detection method for UAV urban patrol based on a multi-scale attention fusion network as described in claim 1, characterized in that, Step 4's learnable channel normalization fusion module replaces static fusion with parameterized weights: for features X from different paths (1) ,X (2) Assign learnable channel scalar vectors respectively Normalization after monotonic activation by Softplus Where ε∈[10] -6 10 -3 To ensure numerical stability, a channel-by-channel scaling was then applied to each path to obtain the desired result. The channel dimension is summed or lightweight spliced ​​according to the shape preservation strategy, and if necessary, a 1×1 convolution correction channel number is added to maintain compatibility with subsequent detection heads. This normalization fusion avoids parameter inflation and gradient competition caused by simple splicing, reduces redundant feature amplification and cross-layer conflict, improves discriminability and convergence speed, and enables threshold pruning of low-weight channels during the quantization / pruning stage to further reduce end-side computation. Its additional calculations are only Level scalar multiplication and constant-order normalization are adapted to low-power processing units in drones.

5. A lightweight target detection method for UAV urban patrol based on a multi-scale attention fusion network as described in claim 1, characterized in that, Step 5's spatial-channel collaborative attention module achieves noise suppression and target enhancement in complex urban backgrounds through bidirectional global statistics and pixel-by-pixel lightweight self-attention: input features Obtained through global average pooling and max pooling splicing [c avg ,c max After linear mapping W ch With activation σ, generate channel weight vector w ch =σ(W ch [c avg ,c max ])∈(0,1] C ; Parallel generation using 1×1 convolution Maintain original spatial resolution; Calculate the simplified dot product for each spatial location (h, w). The spatial weights S are obtained through Softmax. h,w Then construct enhancement terms. By adjusting the residual gating coefficient α SCAM (α SCAM ∈(0,1], can be incremented in stages from 0.2 to 0.8 and added to the input Y=X+α SCAM ·E; The computational complexity of this structure is Avoiding conventional self-attention With a squared space overhead, a combination of deterministic operators, GAP / GMP and 1×1 convolution, is used to ensure cross-device reproducibility and quantization friendliness. At the same time, channel statistics are used to suppress high-frequency background and repetitive textures and enhance the saliency of rare anomalies such as missing manhole covers and roadblock accumulation.

Images