A Multi-Scale Frequency Fusion Method for Small Target Detection in UAV Aerial Photography

Abstract

This invention belongs to the field of deep learning-driven target detection technology and discloses a multi-scale frequency fusion method for detecting small targets in UAV aerial photography. In a one-stage algorithm model composed of a backbone network, a neck network, and a detection head, an adaptive frequency sensing module is set at the beginning of the backbone network; one or more progressive spatially preserved downsampling modules are set in the downsampling layer of the backbone network; and a multi-scale feature fusion upsampling module is set in the upsampling part of the feature pyramid of the neck network, thus constructing a multi-scale frequency fusion UAV aerial photography detection model. This invention can improve mAP50 and mAP50-95 to 0.412 and 0.25 respectively, with relative increases of 8.1% and 9.6%; this improvement verifies the effectiveness and advancement of this invention in addressing the challenge of small target detection in UAV aerial photography.

Description

Technical Field

[0001] This invention belongs to the field of deep learning-driven target detection technology, and relates to a multi-scale frequency fusion method for small target detection in UAV aerial photography. Background Technology

[0002] Object detection is one of the core tasks in computer vision. Its core objective is to accurately identify the category of objects in images or video frames and locate their positions using bounding boxes. This technology has been widely applied in various fields such as security monitoring, autonomous driving, and drone inspection. Deep learning-driven object detection algorithms are mainly divided into two categories: one is two-stage algorithms (such as the R-CNN series and DetNet), which achieve detection by generating candidate boxes and then classifying them; these algorithms have high accuracy but slow inference speed. The other is one-stage algorithms (such as SSD and YOLO series), which directly transform the localization problem into a regression task, balancing speed and accuracy, and have become the mainstream choice for real-time detection scenarios.

[0003] Due to its high flexibility and wide coverage, drone aerial photography technology has been widely used in scenarios such as traffic supervision, environmental monitoring, and emergency rescue. Corresponding small target detection tasks (such as detecting pedestrians, vehicles, obstacles, etc. in aerial images) have become key technologies supporting the above applications.

[0004] In drone aerial photography scenarios, targets (such as vehicles and pedestrians) often appear at a tiny scale (occupying only a few pixels in the image), and there are inherent challenges: drones have a high field of view, and the pixel information of small targets is limited. Key features such as texture and contour are easily masked by background noise, making it difficult for models to extract effective discriminative features; although the large stride downsampling of traditional networks can improve the receptive field, it will cause irreversible loss of spatial information and detailed features of small targets, further increasing the difficulty of detection; the target scale span in aerial photography scenarios is large (such as vehicles at close range and pedestrians at a distance), and existing multi-scale fusion mechanisms are difficult to simultaneously take into account the feature extraction and alignment of targets at different scales; and weather factors such as airflow turbulence, lighting changes, fog, haze, rain and snow during drone flight will cause image blurring and reduced contrast, further reducing the accuracy of small target detection.

[0005] Currently, improvements to small target detection methods for UAVs mainly focus on feature enhancement, downsampling optimization, and multi-scale fusion, but significant limitations remain. First, existing methods primarily focus on spatial domain feature enhancement, failing to fully utilize the complementarity between frequency domain information and edge features, resulting in limited enhancement of blurred features of small targets (Jeon M, Seo J, Min J. Da-raw: Domain adaptive object detection for real-world adverseweather conditions[C] / / 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024: 2013-2020). Furthermore, some algorithms employ fixed kernel functions for feature extraction, which cannot adapt to dynamic scene changes in aerial images, leading to insufficient robustness (Fu C, et al. "Learning Cruxesto Push for Object Detection in Low-Quality Images."IEEE Transactions on Circuits and Systems for Video Technology(2024)).

[0006] Secondly, traditional downsampling methods (such as pooling and large stride convolution) have the problem of "jumping" feature compression, which easily leads to the loss of detailed information of small targets; existing improved downsampling modules mostly adopt a single scale conversion strategy, which fails to achieve progressive adaptation of the number of channels and spatial dimensions, and gradient vanishing is prone to occur during network training (Cha J, Chun S, Lee K, et al. Swad: Domain generalization by seeking flat minima[J]. Advances in NeuralInformation Processing Systems, 2021, 34: 22405-22418).

[0007] Finally, the upsampling process of mainstream algorithms mostly relies on simple interpolation or single-branch convolution, lacking a dynamic adjustment mechanism for the weight allocation of features at different scales, making it difficult to effectively aggregate shallow detail features and deep semantic features of small targets (XiaoY, Jiang A, Ye J, et al. Making of night vision: Object detection under low-illumination[J]. IEEE Access, 2020, 8: 123075-123086); some multi-scale module structures are redundant, resulting in a decrease in model inference speed, which cannot meet the real-time detection requirements of UAVs (Liu W, Ren G, Yu R, et al. Image-adaptive YOLO for object detection in adverse weather conditions[C] / / Proceedings of the AAAI conference on artificial intelligence. 2022, 36(2):1792-1800). Summary of the Invention

[0008] The technical problem solved by this invention is to provide a multi-scale frequency fusion method for detecting small targets in UAV aerial photography, which improves detection accuracy while maintaining real-time performance.

[0009] This invention is achieved through the following technical solution:

[0010] A multi-scale frequency fusion method for small target detection in UAV aerial photography involves setting an adaptive frequency sensing module at the beginning of the backbone network, setting one or more progressive spatial preservation downsampling modules in the downsampling layer of the backbone network, and setting a multi-scale feature fusion upsampling module in the upsampling part of the feature pyramid of the neck network to construct a multi-scale frequency fusion UAV aerial photography detection model.

[0011] A multi-scale frequency fusion UAV aerial photography detection model is trained, and the trained model is used for small target detection in UAV aerial photography.

[0012] The adaptive frequency sensing module includes parallel edge paths and frequency paths, which extract spatial edge features and frequency domain texture information, respectively. The edge path employs a learnable multi-directional adaptive Sobel operator to cover edge information from various angles. The frequency path includes parallel spatial and frequency domain branches. The frequency domain branch generates an adaptive frequency domain filter kernel to adaptively extract frequency domain features, while the spatial domain branch extracts spatial domain features. The output feature maps of the spatial and frequency domain branches are concatenated along the channel dimension. The outputs of the edge and frequency paths are fused using an attention mechanism to serve as the output of the adaptive frequency sensing module.

[0013] The progressive spatial preservation downsampling module includes a progressive downsampling chain composed of multiple SPDStep units based on spatial-to-depth convolution. It maximizes the preservation of spatial details through a multi-step spatial-to-depth transformation strategy, thereby maximizing the preservation of spatial detail information while compressing the feature map size.

[0014] The multi-scale feature fusion upsampling module includes a multi-branch feature extraction layer, a feature fusion layer, and an upsampling output layer. The multi-branch feature extraction layer has multiple branches, each focusing on feature extraction at a different scale. The feature fusion layer and the upsampling output layer fuse features at different scales and restore resolution, respectively.

[0015] Furthermore, the multi-directional adaptive Sobel operator is set at the beginning of the edge path and has multiple convolution kernels with different directions and parallel computation. The convolution kernels include horizontal and vertical convolution kernels, and the remaining convolution kernels are initialized with initial weights using a random initialization method. All elements of these convolution kernels together constitute learnable parameters, and the direction selection is adaptively optimized during network training using backpropagation and gradient descent algorithms.

[0016] Furthermore, in the frequency domain branch, the adaptive frequency domain filter kernel is generated as follows:

[0017] The frequency domain branch obtains the frequency domain statistical features of the image through global average pooling, then generates dynamic weights for the convolution kernel through a fully connected network, and then performs weighted fusion with the basic Laplacian kernel to form an adaptive frequency domain filtering kernel. Finally, the output feature map of the frequency domain branch is obtained through grouped convolution.

[0018] The spatial branch uses a combination of 3×3 convolution and ReLU activation function to extract spatial features;

[0019] The output feature maps of the spatial domain branch and the frequency domain branch are concatenated along the channel dimension to obtain the output feature map of the frequency path.

[0020] Furthermore, the outputs of the edge path and the frequency path are fused using an attention mechanism to obtain:

[0021] First, the edge features and frequency features are concatenated along the channel dimension to form a feature tensor with double the number of channels;

[0022] Then, the weight distribution of each feature is calculated through an attention network that includes spatial attention path and channel attention path. The spatial attention path generates a spatial attention weight map through convolutional layers, and the channel attention path generates a channel attention weight map through global average pooling and fully connected layers.

[0023] Then, the input features, spatial attention weight map, and channel attention weight map are multiplied element-wise and weighted and fused together.

[0024] The weighted fusion output is then processed by a 1x1 convolution to become the final output of the adaptive frequency sensing module.

[0025] Furthermore, the progressive downsampling chain of the progressive spatially preserved downsampling module introduces a space-to-depth convolution instead of a downsampling convolution; for the input feature map x During a single-step spatial transformation operation, the 2×2 adjacent regions are rearranged to the channel dimension:

[0026] (6)

[0027] Where SPD() represents the spatial-to-depth convolution operation. x [::2,::2] means sampling every other pixel starting from the top left corner. After this transformation, the feature map size is reduced by half and the number of channels becomes four times.

[0028] Each step of the downsampling process is represented as follows:

[0029] (7)

[0030] in x i The input feature map is for a single-step spatial transformation. SPD This is a single-step spatial transformation operation; Conv 3×3 () represents a convolution operation with a kernel size of 3×3. x i+1 This is the output feature map of a single-step spatial transformation.

[0031] Furthermore, the progressive spatial retention downsampling module also includes a residual connection path running parallel to the progressive downsampling chain;

[0032] The residual connection path is as follows: when the number or size of input / output channels changes, the residual path uses max pooling downsampling combined with 1×1 convolution to adjust the channels; when the size remains unchanged, the identity mapping is used directly.

[0033] The output feature map of the progressive spatially preserved downsampling module is obtained by adding the output feature map of the progressive downsampling chain path and the output feature map of the residual connection path element by element.

[0034] Furthermore, in the multi-branch feature extraction layer, the first branch uses a 1×1 convolution kernel to capture local detail features; the second branch uses a 3×3 standard convolution to extract contextual information; and the third branch uses dilated convolution to obtain global semantic features.

[0035] The features extracted by the multi-branch feature extraction layer are fused together in the feature fusion layer as follows:

[0036] The multi-scale features extracted from the three branches are spliced ​​along the channel dimension. The splicing process allows features of different scales to interact directly at the channel level, forming a fusion feature with three times the number of channels of the original branches.

[0037] The concatenated features are then subjected to dimensionality reduction and optimization through a 1×1 convolutional layer, followed by batch normalization and SiLU activation function operations.

[0038] The fused features are upsampled in the upsampled output layer using bilinear interpolation to restore the resolution;

[0039] Finally, the output is further refined through a 3×3 convolutional layer to obtain the output of the multi-scale feature fusion upsampling module.

[0040] Compared with the prior art, the present invention has the following beneficial technical effects:

[0041] The multi-scale frequency fusion method for small target detection in UAV aerial photography provided by this invention enhances shallow feature extraction through an adaptive frequency perception module, preserves spatial details through a progressive downsampling module, and optimizes feature pyramid fusion efficiency through a multi-scale fusion upsampling module. This method improves detection accuracy while maintaining real-time performance, providing reliable technical support for UAV applications.

[0042] This invention introduces an Adaptive Frequency Sensing (AFS) module: addressing the issues of insufficient initial feature extraction for small targets in UAV aerial photography and the easy obscuring of key features by the background, this module adopts a dual-path parallel architecture of edge path and frequency path; wherein, the edge path uses a learnable multi-directional Sobel operator to enhance contour feature extraction, and the frequency path uses dynamic convolution kernels to capture frequency domain information; after the outputs of the two paths are fused by attention, the discriminativeness of the initial features of small targets is significantly improved, providing a high-quality feature foundation for subsequent detection.

[0043] This invention constructs a Progressive Space Reservation Convolution (PSC) mechanism. Addressing the issues of lost details and unstable gradient propagation caused by current downsampling, this module employs a multi-step spatial-to-depth transformation strategy. It replaces single-step downsampling with layered compression, maximizing the preservation of spatial details while maintaining the receptive field. A residual connection design ensures effective gradient propagation, mitigating feature decay in deep networks. This mechanism improves the retention rate of key information in feature maps by approximately 40% during downsampling, laying the foundation for accurate localization of small targets in deep networks.

[0044] This invention designs a Multi-scale Feature Fusion Upsampling (MFF) module to address the problem of large target scale span and poor multi-scale feature aggregation effect in UAV aerial photography. This module achieves optimized reconstruction of the feature pyramid through a multi-branch structure, combined with dilated convolution, bidirectional interpolation, and attention mechanisms. Its branch interaction design enhances cross-scale feature consistency, effectively aggregates shallow details and deep semantic features of small targets, improves the network's adaptability and detection capability for small targets of different scales, and maintains a lightweight design while taking into account inference efficiency. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the network structure of the present invention;

[0046] Figure 2 This is a network structure diagram of the adaptive frequency sensing module of the present invention;

[0047] Figure 3 This is a network structure diagram of the progressive spatial retention downsampling module of the present invention;

[0048] Figure 4 This is a network structure diagram of the multi-scale feature fusion upsampling module of the present invention;

[0049] Figure 5 This is a diagram showing the detection results of the model in this invention;

[0050] Figure 6 This is a graph showing the detection results of the baseline model. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to embodiments. These descriptions are for illustrative purposes only and are not intended to limit the scope of the invention.

[0052] Currently, traditional object detection models primarily rely on visual features to identify and locate targets. However, extracting sufficient features from drone aerial images to identify small targets is a challenge. Drone aerial images present problems such as small target size, weak feature representation, and susceptibility to interference from complex backgrounds, making it difficult for current traditional object detection algorithms to meet practical requirements.

[0053] While advanced detectors like YOLOv11 perform excellently in general object detection, they still have significant shortcomings in small target scenarios captured by drones: First, traditional downsampling operations (such as stride convolution) cause severe loss of small target features in deep networks, making the target "disappear" from the feature map; second, the multi-scale fusion mechanism of Feature Pyramid Network (FPN) does not adequately utilize shallow detail features and lacks optimization design for small targets; finally, existing attention mechanisms mostly rely on a single scale or global context, making it difficult to dynamically focus on key regions of small targets. These problems directly lead to low detection accuracy and high false negative rates, making it difficult to meet the real-time and accuracy requirements of drone applications.

[0054] To address the aforementioned bottlenecks, this invention comprehensively enhances the network's ability to detect small targets by adding an Adaptive Frequency Sensing (AFS) module, a Progressive Spatial Preservation Downsampling (PSC) module, and a Multi-Scale Feature Fusion Upsampling (MFF) module to the network.

[0055] See Figure 1 This invention proposes a multi-scale frequency fusion method for detecting small targets in UAV aerial photography. In a one-stage algorithm model consisting of a backbone network, a neck network, and a detection head, an adaptive frequency sensing module is set at the beginning of the backbone network, one or more progressive spatial preservation downsampling modules are set in the downsampling layer of the backbone network, and a multi-scale feature fusion upsampling module is set in the upsampling part of the feature pyramid of the neck network to construct a multi-scale frequency fusion UAV aerial photography detection model.

[0056] A multi-scale frequency fusion UAV aerial photography detection model is trained, and the trained model is used for small target detection in UAV aerial photography.

[0057] The adaptive frequency sensing module includes parallel edge paths and frequency paths, which extract spatial edge features and frequency domain texture information, respectively. The edge path employs a learnable multi-directional adaptive Sobel operator to cover edge information from various angles. The frequency path includes parallel spatial and frequency domain branches. The frequency domain branch generates an adaptive frequency domain filter kernel to adaptively extract frequency domain features, while the spatial domain branch extracts spatial domain features. The output feature maps of the spatial and frequency domain branches are concatenated along the channel dimension. The outputs of the edge and frequency paths are fused using an attention mechanism to serve as the output of the adaptive frequency sensing module.

[0058] The progressive spatial preservation downsampling module includes a progressive downsampling chain composed of multiple SPDStep units based on spatial-to-depth convolution. It maximizes the preservation of spatial details through a multi-step spatial-to-depth transformation strategy, thereby maximizing the preservation of spatial detail information while compressing the feature map size.

[0059] The multi-scale feature fusion upsampling module includes a multi-branch feature extraction layer, a feature fusion layer, and an upsampling output layer. The multi-branch feature extraction layer has multiple branches, each focusing on feature extraction at a different scale. The feature fusion layer and the upsampling output layer fuse features at different scales and restore resolution, respectively.

[0060] The following is a detailed explanation of each part of the model.

[0061] 1. Construct a deep learning-based object detection network model

[0062] See Figure 1 The network consists of a backbone network, a neck network, and a detection head.

[0063] The backbone network, as the core component of feature extraction, adopts a deep convolutional neural network (CNN) structure and is mainly responsible for extracting multi-level features from the input image. It consists of multiple convolutional layers, pooling layers, and feature transformation modules, achieving deep extraction of image features through layer-by-layer abstraction.

[0064] The neck network employs a feature pyramid structure, undertaking the crucial functions of feature fusion and enhancement. This part achieves effective fusion of features at different scales through a bidirectional path of top-down and bottom-up. The neck network organically combines the shallow detail features extracted by the backbone network with deep semantic features, enhancing the network's adaptability to multi-scale targets through information interaction between different levels of the feature pyramid.

[0065] The detection head employs a multi-scale detection strategy, with separate detection heads set up for feature maps at different levels. Each detection head includes three sub-modules: bounding box regression, category classification, and confidence evaluation, enabling simultaneous target localization and recognition. Through a multi-scale collaborative detection mechanism, the detection head can effectively handle targets of different sizes.

[0066] 2. Adaptive Frequency Sensing Module

[0067] See Figure 2The adaptive frequency sensing module employs a dual-path parallel processing architecture, simultaneously extracting spatial edge features and frequency domain texture information to form complementary features and enhance the detailed features of small targets. This module is located at layer zero of the network, and its output features are passed to the progressive spatial preservative downsampling module in layer one for downsampling, providing an enhanced foundation of detailed information for subsequent feature extraction.

[0068] Edge paths enhance contour feature capture capabilities through multi-directional adaptive Sobel operators; the Sobel operator has multiple convolution kernels with different directions and parallel computation.

[0069] Specifically, as a key component of feature extraction, eight convolutional kernels in different directions are used for parallel computation to comprehensively cover edge information from all angles. The first two convolutional kernels are fixed in the horizontal and vertical directions of the traditional Sobel operator, while the remaining six kernels are initialized with random weights.

[0070] All elements of these convolutional kernels together constitute learnable parameters, and the direction selection is adaptively optimized during network training using backpropagation and gradient descent algorithms. This design maintains the stability of traditional edge detection while enhancing adaptability to complex scenes through a data-driven approach, thereby improving model robustness.

[0071] The frequency path is deployed in parallel with the edge path in the adaptive frequency sensing module, and is divided into two branches: the spatial domain branch and the frequency domain branch.

[0072] The frequency domain branch employs a dynamic convolution kernel generation mechanism to achieve adaptive extraction of frequency domain features. This branch obtains the frequency domain statistical features of the image through global average pooling, such as the channel mean (which reflects the overall energy distribution in the frequency domain); then, it generates dynamic weights for the convolution kernel through a fully connected network, and then performs weighted fusion with the basic Laplacian kernel to form an adaptive frequency domain filtering kernel; finally, it obtains the output feature map of the frequency domain branch through grouped convolution.

[0073] The spatial branch uses a combination of 3×3 convolution and ReLU activation function to extract spatial features.

[0074] The output feature maps of the spatial domain branch and the frequency domain branch are concatenated along the channel dimension to obtain the frequency path output feature map.

[0075] The convolution kernel weights are dynamically regenerated as follows:

[0076] (1)

[0077] (2)

[0078] in, x For the input feature map,f generate For weight generation network, GAP For global average pooling, W These are the dynamic weights of the convolutional kernels generated by the weight generation network. α For learnable equilibrium parameters, K base Based on the Laplace nucleus, K final This represents the final convolution kernel after weighted fusion, i.e., the adaptive frequency domain filtering kernel.

[0079] The dual-path features of edge path and frequency path are fused through an attention mechanism. First, the edge features and frequency features are concatenated along the channel dimension to form a feature tensor with double the number of channels.

[0080] Then, the weight distribution of each feature is calculated using an attention network;

[0081] The attention network employs a dual-path mechanism: the spatial attention path generates a spatial attention weight map through convolutional layers, while the channel attention path generates a channel attention weight map through global average pooling and fully connected layers; then, the input features, spatial attention weight map, and channel attention weight map are weighted and fused; the fused output is then processed by a 1×1 convolution (downsampling) and used as the final output of the adaptive frequency perception module.

[0082] The formulas for generating spatial attention weight maps, channel attention weight maps, and weighted fusion are as follows:

[0083] (3)

[0084] (4)

[0085] (5)

[0086] in x For the input feature map, Conv For convolution operations, ReLU It is the ReLU activation function. σ It is the Sigmoid activation function. W s Spatial attention weight map; FC For fully connected operation, W c This represents the channel attention weight map; ⊙ represents element-wise multiplication. x fused This is a weighted fusion weighted graph.

[0087] 3. Progressive Space-Preserving Downsampling Module

[0088] To address the issue of small target feature loss caused by traditional downsampling operations, this invention designs a progressive spatial preservation downsampling module, located in the downsampling layer of the backbone network, which can be configured in multiple ways.

[0089] like Figure 1 The network shown employs two progressive spatial preservation downsampling modules. The first progressive spatial preservation downsampling module (4x downsampling) takes the adaptive frequency sensing module as input and outputs the downsampled features. After feature fusion, the second progressive spatial preservation downsampling module (8x downsampling) is performed.

[0090] This module maximizes the preservation of spatial details through a multi-step space-to-depth transformation strategy, such as... Figure 3 As shown, this module consists of a progressive downsampling chain composed of multiple SPDStep units, combined with a residual connection mechanism, which maximizes the preservation of spatial detail information while compressing the feature map size.

[0091] To address feature loss during downsampling, this module introduces space-to-depth (SPD) convolutions instead of downsampling convolutions. For the input feature map... x During a single-step spatial transformation operation, the 2×2 adjacent regions are rearranged to the channel dimension:

[0092] (6)

[0093] in SPD () represents SPD convolution. x [::2,::2] indicates sampling every other pixel starting from the top left corner; after this transformation, the feature map size is reduced by half, and the number of channels becomes 4 times, achieving the downsampling effect while completely preserving spatial information; to avoid information loss caused by a large downsampling at once, this module adopts a multi-step progressive downsampling strategy; if the downsampling step number is set to k, then the total downsampling factor is 2k.

[0094] Each step of the downsampling process can be represented as:

[0095] (7)

[0096] in x i The input feature map is for a single-step spatial transformation. SPD For a single-step spatial transformation operation, the formula is shown in (6); Conv 3×3 () represents a convolution operation with a kernel size of 3×3. x i+1 This is the output feature map of a single-step spatial transformation.

[0097] To ensure effective gradient propagation and feature reuse, the module employs an intelligent residual connection mechanism. When the number or size of input / output channels changes, the residual path uses max pooling downsampling combined with 1×1 convolution to adjust the channels; when the size remains constant, identity mapping is directly used.

[0098] Specifically, when the sampling factor parameter of the progressive spatial retention downsampling module is... r When = 2 (i.e., when performing 2x downsampling), the residual path uses max pooling downsampling combined with 1×1 convolution for channel adjustment. r When = 1 (i.e., when downsampling is not required), the identity mapping is directly used to ensure the stability of network optimization.

[0099] Finally, the feature maps output by the downsampling path and the residual path are added element by element to obtain the final module's output feature map, as shown in equation (8):

[0100] (8)

[0101] in, x For the input feature map, x out This is the output feature map of the progressive downsampling chain. f shortcut This represents the output feature map of the residual branch. x final This represents the output feature map of the progressive spatially preserved downsampling module.

[0102] 4. Multi-scale feature fusion upsampling module

[0103] To address the issues of semantic inconsistency, loss of details, and low efficiency of multi-scale fusion in the feature pyramid upsampling process, this invention designs a multi-scale feature fusion upsampling module and adopts a multi-branch parallel architecture fusion mechanism.

[0104] Specifically, this module is located in the upsampling part from layer P3 to layer P2 in the Neck network; the input is a feature map, and the output is a feature map with twice the resolution, which provides richer contextual information and more accurate features for detailed texture restoration for subsequent detection, thereby improving detection accuracy.

[0105] like Figure 4 As shown, the module consists of three core parts: a multi-branch feature extraction layer, a feature fusion layer, and an upsampling output layer, forming a complete feature enhancement pipeline.

[0106] The multi-branch feature extraction layer employs a three-branch parallel structure, each focusing on feature extraction at different scales. The first branch uses a 1×1 convolutional kernel to capture local detail features; this operation preserves spatial details through point convolution, which is particularly beneficial for the accurate localization of small targets. The second branch uses a 3×3 standard convolution to extract contextual information, enhancing the perception of the target's surrounding environment by expanding the receptive field. The third branch utilizes dilated convolution (dilation=2) to obtain global semantic features, effectively addressing the feature sparsity problem for distant small targets. This multi-scale design ensures that the module can comprehensively cover feature information from local to global, avoiding feature degradation caused by upsampling in traditional single-path approaches.

[0107] In the feature fusion layer, this module achieves this through channel concatenation and convolution operations of multi-branch output features, as shown in Equation (9):

[0108] (9)

[0109] in x 1. x 2. x 3 are the output feature maps of the three branches of the multi-branch feature extraction layer. concat This is a channel-level splicing operation. Conv2d For convolution operations, BN For batch normalization operations, δ for SiLU Activation function x t This is the fused feature map.

[0110] Specifically, the multi-scale features extracted from the three branches are first concatenated along the channel dimension to form a fused feature with three times the number of channels as the original branches. This concatenation process allows features of different scales to interact directly at the channel level. The concatenated features are then subjected to dimensionality reduction and optimization through a 1×1 convolutional layer, which compresses the number of channels to the original input number, achieving feature fusion and information condensation.

[0111] Then, batch normalization (BatchNorm) and SiLU activation function operations are connected: batch normalization stabilizes the training process by standardizing the feature distribution, and the SiLU activation function (also known as the Swish function) introduces smooth nonlinearity to enhance the feature representation ability. The two work together to alleviate the gradient vanishing or exploding problem.

[0112] The fused features are upsampled in the upsampled output layer to restore resolution through bilinear interpolation. The feature map is then enlarged in the spatial dimension using a linear interpolation algorithm while maintaining edge smoothness. Finally, the output is further refined through a 3×3 convolutional layer to enhance spatial details and adjust the number of channels, ensuring that the upsampled features are compatible with subsequent layers of the network.

[0113] The entire process forms a stable gradient flow through multiple applications of batch normalization and the SiLU activation function (found in each branch and fusion layer), effectively improving the robustness of the module in complex scenarios.

[0114] like Figure 1 As shown, this invention constructs a multi-scale frequency fusion UAV aerial photography detection model based on the baseline model YOLOv11:

[0115] The backbone network includes an adaptive frequency sensing module and two progressive spatial preservation downsampling modules.

[0116] The adaptive frequency sensing module is located at the beginning of the backbone network, and the output of the adaptive frequency sensing module serves as the input of the first progressive spatial retention downsampling module.

[0117] The output of the first progressive spatial preservation downsampling module is the downsampled feature; after feature fusion, the output is sent to the second progressive spatial preservation downsampling module.

[0118] The multi-scale feature fusion upsampling module is located in the upsampling part from layer P3 to layer P2 in the neck network.

[0119] 5. Training the network model

[0120] Dataset selection: VisDrone2019 dataset

[0121] The VisDrone2019 dataset is a large-scale public benchmark dataset specifically designed for drone aerial vision tasks, constructed with full consideration of the uniqueness of the drone's perspective. It contains 10,209 still images (6,471 for training, 548 for validation, and 3,190 for testing) and 288 video sequences, totaling approximately 2.6 million bounding boxes. Image resolutions range from 960×540 to 4000×3000. The images were captured from various drone platforms in diverse scenes across 14 different cities in China, covering city streets, squares, transportation hubs, schools, etc., and include various weather and lighting conditions, reflecting the complexity of real-world drone operating environments. The annotation information includes 10 common categories: pedestrians, people, bicycles, cars, trucks, lorries, tricycles, awning-tricycles, buses, and motorcycles. These categories essentially cover the main target types in drone aerial photography scenes.

[0122] Model evaluation metrics: Metrics including mAP, Parameters, and GFLOPs should be used for model evaluation.

[0123] In order to quantify the effectiveness of the model and compare the merits of different models, a reasonable evaluation metric is needed.

[0124] mAP is the core accuracy metric in object detection, comprehensively reflecting the model's accuracy in object localization and classification. This metric first calculates the AP (Average Precision) for each class, which is the area under the precision-recall curve, and then averages the AP values ​​across all classes to obtain mAP.

[0125] The Parameters metric directly reflects the model's complexity and storage requirements, referring to the sum of all trainable parameters in the model. The number of parameters affects the model's deployment cost and memory usage, which is particularly important for drone platforms with limited computing resources.

[0126] GFLOPs measures the computational cost required during model inference and is a key metric for evaluating algorithm real-time performance. This metric calculates the number of floating-point operations required for the model to process a single image, directly impacting the algorithm's inference speed.

[0127] Training settings:

[0128] Both the base model and the improved model were trained on the VisDrone2019 dataset.

[0129] The specific training settings are as follows: use the stochastic gradient descent (SGD) optimizer, set the momentum parameter to 0.9, the weight decay coefficient to 1e-4, adjust the base learning rate to 0.01 and use a cosine annealing strategy for dynamic adjustment;

[0130] The batch size was set to 16, the total number of training iterations was 300, and an early stopping strategy was adopted (training was terminated when the performance on the validation set did not improve within 10 consecutive cycles) to avoid overfitting.

[0131] In terms of data augmentation, strategies such as random horizontal flipping, color jittering, and multi-scale scaling were applied to improve the model's robustness to changes in illumination and scale differences.

[0132] Regarding the loss function, this invention does not modify the loss function of the baseline model YOLOv11, but maintains its original multi-task loss function combination, including bounding box regression loss (using CIoU loss function), category classification loss (using binary cross-entropy loss) and confidence loss (using focus loss function), which are combined to form the total loss function through weighted summation. The weight parameters are set in the same way as the original YOLOv11 to ensure the stability and consistency of the training process.

[0133] 6. Conduct comparative experiments

[0134] After training, the model is evaluated on the validation set. The detection results of the model of this invention are shown in the figure below. Figure 5 As shown, Figure 6The results are for the baseline model (YOLOv11).

[0135] The comparison results show that the model of this invention successfully detected several small targets that the baseline model missed. For example, in the red box area in the upper right corner of the image, there are three distant pedestrian targets (occupying only about 10×15 pixels of the image). The model of this invention generated accurate bounding boxes and high-confidence detection results, while the baseline model did not recognize the target at all, proving that the model of this invention is significantly better than the baseline model in recognizing small targets in drone aerial images.

[0136] This invention incorporates an adaptive frequency sensing module, a progressive spatial preserving downsampling module, and a multi-scale feature fusion upsampling module into the baseline model YOLOv11, and compares and evaluates them against the baseline model. Training was performed for 300 iterations on the VisDrone2019 dataset using an early stopping strategy. The evaluation metrics were selected as map0.5, map0.5-0.95, parameters, and GFLOPs. The comparative experimental results are shown in Table 1.

[0137] Table 1 Comparison of Experimental Results

[0138]

[0139] The experimental results show that the baseline model's mAP50 and mAP50-95 are 0.381 and 0.228, respectively, while the improved model of this invention (AFS+PSC+MFF) increases to 0.412 and 0.25, respectively, representing a relative increase of 8.1% and 9.6%. This improvement verifies the effectiveness and advancement of this invention in addressing the challenge of small target detection in UAV aerial photography.

[0140] The embodiments given above are preferred examples for implementing the present invention, and the present invention is not limited to the above embodiments. Any non-essential additions or substitutions made by those skilled in the art based on the technical features of the present invention are within the protection scope of the present invention.

Claims

1. A multi-scale frequency fusion method for small target detection in UAV aerial photography, characterized in that, In a one-stage algorithm model consisting of a backbone network, a neck network, and a detection head, an adaptive frequency sensing module is set at the beginning of the backbone network, one or more progressive spatial preservation downsampling modules are set in the downsampling layer of the backbone network, and a multi-scale feature fusion upsampling module is set in the upsampling part from layer P3 to layer P2 in the neck network to construct a multi-scale frequency fusion UAV aerial photography detection model. A multi-scale frequency fusion UAV aerial photography detection model is trained, and the trained model is used for small target detection in UAV aerial photography. The adaptive frequency sensing module includes parallel edge paths and frequency paths, which extract spatial edge features and frequency domain texture information, respectively. The edge path employs a learnable multi-directional adaptive Sobel operator to cover edge information from various angles. The frequency path includes parallel spatial and frequency domain branches. The frequency domain branch generates an adaptive frequency domain filter kernel to adaptively extract frequency domain features, while the spatial domain branch extracts spatial domain features. The output feature maps of the spatial and frequency domain branches are concatenated along the channel dimension. The outputs of the edge and frequency paths are fused using an attention mechanism to serve as the output of the adaptive frequency sensing module. The progressive spatial preserving downsampling module includes a progressive downsampling chain composed of multiple SPDStep units based on spatial-to-depth convolution. This chain maximizes the preservation of spatial details through a multi-step spatial-to-depth transformation strategy, compressing the feature map size while maximizing the retention of spatial detail information. It also includes a residual connection path running parallel to the progressive downsampling chain. The residual connection path adjusts the channels using max-pooling downsampling combined with 1×1 convolution when the number or size of the input / output channels changes; when the size remains constant, it directly uses identity mapping. The multi-scale feature fusion upsampling module includes a multi-branch feature extraction layer, a feature fusion layer, and an upsampling output layer. The multi-branch feature extraction layer has multiple branches, each focusing on feature extraction at a different scale. The feature fusion layer and the upsampling output layer fuse features at different scales and restore resolution, respectively.

2. The multi-scale frequency fusion method for small target detection in UAV aerial photography as described in claim 1, characterized in that, The multi-directional adaptive Sobel operator is set at the beginning of the edge path and has multiple convolution kernels with different directions and parallel computation. The convolution kernels include horizontal and vertical convolution kernels, and the remaining convolution kernels are initialized with initial weights using random initialization. All elements of these convolution kernels together constitute the learnable parameters, and the direction selection is adaptively optimized during network training using backpropagation and gradient descent algorithms.

3. The multi-scale frequency fusion method for small target detection in UAV aerial photography as described in claim 1, characterized in that, In the frequency domain branch, the adaptive frequency domain filter kernel is generated as follows: The frequency domain branch obtains the frequency domain statistical features of the image through global average pooling, then generates dynamic weights for the convolution kernel through a fully connected network, and then performs weighted fusion with the basic Laplacian kernel to form an adaptive frequency domain filtering kernel. Finally, the output feature map of the frequency domain branch is obtained through grouped convolution. The dynamic weights of the convolution kernel are generated as follows: （1） （2） in, x Input feature map ，f generate For weight generation network, GAP For global average pooling, W These are the dynamic weights of the convolutional kernels generated by the weight generation network. α For learnable equilibrium parameters, K base Based on the Laplace nucleus, K final This represents the adaptive frequency domain filter kernel after weighted fusion; The spatial branch uses a combination of 3×3 convolution and ReLU activation function to extract spatial features; The output feature maps of the spatial domain branch and the frequency domain branch are concatenated along the channel dimension to obtain the output feature map of the frequency path.

4. The multi-scale frequency fusion method for small target detection in UAV aerial photography as described in claim 1, characterized in that, The outputs of the edge path and the frequency path are fused together using an attention mechanism to obtain: First, the edge features and frequency features are concatenated along the channel dimension to form a feature tensor with double the number of channels; Then, the weight distribution of each feature is calculated through an attention network that includes spatial attention path and channel attention path. The spatial attention path generates a spatial attention weight map through convolutional layers, and the channel attention path generates a channel attention weight map through global average pooling and fully connected layers. Then, the input features, spatial attention weight map, and channel attention weight map are multiplied element-wise and weighted and fused together. The weighted fusion output is then processed by a 1x1 convolution to become the final output of the adaptive frequency sensing module.

5. The multi-scale frequency fusion method for small target detection in UAV aerial photography as described in claim 4, characterized in that, The spatial attention weight map generation, channel attention weight map generation, and weighted fusion are respectively: （3） （4） （5） in, x For the input feature map, Conv For convolution operations, ReLU It is the ReLU activation function. σ It is the Sigmoid activation function. W s Spatial attention weight map; FC For fully connected operation, W c This represents the channel attention weight map; ⊙ represents element-wise multiplication. x fused This is a weighted fusion weighted graph.

6. The multi-scale frequency fusion method for small target detection in UAV aerial photography as described in claim 1, characterized in that, The progressive downsampling chain of the progressive spatially preserved downsampling module replaces the downsampling convolution with a spatial-to-depth convolution; for the input feature map x During a single-step spatial transformation operation, the 2×2 adjacent regions are rearranged to the channel dimension: （6） Where SPD() represents the spatial-to-depth convolution operation. x [::2,::2] means sampling every other pixel starting from the top left corner. After this transformation, the feature map size is reduced by half and the number of channels becomes four times. Each step of the downsampling process is represented as follows: （7） in x i The input feature map is for a single-step spatial transformation. SPD This is a single-step spatial transformation operation; Conv 3×3 () represents a convolution operation with a kernel size of 3×3. x i+1 This is the output feature map of a single-step spatial transformation.

7. The multi-scale frequency fusion method for small target detection in UAV aerial photography as described in claim 1 or 6, characterized in that, The output feature map of the progressive spatially preserved downsampling module is obtained by element-wise addition of the output feature map of the progressive downsampling chain path and the output feature map of the residual connection path. （8） in, x For the input feature map, x out This is the output feature map of the progressive downsampling chain. f shortcut This is the output feature map of the residual path. x final The output feature map of the downsampling module is preserved for progressive spatial retention.

8. The multi-scale frequency fusion method for small target detection in UAV aerial photography as described in claim 1, characterized in that, In the multi-branch feature extraction layer, the first branch uses a 1×1 convolutional kernel to capture local detail features; the second branch uses a 3×3 standard convolution to extract contextual information; and the third branch uses dilated convolution to obtain global semantic features. The features extracted by the multi-branch feature extraction layer are fused together in the feature fusion layer as follows: The multi-scale features extracted from the three branches are spliced ​​along the channel dimension. The splicing process allows features of different scales to interact directly at the channel level, forming a fusion feature with three times the number of channels of the original branches. The concatenated features are then subjected to dimensionality reduction and optimization through a 1×1 convolutional layer, followed by batch normalization and SiLU activation function operations: （9） in x 1. x 2. x 3 are the output feature maps of the three branches of the multi-branch feature extraction layer. concat This is a channel-level splicing operation. Conv2d For convolution operations, BN For batch normalization operations, δ for SiLU Activation function x t The fused feature map; The fused features are upsampled in the upsampled output layer using bilinear interpolation to restore the resolution; Finally, the output is further refined through a 3×3 convolutional layer to obtain the output of the multi-scale feature fusion upsampling module.

9. The multi-scale frequency fusion method for small target detection in UAV aerial photography as described in claim 1, characterized in that, Training the multi-scale frequency fusion drone aerial photography detection model includes: Using the VisDrone2019 dataset; The model evaluation metrics include mAP, Parameters, and GFLOPs. Training settings: Stochastic gradient descent optimizer was used, with momentum parameter set to 0.9, weight decay coefficient set to 1e-4, base learning rate adjusted to 0.01 and dynamically adjusted using cosine annealing; batch size was set to 16, total training iterations were 300, and an early stopping strategy was adopted. And it employs strategies including random horizontal flipping, color jitter, and multi-scale scaling for data augmentation; The loss function is the same as the loss function of the baseline model YOLOv11, and the total loss function is formed by weighted summation, where the weight parameters are the same as those of the original YOLOv11.

10. The multi-scale frequency fusion method for small target detection in UAV aerial photography as described in claim 1, characterized in that, The backbone network includes one adaptive frequency sensing module and two progressive spatial retention downsampling modules. The adaptive frequency sensing module is located at the beginning of the backbone network, and the output of the adaptive frequency sensing module serves as the input of the first progressive spatial retention downsampling module. The output of the first progressive spatially preserved downsampling module is the downsampled feature; After feature fusion, the output is sent to the second progressive spatially preserved downsampling module; The multi-scale feature fusion upsampling module is located in the upsampling part from layer P3 to layer P2 in the neck network.

Images