Unmanned aerial vehicle small target detection method based on frequency domain enhancement and spatial path refinement

Abstract

The application relates to the field of artificial intelligence and remote sensing technology, in particular to a UAV small target detection method based on frequency domain enhancement and spatial path refinement, which comprises the following steps: acquiring and processing a UAV image small target dataset; constructing a three-branch network MFSNet, which comprises a first branch composed of a backbone network; a second branch composed of the backbone network and a multi-scale frequency domain enhancement module MFEM; and a third branch composed of the backbone network, the MFEM and a spatial path refinement enhancement module SPPRN; wherein the MFEM further comprises a plurality of parallel branches, each branch comprising an ASPP, an FFT module and an adaptive feature enhancement module AFE connected in sequence; the SPPRN is constructed based on a top-down and bottom-up bidirectional feature fusion structure, a dynamic position encoding and a down-sampling module; the MFSNet is trained based on the UAV image small target dataset to obtain a target detection model, which is used for UAV small target detection.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and remote sensing technology, and more specifically, to a method for detecting small targets on unmanned aerial vehicles (UAVs) based on frequency domain enhancement and spatial path refinement. Background Technology

[0002] Small target detection in UAV images has crucial application value in fields such as disaster monitoring, power line inspection, and environmental surveys. Compared with natural scene images, UAV images typically feature complex backgrounds, drastic changes in target scale, and extremely low pixel proportions but dense distribution of small targets, making it difficult for general target detection algorithms to achieve ideal results directly. Therefore, designing detection algorithms specifically for small targets in remote sensing has become a research hotspot in the fields of computer vision and remote sensing.

[0003] Early small object detection methods primarily relied on hand-designed features, such as histograms of oriented gradients (HARQs) and Haar features, combined with sliding windows or image pyramids for searching. However, these methods had weak generalization ability and high computational complexity, making them difficult to handle the diversity of real-world scenes. With the rapid development of deep learning, object detection algorithms based on convolutional neural networks, such as Faster R-CNN and the YOLO series, have made breakthrough progress. Although these general detectors perform well on regular objects, their accuracy in detecting small objects remains unsatisfactory. This is mainly because after multiple layers of downsampling, the feature information of small objects is almost completely lost in the deeper layers of the network.

[0004] To address these issues, researchers have proposed improvement strategies from multiple perspectives, with multi-scale feature fusion being the most effective and mainstream approach. For instance, feature pyramid networks, by fusing high-level semantic features with low-level detail features, significantly improve the detection performance of small targets, becoming the cornerstone of subsequent multi-scale detectors. Building upon this, path aggregation networks further shorten the information transmission path and improve the accuracy of feature localization by adding bottom-up path enhancements.

[0005] In recent years, more efficient structures such as bidirectional feature pyramid networks have been proposed, continuously optimizing the efficiency and effectiveness of feature fusion. Besides network structure optimization, other technical strategies have also driven the development of small target detection. At the data level, data augmentation to increase the number of small target samples is an effective means of alleviating sample imbalance. At the algorithm level, the introduction of FocalLoss effectively solves the problem of imbalance between positive and negative samples and between easy and difficult samples, allowing the model to focus more on learning difficult small target samples. Furthermore, introducing attention mechanisms can enable the model to focus on the target region and suppress background noise; while context-aware modules provide additional clues for small target recognition by capturing the relationship between the target and its surrounding environment.

[0006] Despite significant progress, small object detection still faces three core challenges: First, small object features are easily lost in deep networks, requiring further optimization of feature enhancement and preservation mechanisms. Second, occlusion and dense distribution issues in complex scenes necessitate more effective context modeling and more refined multi-scale feature fusion mechanisms. Third, the conflict between lightweight models and real-time performance requirements is prominent. Therefore, achieving a balance between robust detection of extremely small objects and real-time performance remains a challenging problem to be solved in the future. Summary of the Invention

[0007] In view of this, this application provides a method for detecting small targets on unmanned aerial vehicles based on frequency domain enhancement and spatial path refinement to solve the above problems.

[0008] The technical solution provided in this application is as follows:

[0009] A method for detecting small targets on unmanned aerial vehicles (UAVs) based on frequency domain enhancement and spatial path refinement includes:

[0010] Acquire and process drone image small target datasets;

[0011] A three-branch network MFSNet is constructed, comprising a first branch, a second branch, and a third branch with residual connections; the first branch includes a backbone network; the second branch includes the backbone network and a multi-scale frequency domain enhancement module (MFEM); and the third branch includes the backbone network, the MFEM, and a spatial path refinement enhancement module (SPPRN).

[0012] The MFEM further includes multiple parallel branches, each branch comprising a connected Spatial Pyramid Pooling (ASPP) module, a Fast Fourier Transform (FFT) module, and an Adaptive Feature Enhancement (AFE) module. The outputs of each branch are fused and then enhanced in the multi-scale frequency domain through inverse Fast Fourier Transform and average pooling operations. The SPPRN is constructed based on a top-down and bottom-up bidirectional feature fusion structure, dynamic position encoding, and downsampling modules, and is used to enhance the spatial features of the feature maps at multiple levels output by the backbone network.

[0013] The three-branch network MFSNet is trained based on the UAV image small target dataset to obtain a target detection model for UAV small target detection.

[0014] In one possible implementation, each branch of the MFEM takes the feature maps of each level extracted by the backbone network as input to extract the amplitude spectrum corresponding to the feature maps and enhance the amplitude spectrum; wherein, the method for extracting the amplitude spectrum includes:

[0015] The feature maps of each level are input into the dilated spatial pyramid pooling (ASPP) to generate feature representations of multiple scales based on dilated convolutions with different dilation rates, thereby capturing target context information of different sizes.

[0016] Based on the Fast Fourier Transform (FFT) module, the feature representations at each scale are transformed from the spatial domain to the frequency domain to obtain the corresponding amplitude spectrum.

[0017] In one possible implementation, the adaptive feature enhancement module (AFE) takes the amplitude spectrum output by the FFT module as input and performs a global average pooling operation on the amplitude spectrum to compress the spatial information of each channel into a global descriptor, which is used to capture the average energy distribution of each feature channel in the spatial dimension; based on the global descriptor, a spatially adaptive frequency weight mask and enhancement factor are generated to enhance the amplitude spectrum.

[0018] In one possible implementation, the outputs of each branch of the MFEM include a phase spectrum and an enhanced amplitude spectrum enhanced by the adaptive feature enhancement module (AFE).

[0019] The outputs of the various branches of the MFEM are fused, including:

[0020] The enhanced amplitude spectra obtained by dilated convolutions with different dilation rates are spliced ​​together along the channel dimension to obtain a multi-scale amplitude spectrum.

[0021] Feature fusion is performed on the concatenated multi-scale amplitude spectrum based on 1*1 convolution;

[0022] The fused multiscale amplitude spectrum is split into multiple independent single-scale amplitude spectra according to the number of channels;

[0023] The single-scale amplitude spectrum and the phase spectrum are recombined to reconstruct the frequency domain representation of the feature map.

[0024] In one possible implementation, the SPPRN is used to unify the feature maps of the four layers output by the backbone network to the same number of channels through 1*1 convolution, thereby obtaining channel-aligned features. ;

[0025] For features after channel alignment Based on a top-down feature fusion structure, through upsampling and feature concatenation, high-level semantic information is gradually passed down to lower levels to obtain hierarchical features along the top-down path. ;

[0026] The hierarchical features of the top-down path The features are refined sequentially from bottom to top and from top to bottom to obtain the i-th level features after upsampling. ;

[0027] Fusion of the features corresponding to the lowest level and hierarchical features The second level corresponding to the bottom-up path This yields the fusion of multi-level features corresponding to the lowest level. .

[0028] In one possible implementation, the lowest level corresponding to... Subsequently, the SPPRN is also used to generate dynamic positional codes through depthwise separable convolutions; based on the dynamic positional codes, the lowest-level spatial feature enhancement output with added positional codes is generated.

[0029] In one possible implementation, the downsampling module of the SPPRN is used to generate spatial feature enhancement outputs at each level by progressively downsampling based on the lowest-level spatial feature enhancement output after adding position encoding, thereby constructing a feature pyramid.

[0030] In one possible implementation, the three-branch network MFSNet further includes a detection head. The outputs of the first, second, and third branches are input into the detection head after residual fusion, and are used to output the location coordinates and category information of the detected target.

[0031] This application proposes a UAV small target detection method based on multi-level frequency domain enhancement and spatial path refinement. It utilizes a dual-path architecture to collaboratively process multi-scale features: the frequency domain enhanced path (MFEM) employs an adaptive frequency enhancement mechanism to strengthen the feature representation of small targets in the frequency domain; the spatial path (SPPRN) maintains accurate localization capabilities through progressive refinement; and finally, a residual fusion module integrates and preserves the feature advantages of all three paths by combining the multi-level features extracted from the backbone network. This improves the detection performance of small targets while maintaining computational efficiency. Specifically, compared with existing technologies, the beneficial effects of this application's technical solution are as follows:

[0032] (1) To address the issue of small target features being easily lost in deep networks, we propose a multi-scale frequency domain enhancement module (MFEM). This module captures contextual information from different receptive fields through multi-scale dilated convolutions and performs adaptive frequency enhancement in the frequency domain, effectively improving the model's ability to represent features of small targets. Specifically, we designed a cross-scale frequency domain fusion mechanism to fuse the features of dilated convolution paths with different dilation rates in the frequency domain, thereby enhancing the diversity and discriminativeness of features.

[0033] (2) To address issues such as occlusion and dense distribution in complex scenes, a Spatial Path and Progressive Refinement Module (SPPRN) was designed. This module enhances spatial detail information and improves the multi-scale consistency of the feature pyramid through bidirectional feature fusion from top to bottom and bottom to top, combined with dynamic position encoding. The progressive refinement mechanism optimizes feature representation step by step through multiple upsampling and downsampling iterations, which is particularly beneficial for the accurate regression of the position of small targets.

[0034] (3) An improved adaptive frequency enhancement mechanism is introduced. This application abandons the traditional fixed enhancement factor method and designs a learnable adaptive frequency enhancement mechanism. The enhancement intensity of each frequency component is dynamically learned through the global context of the network, avoiding the tedious manual parameter tuning and improving the model's generalization ability to different datasets.

[0035] (4) A backbone-frequency domain-spatial three-path fusion architecture was constructed. This application innovatively integrates the original features of the backbone network, the frequency domain enhanced features and the spatial refinement features through residual connection and adaptive weights, making full use of the advantages of different feature representations to form a more powerful multi-scale feature representation, which significantly improves the accuracy of small target detection. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the structure of a three-branch network MFSNet provided in this application.

[0037] Figure 2 This is a schematic diagram of the structure of a multi-scale frequency domain enhancement module (MFEM) provided in this application.

[0038] Figure 3 This is a schematic diagram of the adaptive feature enhancement module (AFE) provided in this application.

[0039] Figure 4 A schematic diagram of the Spatial Path Refinement Enhancement Module SPPRN provided in this application.

[0040] Figure 5 This is a flowchart of a UAV small target detection method based on multi-level frequency domain enhancement and spatial path refinement, provided in Embodiment 2 of this application.

[0041] Figure 6 The results are shown below. The technical solution of this application was tested on the small target dataset AI-TODv2.

[0042] Figure 7 The results are shown below, based on the test results of the technical solution of this application on the PLADv4 dataset of small targets powered by unmanned aerial vehicles. Detailed Implementation

[0043] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0044] To address the inherent misalignment between high-resolution spatial details and deep semantic information in existing technologies, this application proposes a UAV small target detection method based on multi-level frequency domain enhancement and spatial path refinement. Considering that the synergistic integration of the frequency and spatial domains across multiple scales is crucial for robust small target detection, this application designs a three-branch network. The input image is processed by a backbone network to extract four-level feature pyramids. Then, the four-level feature maps are processed separately by frequency enhancement and spatial refinement branches. The frequency enhancement path aims to perform adaptive frequency component selection and enhancement at multiple scales, effectively amplifying the discriminative features of small targets while suppressing high-frequency background noise. The spatial refinement path employs a progressive refinement mechanism to preserve and enhance high-resolution spatial details, ensuring accurate localization. Finally, a residual fusion module effectively fuses the original features of the backbone network, the frequency domain features enhanced by MFEM, and the spatial refinement features of the SP path, fully utilizing the advantages of these three different feature representations.

[0045] Specifically, see Figure 1 , Figure 1 This is a schematic diagram of the structure of a three-branch network, MFSNet, provided in this application. Figure 1 As shown, the three branches include a first branch formed by the backbone network, a second branch composed of the backbone network and a multi-scale frequency enhancement module (MFEM), and a third branch composed of the backbone network, the MFEM, and a spatial path progressive refinement network (SPPRN). These three branches are connected using a residual connection method.

[0046] like Figure 1 As shown, the three-branch network MFSNet also includes a detection head. The outputs of the first, second, and third branches are fed into the detection head after residual fusion. Based on the three-branch network structure, this application, while preserving the original feature information, introduces complementary information from frequency domain enhancement features and spatial refinement features, thereby generating a more discriminative and balanced detection feature representation.

[0047] The structure and function of the newly added multi-scale frequency domain enhancement module MFEM and spatial path refinement enhancement module SPPRN in this application will be described in detail below.

[0048] The aforementioned multi-scale frequency domain enhancement module (MFEM) captures contextual information from different receptive fields through multi-scale dilated convolutions and performs adaptive frequency enhancement in the frequency domain. This effectively improves the model's ability to represent features of small targets and addresses the problem of small target features being easily lost in deep networks. Through a cross-scale frequency domain fusion mechanism, features from dilated convolution paths with different dilation rates are fused in the frequency domain, thereby enhancing feature diversity and discriminative power.

[0049] See Figure 2 This is a schematic diagram of the structure of a multi-scale frequency domain enhancement module (MFEM) provided in this application. Figure 2 As shown, the aforementioned multi-scale frequency domain enhancement module (MFEM) comprises multiple parallel branches. Each branch includes a connected spatial pyramid pooling (ASPP) module, a fast Fourier transform (FFT) module, and an adaptive feature enhancement (AFE) module. The outputs of each branch are fused and then subjected to multi-scale frequency domain enhancement through inverse fast Fourier transform and average pooling operations.

[0050] Specifically, the feature map of the i-th level extracted from the backbone network is defined as follows: This approach utilizes a set of parallel convolutional layers to generate multi-scale feature representations, capturing contextual information about targets of different sizes. It efficiently generates multi-scale features based on dilated convolutions with varying dilation rates without reducing spatial resolution, as shown below:

[0051]

[0052] Where s represents the dilation rate of the dilated convolution, which controls the size of the receptive field. This represents the feature map at the s-th scale.

[0053] Subsequently, for the feature map at each scale, a Fast Fourier Transform is applied to transform it from the spatial domain to the frequency domain, as shown below:

[0054]

[0055] in, The 2D Fast Fourier Transform yields a frequency domain representation that includes the amplitude spectrum. | and phase spectrum Amplitude spectrum | | Characterizes the energy distribution of different frequency components in the image, while the phase spectrum Structural location information was preserved.

[0056] Furthermore, this application adaptively modulates the amplitude spectrum at each scale. A learnable frequency weight mask is introduced.

[0057] code (R is a natural number). This mask is dynamically generated using global context information to distinguish the frequency response of the target from the background. The adaptive enhancement module abandons the limitations of traditional methods that rely on manually setting enhancement factors, and dynamically learns the required frequency enhancement intensity for each feature location through a data-driven approach. This mechanism can automatically adjust the enhancement strategy according to the statistical characteristics of the input features, realizing a shift from "manual parameter tuning" to "network self-learning," significantly improving the model's adaptability to different datasets and scenarios.

[0058] See Figure 3 , Figure 3 This is a schematic diagram of the adaptive feature enhancement module (AFE) provided in this application. Figure 3 As shown, the Adaptive Feature Enhancement (AFE) module enhances the amplitude spectrum of the input. Perform global average pooling to compress the spatial information of each channel into a global statistic, capturing the average energy distribution of each feature channel across the entire spatial dimension, thus providing global contextual information for subsequent augmentation decisions. Specifically, this is represented as:

[0059]

[0060] In the formula, This indicates global average pooling.

[0061] A spatially adaptive frequency weight mask is generated using a lightweight attention network. First, the global descriptor undergoes dimensionality reduction and a non-linear transformation. Then, it is upsampled to recover the original spatial size. Finally, the weights are constrained between 0 and 1 using a sigmoid function. This mask identifies the sensitivity of different spatial locations to frequency enhancement. Specifically, it is represented as follows:

[0062]

[0063]

[0064] In the formula, Representing intermediate features, after passing through fully connected layers and activation. , These represent the weights and biases of the first fully connected layer, respectively. , These represent the weights and biases of the second fully connected layer, respectively. This represents the spatially adaptive frequency weight mask for the i-th layer.

[0065] This approach dynamically generates enhancement factors from global descriptors using an independent, learnable network, replacing the traditional method of manually setting hyperparameters. The Softplus activation function ensures that the enhancement factors are always positive, avoiding gradient instability while maintaining a reasonable range for enhancement strength. Specifically:

[0066]

[0067] In the formula, This represents the adaptive enhancement factor, where the enhancement strength is learned independently for each channel. These represent the weights and biases of the enhancement factor generation network, respectively. This represents the activation function, ensuring that the output is always positive.

[0068] The above adaptive enhancement factors Combined with frequency masking, the original amplitude spectrum is enhanced. The enhancement formula employs... The multiplicative form ensures that the enhanced amplitude spectrum does not exhibit amplitude over-enhancement. This form maintains the relative closure of the original features while achieving significant enhancement in important regions. Specifically, the enhanced amplitude spectrum... Represented as:

[0069]

[0070] In the formula, This represents the enhanced amplitude spectrum. This indicates element-wise multiplication, and 1 represents the identity matrix. This indicates adaptive enhancement strength.

[0071] Furthermore, the enhanced amplitude spectra at the three scales are fused and then combined with the original phase spectrum to reconstruct spatial features via inverse Fourier transform. Specifically,

[0072] The enhanced amplitude spectra obtained from three dilated convolutional paths with different dilation rates (s=1,3,5) are concatenated along the channel dimension. The amplitude spectrum at each scale has undergone adaptive frequency enhancement processing, incorporating frequency information from different receptive fields. The result of splicing the amplitude spectra at three scales is specifically represented as follows:

[0073]

[0074] In the formula, This represents the enhancement amplitude spectrum when the expansion rate is 1. This represents the enhancement amplitude spectrum when the expansion rate is 3. This represents the enhancement amplitude spectrum when the expansion rate is 5. This represents the spliced ​​multiscale amplitude spectrum.

[0075] Use 1x1 convolution to stitch together the multi-scale amplitude spectrum Feature fusion is performed to enable information exchange between channels, allowing frequency features at different scales to complement and enhance each other. The fused multi-scale amplitude spectrum is represented as follows:

[0076]

[0077] The fused feature map is split into three independent scale features according to the number of channels. Each feature map contains frequency features enhanced with cross-scale information. The split single-scale amplitude spectrum is represented as follows:

[0078]

[0079] in, This means that the feature map is evenly split according to the channels. Each feature at the split scale is no longer independent, but is integrated with the contextual information of other scales, maintains the spatial structure of the original scale, and enhances the discriminative ability of the features.

[0080] The fused amplitude spectrum is recombine with the original phase spectrum to reconstruct a complex frequency domain representation. The phase spectrum retains the original spatial structure information, while the amplitude spectrum contains the enhanced frequency energy distribution. The frequency domain representation of the reconstructed feature map is shown below. , represented as:

[0081]

[0082] in, Represents the single-scale amplitude spectrum. Represents the phase spectrum. Indicates the imaginary part.

[0083] Based on the reconstructed frequency domain representation, the enhanced frequency domain features are transformed back to the spatial domain using inverse Fourier transform, resulting in a spatial feature map containing rich frequency information. , represented as:

[0084] =

[0085] in, This represents the inverse Fourier transform.

[0086] The reconstructed features at the three scales are averaged and pooled to fuse spatial features from different receptive fields, resulting in the final multi-scale frequency domain enhancement output. , represented as:

[0087]

[0088] The cross-scale fusion and reconstruction mechanism described above in this application enables MFSNet to effectively improve the detection performance of small targets, achieving an organic combination of frequency domain enhancement and spatial features.

[0089] To compensate for the potential loss of spatial detail caused by frequency domain paths and to ensure accurate localization of small targets, this application designs a spatial path based on a progressively refined network (SPPRN). This path maximizes the preservation of original high-resolution spatial information through multi-stage backbone network feature reuse and iterative optimization, solving the problem of wasted spatial information caused by shallow features being used only once in traditional FPN.

[0090] Specifically, such as Figure 4 As shown, the aforementioned Spatial Path Refinement Enhancement Module SPRN is constructed based on a top-down and bottom-up bidirectional feature fusion structure, dynamic position encoding, and downsampling module. It is used to enhance the spatial features of feature maps at multiple levels output by the backbone network.

[0091] In one specific implementation, the four-level feature maps output by the backbone network are first unified to the same number of channels using a 1×1 convolution, resulting in channel-aligned features. This ensures that features at different levels have consistent channel dimensions in subsequent processing, facilitating feature fusion. This application uses the channel-aligned features as { , For example,}

[0092] Starting from the highest level (P5), semantic information from higher levels is gradually transferred to lower levels through upsampling and feature concatenation, thereby enhancing the semantic expressive power of lower-level features. This represents the Pi-level features of the top-down path.

[0093] =

[0094] ( )

[0095] ( )

[0096] ( )

[0097] Building upon top-down fusion, bidirectional feature refinement is performed from both bottom-up and top-down perspectives to enhance feature consistency and expressive power. Specifically, the bottom-up propagation starts from the lowest level P2, and through downsampling and feature concatenation, detailed information is passed to higher levels. This represents the Pi layer features that are passed from bottom to top.

[0098]

[0099]

[0100]

[0101] The top-down propagation, starting again from the highest layer, upsamples and propagates the features enhanced by the bottom-up propagation, resulting in... ;in, Represented as:

[0102]

[0103]

[0104]

[0105] in, This indicates the features after channel alignment. This indicates the fusion of multi-level features, which incorporates... , and Features from three sources are used to maximize the preservation of original high-resolution spatial information through multi-stage backbone network feature reuse and iterative optimization. .

[0106] Furthermore, we obtain the corresponding lowest level. Subsequently, the aforementioned SPPRN module is also used to generate dynamic positional codes through depthwise separable convolution. Based on these dynamic positional codes, the lowest-level spatial feature enhancement output with added positional codes is generated. Specifically, this application generates dynamic positional codes through depthwise separable convolution to enhance the spatial positional sensitivity of features and achieve the target detection task. Specifically, this is expressed as follows:

[0107]

[0108]

[0109] in, () indicates depthwise separable convolution. This indicates dynamic location encoding, which captures spatial location relationships. This represents the positional coding weight coefficient. This represents the second-level spatial feature enhancement output after adding positional encoding.

[0110] Based on the refined P2 features, spatial feature enhancement outputs at other levels are generated through progressive downsampling, constructing a complete feature pyramid. The spatial feature enhancement outputs at each level are represented as follows:

[0111]

[0112]

[0113]

[0114] The aforementioned Spatial Path Refinement Enhancement (SPPRN) module enhances spatial detail information and improves the multi-scale consistency of the feature pyramid through bidirectional feature fusion from top to bottom and bottom to top, combined with dynamic position encoding. Through a progressive refinement mechanism, multiple upsampling and downsampling iterations gradually optimize feature representation, which is particularly beneficial for the accurate regression of small targets and effectively solves the problems of occlusion and dense distribution in complex scenes. Simultaneously, the spatial path and progressive refinement module, complementing the frequency domain path, provide MFSNet with powerful spatial feature representation capabilities, especially excelling in target localization and detail preservation.

[0115] Furthermore, such as Figure 1 As shown, the three-branch network MFSNet also includes a detection head. The detection head includes a first convolutional unit and a second convolutional unit connected in series. The first convolutional unit uses a 3×3 convolutional kernel (Conv 3×3), and the second convolutional unit uses a 3×3 deformable convolutional kernel (DCN 3×3). Figure 1 In this process, the output of the second convolutional unit is also connected to parallel classification and regression branches. The outputs of the first, second, and third branches are then fed into the detection head after residual fusion to output the target location coordinates and category information.

[0116] The UAV small target detection method based on the above target detection model, provided in this application, will be described in detail below through specific embodiments.

[0117] Example 2

[0118] See Figure 5 This is a flowchart of a UAV small target detection method based on multi-level frequency domain enhancement and spatial path refinement, provided in Embodiment 2 of this application. Figure 5 As shown, the specific implementation steps of the above method include:

[0119] Step 101: Obtain and process the UAV image small target dataset, and divide the dataset.

[0120] In this embodiment of the application, two datasets, AI-TODv2 and PLADv4, were selected as experimental data. AI-TODv2 is a dataset for detecting small targets, and PLADv4 is a publicly available dataset of drone aerial images that includes multi-scale power line targets.

[0121] Step 102: Construct the three-branch network MFSNet.

[0122] The three-branch network MFSNet comprises a first branch, a second branch, and a third branch, all with residual connections. The first branch includes a backbone network; the second branch includes a backbone network and a multi-scale frequency domain enhancement module (MFEM); and the third branch includes a backbone network, MFEM, and a spatial path refinement enhancement module (SPPRN). The MFEM further includes multiple parallel branches, each comprising a connected spatial pyramid pooling (ASPP) module, a fast Fourier transform (FFT) module, and an adaptive feature enhancement module (AFE). The outputs of each branch are fused and then subjected to multi-scale frequency domain enhancement through inverse fast Fourier transform and average pooling operations. The SPPRN is constructed based on a top-down and bottom-up bidirectional feature fusion structure, dynamic position encoding, and downsampling modules, used to spatially enhance the feature maps from multiple levels output by the backbone network.

[0123] The specific network structure can be referred to in Embodiment 1, and will not be described in detail in the embodiments of this application.

[0124] In this embodiment, ResNet-50 is used as the backbone network. The input image passes through the backbone network to extract a four-level feature pyramid. The feature map of the i-th level extracted from the backbone network is set as... .

[0125] Step 103: Perform multi-level, multi-scale frequency enhancement on the input image. This mainly includes three steps: multi-scale frequency domain transformation, adaptive frequency selection and enhancement, and cross-scale frequency domain feature interaction and reconstruction.

[0126] The aforementioned multi-scale frequency domain transformation includes generating multi-scale feature representations based on a set of parallel convolutional layers to capture target context information of different sizes. In one specific implementation, each branch of the MFEM takes the feature maps extracted from each level of the backbone network as input to extract the amplitude spectrum corresponding to the feature maps and enhance the amplitude spectrum.

[0127] The methods for extracting the amplitude spectrum include:

[0128] The feature maps at each level are input into the aforementioned dilated spatial pyramid pooling (ASPP) module. Based on dilated convolutions with different dilation rates, feature representations at multiple scales are generated to capture target context information of different sizes. Furthermore, based on the aforementioned Fast Fourier Transform (FFT) module, the feature representations at each scale are transformed from the spatial domain to the frequency domain to obtain the corresponding amplitude spectra.

[0129] The aforementioned adaptive frequency selection and enhancement steps include adaptive modulation of the amplitude spectrum at each scale. Specifically, the adaptive feature enhancement module (AFE) takes the amplitude spectrum output by the FFT module as input and performs global average pooling on it, compressing the spatial information of each channel into a global descriptor to capture the average energy distribution of each feature channel in the spatial dimension. Based on this global descriptor, a spatially adaptive frequency weight mask and enhancement factor are generated to enhance the amplitude spectrum.

[0130] The above-mentioned cross-scale frequency domain feature interaction and reconstruction steps include fusing the enhanced amplitude spectra of the three scales, then combining them with the original phase spectrum, and reconstructing the spatial features through inverse Fourier transform to obtain the final multi-scale frequency domain enhanced output.

[0131] Step 104: Perform progressive thinning enhancement of spatial features on the input image.

[0132] Specifically, this application, based on the SPPRN module, unifies the feature maps of the four layers output by the backbone network to the same number of channels through 1*1 convolution, thus obtaining channel-aligned features. For features after channel alignment Based on a top-down feature fusion structure, through upsampling and feature concatenation, high-level semantic information is gradually passed down to lower levels to obtain hierarchical features along the top-down path. The hierarchical features of the above top-down path. The features are refined sequentially from bottom to top and from top to bottom to obtain the i-th level features after upsampling. Integrate the features corresponding to the lowest level. and hierarchical features The second level corresponding to the bottom-up path To obtain the corresponding lowest level .

[0133] Furthermore, we obtain the corresponding lowest level. Subsequently, SPPRN is also used to generate dynamic positional encodings through depthwise separable convolutions. Based on these dynamic positional encodings, the lowest-level spatial feature enhancement output with added positional encodings is generated. Finally, through progressive downsampling, spatial feature enhancement outputs at each level are generated, constructing a complete feature pyramid.

[0134] Step 105: Three-branch feature residual fusion. Using a residual connection method, complementary information from frequency domain enhancement features and spatial refinement features is introduced while preserving the original feature information, resulting in a more discriminative and balanced detection feature representation.

[0135] Step 106: Calculate the features after fusion of the four levels using a multi-scale detection head, and output the final target location coordinates and category information.

[0136] Step 107: Apply the trained target detection model to small target detection application scenarios such as UAV-based disaster-bearing body monitoring and power target detection to obtain the category, confidence score and corresponding target bounding box coordinate information of each detected target.

[0137] It should be noted that the specific description of each step in Embodiment 2 of this application can be found in Embodiment 1 above, and this Embodiment 2 will not be described in detail.

[0138] Example 3

[0139] Embodiment 3 of this application compares the performance parameters of the technical solution proposed in this application with those of existing mainstream technical solutions in practical applications to verify the feasibility and superiority of the technical solution proposed in this application.

[0140] Specifically, a three-branch network was constructed and network parameters were set according to the methods described in Embodiments 1 and 2 above. In this embodiment, the experiments were conducted within the deep learning framework of PyTorch version 2.0. All experiments were performed using a single RTX4090D graphics card equipped with 24GB of memory. A stochastic gradient descent optimizer was used, with an initial learning rate of 0.005, a minimum learning rate of 0.00001, and 50 training epochs for each iteration. Based on different IoU thresholds, AP, AP50, AP75, APvt, APt, APs, and APm were used as evaluation metrics. AP, AP50, and AP75 correspond to the average mAP and AP with IoU thresholds of 0.5 and 0.75, respectively, while APvt, APt, APs, and APm represent very small, small, small, and medium-scale evaluation metrics, respectively.

[0141] The model was trained and tested using the dataset from step 101 above, and the parameters with the best test results were saved. The test results are shown in Tables 1 and 2. Figure 6 , Figure 7 As shown in the figure. Experimental results demonstrate that, despite having a relatively small number of parameters and FLOPs compared to other networks, our method outperforms other mainstream networks in all evaluation metrics. In particular, for the detection metrics APvt, APt, and APs of very small / tiny / small targets, the performance of the method provided in this application is significantly improved compared to traditional methods.

[0142] Table 1: Performance of the proposed method and other mainstream methods on the AI-TOD-V2 test set

[0143]

[0144] Table 2: Performance of the proposed method and other mainstream methods on the PLADv4 test set

[0145]

[0146] The above results demonstrate that experimental results on the three public datasets AI-TODv2, PLADv4, and Damper_defect show that MFSNet outperforms existing methods on multiple evaluation metrics, including Faster R-CNN, RetinaNet, FCOS, ATSS, YOLO series and its variants, especially in small object detection metrics (APvt, APt, APs), while maintaining reasonable computational complexity. Specifically, on the AI-TODv2 dataset, MFSNet-R50 improves APvt, APt, and APs metrics by 5.1%, 8.1%, and 4.8% compared to Faster R-CNN-R50, and by 1.9%, 3.9%, and 9.1% compared to FCOS-R50; MFSNet-RepVit-TD improves APvt, APt, and APs by 5.9%, 8.6%, and 6.8% compared to LTDNet. On the PLADv4 dataset, the AP50 score is improved by 13.1%, 15.2%, and 4% compared to Faster R-CNN-R50, FCOS-R50, and LTDNet, respectively. On the Damper_defect dataset, the AP50 score is improved by 3%, 14.1%, and 25.3% compared to Faster R-CNN-R50, FCOS-R50, and LTDNet, respectively. Experiments demonstrate that the proposed method MFSNet achieves excellent performance on datasets with small targets, multi-scale, and fine-grained classification. Multiple comparison and ablation experiments fully validate the effectiveness of the proposed method in each module. This application provides a new approach for small target detection in UAV images that balances performance and efficiency.

[0147] Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting small targets on unmanned aerial vehicles (UAVs) based on frequency domain enhancement and spatial path refinement, characterized in that, include: Acquire and process drone image small target datasets; A three-branch network MFSNet is constructed, comprising a first branch, a second branch, and a third branch with residual connections; the first branch includes a backbone network; the second branch includes the backbone network and a multi-scale frequency domain enhancement module (MFEM); and the third branch includes the backbone network, the MFEM, and a spatial path refinement enhancement module (SPPRN). The MFEM further includes multiple parallel branches, each branch comprising a connected Spatial Pyramid Pooling (ASPP) module, a Fast Fourier Transform (FFT) module, and an Adaptive Feature Enhancement (AFE) module. The outputs of each branch are fused and then enhanced in the multi-scale frequency domain using inverse Fast Fourier Transform and average pooling operations. Each branch of the MFEM takes the feature maps extracted from each level of the backbone network as input to extract the amplitude spectrum corresponding to the feature maps and enhances the amplitude spectrum. The amplitude spectrum extraction method includes: The feature maps of each level are input into the dilated spatial pyramid pooling (ASPP) to generate feature representations of multiple scales based on dilated convolutions with different dilation rates, thereby capturing target context information of different sizes. Based on the Fast Fourier Transform (FFT) module, the feature representations at each scale are transformed from the spatial domain to the frequency domain to obtain the corresponding amplitude spectrum. The adaptive feature enhancement module (AFE) takes the amplitude spectrum output by the FFT module as input and performs global average pooling on the amplitude spectrum to compress the spatial information of each channel into a global descriptor, which is used to capture the average energy distribution of each feature channel in the spatial dimension. Based on the global descriptor, a spatially adaptive frequency weight mask and enhancement factor are generated to enhance the amplitude spectrum. The SPPRN is built on a top-down and bottom-up bidirectional feature fusion structure, dynamic position encoding, and downsampling module, and is used to spatially enhance the feature maps of multiple levels output by the backbone network. The SPRN is used to process the feature maps of the four levels output by the backbone network through 1 1. Convolution is unified to the same number of channels to obtain channel-aligned features. ; For features after channel alignment Based on a top-down feature fusion structure, through upsampling and feature concatenation, high-level semantic information is gradually passed down to lower levels to obtain hierarchical features along the top-down path. ; The hierarchical features of the top-down path The features are refined sequentially from bottom to top and from top to bottom, resulting in the upsampled first feature. i Hierarchical features ; Fusion of the features corresponding to the lowest level and hierarchical features The second level corresponding to the bottom-up path This yields the fusion of multi-level features corresponding to the lowest level. ; The three-branch network MFSNet is trained based on the UAV image small target dataset to obtain a target detection model for UAV small target detection.

2. The UAV small target detection method based on frequency domain enhancement and spatial path refinement according to claim 1, characterized in that, The outputs of each branch of the MFEM include the phase spectrum and the enhanced amplitude spectrum after enhancement by the adaptive feature enhancement module (AFE). The outputs of the various branches of the MFEM are fused, including: The enhanced amplitude spectra obtained by dilated convolutions with different dilation rates are spliced ​​together along the channel dimension to obtain a multi-scale amplitude spectrum. Based on 1 1. Convolution is used to fuse features of the spliced ​​multi-scale amplitude spectrum; The fused multiscale amplitude spectrum is split into multiple independent single-scale amplitude spectra according to the number of channels; The single-scale amplitude spectrum and the phase spectrum are recombined to reconstruct the frequency domain representation of the feature map.

3. The UAV small target detection method based on frequency domain enhancement and spatial path refinement according to claim 1, characterized in that, Get the corresponding bottom layer Subsequently, the SPPRN is also used to generate dynamic positional codes through depthwise separable convolutions; based on the dynamic positional codes, the lowest-level spatial feature enhancement output with added positional codes is generated.

4. The UAV small target detection method based on frequency domain enhancement and spatial path refinement according to claim 3, characterized in that, The downsampling module of the SPPRN is used to generate spatial feature enhancement outputs at each level by progressively downsampling the lowest-level spatial feature enhancement output after adding position encoding, thereby constructing a feature pyramid.

5. The UAV small target detection method based on frequency domain enhancement and spatial path refinement according to claim 1, characterized in that, The three-branch network MFSNet also includes a detection head. The outputs of the first, second, and third branches are input into the detection head after residual fusion, and are used to output the location coordinates and category information of the detected target.

Images