Unmanned aerial vehicle image small target detection method based on space-frequency joint and semantic alignment

Abstract

The application discloses a kind of based on space-frequency joint and semantic alignment unmanned aerial vehicle image small target detection method, belong to computer vision and deep learning technical field.The method includes: obtaining image dataset and pre-processing;Build the Transformer model based on space-frequency joint and semantic alignment, model includes backbone network, neck fusion network, detection head and decoder;The model is trained, and small target detection model is obtained;The image to be detected is input into model, and detection result is output.The neck fusion network of the application adopts the cascade closed-loop architecture of local-global-local, integrates multi-scale focusing module, WT-FT-WT cascade space-frequency fusion module and semantic alignment fusion module, enhances space-frequency feature expression through the complementary characteristics of wavelet transform and Fourier transform, and realizes space-frequency semantic alignment through cross-domain channel interaction mechanism.The application effectively solves the problem of image small target detail loss and background interference, and improves detection precision.

Description

Technical Field

[0001] This invention relates to the field of computer vision and deep learning technology, and in particular to a method for small target detection in UAV images based on spatial-frequency joint and semantic alignment. Background Technology

[0002] Target detection in UAV aerial images is an important research direction in the field of computer vision, with broad application prospects in many fields such as military reconnaissance, traffic monitoring, disaster assessment, and agricultural plant protection. However, due to the high flight altitude, varied shooting angles, and large differences in target scale of UAVs, especially for small targets (usually defined as targets with a pixel area of ​​less than 32×32), the discriminative information such as texture and shape contained in the targets is extremely limited. In addition, factors such as complex backgrounds, drastic changes in lighting, and motion blur in aerial images make small target detection in UAV images a very challenging problem.

[0003] Existing small target detection methods mainly face the following technical bottlenecks: First, multiple downsampling operations in deep convolutional networks cause small target features to be almost completely lost in the spatial dimension. The downsampling factor of up to 32 times in traditional backbone networks (such as ResNet, VGG, etc.) means that targets smaller than 32×32 pixels occupy less than 1 pixel in the deep feature map, making them difficult to detect effectively. Second, small targets are extremely sensitive to positional deviations. Traditional anchor point matching mechanisms are prone to causing a serious imbalance between positive and negative samples, with a large number of small targets being misclassified as background. Third, background clutter in aerial images (such as building textures, tree shadows, etc.) overlaps with target features in the frequency domain, making it difficult for simple spatial convolution to effectively distinguish between targets and background.

[0004] In recent years, researchers have proposed various improvement methods. For example, Feature Pyramid Networks (FPN) and its variants alleviate the problem of small target information loss to some extent by fusing deep semantic information with shallow detail information. However, these methods are still limited by the local receptive field of convolutional neural networks, making it difficult to capture the global contextual dependencies of images. The Transformer architecture, with its self-attention mechanism, demonstrates powerful capabilities in global modeling, but when directly applied to small target detection by UAVs, it often fails to meet real-time requirements due to excessive computation. Furthermore, existing methods mostly focus on enhancing spatial features, neglecting the unique value of frequency domain information in target-background differentiation. Wavelet transform can decompose images into sub-bands of different frequencies, where high-frequency components contain detailed information such as edges and corners, which is crucial for small target localization; Fourier transform can reveal the global periodicity and overall structural information of images. However, how to effectively combine these two types of frequency domain transforms with spatial convolutional networks and achieve semantic alignment of spatial and frequency features still lacks a systematic solution.

[0005] Furthermore, some existing works have attempted to introduce wavelet transform into deep learning models, for example, using wavelet transform to replace traditional stride convolution or pooling operations for downsampling to reduce information loss. However, these methods often treat wavelet transform merely as a fixed downsampling tool, failing to fully explore its synergistic enhancement potential with learnable convolution and global frequency domain transform. Simultaneously, spatial and frequency domain features differ significantly in their distribution characteristics: spatial features possess clear physical location information but lack a global perspective, while frequency domain features contain global statistical information but lack spatial localization capabilities. Simply concatenating or adding the two leads to semantic misalignment, failing to fully leverage their respective advantages. Summary of the Invention

[0006] To address the problems of insufficient spatial-frequency feature fusion, easy loss of small target details, and difficulty in balancing detection accuracy and real-time performance in existing UAV image small target detection methods, this invention provides a UAV image small target detection method based on spatial-frequency joint and semantic alignment. Through an innovative multi-scale focusing architecture, wavelet-convolution hybrid module, and channel attention weighted fusion module, it achieves efficient extraction and accurate matching of small target features, improving detection accuracy while ensuring real-time processing capabilities.

[0007] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0008] A method for small target detection in UAV images based on spatial-frequency joint and semantic alignment includes the following steps:

[0009] Step S1: Obtain the drone aerial image dataset and perform preprocessing;

[0010] Step S2: Construct a Transformer model based on spatial-frequency integration and semantic alignment. The model includes a backbone network, a neck fusion network, a detection head, and a decoder.

[0011] The neck fusion network adopts a cascaded closed-loop architecture of local-global-local, integrating a multi-scale focusing module, a WT-FT-WT cascaded space-frequency fusion module, and a semantic alignment fusion module for deep fusion and semantic alignment of space-frequency domain features; the WT-FT-WT is a local-global-local architecture.

[0012] Step S3: Train the Transformer model using the preprocessed dataset to obtain a small target detection model for UAV images;

[0013] Step S4: Input the image to be detected into the UAV image small target detection model and output the target detection result.

[0014] A further improvement to the above technical solution is as follows:

[0015] Preferably, the multi-scale focusing module is used to perform lossless intermittent sampling of shallow features, introduces a channel adjustment module to adjust the channel dimension, and performs dual-path feature fusion.

[0016] Preferably, the WT-FT-WT cascaded space-frequency fusion module includes a local frequency domain dimensionality reduction unit, a global frequency domain sensing unit, and a space-frequency collaborative reconstruction unit;

[0017] The local frequency domain dimensionality reduction unit is a wavelet transform downsampling module, which uses a three-way parallel branch structure to downsample the input features and convert spatial location information into channel frequency information.

[0018] The global frequency domain sensing unit is used to capture global context dependencies: it performs a two-dimensional fast Fourier transform on the input features to map them to the frequency domain, then performs global weighting, and then restores them to real-valued feature maps through a two-dimensional inverse Fourier transform.

[0019] The space-frequency collaborative reconstruction unit uses an inverse wavelet transform module to reconstruct the enhanced frequency domain features and the retained high-frequency details back into the spatial domain, thereby restoring the spatial resolution.

[0020] Preferably, the semantic alignment fusion module includes a channel attention weighted fusion module and a feature fusion convolution module; the channel attention weighted fusion module is used to perform horizontal and vertical average pooling on the dual-path features, generate channel attention weights through a fully connected layer, and perform cross-modulation and channel concatenation; the feature fusion convolution module consists of 1×1 convolution, 3×3 and 5×5 parallel convolution branches, and 1×1 convolution, and is used to extract multi-scale details and restore channel dimensions.

[0021] Preferably, step S4 specifically involves: inputting the image to be detected into a small target detection model, whereby the backbone network extracts multi-scale feature maps; inputting the multi-scale feature maps into a neck fusion network for feature enhancement and fusion; and inputting the fused features into the detection head for target classification and regression.

[0022] The feature enhancement and fusion are as follows: First, the feature map is input into the multi-scale focusing module for lossless intermittent sampling and multi-scale fusion; in the WT-FT-WT cascaded space-frequency fusion module, the feature stream is sequentially processed through local wavelet decomposition, global Fourier enhancement, and local wavelet reconstruction to obtain a feature map containing global semantic enhancement. The frequency components of the enhanced feature map in the channel dimension are remapped back to the spatial dimension to restore the spatial resolution of the feature map and generate an output feature map that fuses local high-frequency details and global low-frequency semantics; in the semantic alignment fusion module, the spatial domain features and frequency domain reconstructed features are fused.

[0023] Preferably, the backbone network extracts multi-scale features from layers S2 and S3, and inputs the feature maps of layers S2 and S3 into a multi-scale focusing module for lossless interval sampling and multi-scale fusion. The multi-scale focusing module performs multi-level interval sampling on the S2 layer features, adjusts the channel dimensions through a channel adjustment module, performs dual-path feature fusion, and generates features with a resolution of 160×160. Figure I And feature map II with 80×80 resolution.

[0024] Preferably, the features Figure I The input spatial feature map II is fed into the WT-FT-WT cascaded space-frequency fusion module, which processes it sequentially through a local frequency domain dimensionality reduction unit, a global frequency domain perception unit, and a space-frequency collaborative reconstruction unit. First, the input spatial feature map is decomposed into low-frequency and high-frequency components with spatial locality through discrete wavelet transform. Then, a global fast Fourier transform is performed on the low-frequency components rich in semantic information to capture the global contextual dependencies of the image in the frequency domain and enhance its features. Finally, the enhanced global frequency domain features are restored through inverse Fourier transform and reconstructed with the retained high-frequency detail components through inverse wavelet transform.

[0025] Preferably, the channel attention weighted fusion module is specifically implemented as follows: Let the spatial domain features be... Frequency domain reconstruction features are Perform concatenation along the channel dimension to obtain a hybrid feature map. Global average pooling is performed on the hybrid feature map to obtain the channel descriptor z; z is input into two fully connected layers. The first fully connected layer compresses the number of channels and activates them with ReLU, and the second fully connected layer restores the number of channels and activates them with Sigmoid to generate a weight vector s; the weight vector s is multiplied with the original hybrid feature map channel by channel to obtain the fused feature.

[0026] Preferably, in step S4, the decoder is a multi-layer Transformer decoder structure, each layer containing multi-head self-attention, multi-scale deformable cross-attention, and a feedforward network; the input of the decoder is the object query vector filtered by the uncertainty minimization mechanism and the semantic enhancement feature sequence output by the semantic alignment fusion module. The association between queries is established through self-attention, the target-related features are extracted from the fused features through cross-attention, and finally the prediction results are output through classification and regression branches.

[0027] The small target detection method for UAV images based on space-frequency joint and semantic alignment provided by this invention has the following advantages compared with the prior art:

[0028] (1) The UAV image small target detection method based on spatial-frequency joint and semantic alignment of the present invention significantly improves the signal-to-noise ratio and reconstruction quality of UAV small target features. Existing technologies usually adopt a dual-stream structure with parallel spatial and frequency domains, which easily leads to semantic misalignment when the features of the two domains are fused. The WT-FT-WT cascaded architecture proposed in this invention utilizes the mathematical complementarity between the local time-frequency characteristics of wavelet transform (WT) and the global frequency statistical characteristics of Fourier transform (FT).

[0029] (2) The UAV image small target detection method based on space-frequency joint and semantic alignment of the present invention uses a multi-scale focusing (MF) module to retain rich low-dimensional features of small targets with low computational complexity through innovative feature sampling and fusion strategies. Compared with traditional focusing modules, it can capture more comprehensive multi-scale feature representations and provide sufficient detailed information for small target detection.

[0030] (3) The UAV image small target detection method based on space-frequency joint and semantic alignment of the present invention fully leverages the time-frequency localization characteristics of wavelet transform and the global frequency statistical advantages of Fourier transform in the WT+FT+WT cascade architecture, complementarily integrates local details and global structural information, effectively improving the recognition of small target features. At the same time, compared with the downsampling method based on Fourier transform, the WTDS module reduces the number of parameters while preserving frequency domain information.

[0031] (4) The UAV image small target detection method based on spatial-frequency joint and semantic alignment of the present invention achieves adaptive channel attention weighting through the SE+Cat module by improving SEnetv2 and combining the multi-scale feature extraction capability of the FFC module, realizing accurate semantic alignment of spatial-frequency domain features, effectively suppressing background interference, improving the accuracy of small target detection, and alleviating overfitting in the later stage of model training.

[0032] (5) The UAV image small target detection method based on spatial-frequency joint and semantic alignment of the present invention, by embedding Fourier global enhancement in the wavelet decomposition domain, is equivalent to constructing a filter with a specific passband. It not only uses FFT to make up for the lack of global receptive field of wavelet transform, but also uses DWT to protect the high-frequency tiny pixel details that are easily lost when small targets are transformed by FFT. Compared with simple spatial domain convolution or simple frequency domain transformation, this cascaded structure effectively reduces feature reconstruction error and significantly improves the accuracy of small target detection in complex backgrounds. Attached Figure Description

[0033] Figure 1 This is a diagram of the Transformer model based on spatial-frequency joint and semantic alignment of the present invention. Detailed Implementation

[0034] The following provides a detailed description of specific embodiments of the present invention. It should be understood that the specific embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of the invention.

[0035] like Figure 1 As shown, the UAV image small target detection method based on space-frequency joint and semantic alignment of the present invention includes the following steps:

[0036] Step S1: Collect aerial image datasets using drones and preprocess the datasets. The preprocessed datasets are divided into training, validation, and test sets.

[0037] Data sets were collected from various scenarios, including people, cars, trucks, and buses. Preprocessing methods included image enhancement, geometric transformation, data augmentation and annotation, normalization, and scaling. Small object standards were defined according to international dataset requirements.

[0038] Step S2: Design a Transformer model based on spatial-frequency joint and semantic alignment.

[0039] The Transformer model based on spatial-frequency integration and semantic alignment, namely the UAV-DETR+ detection framework, is an improvement on the RT-DETR and UAV-DETR architectures. It consists of a backbone network, a neck fusion network, a head, and a decoder.

[0040] S2-1, the backbone network is used to extract multi-scale basic features of the input UAV image. ResNet18 or ResNet50 is preferred. The input is the preprocessed UAV aerial image, and the output is a multi-scale feature map. It mainly uses the S2 layer (high-resolution shallow features, 320×320×64) and the S3 layer (medium-resolution features, 160×160×128).

[0041] S2-2, the neck fusion network, is the core component. It employs a cascaded closed-loop architecture of local-global-local, integrating a multi-scale focusing module (MF), a WT-FT-WT cascaded space-frequency fusion module, and a semantic alignment fusion module to achieve deep fusion and semantic alignment of space-frequency domain features. The neck fusion network specifically includes the following modules:

[0042] S2-2-1, the Multi-scale Focus (MF) module, is used to perform lossless point sampling of shallow features, preserving tiny target pixels.

[0043] A Channel Adjust (CA) module is introduced to dynamically adjust the number of channels through 1×1 convolution; dual-path feature fusion is performed to generate feature maps.

[0044] S2-2-2, Local Frequency Domain Dimensionality Reduction Unit (Wavelet Transform Downsampling Module, WTDS), is used to convert spatial location information into channel frequency information to achieve lossless downsampling.

[0045] The local frequency domain dimensionality reduction unit is a three-way parallel branch structure built based on wavelet transform convolution (WTConv). Branch 1 preserves local fine-grained features (convolutional layer); branch 2 includes max pooling and wavelet transform convolution (WTConv); and branch 3 is downsampling with a stride of 2.

[0046] Wavelet transform convolution (WTConv) uses a fixed Haar wavelet basis and fuses high-frequency details and low-frequency approximation information through cascaded operations of wavelet decomposition, convolution filtering, inverse wavelet reconstruction, and residual connection.

[0047] S2-2-3, Global Frequency Domain Aware Unit (Multi-Scale Frequency Enhancement Fusion Module, MFFF / FT), is used to capture global contextual dependencies and enhance feature recognition.

[0048] Perform a two-dimensional fast Fourier transform (FFT2D) on the downsampled feature map to map it to the frequency domain; use a learnable spectral filter to globally weight the frequency domain features; and restore it to a real feature map through an inverse two-dimensional Fourier transform (IFFT2D).

[0049] S2-2-4, the spatial-frequency collaborative reconstruction unit, is used to restore spatial resolution by fusing local high-frequency details with global low-frequency semantics.

[0050] The inverse wavelet transform (IDWT) module is used to reconstruct the enhanced frequency domain features and the preserved high-frequency details back into the spatial domain through inverse wavelet transform.

[0051] S2-2-5, the semantic alignment fusion module, is used to solve the semantic misalignment problem when fusing spatial and frequency domain features, and to realize cross-domain channel interaction and recalibration.

[0052] The semantic alignment fusion module includes the channel attention weighted fusion module (SE+Cat) and the feature fusion convolution (FFC).

[0053] SE+Cat (an improved version of SEnetV2) is used to perform horizontal and vertical average pooling on dual-path features, generate channel attention weights through fully connected layers, and perform cross-modulation and channel concatenation.

[0054] Feature Fusion Convolution (FFC) consists of a 1×1 convolution, a 3×3+5×5 parallel convolution branch, and a 1×1 convolution, and is used to extract multi-scale details and restore channel dimensions.

[0055] S2-3, the head and decoder are used to receive the fused features of the neck network output.

[0056] The system completes target classification and bounding box regression through a multi-head self-attention mechanism and ensemble prediction strategy. It does not require non-maximum suppression (NMS) post-processing and directly outputs the detection results, which include target category, location coordinates, and confidence score.

[0057] Step S3: Train the Transformer model based on spatial-frequency joint and semantic alignment using the training set to obtain a small target detection model for UAV images.

[0058] S3-1, Prepare data: Input the drone aerial images and labeled target boxes (category, location) into the model.

[0059] S3-2, Forward Propagation: The image passes sequentially through the backbone network, MF module, WT-FT-WT cascade fusion, semantic alignment module, and decoder, and outputs the prediction result (target category and location).

[0060] S3-3, Calculate the loss: Match the prediction results with the actual labels and calculate the error.

[0061] S3-4, Backpropagation and Optimization: Update model parameters based on loss.

[0062] S3-5, Iterative Training: Repeat the above steps for the preset period, evaluate on the validation set in each round, and save the model with the best performance.

[0063] S3-6, Obtain the model: After training, a model that can be used for small target detection in UAV images is obtained.

[0064] Step S4: Output the target detection results.

[0065] Specifically, the following steps are included:

[0066] S4-1, backbone network extracts multi-scale features.

[0067] The image is input into the backbone network (ResNet18 or ResNet50), and after forward propagation, feature maps of two key layers are extracted. Together, they cover the multi-scale feature range of small targets in the UAV image.

[0068] The S2 layer has a resolution of 320×320 and 64 channels. This layer is a high-resolution shallow feature layer that preserves rich spatial details (edges, textures, etc.) and is sensitive to small targets.

[0069] The S3 layer has a resolution of 160×160 and 128 channels. This layer is a medium-resolution feature that balances details and preliminary semantic information.

[0070] S4-2, perform multi-scale feature fusion.

[0071] The S2 and S3 feature maps are input into the MF module for lossless, spaced-point sampling and multi-scale fusion, achieving grid sampling and fusion of low-dimensional tensors. Specifically:

[0072] S4-2-1, Perform multi-level focus sampling operation: Perform sampling processing on the features of layer S2 and layer S3 respectively.

[0073] Two-level interval sampling is performed on the S2 layer features, and one-level interval sampling is performed on the S3 layer features to achieve alignment of features at different scales. This is represented as:

[0074] (1)

[0075] in, This indicates a sampling at an interval. This represents the feature tensor obtained after performing the first-level interval sampling on the features of layer S2. This indicates the output features of the first level. The feature tensor obtained after sampling again (second-level interval sampling). This represents the feature tensor obtained after performing first-level interval sampling on the features of layer S3.

[0076] In S4-2-2, the CA module performs channel dimension calibration on the S2 layer features and the original S3 layer features after multi-level interval sampling operations. It dynamically adjusts the number of channels for each feature through a 1×1 convolution kernel to keep the channel dimension of features at different scales consistent.

[0077] S4-2-3, Perform dual-path feature fusion: Path 1 stacks the first-sampled S2 features calibrated by the channel adjustment module with the S3 layer features to generate 160×160 resolution features. Figure I Path two stacks the secondary sampled S2 features calibrated by the channel module with the primary sampled S3 features to generate an 80×80 resolution feature map II, represented as:

[0078] (2)

[0079] (3)

[0080] in, This represents the 160×160 resolution feature tensor generated by the first fusion path; This indicates a concatenation operation performed along the channel dimension. This represents the single-sampled features of layer S2. The feature tensor output via channel calibration through the CA module; This represents the feature tensor output directly from the original features of layer S3 via channel calibration using the CA module; This represents the 80×80 resolution feature tensor generated by the second fusion path; This represents the secondary sampling features of layer S2. The feature tensor output via channel calibration through the CA module; This represents the single-sampled features of layer S3. The feature tensor is output via channel calibration through the CA module.

[0081] The Focus operation is implemented through two-dimensional stride sampling, simultaneously sampling the tensor along both the horizontal and vertical directions with a stride of 2, selecting every other element in each spatial direction to minimize computational overhead while achieving downsampling. The CA module dynamically adjusts the channel dimensions through convolution operations, optimizing feature compatibility between network branches. The output consists of two downsampled tensors with dimensions of 160×160 and 80×80, respectively, preserving rich details of small targets and multi-scale feature representations.

[0082] S4-3, WT-FT-WT cascaded space-frequency fusion.

[0083] First, the input spatial feature map is decomposed into low-frequency and high-frequency components with spatial locality through discrete wavelet transform. Then, global fast Fourier transform (FFT) is performed only on the low-frequency components rich in semantic information to capture the global contextual dependencies of the image in the frequency domain and perform feature enhancement. Finally, the enhanced global frequency domain features are restored through inverse Fourier transform and reconstructed with the preserved high-frequency detail components through inverse wavelet transform.

[0084] This cascaded structure differs from the traditional parallel structure with two branches in the spatial and frequency domains. It forces the feature flow to sequentially undergo local spatial decoupling and global frequency domain integration to achieve deep complementarity of spatial and frequency information.

[0085] S4-3-1, Feature enhancement via wavelet transform convolution, uses a fixed Haar wavelet basis and fuses high-frequency details with low-frequency approximation information. The calculation formula is as follows:

[0086] (4)

[0087] in, This is a wavelet transform operation used to decompose multi-scale frequency domain sub-features; This indicates a convolution operation used to extract detailed information from sub-features; This represents the feature tensor input to this wavelet transform convolution module; This involves using inverse wavelet transform to reconstruct frequency domain features. By fusing convolutional features with wavelet / inverse wavelet transform features through residual connections, the local feature extraction capability and high reconstructivity of wavelet transform are preserved, while flexibly enhancing the information representation of different channels.

[0088] Two-dimensional wavelet decomposition is performed using the Haar wavelet basis to transform the input feature tensor The input is fed into a 2D discrete wavelet transform module. Using the Haar wavelet basis or other orthogonal wavelet basis, the spatial features are decomposed into four components: an approximate component and three detail components. The approximate component LL represents low-frequency semantics; the three detail components LH, HL, and HH represent high-frequency edge details along the horizontal, vertical, and diagonal lines. This step preserves the physical location information of small targets. The convolutional layer uses a 3×3 kernel, with the number of channels consistent with the input features (128 channels), and the activation function is ReLU. After inverse wavelet reconstruction, a residual connection is made with the original input features to avoid gradient vanishing.

[0089] S4-3-2, Feature compression of the wavelet transform downsampling module: The wavelet transform downsampling module is constructed based on the wavelet transform convolution module, and a three-way parallel branch structure is used to compress the features output by the multi-scale focusing module. Figure I B is downsampled.

[0090] Branch 1: After multiplying the input feature map by the coefficient (1-e), it is passed through a 3×3 convolutional layer (Conv1) and a 1×1 convolutional layer (Conv2) in sequence to preserve local fine-grained features.

[0091] Branch 2: After multiplying the input feature map by the coefficient e, it is split and fed into a 5×5 max pooling layer (MaxPool) and a WTConv module respectively. After adjusting the channel dimensions through a 1×1 convolutional layer (Conv3), the features are added together.

[0092] Branch 3: Performs a downsampling operation with a step size of 2 on the input feature map to compress the spatial dimension.

[0093] The output features of branch 2 and branch 3 are added element-wise, and then concatenated with the output features of branch 1 to obtain the output of the WTDS module. The calculation formula is as follows:

[0094]

[0095]

[0096] (5)

[0097] in, This represents the local fine-grained feature tensor extracted from branch 1 (the first branch). This represents the feature tensor output after the fusion of pooling and wavelet transform in branch 2 (the second branch); This represents the original feature tensor input to the wavelet transform downsampling module; This represents the scaling factor used to assign feature weights (therefore) (This represents the complementary weight coefficient of the other branch). This indicates the first convolution operation (i.e., a 3×3 convolutional layer), used for initial feature extraction; This indicates the second convolution operation (i.e., a 1×1 convolutional layer), used to further process fine-grained features; This indicates the third convolution operation (i.e., the 1×1 convolutional layer), used to adjust the channel dimension of the features after max pooling; This indicates a max pooling operation (i.e., a maximum 5×5 pooling layer), used to extract local salient features and reduce spatial resolution; This represents the wavelet transform convolution operation, used to extract and fuse high-frequency details and low-frequency approximation information in the frequency domain; This indicates a concatenation operation performed along the channel dimension; Indicate the characteristic path of branch 1 and branch 2 The final downsampled output feature tensor obtained after concatenation.

[0098] In this embodiment, the coefficient e is set to 0.6 to balance the weights of local features and frequency domain features; Conv1 is a 3×3 convolution with a stride of 2; Conv2 is a 1×1 convolution; Conv3 is a 1×1 convolution, and the number of output channels is 64 for each layer; the MaxPool layer uses a 5×5 pooling kernel with a stride of 2; the downsampling operation is implemented by a 3×3 convolution with a stride of 2, and the final number of output channels is 128.

[0099] S4-3-3 performs global Fourier transform fusion, inputting the features output from the WTDS module into the multi-scale frequency enhancement fusion (MFFF) module. Global Fourier transform extracts global frequency domain statistics, which complement the local frequency domain features extracted by wavelet transform, forming a local-global-local WT-FT-WT cascaded processing architecture. This simultaneously improves the signal-to-noise ratio, calculated using the following formula:

[0100] (6)

[0101] in, The effective bandwidth of wavelet decomposition, For the full bandwidth of the Fourier transform, Signal-to-noise ratio is improved by bandwidth limitation and orthogonal noise suppression; This represents the system output signal-to-noise ratio after the cascaded wavelet transform and global Fourier transform are fused. This indicates that the local high-frequency sub-bands extracted by wavelet transform are at the frequency... The characteristic spectral amplitude at that location; This indicates that the global features extracted by the global Fourier transform (FFT) are in frequency... The characteristic spectral amplitude at that location; This represents the angular frequency in frequency domain analysis; This represents the baseline signal-to-noise ratio when processing solely using the global Fourier transform. This represents the standard deviation of background noise in the feature map or input signal; This represents the L2-norm operation, used to calculate the signal energy of the corresponding frequency band in the frequency domain.

[0102] This module specifically receives the downsampled feature map output from the previous wavelet transform as input. First, the downsampled feature map is input into the global frequency domain (FT) to perform a two-dimensional fast Fourier transform (FFT), mapping the feature data from the spatial domain to the complex frequency space to obtain frequency domain features. Next, in the frequency domain space, a learnable spectral filter is used to perform point-by-point multiplication with the frequency domain features to obtain modulated spectral features. Finally, a two-dimensional inverse Fourier transform (IFFT) is performed on the modulated spectral features to restore them to a feature map in the real number space, resulting in a feature map containing global semantic enhancement.

[0103] S4-4, Space-Frequency Coordinated Reconfiguration.

[0104] The enhanced feature map output from S4-3-3 is input into the inverse wavelet transform module. According to the inverse operation rules of discrete wavelet transform, the frequency components of the enhanced feature map in the channel dimension are remapped back to the spatial dimension to restore the spatial resolution of the feature map, thereby generating the final output feature map that integrates local high-frequency details and global low-frequency semantics.

[0105] S4-5, Semantic alignment fusion.

[0106] To address the semantic misalignment problem that arises during direct fusion of frequency domain features (due to their global distribution and lack of spatial localization) and spatial domain features (due to their local localization and lack of global perspective), this invention proposes a Semantic Alignment Module (SE-Alignment Module). This module goes beyond simple linear superposition; instead, it achieves nonlinear alignment of cross-domain information through a channel attention recalibration mechanism.

[0107] To achieve efficient fusion and semantic alignment of spatial frequency domain features, a channel attention weighted fusion module (SE+Cat) and a feature fusion convolution (FFC) module are designed. The specific fusion method is as follows:

[0108] S4-5-1, Unify channel dimensions: Adjust the channel dimensions of the two enhanced features output by the WTDS module to ensure that the number of feature channels in the two inputs to the SE+Cat module is consistent.

[0109] S4-5-2, Cross-domain concatenation: Suppose that among the two branch features after the above channel unification, the input feature from the regular convolution operation branch (i.e., the branch that extracts local details) is the spatial domain feature. The input features from the wavelet transform convolution branch (i.e., the branch that extracts multi-scale frequency information) are frequency domain reconstructed features. First, a preliminary concatenation operation is performed on the two feature paths along the channel dimension to obtain a mixed feature map. .at this time, It contains physical location information and frequency statistics, but the two are in a messy, misaligned state.

[0110] S4-5-3, Perform global information compression: for hybrid features Perform global average pooling operation, spatial dimensions compressed to We obtain the channel descriptor z and calculate the response intensity of each channel across the entire graph.

[0111] S4-5-4, cross-channel interactive modeling: z is input into two fully connected (FC) layers. The first FC layer compresses the number of channels and introduces nonlinearity using the ReLU activation function; the second FC layer restores the number of channels and generates a weight vector s using the Sigmoid activation function. The generated weight s is calculated based on joint spatial-frequency information, which can reflect the model's attention difference between spatial and frequency domain information.

[0112] S4-5-5, Perform feature recalibration: Combine the weight vector s with the original mixed features Element-wise product is performed channel-by-channel to obtain the final fused feature. The calibrated channel attention weight SEweight is calculated using the following formula:

[0113] (7)

[0114] in, Indicates input features The calculated channel attention calibration weight vector; This represents the branch feature tensor (i.e., the feature tensor mentioned later) input to this weight calculation module. or ); This indicates a fully connected layer operation, used for mapping and information exchange along the feature channel dimensions; Indicates input features Parallel average pooling or max pooling operations are performed in both horizontal and vertical dimensions to extract global channel descriptors; ∑ represents the element-wise addition and fusion operation of features extracted by different pooling methods (or along different horizontal and vertical directions) after processing by a fully connected layer.

[0115] S4-5-6, the calculated channel attention weights are used to cross-modulate the two feature streams, and then the channels are concatenated to obtain the interactive feature. The calculation formula is:

[0116] (8)

[0117] in, This indicates a concatenation operation performed along the channel dimension; This represents the first input feature tensor of the module, i.e., the spatial domain feature after channel unification mentioned earlier. ; This represents the second input feature tensor of the module, i.e., the frequency domain reconstructed feature after channel unification mentioned earlier. ; and They respectively represent: based on formula (7) for the features and characteristics Channel attention weights calculated separately; This represents the interactive fusion feature tensor obtained after the two features are subjected to attention-based cross-modulation and channel concatenation.

[0118] In S4-5-7, Output1 is input into the FFC module. First, channel fusion is completed through 1×1 convolution. Then, multi-scale detail information is obtained through the feature extraction branch composed of 3×3 convolution and 5×5 convolution. Finally, the channel dimension is restored through 1×1 convolution to obtain the fused feature Output2 after spatial frequency information alignment. This feature is then input into the Feature Pyramid Network (FPN) for processing.

[0119] S4-6, Target Detection and Output.

[0120] The Output2 is input into the Transformer decoder, which completes target classification and bounding box regression through a multi-head self-attention mechanism and ensemble prediction strategy. The entire detection process does not require non-maximum suppression (NMS) post-processing and directly outputs the small target detection results in the UAV image, including target category, location coordinates and confidence score.

[0121] Experimental verification:

[0122] The model training and testing of this invention are based on an NVIDIA GeForce RTX3090 graphics card, and the software environment includes Python 3.8, PyTorch 1.13, CUDA 11.8, and cuDNN 8.9.2. During training, 400 training epochs are set, the batch size is 4, and an early stopping mechanism (patience=30) is used to avoid overfitting. The optimizer is AdamW, with a learning rate of 0.0001 and a momentum of 0.9. The input image scale is uniformly adjusted to 640×640, and data augmentation strategies such as Mosaic (parameter=1) and mixup (parameter=0.2) are used to improve the model's generalization ability. The model performance is evaluated using the COCO standard metrics, including average accuracy (AP) and AP50 (average accuracy when the IoU threshold is 0.5).

[0123] Two public datasets were used:

[0124] (1) VisDrone-2019-DET (General UAV Scene) dataset: contains 10 target categories, 6471 images in the training set, 548 images in the validation set, and 3190 images in the test set, used to evaluate the performance of the model in the general UAV small target detection scene;

[0125] (2) UAVVaste (specific small target scene) dataset: contains 772 drone aerial images with a total of 3718 bounding boxes. The target category is garbage, which is used to verify the generalization ability and transfer performance of the model.

[0126] Experimental results on the VisDrone-2019-DET test set show that the detection method of the present invention is significantly better than the existing mainstream algorithms in terms of detection accuracy. The data is shown in Table 1. The larger the AP and AP50, the better the model.

[0127] Table 1. Detection performance data of various models and detectors

[0128]

[0129] The specific data is as follows:

[0130] (1) Compared to the baseline model, the present invention improves the average accuracy (AP) by 1.3% to 1.9%;

[0131] (2) In particular, in terms of the detection metric (AP) for tiny objects (pixel area less than 32×32), the present invention improves the detection network by about 2.4% compared with the detection network that simply uses spatial convolution;

[0132] (3) On the UAVVaste dataset, the present invention demonstrates strong generalization ability and can effectively identify tiny targets in extremely cluttered backgrounds, verifying the significant role of global frequency domain perception in suppressing background noise.

[0133] To further demonstrate the independent contribution of each innovative module in this invention, ablation experiments were conducted item by item, and the results proved the indispensability of each module:

[0134] (1) Verification of the effectiveness of the WT-FT-WT cascade structure: Experiments show that if only wavelet downsampling (WTDS) is used and the intermediate Fourier global perception (FT) stage is removed, the model’s false negative rate for large-scale targets increases significantly; if only Fourier transform is used and wavelet is removed to preserve details, the localization accuracy (AP50) of small targets decreases by about 1.1%. When the WT-FT-WT cascade structure proposed in this invention is fully adopted, the model achieves the best balance between preserving high-frequency details and acquiring global semantics, and the AP value reaches the optimal value.

[0135] (2) Verification of the effectiveness of the semantic alignment module: Comparative experiments show that the detection accuracy is improved by 0.8% and 0.5% respectively compared with simple channel concat or direct addition using the semantic alignment fusion module of the present invention. This confirms the necessity of actively eliminating spatial-frequency domain semantic misalignment through the channel interaction mechanism.

[0136] The above embodiments are merely preferred examples of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A method for small target detection in UAV images based on spatial-frequency joint and semantic alignment, characterized in that, Includes the following steps: Step S1: Obtain the drone aerial image dataset and perform preprocessing; Step S2: Construct a Transformer model based on spatial-frequency integration and semantic alignment. The model includes a backbone network, a neck fusion network, a detection head, and a decoder. The neck fusion network adopts a cascaded closed-loop architecture of local-global-local, integrating a multi-scale focusing module, a WT-FT-WT cascaded space-frequency fusion module, and a semantic alignment fusion module for deep fusion and semantic alignment of space-frequency domain features; the WT-FT-WT is a local-global-local architecture. Step S3: Train the Transformer model using the preprocessed dataset to obtain a small target detection model for UAV images; Step S4: Input the image to be detected into the UAV image small target detection model and output the target detection result.

2. The method for small target detection in UAV images based on space-frequency joint and semantic alignment according to claim 1, characterized in that, The multi-scale focusing module is used to perform lossless intermittent sampling of shallow features, and the channel adjustment module is introduced to adjust the channel dimension and perform dual-path feature fusion.

3. The method for small target detection in UAV images based on space-frequency joint and semantic alignment according to claim 2, characterized in that, The WT-FT-WT cascaded space-frequency fusion module includes a local frequency domain dimensionality reduction unit, a global frequency domain sensing unit, and a space-frequency collaborative reconstruction unit. The local frequency domain dimensionality reduction unit is a wavelet transform downsampling module, which uses a three-way parallel branch structure to downsample the input features and convert spatial location information into channel frequency information. The global frequency domain sensing unit is used to capture global context dependencies: it performs a two-dimensional fast Fourier transform on the input features to map them to the frequency domain, then performs global weighting, and then restores them to real-valued feature maps through a two-dimensional inverse Fourier transform. The space-frequency collaborative reconstruction unit uses an inverse wavelet transform module to reconstruct the enhanced frequency domain features and the retained high-frequency details back into the spatial domain, thereby restoring the spatial resolution.

4. The method for small target detection in UAV images based on space-frequency joint and semantic alignment according to claim 3, characterized in that, The semantic alignment fusion module includes a channel attention weighted fusion module and a feature fusion convolution module. The channel attention weighted fusion module is used to perform horizontal and vertical average pooling on the dual-path features, generate channel attention weights through a fully connected layer, and perform cross-modulation and channel concatenation. The feature fusion convolution module consists of 1×1 convolutions, 3×3 and 5×5 parallel convolution branches, and 1×1 convolutions, and is used to extract multi-scale details and restore channel dimensions.

5. The method for small target detection in UAV images based on space-frequency joint and semantic alignment according to claim 4, characterized in that, Step S4 specifically involves: inputting the image to be detected into a small target detection model, where the backbone network extracts multi-scale feature maps; inputting the multi-scale feature maps into a neck fusion network for feature enhancement and fusion; and inputting the fused features into the detection head for target classification and regression. The feature enhancement and fusion are specifically as follows: First, the feature map is input into the multi-scale focusing module for lossless intermittent sampling and multi-scale fusion; in the WT-FT-WT cascaded space-frequency fusion module, the feature stream is sequentially processed by local wavelet decomposition, global Fourier enhancement, and local wavelet reconstruction to obtain a feature map containing global semantic enhancement. The frequency components of the enhanced feature map in the channel dimension are remapped back to the spatial dimension to restore the spatial resolution of the feature map and generate an output feature map that fuses local high-frequency details and global low-frequency semantics. In the semantic alignment fusion module, spatial domain features and frequency domain reconstructed features are fused.

6. The method for small target detection in UAV images based on space-frequency joint and semantic alignment according to claim 5, characterized in that, The backbone network extracts multi-scale features from layers S2 and S3, and inputs the feature maps of layers S2 and S3 into the multi-scale focusing module for lossless intermittent sampling and multi-scale fusion. The multi-scale focusing module performs multi-level intermittent sampling on the features of layer S2, adjusts the channel dimension through the channel adjustment module, performs dual-path feature fusion, and generates feature map I with a resolution of 160×160 and feature map II with a resolution of 80×80.

7. The method for small target detection in UAV images based on space-frequency joint and semantic alignment according to claim 6, characterized in that, Feature map I and feature map II are input into the WT-FT-WT cascaded space-frequency fusion module, and are processed sequentially through a local frequency domain dimensionality reduction unit, a global frequency domain sensing unit, and a space-frequency collaborative reconstruction unit. First, the input spatial feature map is decomposed into low-frequency and high-frequency components with spatial locality through discrete wavelet transform. Then, a global fast Fourier transform is performed on the low-frequency components rich in semantic information to capture the global contextual dependencies of the image in the frequency domain and perform feature enhancement. Finally, the enhanced global frequency domain features are restored by inverse Fourier transform and reconstructed with the preserved high-frequency detail components by inverse wavelet transform.

8. The method for small target detection in UAV images based on space-frequency joint and semantic alignment according to claim 5, characterized in that, The specific implementation of the channel attention weighted fusion module is as follows: Let the spatial domain feature be... Frequency domain reconstruction features are Perform concatenation along the channel dimension to obtain a hybrid feature map. Global average pooling is performed on the hybrid feature map to obtain the channel descriptor z; z is input into two fully connected layers. The first fully connected layer compresses the number of channels and activates them with ReLU, and the second fully connected layer restores the number of channels and activates them with Sigmoid to generate a weight vector s; the weight vector s is multiplied with the original hybrid feature map channel by channel to obtain the fused feature.

9. The method for small target detection in UAV images based on space-frequency joint and semantic alignment according to claim 1, characterized in that, In step S4, the decoder is a multi-layer Transformer decoder structure, with each layer containing multi-head self-attention, multi-scale deformable cross-attention, and a feedforward network. The input to the decoder is the object query vector filtered by the uncertainty minimization mechanism and the semantic enhancement feature sequence output by the semantic alignment fusion module. The association between queries is established through self-attention, the target-related features are extracted from the fused features through cross-attention, and finally the prediction results are output through classification and regression branches.

Images