Aerial image small target detection method and system based on mixed lossless spatial frequency domain perception

Abstract

The application provides a kind of aerial image small target detection method and system based on mixed lossless spatial frequency domain perception, it is related to computer vision and deep learning technical field.The method solves the problem of small target information loss and background interference caused by down-sampling, which includes: normalizing aerial image and constructing sample set by CP-Mosaic enhancement;Input detection Transformer model, retain small target fine-grained features through LFR-LSKNet backbone network lossless down-sampling, enhance small target high-frequency features through SDF-CCFF module, and output category and bounding box by prediction head;Adopt Varifocal Loss and GIoU Loss combination loss function training;Finally, the detection result is output by confidence threshold value.The application significantly improves the small target detection accuracy and anti-interference ability in complex scene through whole-process collaborative design.

Description

Technical Field

[0001] This application relates to the fields of computer vision and deep learning technology, and in particular to a method and system for small target detection in aerial images based on hybrid lossless spatial frequency domain perception. Background Technology

[0002] With the rapid development of drone technology, aerial image target detection is playing an increasingly important role in fields such as traffic monitoring, urban planning, disaster relief, and precision agriculture. Compared to general object detection tasks, small target detection in drone aerial photography scenarios has higher requirements for preserving fine-grained features and suppressing background noise. General object detection tasks process images with relatively low resolution, and the target size is generally larger than 32×32 pixels with clear features and textures. However, in drone aerial photography, images typically have higher resolution, and the size of small targets is mostly in the range of 2×2 to 32×32 pixels. The optimal anchor point settings and parameters of general object detection models are not applicable to small target detection in aerial photography. The model transfer is too large and cannot be directly applied to small target detection in aerial images.

[0003] From the perspective of the technological evolution of mainstream object detection models, they can be mainly divided into anchor-based detection models and non-anchor-based and end-to-end detection models. Anchor-based detection models perform classification and boundary regression by pre-setting a large number of candidate boxes of different scales and aspect ratios on the image, such as Faster R-CNN and YOLOv3. When dealing with small targets in aerial photography, these methods are difficult to accurately cover all targets due to the varied shapes and dense distribution of small targets. They also rely heavily on fine-tuning of hyperparameters, resulting in high computational redundancy and a tendency to miss detections on extremely small targets. Non-anchor-based and end-to-end detection models include methods represented by keypoint prediction and the recently emerging Transformer-based detector DETR. Among them, RT-DETR, as an end-to-end real-time detector based on DETR improvement, processes multi-scale features through a hybrid encoder, while abandoning the traditional non-maximum suppression post-processing. It directly predicts the object set using a query selection mechanism and the Hungarian matching algorithm, achieving outstanding results in the field of object detection.

[0004] While RT-DETR performs well in general scenarios, its direct application to small target detection in high-resolution drone aerial photography with complex backgrounds still presents some technical limitations. Aerial images typically contain small targets with varying shapes, as well as complex background noise such as dense tree textures, building edges, and water surfaces. The RT-DETR backbone network uses strided convolutional or pooling layers for continuous downsampling. For tiny targets smaller than 32×32 pixels, their spatial information may be completely filtered out after multiple downsampling steps, resulting in subsequent feature fusion failing to capture effective small target features. Furthermore, the RT-DETR encoder only extracts features in the spatial domain, making it difficult to distinguish high-frequency target textures from complex background noise, causing the model to easily misdetect background textures as targets. Summary of the Invention

[0005] This application provides a method and system for small target detection in aerial images based on hybrid lossless spatial-frequency domain perception. It is based on a detection Transformer (HLSF-DETR) architecture designed with hybrid lossless spatial-frequency domain perception, which aims to improve the feature discrimination capability of small targets in complex backgrounds by integrating CP-Mosaic data augmentation, lossless feature preservation backbone network and spatial-dynamic-frequency feature coding module.

[0006] In a first aspect, this application provides a method for small target detection in aerial images based on hybrid lossless spatial frequency domain sensing, the method comprising: Acquire drone aerial image data and preprocess it to obtain a preprocessed dataset; Data augmentation is performed on the preprocessed dataset to generate a training image set; The training image set is input into a detection Transformer model based on hybrid lossless spatial-frequency domain awareness to generate predicted class probabilities and bounding box locations for targets. The detection Transformer model includes a lossless feature-preserving backbone network LFR-LSKNet, an efficient hybrid encoder, a Transformer decoder, and a prediction head. The lossless feature-preserving backbone network LFR-LSKNet extracts multi-scale features from the input images. The efficient hybrid encoder fuses and enhances the multi-scale features output from the backbone network. The Transformer decoder receives the fused features from the efficient hybrid encoder and generates a target query. The prediction head generates prediction results based on the target query output from the decoder. Calculate the loss between the predicted target and the real target, update the model parameters by combining the loss functions, and obtain the trained target detection model through iterative training. The aerial image to be detected is input into the trained target detection model, and the predicted category and bounding box of the target are output according to the decision rules.

[0007] As a preferred technical solution, methods for acquiring drone aerial image data and preprocessing it to obtain the preprocessed dataset include: For any input image , For the real number space, H raw The height of the image. W raw To adjust the image to a uniform network input size using bilinear interpolation, the pixel values ​​are normalized. The calculation formula is: in, For the image in coordinates First Pixel values ​​of the channel, and These represent the mean and standard deviation of the dataset across the three RGB channels, respectively.

[0008] As a preferred technical solution, data augmentation is performed on the preprocessed dataset to generate a training image set, including: Building a Small Goal Instance Library ,in, s 1. s 2 and s N They represent the first, the second, and the third, respectively. N A set of small target image patches extracted from the training set; at least two images are randomly selected from the dataset, and copy and paste operations are performed on each image to generate an enhanced source image set; For the input image I k and its corresponding set of original bounding boxes B k ,from S obj Randomly selected m There are 1 target samples, and for each sample s j Randomly generate a paste coordinate Form candidate bounding boxes And satisfy the Intersection over Union (IoU) ratio between the candidate location and all existing bounding boxes. (b) Constraints where b) is 0; Enhanced image generated through pixel-level blending : in, For the goal In position The binary mask at the specified location has a value of 1 in the target area and 0 elsewhere. The bounding box set is then updated. ; For a source image set consisting of enhanced images, a center stitching point is randomly generated. Place multiple images separately in Of the four regions divided, for the first... Zhang Image Its effective region after mapping for: in, For the first Each quadrant corresponds to a clipping box; The original bounding box set for each image Based on the image's position in the Mosaic composite image, for each bounding box in the set Perform translation and clipping to generate the transformed coordinates. If the ratio of the area retained by the transformed bounding box to the original area is less than a set threshold... If the invalid bounding box is removed, the bounding box coordinates of all valid targets are converted to a normalized center point coordinate format. The training image set is obtained. .

[0009] As a preferred technical solution, the lossless feature-preserving backbone network LFR-LSKNet extracts multi-scale features from the input image in the following manner: Input the trained image set into LFR-LSKNet; The LFR-LSKNet adopts a hierarchical architecture, consisting of four stages, Stage 1 to Stage 4. For each stage, the SPD-Conv module first performs lossless downsampling on the input feature map, and then multiple cascaded LSKBlocks refine the features. In the SPD-Conv module, based on the acquired input feature map... X According to the set scaling factor scale Slicing and sampling are performed to generate sub-feature sequences. Defined as: in, ; f i,j [ x , y ] as Using this as the starting point for offset, the sub-feature maps extracted by slicing at a certain step size are located in the coordinate system. Features x The row coordinates of the elements in the feature map. y The column coordinates of the elements in the feature map. i The starting point of the row offset. j This is the starting point of the column offset; The generated The individual feature maps are concatenated along the channel dimension to obtain the intermediate feature map. And by applying non-stepping convolutional layers, the number of channels is adjusted to the target output dimension. This completes the lossless downsampling in this stage, yielding the output features of the SPD-Conv module; In each LSKBlock, the output features of the SPD-Conv module are processed as follows: Features with different receptive fields are generated by decomposing large-kernel depthwise convolution sequences: in, and For two depthwise convolutions with different kernel sizes and dilation rates; X is the output feature of the SPD-Conv module, U1 is the first feature, and U2 is the second feature; The channel dimension is adjusted for the first and second features using the following formula: Among them, U i This indicates characteristics with different receptive fields. This represents the features after channel dimension adjustment. Represents unit convolution; Multi-scale information is fused through a spatial kernel selection mechanism to obtain spliced ​​features. Using average pooling and max pooling Spatial relation descriptors are extracted, pooled features are concatenated, and then passed through a convolutional layer. The Sigmoid activation function is converted into two spatial selection masks. And by fusing convolutional layers Obtain weighted attention features : The formula for calculating the spatial selection mask is: The output features of the current LSKBlock are obtained through residual connections. ; After the above four stages of processing, the backbone network finally outputs multi-scale feature maps generated by Stage 2, Stage 3, and Stage 4, denoted as follows: .

[0010] As a preferred technical solution, the high-efficiency hybrid encoder processes the multi-scale features output by the backbone network in the following manner. Integration and enhancement: The highest level features S 5. Input the attention-based intra-scale feature interaction module AIFI, which captures the global contextual information of the image through Transformer self-attention calculation and outputs the enhanced features; Will S 3. S 4. The enhanced spatial-dynamic-frequency feature encoding module SDF-CCFF is input together to perform cross-scale feature fusion; the spatial-dynamic-frequency feature encoding module includes multiple fusion modules such as Fusion, convolutional downsampling and dynamic upsampling; For any feature map input to the Fusion module, it is first divided into principal components along the channel dimension. and reserved copies ;right The frequency domain processing and spatial processing branches are run in parallel. In the frequency domain branch, FFT is used to extract features. Transform to the frequency domain and introduce the Spectral Channel Attention (SCA) mechanism to first target the features. A global average pooling (GAP) algorithm is performed, generating channel attention vectors through a convolutional layer. These channel attention vectors are then applied to the frequency domain features, and finally, an IFFT is performed to obtain the output frequency domain features. ; In the spatial branch, local spatial details are extracted through two parallel convolutions to obtain spatial features. X spatial ; Introducing learnable coefficients Fusion of frequency domain features and spatial features: in, Features of fusion; Will and Feature concatenation is performed, and channel mixing is carried out through unit convolution to obtain the output features of the fusion module. : The SDF-CCFF module utilizes the aforementioned fusion module to refine the features, combining downsampling with convolutions of a set stride and dynamic upsampling based on the DySample operator to obtain fused multi-scale features. The dynamic upsampling first uses unit convolutional layers to adjust the channel dimensions, and then uses the DySample operator to dynamically generate pixel-by-pixel sampling offsets based on the input feature content. Each position of the fused multi-scale features is obtained by irregular sampling on the input features using the Grid Sample function; wherein, the input features are the features after downsampling the output features of the fusion module through convolution with a set stride. The SDF-CCFF module outputs multi-scale features after spatial and frequency domain enhancement and fusion, which are then unfolded and stitched together as fusion features of the high-efficiency hybrid encoder.

[0011] As a preferred technical solution, the Transformer decoder receives fused features from the high-efficiency hybrid encoder and generates a target query through the following data flow: Classification and regression losses are applied to supervise the fusion features of the efficient hybrid encoder, and an uncertainty score for each feature is calculated. The uncertainty score is the distribution of the feature prediction category. Distribution of predicted positioning quality Differences between them: The initial object query and the fused features from the efficient hybrid encoder are input into a multi-layer decoding layer. Each decoding layer contains a multi-head self-attention mechanism, a cross-attention mechanism, and a feedforward neural network (FFN). The initial object query first interacts with each other through the multi-head self-attention mechanism, then interacts with multi-scale deformable attention and multi-scale image features through the cross-attention mechanism, and finally is updated by the FFN, outputting the updated object query as the target query.

[0012] As a preferred technical solution, the prediction head generates the prediction result based on the target query output by the decoder in the following ways: The updated object query output from the last layer of the Transformer decoder is input into the classification branch and the regression branch respectively. The classification branch consists of linear layers and outputs the probability distribution of the target's category. The regression branch consists of three layers of multilayer perceptron (MLP) and outputs the normalized center coordinates and width and height of the target.

[0013] As a preferred technical solution, the loss between the predicted target and the real target is calculated, the model parameters are updated by combining loss functions, and the trained target detection model is obtained through iterative training, including: The Hungarian matching algorithm is used to perform one-to-one matching between predicted bounding boxes and ground truth bounding boxes; for the th Prediction boxes and the A real frame The matching cost is determined by the following formula. : in, These are classification cost, L1 regression cost, and GIoU cost, respectively. They are respectively The weights; Determine the combined loss function, the combined loss function Decoder loss Encoder auxiliary loss composition: in, The number of multi-layer decoding layers; For the output of any layer of the decoder or the encoder, the detection loss is... Classification loss L1 regression loss and GloU loss Weighted composition: in, , and These are the weights of the classification loss, L1 regression loss, and GIoU loss, respectively. The formula for calculating the classification loss is as follows: in, The classification score predicted by the model. ; The target score; These are different weight hyperparameters; The formula for calculating the L1 regression loss is as follows: in, For the set of matched positive samples, The total number of positive samples. For the coordinates of the predicted bounding box, The coordinates of the true bounding box; The formula for calculating the GloU loss is as follows: in, MER is the smallest bounding rectangle containing both the predicted bounding box and the ground truth bounding box; By gradually updating the model parameters through a combination of loss functions and undergoing multiple rounds of iterative training, a trained object detection model with the minimum total loss is obtained.

[0014] As a preferred technical solution, the aerial image to be detected is input into the trained target detection model, and the predicted category and bounding box of the target are output according to the judgment rules, including: Image to be detected Input the trained object detection model, output For each prediction result, a confidence threshold is set. The final prediction result, Result, is obtained using the following formula: in, and These are the predicted category and bounding box location of the target, respectively. The confidence score is... and These represent the predicted class probability of the target and the cross-union ratio between the predicted bounding box and the ground truth bounding box, respectively.

[0015] Secondly, this application provides a small target detection system for aerial images based on hybrid lossless spatial frequency domain sensing, the system comprising: The data preprocessing module is configured to acquire drone aerial image data and perform preprocessing to obtain a preprocessed dataset. The training image generation module is configured to perform data augmentation on the preprocessed dataset to generate a training image set. The model prediction module is configured to input the training image set into a detection Transformer model based on hybrid lossless spatial-frequency domain awareness, and generate the predicted class probability and bounding box location of the target. The detection Transformer model includes a lossless feature-preserving backbone network LFR-LSKNet, an efficient hybrid encoder, a Transformer decoder, and a prediction head. The lossless feature-preserving backbone network LFR-LSKNet is used to extract multi-scale features from the input image. The efficient hybrid encoder is used to fuse and enhance the multi-scale features output by the backbone network. The Transformer decoder is used to receive the fused features from the efficient hybrid encoder and generate a target query. The prediction head is used to generate a prediction result based on the target query output by the decoder. The model training module is configured to calculate the loss between the predicted target and the real target, update the model parameters by combining loss functions, and obtain the trained target detection model through iterative training. The target prediction module is configured to input the aerial image to be detected into the trained target detection model and output the predicted category and bounding box of the target according to the judgment rules.

[0016] The method and system for small target detection in aerial images based on hybrid lossless spatial frequency domain sensing provided in this application have at least the following beneficial effects: This application proposes a lossless feature-preserving backbone network, LFR-LSKNet, which integrates the spatial-depth transformation convolutional module SPD-Conv with the large-kernel selective convolutional network LSKNet. For targets in aerial images with a pixel area smaller than 32×32 pixels, traditional convolutional networks often lose key feature information during downsampling due to strided convolutions or pooling operations. This application utilizes SPD-Conv to slice and reconstruct spatial dimension information into channel dimension, avoiding the loss of physical pixels and achieving lossless downsampling. Simultaneously, combined with LSKNet's large-kernel selective attention mechanism, it can dynamically adjust the receptive field according to the target scale, thus preserving fine-grained features and contextual information of small targets even in deep feature maps. The Spatial-Dynamic-Frequency Feature Encoding Module (SDF-CCFF) designed in this application introduces a frequency domain awareness mechanism. It uses Fast Fourier Transform (FFT) to map the feature map to the frequency domain and leverages the global characteristics of the frequency domain to separate the high-frequency randomly distributed background noise from the target edge texture. At the same time, it combines the dynamic upsampling operator DySample to reconstruct the target edge contour by using content-based point sampling. This avoids the aliasing of semantic information and enhances the model's anti-interference ability in complex backgrounds such as cities and sea surfaces, effectively reducing the false detection rate. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] Figure 1 A flowchart illustrating an aerial image small target detection method based on hybrid lossless spatial frequency domain sensing, provided for embodiments of this application; Figure 2 A flowchart illustrating the specific implementation of a small target detection method for aerial images based on hybrid lossless spatial frequency domain sensing, provided in this application embodiment; Figure 3 A schematic diagram of the structure of the lossless feature-preserving backbone network LFR-LSKNet and LSK Block provided in the embodiments of this application; Figure 4 A schematic diagram of the spatial-dynamic-frequency feature coding module SDF-CCFF and the fusion module Fusion provided in the embodiments of this application; Figure 5This is a structural diagram of an aerial image small target detection system based on hybrid lossless spatial frequency domain sensing, provided in an embodiment of this application.

[0019] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0021] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0022] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0023] This application provides a method for small target detection in aerial images based on hybrid lossless spatial frequency domain sensing. Figure 1 The overall flowchart of this method is as follows: The input image is first fed into the lossless feature preservation backbone network LFR-LSKNet for multi-scale feature extraction. The extracted features are then processed by the attention-based intra-scale feature interaction module AIFI and the spatial-dynamic-frequency feature encoding module SDF-CCFF, and then input into the high-efficiency hybrid encoder for feature encoding. The encoded features are then fed into the Transformer decoder, which consists of multiple decoding layers. Each decoding layer is connected to a corresponding prediction head, and the prediction heads work together to output the final detection result. The entire architecture forms an end-to-end detection process from image input to detection result output.

[0024] Specifically, please combine Figure 2 As shown, the aerial image small target detection method based on hybrid lossless spatial frequency domain perception specifically includes the following steps S10-S50.

[0025] S10: Acquire drone aerial image data and preprocess it to obtain a preprocessed dataset.

[0026] In some embodiments, aerial image data from drones is acquired, and the acquired images undergo uniform size adjustment and normalization preprocessing to construct a preprocessed dataset as the original aerial image dataset. Specific implementation methods are as follows: A drone equipped with a high-resolution camera was used to collect visible light images of urban, sea, and wilderness scenes at different altitudes (50m-200m) and angles (30°-90°). Let the original set of collected images be... ,in The total number of images, I 1. I 2 and I N These are the 1st, 2nd, and Nth original images, respectively. For any input image... , For the real number space, H raw The height of the image. W raw Width of the image: 1) Use bilinear interpolation to adjust the image to a uniform network input size. ; 2) Standardize the image pixels to accelerate network convergence. Let... For the image in coordinates First Pixel values ​​of the channel, normalized pixel values The calculation formula is: in, and These represent the mean and standard deviation of the dataset across the three RGB channels, respectively.

[0027] S20: Perform data augmentation on the preprocessed dataset to generate a training image set.

[0028] In some embodiments, the preprocessed dataset is augmented with CP-Mosaic data according to a preset probability to increase sample diversity and coverage of complex backgrounds, thereby generating a training image set. Specific implementation methods are as follows: First, build a small goal instance library. Each of them This represents a small image patch of a target extracted from the training set. Four images are randomly selected from the dataset. Perform copy and paste operations on the input image respectively. and its corresponding set of original bounding boxes Perform the following steps: from Randomly selected For each target sample, Randomly generate a paste coordinate Form candidate bounding boxes To ensure that the newly pasted target does not obscure the existing target, the candidate positions must satisfy the following constraints: That is, the intersection-union ratio between the bounding box of the new target and all existing bounding boxes in the graph is 0.

[0029] Enhanced image generated through pixel-level blending : in, For the goal In position The binary mask is used, with mask values ​​of 1 in the target region and 0 elsewhere. The bounding box set is updated to the union of the original set and the newly added target set. Generate an enhanced source image set .

[0030] For the source image set Randomly generate a center splicing point The coordinates of this point are in the output image size. Floating within the central area. Place the four images respectively... In the four regions divided into top left, top right, bottom left, and bottom right, for the first... Zhang Image Its effective region after mapping for: in, For the first The clipping frames corresponding to each quadrant. Any portion exceeding the boundaries will be discarded, and insufficient portions will be padded with gray values. Correspondingly, for each image, the original set of bounding boxes... Based on the image's position in the Mosaic composite image, for each bounding box in the set Perform translation and clipping to generate the transformed coordinates. If the ratio of the area retained by the transformed bounding box to the original area is less than a threshold... If an invalid bounding box is found, it is removed to prevent the generation of noise labels. Then, the bounding box coordinates of all valid targets are converted from the pixel domain to a normalized center point coordinate format. .

[0031] In constructing the training image set During the traversal process, in order to ensure that the model can balance its ability to analyze complex scenes with its ability to generalize to real natural images, this method adopts a method based on a preset trigger probability. The system employs a dynamic enhancement mechanism. Each time a training sample is read, the system generates a set of parameters that conform to... Uniformly distributed random numbers : 1) If If this occurs, the CP-Mosaic enhancement mechanism is triggered, randomly selecting 4 images from the dataset and performing the above copy-paste, stitching, and label recalculation process to generate a large-format enhanced image. 2) If If the CP-Mosaic mechanism is skipped, only one image is extracted from the dataset without any enhancement.

[0032] The above-mentioned triggering logic continuously traverses the original dataset, dynamically generating a training image set containing diverse scene features within each training iteration cycle. .

[0033] S30: Input the training image set into the detection Transformer model based on hybrid lossless spatial-frequency domain awareness to generate the predicted class probability and bounding box location of the target. The detection Transformer model includes a lossless feature-preserving backbone network LFR-LSKNet, an efficient hybrid encoder, a Transformer decoder, and a prediction head. The lossless feature-preserving backbone network LFR-LSKNet is used to extract multi-scale features from the input image. The efficient hybrid encoder is used to fuse and enhance the multi-scale features output by the backbone network. The Transformer decoder is used to receive the fused features from the efficient hybrid encoder and generate a target query. The prediction head is used to generate a prediction result based on the target query output by the decoder.

[0034] In some embodiments, the training image set is input into the detection Transformer model HLSF-DETR based on hybrid lossless spatial frequency domain awareness to generate the predicted class probability and bounding box location of the target. This detection Transformer model mainly consists of a lossless feature-preserving backbone network LFR-LSKNet, an efficient hybrid encoder, a Transformer decoder, and a prediction head. Its data processing flow is shown in steps S301-S304 below.

[0035] S301: Input the training image set obtained in step S20 into the backbone network for multi-scale feature extraction. This backbone network adopts a hierarchical architecture, comprising four stages (Stage 1 to Stage 4). Each stage consists of two core components: an SPD-Conv module and an LSKBlock stacked module. The overall structure is as follows: Figure 3 As shown, LFR-LSKNet consists of four stages, from Stage 1 to Stage 4. Each stage begins with an SPD Layer. The scaling factor of the SPD Layer in Stage 1 is 4, while the scaling factor of the SPD Layers in Stages 2 through 4 is 2. The SPD Layer in Stage 1 is followed by a 3×3 convolution with a stride of 1, and the remaining SPD Layers are followed by a 1×1 convolution with a stride of 1. After the convolutions, different numbers of LSK Blocks are stacked: 3 in Stage 1, 4 in Stage 2, 6 in Stage 3, and 3 in Stage 4. Each stage outputs features S2, S3, and S4, respectively. Inside the LSK Block, the input is first processed by a 5×5 depthwise convolution with a dilation rate of 1. The resulting features are then divided into two branches. The first branch directly performs a 1×1 convolution with a stride of 1, while the second branch further processes the input by a 7×7 depthwise convolution with a dilation rate of 3, followed by a 1×1 convolution with a stride of 1. Subsequently, the features from the two branches are concatenated and then sequentially processed using average pooling and max pooling combined with a 7×7 convolution and a sigmoid activation function to generate spatial selection weights. These weights are then sliced ​​according to their respective branches, multiplied and added to the features of the two branches, and then fused together using a 1×1 convolution. Finally, this fused feature is multiplied element-wise with the original input features to output the processed feature. These components work together to achieve lossless extraction of fine-grained features from small targets.

[0036] Specifically, SPD-Conv is located at the beginning of each Stage, and the input feature map (i.e., a certain image in the training image set) is... The size is For Stages 1-4, set the scaling factor. According to the set scaling factor Perform slice sampling for any position on the output feature map. and the Each sub-image channel has pixel values ​​derived from specific locations in the input feature map. The generated sub-feature sequence Defined as: in, ; f i,j [x , y ] as Using this as the starting point for offset, the sub-feature maps extracted by slicing at a certain step size are located in the coordinate system. Features x The row coordinates of the elements in the feature map. y The column coordinates of the elements in the feature map. i The starting point of the row offset. j This is the starting point of the column offset.

[0037] The generated The individual feature maps are concatenated along the channel dimension to obtain the intermediate feature map. Its size becomes This step ensures that all pixel information in physical space is transferred to the channel dimension without loss. Subsequently, a non-stretched convolutional layer with a stride of 1 is applied, and the kernel size is defined. for: Use this convolutional layer to increase the number of channels from Adjust to target output dimension This completes the fusion and dimensionality reduction of features.

[0038] The LSK Block stack module is located after the SPD-Conv layer and consists of multiple LSK Blocks connected in series. For Stages 1-4, the number of LSK Blocks stacked is set. For each LSK block, the process includes two parts: large kernel decomposition and spatial kernel selection. To obtain a large receptive field while maintaining low computational cost, [the following is omitted as the text is incomplete and requires further context]. The large kernel convolution decomposes into two depthwise convolutions. and Let the first Deep convolution The kernel size is The expansion rate is , Then we have: Input features (i.e., the features output by SPD-Conv) are sequentially processed through sequence convolution to generate features with different receptive fields. : U1 is the first feature, and U2 is the second feature.

[0039] For each output through Convolution adjusts channel dimensions: Among them, U i This indicates characteristics with different receptive fields. This represents the features after channel dimension adjustment. This represents cell convolution.

[0040] In spatial kernel selection, all features are first concatenated to obtain... Using average pooling and max pooling Spatial relation descriptors are extracted, pooled features are concatenated, and then passed through a convolutional layer. The Sigmoid activation function is converted into two spatial selection masks. Then by fusing convolutional layers Obtain weighted attention features : in, Represents element-wise product. The final output feature Y is: Finally, the multi-scale features output from Stage 2 to Stage 4 are extracted through the backbone network. .

[0041] S302: The high-efficiency hybrid encoder is responsible for processing the multi-scale features output by the backbone network. The encoder is composed of two parts: an attention-based intra-scale feature interaction module (AIFI) and a spatial-dynamic-frequency feature encoding module (SDF-CCFF). The encoder is fused and enhanced.

[0042] In the AIFI phase, the highest level of features Transformer self-attention computation is performed to capture global contextual information of the image. In the SDF-CCFF stage, cross-scale feature fusion is performed using FPN-like and PANet-like structures, including the fusion module Fusion, convolutional downsampling, and dynamic upsampling. The overall structure is as follows: Figure 4 As shown. The SDF-CCFF receives multi-scale features such as input S3, input S4, and input S5. Input features S3 and S4 are first processed by the Fusion module. S5 is first processed by a 1×1 convolution with a stride of 1 before being sent to the Fusion module. Internally, the Fusion module divides the input features into principal and retained parts. The principal features are sent to a frequency domain processing branch consisting of FFT, channel attention, and IFFT, and to a spatial processing branch consisting of parallel 3×3 and 1×1 convolutions. The features processed by both paths are then compared with learnable coefficients. α , βAfter weighting and GELU activation, refined features are obtained. The refined features and the retained features are concatenated and then output as fused features through 1×1 convolution. SDF-CCFF also has multiple sets of 3×3 convolution downsampling components with a stride of 2 and dynamic upsampling components composed of 1×1 convolution combined with the DySample operator. The downsampling and upsampling components are cross-connected to the output and input of each Fusion module. Multiple Fusion modules and sampling components work together to complete the fusion of spatial and frequency domain features across scales. Finally, the fused multi-scale features are output for subsequent decoder processing.

[0043] The fusion module Fusion first combines the input feature maps (multi-scale features output by the backbone network) into a single module. (And features that need to be fused after upsampling or downsampling) are divided into two parts in the channel dimension: principal component One-quarter of the total channels are used for computationally intensive spatial and frequency domain processing; reserved portions... These comprise 3 / 4 of the total channels and are reserved for subsequent skip-joining. The frequency domain processing and spatial processing branches are run in parallel. In the frequency domain branch, FFT is used to process the features. Transform to the frequency domain and introduce the Spectral Channel Attention (SCA) mechanism to first target the features. Perform global average pooling (GAP) through a The convolutional layer generates a channel attention vector, which is then applied to the frequency domain features. Finally, an IFFT is performed to obtain the output frequency domain features. The calculation formula is as follows: In the spatial branching, through parallel... and Convolution extracts local spatial details: Introducing learnable coefficients Merge the results from the two branches: Finally and Perform feature splicing, through Convolution performs channel blending to obtain the output features of the fusion module. : Convolutional downsampling uses a stride of 2. The convolutional layer performs downsampling. Dynamic upsampling first uses a stride of 1. The convolutional layer adjusts the channel dimensions, and then upsampling is performed using the DySample operator. The DySample operator dynamically generates pixel-by-pixel sampling offsets based on the input feature content through a lightweight linear layer. Output feature map Each position is sampled using the Grid Sample function in the input features. Obtained by irregular sampling: Finally, the multi-scale features output after SDF-CCFF processing are flattened and stitched together, and then input into the Transformer decoder.

[0044] S303: The Transformer decoder includes minimum uncertainty query selection and multiple decoding layers. Minimum uncertainty query selection uses features with minimal uncertainty (i.e., high classification scores and accurate localization) as the initial query. Defined as the distribution of the predicted categories of this feature Distribution of predicted positioning quality Differences between them: In the implementation, classification and regression loss supervision are applied to the output features of the hybrid encoder, uncertainty scores are calculated, and the top-K features are selected as the initial query queries.

[0045] The multi-layer decoding layer consists of six Transformer Decoder Layers. Each decoding layer includes a multi-head self-attention mechanism, a cross-attention mechanism, and a feedforward neural network (FFN). The multi-head self-attention mechanism enables interaction between object queries, the cross-attention mechanism utilizes multi-scale deformable attention to enable interaction between object queries and multi-scale image features, and the FFN uses a standard multilayer perceptron (MLP) structure to output updated object queries.

[0046] S304: The prediction head outputs the target's class probability distribution and the target's normalized center coordinates and dimensions (width and height) through classification and regression branches, respectively. The classification branch consists of linear layers, and the regression branch consists of 3 MLP layers. Each layer of the multi-layer decoding outputs the updated object query and predicts new classification scores and bounding box coordinates through the prediction head.

[0047] S40: Calculate the loss between the predicted target and the real target, update the model parameters by combining the loss functions, and obtain the trained target detection model through iterative training.

[0048] In some embodiments, the loss between the predicted target and the real target is calculated, the model parameters are updated by combining the loss functions, and after iterative training, the trained target detection model is obtained. The specific implementation is as follows: Before calculating the loss, the Hungarian matching algorithm is first used to perform a one-to-one matching between the predicted bounding boxes and the ground truth bounding boxes. The regression loss is calculated for predicted bounding boxes that match as positive samples, and the classification loss is calculated for all predicted bounding boxes. For the ... Prediction boxes and the A real frame Matching cost Defined as: in, These are the classification cost, L1 regression cost, and GIoU cost, respectively. These represent the weights of each cost.

[0049] Combination loss function Decoder loss Encoder auxiliary loss composition: in, This refers to the number of decoding layers. The output of any layer in the decoder or encoder consists of detection loss. Classification loss L1 regression loss and GloU loss Weighted composition: in, , and These are the weights of the classification loss, L1 regression loss, and GIoU loss, respectively.

[0050] The classification loss uses Varifocal Loss to train the classification branch to predict the confidence score of IoU perception. The calculation formula is as follows: The classification score predicted by the model. ; The target score. These are different weight hyperparameters.

[0051] The regression loss uses L1 Loss to calculate the L1 error of the bounding box coordinates for the matched positive samples: in, For the set of matched positive samples, The total number of positive samples. For the coordinates of the predicted bounding box, These are the coordinates of the true bounding box.

[0052] The GIoU loss is calculated on the matched positive samples to optimize overlap and address the gradient problem of non-overlapping boxes. The calculation formula is as follows: in, It is the smallest bounding rectangle that contains both the predicted bounding box and the ground truth bounding box.

[0053] By gradually updating the model parameters through a combination of loss functions and undergoing multiple rounds of iterative training, the optimal object detection model with the minimum total loss after training is finally obtained.

[0054] S50: Input the aerial image to be detected into the trained target detection model, and output the predicted category and bounding box of the target according to the judgment rules.

[0055] In some embodiments, the aerial image to be detected is input into a trained target detection model, and the model outputs the predicted category and bounding box of the target according to a decision rule including a confidence threshold, thus achieving end-to-end target detection. Specific implementation methods are as follows: Image to be detected Input a pre-trained HLSF-DETR model, output For each prediction result, a confidence threshold is set. Select the fraction with a value greater than 1. The prediction bounding box is used to obtain the final prediction result Result: in, and These are the predicted category and bounding box location of the target, respectively. The confidence score is... and These represent the predicted class probability of the target and the cross-union ratio between the predicted bounding box and the ground truth bounding box, respectively.

[0056] This application also provides a small target detection system for aerial images based on hybrid lossless spatial frequency domain sensing, such as... Figure 5 As shown, the aerial image small target detection system based on hybrid lossless spatial frequency domain sensing includes: The data preprocessing module 501 is configured to acquire drone aerial image data and perform preprocessing to obtain a preprocessed dataset. The training image generation module 502 is configured to perform data augmentation on the preprocessed dataset to generate a training image set. The model prediction module 503 is configured to input the training image set into a detection Transformer model based on hybrid lossless spatial-frequency domain awareness, and generate the predicted class probability and bounding box location of the target. The detection Transformer model includes a lossless feature-preserving backbone network LFR-LSKNet, an efficient hybrid encoder, a Transformer decoder, and a prediction head. The lossless feature-preserving backbone network LFR-LSKNet is used to extract multi-scale features from the input image. The efficient hybrid encoder is used to fuse and enhance the multi-scale features output by the backbone network. The Transformer decoder is used to receive the fused features from the efficient hybrid encoder and generate a target query. The prediction head is used to generate a prediction result based on the target query output by the decoder. The model training module 504 is configured to calculate the loss between the predicted target and the real target, update the model parameters by combining the loss function, and obtain the trained target detection model through iterative training. The target prediction module 505 is configured to input the aerial image to be detected into the trained target detection model and output the predicted category and bounding box of the target according to the judgment rules.

[0057] It should be noted that the above-mentioned aerial image small target detection system based on hybrid lossless spatial frequency domain perception belongs to the same technical concept as the prior method and can achieve the same technical effect, so it will not be elaborated here.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for small target detection in aerial images based on hybrid lossless spatial frequency domain sensing, characterized in that, The method includes: Acquire drone aerial image data and preprocess it to obtain a preprocessed dataset; Data augmentation is performed on the preprocessed dataset to generate a training image set; The training image set is input into a detection Transformer model based on hybrid lossless spatial-frequency domain awareness to generate predicted class probabilities and bounding box locations for targets. The detection Transformer model includes a lossless feature-preserving backbone network LFR-LSKNet, an efficient hybrid encoder, a Transformer decoder, and a prediction head. The lossless feature-preserving backbone network LFR-LSKNet extracts multi-scale features from the input images. The efficient hybrid encoder fuses and enhances the multi-scale features output from the backbone network. The Transformer decoder receives the fused features from the efficient hybrid encoder and generates a target query. The prediction head generates prediction results based on the target query output from the decoder. Calculate the loss between the predicted target and the real target, update the model parameters by combining the loss functions, and obtain the trained target detection model through iterative training. The aerial image to be detected is input into the trained target detection model, and the predicted category and bounding box of the target are output according to the decision rules.

2. The method according to claim 1, characterized in that, Methods for acquiring drone aerial image data and preprocessing it to obtain the preprocessed dataset include: For any input image , For the real number space, H raw The height of the image. W raw To adjust the image to a uniform network input size using bilinear interpolation, the pixel values ​​are normalized. The calculation formula is: in, For the image in coordinates First Pixel values ​​of the channel, and These represent the mean and standard deviation of the dataset across the three RGB channels, respectively.

3. The method according to claim 1, characterized in that, Data augmentation is performed on the preprocessed dataset to generate a training image set, including: Building a Small Goal Instance Library ,in, s 1. s 2 and s N They represent the first, the second, and the third, respectively. N A set of small target image patches extracted from the training set; at least two images are randomly selected from the dataset, and copy and paste operations are performed on each image to generate an enhanced source image set; For the input image I k and its corresponding set of original bounding boxes B k ,from S obj Randomly selected m There are 1 target samples, and for each sample s j Randomly generate a paste coordinate Form candidate bounding boxes And satisfy the Intersection over Union (IoU) ratio between the candidate location and all existing bounding boxes. (b) Constraints where b) is 0; Enhanced image generated through pixel-level blending : in, For the goal In position The binary mask at the specified location has a value of 1 in the target area and 0 elsewhere. The bounding box set is then updated. ; For a source image set consisting of enhanced images, a center stitching point is randomly generated. Place multiple images separately in Of the four regions divided, for the first... Zhang Image Its effective region after mapping for: in, For the first Each quadrant corresponds to a clipping box; The original bounding box set for each image Based on the image's position in the Mosaic composite image, for each bounding box in the set Perform translation and clipping to generate the transformed coordinates. If the ratio of the area retained by the transformed bounding box to the original area is less than a set threshold... If the invalid bounding box is removed, the bounding box coordinates of all valid targets are converted to a normalized center point coordinate format. The training image set is obtained. .

4. The method according to claim 1, characterized in that, The lossless feature-preserving backbone network LFR-LSKNet extracts multi-scale features from the input image in the following manner: Input the trained image set into LFR-LSKNet; The LFR-LSKNet adopts a hierarchical architecture, consisting of four stages, Stage 1 to Stage 4. For each stage, the SPD-Conv module first performs lossless downsampling on the input feature map, and then multiple cascaded LSKBlocks refine the features. In the SPD-Conv module, based on the acquired input feature map... X According to the set scaling factor scale Slicing and sampling are performed to generate sub-feature sequences. Defined as: in, ; f i,j [ x , y ] as Using this as the starting point for offset, the sub-feature maps extracted by slicing at a certain step size are located in the coordinate system. Features x The row coordinates of the elements in the feature map. y The column coordinates of the elements in the feature map. i The starting point of the row offset. j This is the starting point of the column offset; The generated The individual feature maps are concatenated along the channel dimension to obtain the intermediate feature map. And by applying non-stepping convolutional layers, the number of channels is adjusted to the target output dimension. This completes the lossless downsampling in this stage, yielding the output features of the SPD-Conv module; In each LSKBlock, the output features of the SPD-Conv module are processed as follows: Features with different receptive fields are generated by decomposing large-kernel depthwise convolution sequences: in, and For two depthwise convolutions with different kernel sizes and dilation rates; X is the output feature of the SPD-Conv module, U1 is the first feature, and U2 is the second feature; The channel dimension is adjusted for the first and second features using the following formula: Among them, U i This indicates characteristics with different receptive fields. This represents the features after channel dimension adjustment. Represents unit convolution; Multi-scale information is fused through a spatial kernel selection mechanism to obtain spliced ​​features. Using average pooling and max pooling Spatial relation descriptors are extracted, pooled features are concatenated, and then passed through a convolutional layer. The Sigmoid activation function is converted into two spatial selection masks. And by fusing convolutional layers Obtain weighted attention features : The formula for calculating the spatial selection mask is: The output features of the current LSKBlock are obtained through residual connections. ; After the above four stages of processing, the backbone network finally outputs multi-scale feature maps generated by Stage 2, Stage 3, and Stage 4, denoted as follows: .

5. The method according to claim 1 or 4, characterized in that, The high-efficiency hybrid encoder processes the multi-scale features output by the backbone network in the following manner. Integration and enhancement: The highest level features S 5. Input the attention-based intra-scale feature interaction module AIFI, which captures the global contextual information of the image through Transformer self-attention calculation and outputs the enhanced features; Will S 3. S 4. The enhanced spatial-dynamic-frequency feature encoding module SDF-CCFF is input together to perform cross-scale feature fusion; the spatial-dynamic-frequency feature encoding module includes multiple fusion modules such as Fusion, convolutional downsampling and dynamic upsampling; For any feature map input to the Fusion module, it is first divided into principal components along the channel dimension. and reserved copies ;right The frequency domain processing and spatial processing branches are run in parallel. In the frequency domain branch, FFT is used to extract features. Transform to the frequency domain and introduce the Spectral Channel Attention (SCA) mechanism to first target the features. A global average pooling (GAP) algorithm is performed, generating channel attention vectors through a convolutional layer. These channel attention vectors are then applied to the frequency domain features, and finally, an IFFT is performed to obtain the output frequency domain features. ; In the spatial branch, local spatial details are extracted through two parallel convolutions to obtain spatial features. X spatial ; Introducing learnable coefficients Fusion of frequency domain features and spatial features: in, Features of fusion; Will and Feature concatenation is performed, and channel mixing is carried out through unit convolution to obtain the output features of the fusion module. : The SDF-CCFF module utilizes the aforementioned fusion module to refine the features, combining downsampling with convolutions of a set stride and dynamic upsampling based on the DySample operator to obtain fused multi-scale features. The dynamic upsampling first uses unit convolutional layers to adjust the channel dimensions, and then uses the DySample operator to dynamically generate pixel-by-pixel sampling offsets based on the input feature content. Each position of the fused multi-scale features is obtained by irregular sampling on the input features using the Grid Sample function; wherein, the input features are the features after downsampling the output features of the fusion module through convolution with a set stride. The SDF-CCFF module outputs multi-scale features after spatial and frequency domain enhancement and fusion, which are then unfolded and stitched together as fusion features of the high-efficiency hybrid encoder.

6. The method according to claim 1 or 5, characterized in that, The Transformer decoder receives fused features from the efficient hybrid encoder and generates the target query through the following data flow: Classification and regression losses are applied to supervise the fusion features of the efficient hybrid encoder, and an uncertainty score for each feature is calculated. The uncertainty score is the distribution of the feature prediction category. Distribution of predicted positioning quality Differences between them: The initial object query and the fused features from the efficient hybrid encoder are input into a multi-layer decoding layer. Each decoding layer contains a multi-head self-attention mechanism, a cross-attention mechanism, and a feedforward neural network (FFN). The initial object query first interacts with each other through the multi-head self-attention mechanism, then interacts with multi-scale deformable attention and multi-scale image features through the cross-attention mechanism, and finally is updated by the FFN, outputting the updated object query as the target query.

7. The method according to claim 1 or 6, characterized in that, The prediction head generates prediction results based on the target query output by the decoder in the following ways: The updated object query output from the last layer of the Transformer decoder is input into the classification branch and the regression branch respectively. The classification branch consists of linear layers and outputs the probability distribution of the target's category. The regression branch consists of three layers of multilayer perceptron (MLP) and outputs the normalized center coordinates and width and height of the target.

8. The method according to claim 1, characterized in that, The loss between the predicted target and the true target is calculated, the model parameters are updated by combining the loss functions, and the trained target detection model is obtained through iterative training, including: The Hungarian matching algorithm is used to perform one-to-one matching between predicted bounding boxes and ground truth bounding boxes; for the th Prediction boxes and the A real frame The matching cost is determined by the following formula. : in, These are classification cost, L1 regression cost, and GIoU cost, respectively. They are respectively The weights; Determine the combined loss function, the combined loss function Decoder loss Encoder auxiliary loss composition: in, The number of multi-layer decoding layers; For the output of any layer of the decoder or the encoder, the detection loss is... Classification loss L1 regression loss and GloU loss Weighted composition: in, , and These are the weights of the classification loss, L1 regression loss, and GIoU loss, respectively. The formula for calculating the classification loss is as follows: in, The classification score predicted by the model. ; The target score; These are different weight hyperparameters; The formula for calculating the L1 regression loss is as follows: in, For the set of matched positive samples, The total number of positive samples. For the coordinates of the predicted bounding box, The coordinates of the true bounding box; The formula for calculating the GloU loss is as follows: in, MER is the smallest bounding rectangle containing both the predicted bounding box and the ground truth bounding box; By gradually updating the model parameters through a combination of loss functions and undergoing multiple rounds of iterative training, a trained object detection model with the minimum total loss is obtained.

9. The method according to claim 1, characterized in that, The aerial image to be detected is input into the trained object detection model, which outputs the predicted category and bounding box of the object according to the decision rules, including: Image to be detected Input the trained object detection model, output For each prediction result, a confidence threshold is set. The final prediction result, Result, is obtained using the following formula: in, and These are the predicted category and bounding box location of the target, respectively. The confidence score is... and These represent the predicted class probability of the target and the cross-union ratio between the predicted bounding box and the ground truth bounding box, respectively.

10. A small target detection system for aerial images based on hybrid lossless spatial frequency domain sensing, characterized in that, The system includes: The data preprocessing module is configured to acquire drone aerial image data and perform preprocessing to obtain a preprocessed dataset. The training image generation module is configured to perform data augmentation on the preprocessed dataset to generate a training image set. The model prediction module is configured to input the training image set into a detection Transformer model based on hybrid lossless spatial-frequency domain awareness, and generate the predicted class probability and bounding box location of the target. The detection Transformer model includes a lossless feature-preserving backbone network LFR-LSKNet, an efficient hybrid encoder, a Transformer decoder, and a prediction head. The lossless feature-preserving backbone network LFR-LSKNet is used to extract multi-scale features from the input image. The efficient hybrid encoder is used to fuse and enhance the multi-scale features output by the backbone network. The Transformer decoder is used to receive the fused features from the efficient hybrid encoder and generate a target query. The prediction head is used to generate a prediction result based on the target query output by the decoder. The model training module is configured to calculate the loss between the predicted target and the real target, update the model parameters by combining loss functions, and obtain the trained target detection model through iterative training. The target prediction module is configured to input the aerial image to be detected into the trained target detection model and output the predicted category and bounding box of the target according to the judgment rules.

Images