Unmanned aerial vehicle image detection method and system based on frequency domain multi-perception collaborative fusion

Abstract

The present application relates to the technical field of image analysis, in particular to a UAV image detection method and system based on frequency domain multi-perception collaborative fusion; the method of the present application first extracts low-level features from a UAV image, and obtains four layers of low-level spatial features through processing by different times of residual modules; after multi-head attention processing is performed on the fourth layer of features, gradient attention sampling and multi-layer low-level feature layer-by-layer fusion are performed, combined with frequency multi-perception enhancement and down-sampling fusion to obtain medium-high layer features, and then semantic alignment fusion multi-features are formed to obtain frequency domain perception fusion features, and finally target detection is completed; the method of the present application can greatly improve the capture ability of small object detail features in a scene with small object size and complex background, thereby significantly improving the detection accuracy of small objects.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, specifically to a method and system for UAV image detection based on frequency domain multi-sensor collaborative fusion. Background Technology

[0002] Image-based UAV target detection has been widely applied in recent years in fields such as urban planning, environmental monitoring, and disaster response. However, traditional image-based target detection methods mainly rely on manually designed features and rules, which have limitations when dealing with dynamically changing aerial scenes. Especially when detecting small objects and occluded objects, traditional methods often perform poorly. Therefore, target detection models utilizing global feature information have become an important direction for solving this problem.

[0003] Traditional image analysis methods for UAV target detection typically rely on multi-stage detection processes. While these methods have achieved some accuracy, their high computational complexity makes them unsuitable for real-time performance. With the development of deep learning technology, single-stage detection models based on CNNs and Transformers have gradually become mainstream. These models achieve a better balance between accuracy and real-time performance through global information modeling and efficient feature extraction. However, most existing Transformer models still face the problems of high computational cost and high model complexity, making them difficult to deploy efficiently on resource-constrained UAV hardware platforms.

[0004] Furthermore, most existing image analysis object detection methods rely on fixed-scale feature extraction and fixed anchor box design, which leads to poor performance when dealing with objects with large scale variations. Especially in UAV images, due to the small size of objects and complex backgrounds, how to capture detailed features and improve the detection accuracy of small objects has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for UAV image detection based on frequency domain multi-sensor collaborative fusion.

[0006] The technical solution of this invention is as follows: A UAV image detection method based on frequency domain multi-sensor collaborative fusion includes the following operations: S1. After low-level feature extraction, the UAV image is processed by residual modules once, twice, three times, and four times to obtain the first low-level spatial feature map, the second low-level spatial feature map, the third low-level spatial feature map, and the fourth low-level spatial feature map. The fourth low-level spatial feature map is processed by a multi-head attention mechanism to obtain the fourth low-level spatial attention feature map. S2. The fourth low-level spatial attention feature map is processed by gradient attention sampling to obtain the first gradient attention sampling map. This map is then fused with the third low-level spatial feature map to obtain the first sampled fused feature map. The first sampled fused feature map is processed by gradient attention sampling to obtain the second gradient attention sampling map. This map, along with the first and second low-level spatial feature maps, is processed by frequency multi-sensor enhancement to obtain the frequency-aware feature map. The frequency-aware feature map is downsampled and then fused with the second gradient attention sampling map to obtain the middle-level feature map. The middle-level feature map is downsampled and then fused with the first gradient attention sampling map to obtain the high-level feature map. The frequency-aware feature map and the high-level feature map are semantically aligned to obtain the semantically aligned feature map. The middle-level feature map, the high-level feature map, and the semantically aligned feature map are then fused to obtain the frequency domain-aware fused feature map. S3. The frequency domain sensing fusion feature map is processed by target detection to obtain the UAV image detection result.

[0007] The gradient attention sampling process in S2 is as follows: The horizontal and vertical gradients of the fourth low-level spatial attention feature map are obtained, the gradient magnitudes are calculated, and a confidence map is obtained after normalization. The fourth low-level spatial attention feature map is then processed using a self-attention mechanism to obtain a self-attention feature map. The confidence map and the self-attention feature map are then weighted and fused to obtain a weighted coefficient feature map. Based on the weighted coefficient feature map and a preset scaling factor, the fourth low-level spatial attention feature map is bilinearly interpolated to obtain a linear interpolation feature map. The image offset of the fourth low-level spatial attention feature map is obtained, rearranged by pixels, and used as the feature map offset of the linear interpolation feature map. Based on the weighted coefficient feature map and a preset sampling grid, the linear interpolation feature map is pixel-level sampled to obtain an initial attention sampling feature map. The initial attention sampling feature map is then processed by convolution, batch normalization, and the SiLU activation function to obtain the first gradient attention sampling map.

[0008] The frequency multi-sensor enhancement processing in S2 is as follows: The second gradient attention sampling map is fused with the first and second low-level spatial feature maps, and then convolved to obtain a fused sampling space feature map. This fused sampling space feature map is then convolved and average pooled to obtain a fused sampling space pooled map. This pooled map is then subjected to horizontal and vertical convolutions, followed by convolution and activation function processing to obtain a spatial context feature map. Finally, the fused sampling space pooled map undergoes a Fourier transform, and the absolute value of the Fourier transform result is taken, followed by convolution. The frequency domain feature map is obtained. The spatial context feature map and the frequency domain feature map are multiplied element-wise, and then an inverse Fourier transform is performed. After convolution and activation function processing, a spatially sensed feature map is obtained. The sampled spatial fusion pooling map is convolved to obtain feature maps of different channels. These feature maps are then subjected to depthwise separable convolutions with 3, 5, and 7 channels, respectively. After convolution processing, they are added element-wise and then convolved to obtain channel-aware feature maps. The channel-aware feature maps and the spatially sensed feature maps are weighted, fused, and convolved to obtain the frequency-aware feature map.

[0009] In S2, the feature fusion process between the first gradient attention sampling map and the third low-level spatial feature map is as follows: the first gradient attention sampling map and the third low-level spatial feature map are convolved and then superimposed on the first gradient attention sampling map and the third low-level spatial feature map, and then processed by the SiLU activation function to obtain the first sampling fusion feature map.

[0010] In S1, low-level feature processing operations can be implemented through convolutional layers, batch normalization layers, and activation functions.

[0011] In S1, the residual module consists of convolutional layers, batch normalization layers, activation functions, and residual connections.

[0012] A UAV image detection system based on frequency domain multi-sensor collaborative fusion, used to implement the aforementioned UAV image detection method based on frequency domain multi-sensor collaborative fusion, includes: The low-level feature extraction module is used to process the UAV image after low-level feature extraction by performing residual module processing one, two, three, and four times respectively to obtain the first low-level spatial feature map, the second low-level spatial feature map, the third low-level spatial feature map, and the fourth low-level spatial feature map; the fourth low-level spatial feature map is processed by a multi-head attention mechanism to obtain the fourth low-level spatial attention feature map. The frequency domain-aware fusion feature map generation module is used to process the fourth low-level spatial attention feature map through gradient attention sampling to obtain a first gradient attention sampling map, which is then fused with the third low-level spatial feature map to obtain a first sampled fusion feature map. The first sampled fusion feature map is then processed through gradient attention sampling to obtain a second gradient attention sampling map, which, along with the first and second low-level spatial feature maps, undergoes frequency multi-sensor enhancement processing to obtain a frequency-aware feature map. The frequency-aware feature map is then downsampled and fused with the second gradient attention sampling map to obtain a mid-level feature map. The mid-level feature map is then downsampled and fused with the first gradient attention sampling map to obtain a high-level feature map. The frequency-aware feature map and the high-level feature map are semantically aligned to obtain a semantically aligned feature map. Finally, the mid-level feature map, the high-level feature map, and the semantically aligned feature map are fused to obtain the frequency domain-aware fusion feature map. The UAV image detection result generation module is used to process the frequency domain perception fusion feature map into target detection results to obtain UAV image detection results.

[0013] A drone image detection device based on frequency domain multi-sensor collaborative fusion includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the aforementioned drone image detection method based on frequency domain multi-sensor collaborative fusion.

[0014] A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described UAV image detection method based on frequency domain multi-sensor collaborative fusion.

[0015] The beneficial effects of this invention are as follows: This invention provides a UAV image detection method based on frequency domain multi-sensor collaborative fusion. First, low-level features are extracted from the UAV image, and then processed by residual modules at different times to obtain four layers of low-level spatial features. After multi-head attention processing of the fourth layer features, gradient attention sampling is used to fuse the features layer by layer with the multi-layer low-level features. Combined with frequency multi-sensor enhancement and downsampling fusion, mid-to-high-level features are obtained. Finally, semantic alignment is used to fuse multiple features to form frequency domain perception fusion features, ultimately completing target detection. This method preserves low-level detail features at different scales through multi-layer residuals, accurately focuses effective feature information through gradient attention sampling, strengthens frequency domain detail expression through frequency multi-sensor enhancement, and effectively integrates feature information at different levels through multi-scale feature fusion and semantic alignment. This significantly improves the ability to capture small object detail features in scenarios with small object sizes and complex backgrounds, thereby significantly improving the detection accuracy of small objects. This invention provides a UAV image detection method based on frequency domain multi-sensor collaborative fusion. Through innovative design of multi-scale feature extraction, gradient attention sampling processing, and frequency multi-sensor enhancement processing, it significantly improves the detection capability of small objects and occluded objects. This invention combines the latest technologies in deep learning to overcome the limitations of traditional target detection methods in dynamic aerial scenes, especially the bottleneck in complex backgrounds and small object detection. It provides an efficient and accurate solution for UAV image target detection and promotes the widespread application of UAV intelligent systems in complex environments. This invention provides a UAV image detection method based on frequency domain multi-sensor collaborative fusion. Unlike traditional multi-stage detection methods, this invention adopts a single-stage detection framework based on Transformer, which significantly reduces computational complexity and improves inference speed. This design ensures high detection accuracy while optimizing the model's inference efficiency, ensuring that UAV image target detection can run efficiently on resource-constrained platforms. Attached Figure Description

[0016] The solutions and advantages of this application will become clear to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0017] In the attached diagram: Figure 1 This is a schematic diagram of the process structure of the method in this embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the exemplary embodiments of this application clearer, the technical solutions in the exemplary embodiments of this application are described clearly and completely below. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0019] This embodiment provides a UAV image detection method based on frequency domain multi-sensor collaborative fusion, see [link to relevant documentation]. Figure 1 This includes the following operations: S1. After low-level feature extraction, the UAV image is processed by residual modules once, twice, three times, and four times to obtain the first low-level spatial feature map, the second low-level spatial feature map, the third low-level spatial feature map, and the fourth low-level spatial feature map. The fourth low-level spatial feature map is processed by a multi-head attention mechanism to obtain the fourth low-level spatial attention feature map. S2. The fourth low-level spatial attention feature map is processed by gradient attention sampling to obtain the first gradient attention sampling map. This map is then fused with the third low-level spatial feature map to obtain the first sampled fused feature map. The first sampled fused feature map is processed by gradient attention sampling to obtain the second gradient attention sampling map. This map, along with the first and second low-level spatial feature maps, is processed by frequency multi-sensor enhancement to obtain the frequency-aware feature map. The frequency-aware feature map is downsampled and then fused with the second gradient attention sampling map to obtain the middle-level feature map. The middle-level feature map is downsampled and then fused with the first gradient attention sampling map to obtain the high-level feature map. The frequency-aware feature map and the high-level feature map are semantically aligned to obtain the semantically aligned feature map. The middle-level feature map, the high-level feature map, and the semantically aligned feature map are then fused to obtain the frequency domain-aware fused feature map. S3. The frequency domain sensing fusion feature map is processed by target detection to obtain the UAV image detection result.

[0020] The specific steps are detailed below.

[0021] S1. After low-level feature extraction, the UAV image is processed by residual modules once, twice, three times, and four times to obtain the first low-level spatial feature map, the second low-level spatial feature map, the third low-level spatial feature map, and the fourth low-level spatial feature map. The fourth low-level spatial feature map is processed by a multi-head attention mechanism to obtain the fourth low-level spatial attention feature map.

[0022] First, to extract low-level spatial features from the UAV image, low-level feature extraction is performed on the UAV image to obtain an initial low-level spatial feature map. Low-level feature extraction and processing can be achieved through convolutional layers, batch normalization layers, and activation functions (preferably ReLU activation function).

[0023] Then, the initial low-level spatial feature map is processed by the residual module once, twice, three times, and four times to extract spatial feature maps at different levels, resulting in the first low-level spatial feature map, the second low-level spatial feature map, the third low-level spatial feature map, and the fourth low-level spatial feature map.

[0024] The residual module consists of convolutional layers, batch normalization layers, activation functions, and residual connections. The processing flow within the residual module is as follows: the input is processed by the convolutional layers, batch normalization layers, and activation functions, and then fed back through residual connections to enhance the graph's expressive power, resulting in the output. Residual connections also effectively avoid the vanishing gradient problem in deep networks, facilitating efficient subsequent training.

[0025] Finally, the fourth low-level spatial feature map is processed by a multi-head attention mechanism to obtain the fourth low-level spatial attention feature map.

[0026] S2. The fourth low-level spatial attention feature map is processed by gradient attention sampling to obtain the first gradient attention sampling map. This map is then fused with the third low-level spatial feature map to obtain the first sampled fused feature map. The first sampled fused feature map is processed by gradient attention sampling to obtain the second gradient attention sampling map. This map is then processed with the first and second low-level spatial feature maps by frequency multi-sensor enhancement to obtain the frequency-aware feature map. The frequency-aware feature map is downsampled and then fused with the second gradient attention sampling map to obtain the middle-level feature map. The middle-level feature map is downsampled and then fused with the first gradient attention sampling map to obtain the high-level feature map. The frequency-aware feature map and the high-level feature map are semantically aligned to obtain the semantically aligned feature map. The middle-level feature map, the high-level feature map, and the semantically aligned feature map are then fused to obtain the frequency domain-aware fused feature map.

[0027] First, the fourth low-level spatial attention feature map is processed by gradient attention sampling to obtain the first gradient attention sampling map, which is then processed with the third low-level spatial feature map to obtain the first sampling fusion feature map.

[0028] The gradient attention sampling process is performed as follows.

[0029] Step 1: Use the Sobel operator to calculate the gradient information of the fourth low-level spatial attention feature map, obtain the horizontal and vertical gradients, calculate the gradient magnitude, and obtain the confidence map after normalization, which is used to represent the importance of each pixel in the feature map.

[0030] Normalization can be achieved using the following formula: , This represents the confidence level of a pixel location in the confidence map, ranging from [0,1]. For gradient magnitude, , These are the minimum and maximum values ​​of the gradient magnitude, respectively.

[0031] Simultaneously, the fourth low-level spatial attention feature map is processed using a self-attention mechanism to obtain a self-attention feature map.

[0032] Step 2: The confidence map and the self-attention feature map are weighted and fused to obtain a weighted coefficient feature map, which is used to adjust the sampling position and intensity of the image. Based on the weighted coefficient feature map and the preset scaling factor, the fourth low-level spatial attention feature map is bilinearly interpolated to obtain a high-resolution linear interpolation feature map.

[0033] Step 3: Obtain the image offset of the fourth low-level spatial attention feature map, perform pixel shaving (PixelShuffle) and use it as the feature map offset of the high-resolution linear interpolation feature map; based on the weighted coefficient feature map and the preset sampling grid, perform pixel-level sampling on the high-resolution linear interpolation feature map to obtain the initial attention sampling feature map.

[0034] Step 4: The initial attention sampling feature map is processed by convolution, batch normalization and SiLU activation function to obtain the first gradient attention sampling map.

[0035] Gradient attention sampling processing uses gradients and attention mechanisms to select and adjust the sampling of important regions in an image. It optimizes the detection effect by dynamically adjusting the pixel sampling position and intensity, thereby enhancing the detection capability of small objects and occluded objects.

[0036] The above feature fusion operation is as follows: the first gradient attention sampling map and the third low-level spatial feature map are respectively convolved and then superimposed with the first gradient attention sampling map and the third low-level spatial feature map. After processing by the SiLU activation function, the first sampling fusion feature map is obtained.

[0037] Then, the first sampled fusion feature map is processed by gradient attention sampling to obtain the second gradient attention sampling map. Together with the first low-level spatial feature map and the second low-level spatial feature map, it is processed by frequency multi-sensory enhancement to enhance the perception capability of the input data and finally generate a richer feature representation, resulting in a frequency-aware feature map.

[0038] The specific operations of frequency multi-sensor enhancement processing are as follows.

[0039] Step 1: After fusing the second gradient attention sampling map with the first low-level spatial feature map and the second low-level spatial feature map, the sampling space fusion feature map is obtained by convolution.

[0040] Step 2: The sampled spatial fusion feature map is subjected to convolution and average pooling to obtain a sampled spatial fusion pooled map. The sampled spatial fusion pooled map is then subjected to horizontal convolution (convolution scale of 1×7) and vertical convolution (convolution scale of 7×1) to capture spatial context information. After convolution processing and activation function processing (preferably Sigmoid function), multi-channel information is extracted and fused to obtain a spatial context feature map. The sampled spatial fusion pooled map is then transformed to the frequency domain by 2D Fourier transform to extract global information in the frequency domain. The absolute value of the Fourier transform result is then taken and processed by 1x1 convolution to obtain a frequency domain feature map. The spatial context feature map and the frequency domain feature map are multiplied element-wise and then subjected to inverse Fourier transform to transform back to the spatial domain. After processing by 1x1 convolution and SiLU activation function, a spatially aware feature map is obtained.

[0041] Meanwhile, to further extract features at different scales, the sampling space fusion pooling map is convolved to obtain feature maps of different channels. These feature maps are then subjected to 3x3 depthwise separable convolutions with 3, 5, and 7 channels, and then further processed by 1x1 convolutions. After element-wise addition and 1x1 convolution processing, detailed information under different receptive fields is captured to obtain channel-aware feature maps.

[0042] Step 3: Perform weighted fusion and 1x1 convolution on the channel-aware feature map and the spatial-aware feature map to obtain the frequency-aware feature map.

[0043] Next, the frequency-aware feature map is downsampled and then fused with the second gradient attention sampling map to obtain the middle-layer feature map; the middle-layer feature map is downsampled and then fused with the first gradient attention sampling map to obtain the high-layer feature map.

[0044] Subsequently, the frequency-aware feature map and the high-level feature map are semantically aligned to obtain a semantically aligned feature map.

[0045] The semantic alignment operation is as follows: the frequency-aware feature map is processed by 3x3 convolution to obtain the frequency-aware convolutional feature map; the high-level feature map is processed by 3x3 convolution and upsampling to obtain the high-level upsampled feature map; the high-level upsampled feature map is Fourier transformed and then added element-wise with the high-level upsampled feature map to obtain the high-level sampled fusion feature map; the high-level sampled fusion feature map and the frequency-aware convolutional feature map are weighted and fused to achieve semantic alignment, resulting in the semantically aligned feature map.

[0046] Finally, the mid-level feature map, high-level feature map, and semantically aligned feature map are fused to obtain the frequency domain-aware fused feature map.

[0047] S3. The frequency domain sensing fusion feature map is processed by target detection to obtain the UAV image detection result.

[0048] The above object detection processing can be implemented using an object detector, which mainly consists of a stack of multiple Transformer decoder layers and a final Softmax layer. Each Transformer decoder layer contains two main components: a self-attention layer (used to capture the relationship between different positions in the input features) and a cross-attention layer (matching the encoder output with the decoder query for object detection).

[0049] The final Softmax layer uses the Softmax function to classify the output of each query to obtain the category of the target. The calculation formula is as follows: , in, For the category of the target, It is the final output feature of the decoder. and These are the weights and biases for class prediction, respectively.

[0050] For each query, predict the coordinates of a bounding box. The calculation formula is as follows: , in It is the regression result of the target bounding box, representing the location of the target bounding box (e.g., the coordinates of the top left and bottom right corners). and These are the weights and biases for bounding box prediction, respectively.

[0051] To verify the effectiveness of the method in this embodiment, the following experiment was conducted, comparing the detection performance of the method in this embodiment with existing advanced detection models (including YOLOv8-X, YOLOv10-X, RT-DETR-R34, and FBRT-YOLO-X). The experimental results are shown in Table 1, with values ​​for Params(M), GFLOPs, AP, and other parameters. FPS represents the number of parameters, computational complexity, average precision, precision at IoU=0.5, and frames per second, respectively.

[0052] Table 1. Summary of the detection results of the method in this embodiment and existing methods

[0053] As shown in Table 1, the computational complexity (parameters and GFLOPs) of the method in this embodiment is relatively low. This embodiment has only 21.01M parameters and a computational complexity of 67.5 GFLOPs, significantly lower than other existing comparative methods. For example, YOLOv8-X and FBRT-YOLO-X have 68.2M and 22.8M parameters respectively, but their GFLOPs are 257.8 and 185.8. The lower computational complexity means that the method in this embodiment not only reduces memory usage but also makes the model more suitable for resource-constrained environments.

[0054] In terms of inference speed, the method in this embodiment demonstrates a significant improvement in FPS (frames per second), reaching 99 FPS, which is more than double that of YOLOv8-X (47 FPS) and YOLOv10-X (49 FPS). This gives the method a clear advantage in real-time applications, especially in scenarios requiring high-speed processing (such as video surveillance and autonomous driving).

[0055] Furthermore, in addition to its advantages in computational complexity and inference speed, the method in this embodiment also demonstrates excellent performance in AP (Average Precision) and AP50 (precision at IoU=0.5). This embodiment achieves an AP of 32.9, significantly outperforming YOLOv8-X (28.9) and YOLOv10-X (28.7). Regarding AP50, this embodiment also leads YOLOv8-X (46.8), achieving an AP50 of 52, demonstrating its capability in high-quality detection tasks.

[0056] This embodiment also provides a UAV image detection system based on frequency domain multi-sensor collaborative fusion, used to implement the above-mentioned UAV image detection method based on frequency domain multi-sensor collaborative fusion, including: The low-level feature extraction module is used to process the UAV image after low-level feature extraction by performing residual module processing one, two, three, and four times respectively to obtain the first low-level spatial feature map, the second low-level spatial feature map, the third low-level spatial feature map, and the fourth low-level spatial feature map; the fourth low-level spatial feature map is processed by a multi-head attention mechanism to obtain the fourth low-level spatial attention feature map. The frequency domain-aware fusion feature map generation module is used to process the fourth low-level spatial attention feature map through gradient attention sampling to obtain a first gradient attention sampling map, which is then fused with the third low-level spatial feature map to obtain a first sampled fusion feature map. The first sampled fusion feature map is then processed through gradient attention sampling to obtain a second gradient attention sampling map, which, along with the first and second low-level spatial feature maps, undergoes frequency multi-sensor enhancement processing to obtain a frequency-aware feature map. The frequency-aware feature map is then downsampled and fused with the second gradient attention sampling map to obtain a mid-level feature map. The mid-level feature map is then downsampled and fused with the first gradient attention sampling map to obtain a high-level feature map. The frequency-aware feature map and the high-level feature map are semantically aligned to obtain a semantically aligned feature map. Finally, the mid-level feature map, the high-level feature map, and the semantically aligned feature map are fused to obtain the frequency domain-aware fusion feature map. The UAV image detection result generation module is used to process the frequency domain perception fusion feature map into target detection results to obtain UAV image detection results.

[0057] In this embodiment, the specific training method for training the network model corresponding to the above system is as follows: The framework was trained for 400 epochs with a batch size of 4, and 4 images were input for each training iteration. To prevent overfitting, an early stopping mechanism was used, and the patience value was set to 20, meaning that training would automatically stop if the performance on the validation set did not improve within 20 consecutive epochs. During the optimization process, the AdamW optimizer was used, which can effectively handle the training of deep neural networks. The learner rate of the optimizer was set to 0.0001, and the momentum was set to 0.9, which helps to accelerate the training process, reduce oscillations, and improve the convergence speed. To ensure that the model can effectively handle images of various sizes, all input images were uniformly scaled to 640×640 pixels, a size that maintains sufficient image detail while ensuring training efficiency. The trained object detection framework can be used for object recognition of new drone images. For a given new drone image, it is first preprocessed and then input into the trained model, which outputs the category and location (i.e., bounding box) of each object in the image. The recognition results can be used by drones for tasks such as real-time monitoring, inspection, and map generation, especially for target detection in complex environments (such as cities, forests, and farmlands).

[0058] The training dataset employed Mosaic and Mixup techniques. The application probability of Mosaic was set to 1, meaning that Mosaic enhancement was applied to each training sample, stitching together different image regions to increase the diversity of training samples and improve the model's adaptability to different contextual information. Simultaneously, the application probability of Mixup was set to 0.2, indicating that approximately 20% of the training samples would be linearly weighted and mixed from two images and their labels, further enhancing data diversity and helping to improve the model's robustness and generalization ability.

[0059] This embodiment also provides a UAV image detection device based on frequency domain multi-sensor collaborative fusion, including a processor and a memory, wherein the processor executes the computer program stored in the memory to implement the above-mentioned UAV image detection method based on frequency domain multi-sensor collaborative fusion.

[0060] This embodiment also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described UAV image detection method based on frequency domain multi-sensor collaborative fusion.

[0061] This embodiment provides a UAV image detection method based on frequency domain multi-sensor collaborative fusion. First, low-level features are extracted from the UAV image, and then processed by residual modules at different times to obtain four layers of low-level spatial features. After multi-head attention processing of the fourth layer features, gradient attention sampling is used to fuse the features layer by layer with the multi-layer low-level features. Combined with frequency multi-sensor enhancement and downsampling fusion, mid-to-high-level features are obtained. Finally, semantic alignment is used to fuse multiple features to form frequency domain perception fusion features, ultimately completing target detection. This method preserves low-level detail features at different scales through multi-layer residuals, accurately focuses effective feature information with gradient attention sampling, strengthens frequency domain detail expression with frequency multi-sensor enhancement, and effectively integrates feature information at different levels through multi-scale feature fusion and semantic alignment. This significantly improves the ability to capture small object detail features in scenarios with small object sizes and complex backgrounds, thereby significantly improving the detection accuracy of small objects.

[0062] This embodiment provides a UAV image detection method based on frequency domain multi-sensor collaborative fusion. Through innovative design of multi-scale feature extraction, gradient attention sampling processing, and frequency multi-sensor enhancement processing, it significantly improves the detection capability of small objects and occluded objects. The method of this invention combines the latest technologies in deep learning and overcomes the limitations of traditional target detection methods in dynamic aerial scenes, especially the bottleneck in complex backgrounds and small object detection. It provides an efficient and accurate solution for UAV image target detection and promotes the widespread application of UAV intelligent systems in complex environments.

[0063] This embodiment provides a UAV image detection method based on frequency domain multi-sensor collaborative fusion. Unlike traditional multi-stage detection methods, this invention adopts a single-stage detection framework based on Transformer, which significantly reduces computational complexity and improves inference speed. This design ensures high detection accuracy while optimizing the model's inference efficiency, ensuring that UAV image target detection can run efficiently on resource-constrained platforms.

[0064] While exemplary embodiments of the invention have been described herein, many other variations or modifications conforming to the principles of the invention can be directly determined or derived from the disclosure of this invention without departing from its spirit and scope. Therefore, the scope of the invention should be understood and recognized to cover all such other variations or modifications.

Claims

1. A UAV image detection method based on frequency domain multi-sensor collaborative fusion, characterized in that, This includes the following operations: S1. After low-level feature extraction, the UAV image is processed by the residual module once, twice, three times and four times respectively to obtain the first low-level spatial feature map, the second low-level spatial feature map, the third low-level spatial feature map and the fourth low-level spatial feature map. The fourth low-level spatial feature map is processed by a multi-head attention mechanism to obtain the fourth low-level spatial attention feature map; S2. The fourth low-level spatial attention feature map is processed by gradient attention sampling to obtain the first gradient attention sampling map, which is then processed with the third low-level spatial feature map to obtain the first sampling fusion feature map. The first sampled fusion feature map is processed by gradient attention sampling to obtain the second gradient attention sampling map. The first low-level spatial feature map and the second low-level spatial feature map are then processed by frequency multi-sensor enhancement to obtain the frequency-sensor feature map. After downsampling, the frequency-aware feature map is fused with the second gradient attention sampling map to obtain the mid-level feature map. After downsampling, the mid-level feature map is fused with the first gradient attention sampling map to obtain the high-level feature map; Semantically align the frequency-aware feature map and the high-level feature map to obtain a semantically aligned feature map; The mid-level feature map, high-level feature map, and semantically aligned feature map are fused to obtain a frequency domain-aware fused feature map; S3. The frequency domain sensing fusion feature map is processed by target detection to obtain the UAV image detection result.

2. The UAV image detection method based on frequency domain multi-sensor collaborative fusion according to claim 1, characterized in that, The gradient attention sampling process in S2 is as follows: Obtain the horizontal and vertical gradients of the fourth low-level spatial attention feature map, calculate the gradient magnitude, and obtain the confidence map after normalization. The fourth low-level spatial attention feature map is processed by a self-attention mechanism to obtain a self-attention feature map; The confidence map and the self-attention feature map are weighted and fused to obtain a weighted coefficient feature map; based on the weighted coefficient feature map and a preset scaling factor, the fourth low-level spatial attention feature map is bilinearly interpolated to obtain a linear interpolation feature map. The image offset of the fourth low-level spatial attention feature map is obtained, and after pixel rearrangement, it is used as the feature map offset of the linear interpolation feature map. Based on the weighted coefficient feature map and the preset sampling grid, the linear interpolation feature map is sampled at the pixel level to obtain the initial attention sampling feature map. The initial attention sampling feature map is processed by convolution, batch normalization, and SiLU activation function to obtain the first gradient attention sampling map.

3. The UAV image detection method based on frequency domain multi-sensor collaborative fusion according to claim 1, characterized in that, The operation of frequency multi-sensor enhancement processing in S2 is as follows: The second gradient attention sampling map is fused with the first low-level spatial feature map and the second low-level spatial feature map, and then processed by convolution to obtain the sampling space fused feature map. The sampled spatial fusion feature map is convolutional and average pooling to obtain the sampled spatial fusion pooling map; The sampled spatial fusion pooling map is sequentially subjected to horizontal and vertical convolutions, followed by convolution and activation function processing to obtain the spatial context feature map; The sampled spatial fusion pooling map is subjected to Fourier transform, and the absolute value of the Fourier transform result is taken and then convolved to obtain the frequency domain feature map. After element-wise multiplication of the spatial context feature map and the frequency domain feature map, an inverse Fourier transform is performed, followed by convolution and activation function processing to obtain the spatial perception feature map. The sampling space fusion pooling map is convolved to obtain feature maps of different channels. These feature maps are then subjected to depthwise separable convolutions with 3, 5, and 7 channels, respectively. After further convolution, each feature map is added element-wise and then convolved again to obtain the channel-aware feature map. The channel-aware feature map and the spatial-aware feature map are weighted, fused, and convolved to obtain the frequency-aware feature map.

4. The UAV image detection method based on frequency domain multi-sensor collaborative fusion according to claim 1, characterized in that, In S2, the feature fusion operation between the first gradient attention sampling map and the third low-level spatial feature map is as follows: The first gradient attention sampling map and the third low-level spatial feature map are convolved and then superimposed on the first gradient attention sampling map and the third low-level spatial feature map. After being processed by the SiLU activation function, the first sampling fusion feature map is obtained.

5. The UAV image detection method based on frequency domain multi-sensor collaborative fusion according to claim 1, characterized in that, In S1, low-level feature processing operations can be implemented through convolutional layers, batch normalization layers, and activation functions.

6. The UAV image detection method based on frequency domain multi-sensor collaborative fusion according to claim 1, characterized in that, In S1, the residual module consists of convolutional layers, batch normalization layers, activation functions, and residual connections.

7. A UAV image detection system based on frequency domain multi-sensor collaborative fusion, used to implement the UAV image detection method based on frequency domain multi-sensor collaborative fusion as described in claim 1, characterized in that, include: The low-level feature extraction module is used to process the UAV image after low-level feature extraction by performing residual module processing one, two, three, and four times respectively to obtain the first low-level spatial feature map, the second low-level spatial feature map, the third low-level spatial feature map, and the fourth low-level spatial feature map; the fourth low-level spatial feature map is processed by a multi-head attention mechanism to obtain the fourth low-level spatial attention feature map. The frequency domain sensing fusion feature map generation module is used to process the fourth low-level spatial attention feature map through gradient attention sampling to obtain the first gradient attention sampling map, and then process it with the third low-level spatial feature map through feature fusion to obtain the first sampling fusion feature map. The first sampled fusion feature map is processed by gradient attention sampling to obtain the second gradient attention sample map. This second gradient attention sample map is then processed with the first low-level spatial feature map and the second low-level spatial feature map by frequency multi-sensor enhancement to obtain the frequency-aware feature map. The frequency-aware feature map is then downsampled and fused with the second gradient attention sample map to obtain the mid-level feature map. The mid-level feature map is then downsampled and fused with the first gradient attention sample map to obtain the high-level feature map. Semantically align the frequency-aware feature map and the high-level feature map to obtain a semantically aligned feature map; The mid-level feature map, high-level feature map, and semantically aligned feature map are fused to obtain a frequency domain-aware fused feature map; The UAV image detection result generation module is used to process the frequency domain perception fusion feature map into target detection results to obtain UAV image detection results.

8. A UAV image detection device based on frequency domain multi-sensor collaborative fusion, characterized in that, It includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the UAV image detection method based on frequency domain multi-sensor collaborative fusion as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the UAV image detection method based on frequency domain multi-sensor collaborative fusion as described in any one of claims 1-6.

Images