A small target detection method based on Mamba enhanced spatial frequency domain fusion

Abstract

The application discloses a small target detection method based on Mamba enhanced space frequency domain fusion and relates to the technical fields of computer vision, artificial intelligence and image processing. The method comprises the following steps: a detection network composed of a ResNet18 backbone network, a hybrid encoder, an IoU perception query selection module and a decoder is constructed, multi-scale features of an input image are extracted through the backbone network, the multi-scale features are sequentially subjected to AIFI scale internal interaction, PMFEM feature enhancement and MECSFF space-frequency domain fusion, and finally, a detection result is output through query screening and the decoder. The application solves the problems of feature loss, poor fusion effect, insufficient global modeling, insufficient frequency domain utilization and low positioning accuracy in the existing small target detection, realizes high-precision and lightweight detection of small targets in complex scenes, and adapts to real-time deployment requirements in multiple fields.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, artificial intelligence and image processing technology, and more specifically to a small target detection method based on Mamba-enhanced spatial frequency domain fusion. Background Technology

[0002] Object detection technology is an important research direction in the field of computer vision, and it is widely used in scenarios such as security monitoring, autonomous driving, industrial inspection, drone inspection, and remote sensing analysis. Existing object detection methods mainly include detection methods based on convolutional neural networks and detection methods based on Transformer structures.

[0003] Among these, methods based on convolutional neural networks extract image features through local convolution operations. Representative methods include the YOLO series, Faster R-CNN, and RetinaNet. These methods have advantages such as fast inference speed and convenient deployment, and are widely used in real-time detection tasks.

[0004] With the development of visual Transformers, DETR and its improved methods have introduced attention mechanisms into object detection tasks, enhancing feature representation capabilities through global modeling and simplifying candidate box design and non-maximum suppression steps in traditional detection processes. RT-DETR and RT-DETRv2 further achieve a balance between real-time performance and detection accuracy, becoming important solutions for end-to-end real-time detection.

[0005] In multi-scale object detection tasks, existing methods typically employ Feature Pyramid Networks (FPN), Path Aggregation Networks (PANet), or Bidirectional Feature Pyramid Networks (BiFPN) for cross-layer feature fusion to enhance the detection capability of objects at different scales. Some studies further introduce attention mechanisms, multi-branch convolutional structures, and context modeling modules to improve the performance of small object recognition in complex scenes.

[0006] In addition, some studies have attempted to introduce frequency domain transformation methods into visual tasks, extracting high-frequency information from images through Fourier transform, wavelet transform, and other methods to enhance texture, edge, or detail features, thereby improving the detection performance of small targets.

[0007] In small target detection scenarios, existing technologies still have the following problems: 1. Feature information is easily lost: Small targets occupy a small area in the input image. After being downsampled multiple times by the backbone network, their texture, edge, and structural information is easily attenuated, resulting in insufficient features that can be used for recognition.

[0008] 2. The effect of multi-scale feature fusion is limited: Low-level features have high spatial resolution but weak semantic information; high-level features have strong semantic information but lack localization details. Existing fusion structures struggle to achieve an effective balance between semantic and detailed information.

[0009] 3. Insufficient global dependency modeling: Convolution-based feature extraction methods mainly rely on local receptive fields, which do not make sufficient use of the correlation information between distant regions. In cases of dense targets, occlusion, or complex backgrounds, the detection stability is limited.

[0010] 4. Highly complex structures are not conducive to real-time deployment: While some detection methods employing global attention mechanisms enhance the ability to model long-distance dependencies, they involve significant computational costs and struggle to meet real-time requirements in scenarios with high-resolution input or deployment on edge devices.

[0011] 5. Insufficient utilization of frequency domain features: Most existing methods mainly perform feature extraction and fusion in the spatial domain, failing to fully utilize the high-frequency information corresponding to the edge details of the target, thus affecting the ability to recognize small-sized targets.

[0012] 6. The localization loss function lacks adaptability: Existing bounding box regression loss functions are quite sensitive to small target position deviations; even a small pixel shift can cause significant loss fluctuations, thus affecting positioning accuracy.

[0013] Therefore, proposing a small target detection method based on Mamba-enhanced spatial-frequency domain fusion to address the difficulties in existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0014] In view of this, the present invention provides a small target detection method based on Mamba-enhanced spatial-frequency domain fusion to solve the technical problems existing in the prior art.

[0015] To achieve the above objectives, the present invention provides the following technical solution: A small target detection method based on Mamba-enhanced spatial frequency domain fusion includes the following steps: A detection network is constructed, which consists of a backbone network, a Transformer-Mamba hybrid encoder, an IoU-aware query selection module, and a decoder. The backbone network adopts ResNet18, and the Transformer-Mamba hybrid encoder includes: the intra-scale interaction module AIFI, the parallel multi-kernel feature enhancement module PMFEM, and the Mamba-enhanced cross-scale feature fusion module MECSFF. The Mamba-enhanced cross-scale feature fusion module MECSFF includes: the Mamba-based semantic fusion module MSF and the dual-domain feature fusion module SFDDF; The decoder uses the WGIoU loss function, which is obtained by co-optimization of NWD and GIoU. Input the image to be detected, and extract multi-scale features through the backbone network to obtain feature maps S2, S3, S4, and S5; Feature map S5 is input to the intra-scale interaction module AIFI for intra-scale interaction, and feature maps S2, S3, and S4 are input to the parallel multi-kernel feature enhancement module PMFEM for parallel multi-kernel feature enhancement. The enhanced feature input is based on the Mamba-enhanced cross-scale feature fusion module MECSFF. Semantic fusion is performed through the Mamba-based semantic fusion module MSF, and spatial and frequency domain features are collaboratively fused through the dual-domain feature fusion module SFDDF to obtain fused features. The feature IoU perception query selection module filters query features, inputs them into the decoder, optimizes bounding box regression through the WGIoU loss function, and outputs small target detection results.

[0016] Optionally, the processing procedure of the parallel multi-core feature enhancement module PMFEM is as follows: The input feature map is first halved in channel dimension by a 1×1 convolution, and then input into three parallel branches; The first branch uses 1×1 convolution to linearly reorganize the channel dimensions; the second branch uses 3×3 windmill convolution PConv to extract features. The windmill convolution PConv uses 4 sets of asymmetric padding combined with horizontal and vertical convolution kernels to capture local features in four directions, and the receptive field decays in a Gaussian distribution; the third branch uses 5×5 depthwise separable convolution to extract features. After the output features of the three branches are concatenated, they are processed by a feature focusing structure activated by global average pooling (GAP) and sigmoid, which adaptively enhances key feature channels and suppresses redundant information. Finally, the output is connected via residuals.

[0017] Optionally, the processing procedure of the Mamba-based Semantic Fusion Module (MSF) is as follows: The input feature X is preprocessed by mapping the original feature to the hidden dimension through a 1×1 convolutional layer and combining the SiLU activation function to enhance the nonlinear expression, thus obtaining the preprocessed feature X'. The preprocessed feature X' is subjected to 1×1 convolution, separable convolution with different sizes and depths, and GELU activation to enhance local features. It is then mapped back to the original channel dimension by 1×1 convolution to obtain global features X1' at multiple scales. Global features X1' at multiple scales are input into the SS2D module after layer normalization. The SS2D module expands the input image into a sequence by traversing four directions. Each sequence is processed by an independent S6 block, then scanned, merged and reconstructed into an image form, and outputs feature X2'. After feature X2' is normalized by the layer, it is expanded through a 1×1 convolution and split to obtain X. A X B ;X A Separable convolution with depth 3×3 and X A Residual connections, activated by GELU, yield X. A '; X A 'with X B Element-wise multiplication is performed, and the result is convolved with the X2' residual after 1×1 convolution to output the final fused feature Y.

[0018] Optionally, the processing procedure of the dual-domain feature fusion module SFDDF is as follows: The low-level feature map is downsampled by the Haar wavelet downsampling module (HWD) to align the high- and low-level features. The aligned features are then added together to obtain the fused feature X. merge ; Fusion Feature X merge The input spatial domain branch is processed by the Mamba-based semantic fusion module MSF to capture spatial domain information, and the output spatial domain information weight ω is generated. s ; Fusion Feature X merge The input frequency branch first undergoes a 1×1 convolution and GELU activation to introduce nonlinearity, and the absolute value is taken to highlight high-frequency information. Then, it undergoes two independent 1×1 convolutions, followed by Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT) respectively. Combined with learnable parameters α and β, the frequency domain feature X is obtained. f ''; The frequency branch is processed by a 1×1 depthwise separable convolution, ReLU activation, and a 1×1 convolution to output the frequency domain information weights ω. f High- and low-level features are respectively related to the spatial domain information weight ω s Frequency domain information weight ω f After multiplying and concatenating, we obtain feature X. concat Feature X concat After channel attention recalibration, 1×1 convolution, batch normalization, and ReLU activation, the final dual-domain fusion feature is output.

[0019] Optionally, the formula for the SFDDF processing procedure of the dual-domain feature fusion module is as follows:

[0020] Where, ω s Represents the spatial domain information weights. Represents the frequency domain information weights. This represents a dual-mapping feature. This indicates a concatenation operation along the channel dimension. Indicates channel attention weights. Indicates the height of the feature map. Indicates the width of the feature map. An index representing the spatial height direction. Indicates the index in the direction of spatial width. Indicates the characteristics after fusion. Indicates the batch normalization layer. This represents a convolution operation with a kernel size of 1×1. Indicates that the convolution kernel is k 1D convolution, It adapts to the number of input channels.

[0021] Optionally, the WGIoU loss function can be calculated as follows:

[0022] Where α is the learnable weight parameter, NWD is the normalized Wasserstein distance, the bounding box is modeled as a Gaussian distribution and the Wasserstein distance between distributions is calculated, and GIoU is the intersection-union ratio loss function.

[0023] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a small target detection method based on Mamba-enhanced spatial-frequency domain fusion, the beneficial effects of which are: 1) The Mamba-based semantic fusion module MSF is lightweight and slightly improves accuracy, the parallel multi-kernel feature enhancement module PMFEM enhances feature representation, the dual-domain feature fusion module SFDDF focuses on improving the detection effect of small targets (APsmall reaches 21.4%), and WGIoU optimizes bounding box localization. The modules complement each other and work together to improve the overall performance. 2) Robustness in complex scenarios: In extreme scenarios such as dense targets, severe occlusion, low resolution, low light, and strong reflection, it can accurately locate the boundary of small targets, reduce the false detection rate, maintain high detection confidence and accuracy, and is suitable for complex application scenarios such as drone inspection and security. 3) Adaptable to real-time deployment in multiple fields: The computing cost is controllable, balancing detection accuracy and inference speed. It can be deployed on edge devices such as drones, monitoring equipment, vehicle systems, and industrial cameras. It supports continuous detection of single images and video streams, covering the small target recognition needs of multiple industries. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0025] Figure 1 The overall flowchart of a small target detection method based on Mamba-enhanced spatial frequency domain fusion provided by the present invention; Figure 2 This is an overall structural diagram of the parallel multi-core feature enhancement module PMFEM provided by the present invention; Figure 3 The overall structure diagram of the Mamba-based semantic fusion module MSF provided by this invention; Figure 4 This is an overall structural diagram of the dual-domain feature fusion module SFDDF provided by the present invention; Figure 5 The image shows the detection results of a typical scene selected on the VisDrone dataset for this invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] See Figure 1 As shown, this invention discloses a small target detection method based on Mamba-enhanced spatial-frequency domain fusion, comprising the following steps: A detection network is constructed, which consists of a backbone network, a Transformer-Mamba hybrid encoder, an IoU-aware query selection module, and a decoder. The backbone network adopts ResNet18, and the Transformer-Mamba hybrid encoder includes: the intra-scale interaction module AIFI, the parallel multi-kernel feature enhancement module PMFEM, and the Mamba-enhanced cross-scale feature fusion module MECSFF. The Mamba-enhanced cross-scale feature fusion module MECSFF includes: the Mamba-based semantic fusion module MSF and the dual-domain feature fusion module SFDDF; The decoder uses the WGIoU loss function, which is obtained by co-optimization of NWD and GIoU. Input the image to be detected, and extract multi-scale features through the backbone network to obtain feature maps S2, S3, S4, and S5; Feature map S5 is input to the intra-scale interaction module AIFI for intra-scale interaction, and feature maps S2, S3, and S4 are input to the parallel multi-kernel feature enhancement module PMFEM for parallel multi-kernel feature enhancement. The enhanced feature input is based on the Mamba-enhanced cross-scale feature fusion module MECSFF. Semantic fusion is performed through the Mamba-based semantic fusion module MSF, and spatial and frequency domain features are collaboratively fused through the dual-domain feature fusion module SFDDF to obtain fused features. The feature IoU perception query selection module filters query features, inputs them into the decoder, optimizes bounding box regression through the WGIoU loss function, and outputs small target detection results.

[0028] Further, see Figure 2 As shown, the processing procedure of the parallel multi-core feature enhancement module PMFEM is as follows: The input feature map is first halved in channel dimension by a 1×1 convolution, and then input into three parallel branches; The first branch uses 1×1 convolution to linearly reorganize the channel dimensions; the second branch uses 3×3 windmill convolution PConv to extract features. The windmill convolution PConv uses 4 sets of asymmetric padding combined with horizontal and vertical convolution kernels to capture local features in four directions, and the receptive field decays in a Gaussian distribution; the third branch uses 5×5 depthwise separable convolution to extract features. After the output features of the three branches are concatenated, they are processed by a feature focusing structure activated by global average pooling (GAP) and sigmoid, which adaptively enhances key feature channels and suppresses redundant information. Finally, the output is connected via residuals.

[0029] Specifically, given an input feature map, it is first transformed by a 1×1 convolution to half the original channel dimension. Then, the transformed feature map is input into three parallel branches for convolution operations to extract feature information at different scales. After that, a feature focusing structure is used to adaptively enhance key feature channels and suppress redundant information.

[0030] The first branch uses 1×1 convolutions to linearly reorganize the input feature map along the channel dimension, enhancing information interaction between channels and preserving key feature information of small targets. The second branch uses a 3×3 windmill convolution (PConv) to enhance feature capture of small targets, thus improving the model's sensitivity to them. The windmill PConv uses four sets of asymmetric padding with horizontal and vertical convolution kernels to capture local features on the horizontal right, horizontal left, vertical bottom, and vertical top, respectively. The receptive field decay of this diffuse convolution kernel exhibits a Gaussian distribution, with its perceptual intensity gradually decreasing from the center to the edges. This characteristic naturally matches the spatial Gaussian distribution of small target pixels, effectively strengthening the feature representation of small targets through spatial distribution alignment. For computational cost considerations, the third branch uses a 5×5 depthwise separable convolution. In this branch, the 5×5 convolution kernel provides a larger receptive field, capturing broader contextual information while maintaining a relatively lightweight approach and compensating for the shortcomings of the first two branches in multi-scale space.

[0031] Further, see Figure 3 As shown, the processing procedure of the Mamba-based Semantic Fusion Module (MSF) is as follows: The input feature X is preprocessed by mapping the original feature to the hidden dimension through a 1×1 convolutional layer and combining the SiLU activation function to enhance the nonlinear expression, thus obtaining the preprocessed feature X'. The preprocessed feature X' is subjected to 1×1 convolution, separable convolution with different sizes and depths, and GELU activation to enhance local features. It is then mapped back to the original channel dimension by 1×1 convolution to obtain global features X1' at multiple scales. Global features X1' at multiple scales are input into the SS2D module after layer normalization. The SS2D module expands the input image into a sequence by traversing four directions. Each sequence is processed by an independent S6 block, then scanned, merged and reconstructed into an image form, and outputs feature X2'. After feature X2' is normalized by the layer, it is expanded through a 1×1 convolution and split to obtain X. A X B ;X A Separable convolution with depth 3×3 and X A Residual connections, activated by GELU, yield X. A '; X A 'with X B Element-wise multiplication is performed, and the result is convolved with the X2' residual after 1×1 convolution to output the final fused feature Y.

[0032] Specifically, the Mamba-based Semantic Fusion Module (MSF) is a multi-scale feature fusion module designed for small object detection tasks. Its design goal is to promote semantic fusion of small objects at different scales through an efficient local-global feature interaction mechanism. Leveraging the advantages of the Mamba architecture, it achieves efficient fusion of multi-scale features.

[0033] The Mamba-based semantic fusion module MSF performs input feature processing. Preprocessing is performed by mapping the original features to a hidden dimension using a 1×1 convolutional layer, and then combining this with the SiLU activation function to enhance the model's non-linear expressive power. The preprocessed features... Depthwise separable convolutions with kernels of varying sizes enhance local feature representation across different scales. A 1×1 convolutional layer enables efficient interaction of information between channels, and a non-linear activation function (GELU) further enhances feature representation. Subsequently, another 1×1 convolutional layer maps the features back to the original channel dimension, yielding global features across multiple scales. Global features at multiple scales The SS2D module effectively establishes a global receptive field, capturing overall structural and contextual semantic information. First, the input image is expanded into a sequence by traversing four directions: top-down, bottom-up, left-to-right, and right-to-left. This comprehensively covers all regions of the input image and achieves multi-directional feature capture. Then, the sequence corresponding to each scan path is processed using an independent S6 block. Finally, the four processed sequences are scanned, merged, and reconstructed into an image for further processing.

[0034]

[0035] Subsequently, the output characteristics of the SS2D module were analyzed. Further feature enhancement is performed. First, a 1×1 convolution is used to enhance the features. The number of channels was doubled, and then split into channels. and Two parts, After passing through a convolution kernel with a depth of 3×3, the convolution and... The residual connection, after passing through the GELU activation function, yields... Then, with dynamic gating branches Element-wise multiplication allows the module to adaptively and dynamically adjust feature importance, amplifying key features and suppressing redundant information. The specific calculation process is as follows.

[0036] Finally, the final output is obtained through a residual connection. .

[0037] Further, see Figure 4 As shown, the processing procedure of the dual-domain feature fusion module SFDDF is as follows: The low-level feature map is downsampled by the Haar wavelet downsampling module (HWD) to align the high- and low-level features. The aligned features are then added together to obtain the fused feature X. merge ; Fusion Feature X merge The input spatial domain branch is processed by the Mamba-based semantic fusion module MSF to capture spatial domain information, and the output spatial domain information weight ω is generated. s ; Fusion Feature X merge The input frequency branch first undergoes a 1×1 convolution and GELU activation to introduce nonlinearity, and the absolute value is taken to highlight high-frequency information. Then, it undergoes two independent 1×1 convolutions, followed by Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT) respectively. Combined with learnable parameters α and β, the frequency domain feature X is obtained. f ''; The frequency branch is processed by a 1×1 depthwise separable convolution, ReLU activation, and a 1×1 convolution to output frequency domain information weights ω. f High- and low-level features are respectively related to the spatial domain information weight ω s Frequency domain information weight ω f After multiplying and concatenating, we obtain feature X. concat Feature X concat After channel attention recalibration, 1×1 convolution, batch normalization, and ReLU activation, the final dual-domain fusion feature is output.

[0038] Furthermore, the formula for the SFDDF processing procedure of the dual-domain feature fusion module is as follows:

[0039] Where, ω s Represents the spatial domain information weights. Represents the frequency domain information weights. This represents a dual mapping feature. This indicates a concatenation operation along the channel dimension. Indicates channel attention weights. Indicates the height of the feature map. Indicates the width of the feature map. An index representing the spatial height direction. Indicates the index in the direction of spatial width. Indicates the characteristics after fusion. Indicates the batch normalization layer. This represents a convolution operation with a kernel size of 1×1. Indicates that the convolution kernel is k 1D convolution, It adapts to the number of input channels.

[0040] Specifically, the dual-domain feature fusion module SFDDF employs a parallel dual-branch architecture to achieve the collaborative fusion of spatial and frequency domain features. First, HWD is used to downsample the low-level feature map to facilitate feature alignment. HWD is a downsampling module based on Haar wavelets, which greatly preserves high-frequency detail information (such as edges and textures) and spatial location accuracy of small targets, mitigating the information loss caused by traditional downsampling methods. Aligned features... and After adding, we get The inputs are fed into a dual-branch network for feature extraction in the spatial and frequency domains, respectively. In the spatial domain branch, considering that Mamba's selective mechanism better suits the feature sparsity of the image spatial domain and surpasses traditional CNNs and Transformers in long-range dependency modeling and computational efficiency, [the following is used:] Spatial domain information is captured by the Mamba-based semantic fusion module MSF and spatial domain information weights are obtained. .

[0041]

[0042] The frequency branch design employs a frequency domain transformation approach based on Fourier transform to achieve efficient feature representation. First, non-linearity is introduced through 1×1 convolution and the GELU activation function to enhance the expressive power of the model.

[0043]

[0044] Small targets are easily obscured by the background. To extract key features more stably and reduce noise interference, [further measures are needed]. The modulo operation highlights high-frequency information such as edges and textures of small targets. The processed features are fed into two independent 1×1 convolutions, and then a Fast Fourier Transform (FFT) is used to obtain the frequency domain input. The specific frequency domain feature interaction is as follows:

[0045] Here, FFT and IFFT represent Fast Fourier Transform and Inverse Fast Fourier Transform, respectively. Cross-domain feature fusion is achieved by element-wise multiplication of spatial and frequency domain features, enhancing the response intensity of the target region. Learnable parameters α and β dynamically adjust the weight ratio between the frequency domain enhancement features and the original input, achieving adaptive residual fusion that injects frequency domain prior knowledge while preserving the original structural information.

[0046]

[0047] Another Using a 1×1 depthwise separable convolution to focus on local spatial features, and frequency domain features. This creates complementary enhancement. The frequency branch output is obtained after ReLU activation and convolution adjustment. After obtaining the spatial domain information weights and frequency domain information weights Then they are mapped to features respectively. and Then, these two branches are connected together to obtain a dual mapping feature in both the spatial and frequency domains. Simple concatenation can lead to a lack of differentiation in the importance of dual-domain features. Channel attention is used to recalibrate the dual-domain features, enhancing the edge information of small targets in the frequency domain and reducing interference from spatial background features. The channel attention weight g is then combined with... After multiplication, a 1×1 convolution is used to adjust the number of channels, followed by batch normalization (BN) and ReLU activation to obtain the final output features that fuse dual-domain features.

[0048] Furthermore, the formula for calculating the WGIoU loss function is as follows:

[0049] Where α is the learnable weight parameter, NWD is the normalized Wasserstein distance, the bounding box is modeled as a Gaussian distribution and the Wasserstein distance between distributions is calculated, and GIoU is the intersection-union ratio loss function.

[0050] Specifically, GIoU has significant limitations in small object detection scenarios. Small objects have small bounding box areas and are highly sensitive to positional deviations; relying solely on GIoU is insufficient to adequately optimize regression performance for small objects. This paper employs a strategy of co-optimizing NWD (Normalized Wasserstein Distance) with GIoU. NWD addresses the IoU sensitivity issue caused by pixel-level positional deviations in small object detection by modeling the bounding boxes as Gaussian distributions and calculating the Wasserstein distance between distributions, thereby improving the model's ability to recognize tiny objects.

[0051] To optimize the scale invariance and location robustness of bounding box regression loss, a WGIoU loss function is proposed by leveraging the complementary properties of GIoU and NWD.

[0052] In one specific embodiment, the following is included: Dataset: To evaluate the model's ability to detect small targets, the experiments used VisDrone, a mainstream dataset in the object detection field that contains a large number of small targets. The VisDrone-2019-DET dataset contains 8629 images, of which 6471 were used as the training set, 548 as the validation set, and 1610 as the test set. This dataset includes ten categories: pedestrians, people, bicycles, cars, vans, trucks, tricycles, awning tricycles, buses, and motorcycles. The experiments used the VisDrone-2019-DET training set for training and the validation set for testing.

[0053] Implementation Details: The experimental environment for this invention used Ubuntu 22.04 operating system, CUDA version 12.4, NVIDIA GeForce RTX 2080Ti GPU, Python 3.9.21, and PyTorch 2.4.0. RT-DETRv2 with ResNet18 as the backbone network was used as the baseline model. Training used the AdamW optimizer, with a batch size of 4 and 200 training epochs; all other settings were default.

[0054] To verify the effectiveness of the proposed method for detecting small-scale objects, comparisons were made with state-of-the-art YOLO models and Transformer-based object detection models on the VisDrone-2019-DET dataset. The results are shown in Table 1. The data demonstrates that the baseline model RT-DETRv2-R18 shows a significant improvement in detection accuracy compared to its predecessor, RT-DETR-R18, with a 1% increase in AP and a 1.6% increase in AP50. Furthermore, the proposed MESF-DETR-R18, due to its more efficient feature fusion method, exhibits even better performance than the baseline model. Under controllable computational costs, the proposed method achieves a detection accuracy of 30.7% AP and 51.0% AP50, representing improvements of 1.7% and 2.1% respectively compared to the baseline model. Similarly, MESF-DETR-R50 shows improvements in both AP and AP50 compared to the baseline model. It is worth noting that compared with advanced YOLO models, MESF-DETR-R18 also maintains a significant advantage in detection accuracy, especially in complex backgrounds and small target scenarios, significantly alleviating the YOLO series' problem of missing small targets. Compared with YOLOv8-L and YOLOv11-L, which have larger parameter counts and computational requirements, APsmall improves by 6.5% and 6.1%, respectively.

[0055] Table 1 Comparison of different detection models and MESF-DETR on the VisDrone-2019-DET dataset

[0056] To further verify its effectiveness, a series of ablation experiments were conducted to investigate the impact of each module of MESF-DETR on model performance. The experimental results are shown in Table 2 below.

[0057] Adding the Mamba-based semantic fusion module MSF to the baseline model improved the AP from 29.0% to 29.2%, while reducing the number of parameters and computational cost, indicating that this module balances accuracy and lightweight design. Adding the parallel multi-kernel feature enhancement module PMFEM further improved the AP to 29.5%, demonstrating its ability to enhance feature representation. Adding the dual-domain feature fusion module SFDDF improved the AP to 30.3%, and APsmall to 21.4%, indicating its most significant effect on small object detection. Using WGIoU further improved the AP to 29.5%, validating its ability to optimize localization and regression performance.

[0058] When multiple modules are used in combination, the model performance is further improved. After integrating all modules, the model achieves the best results: AP reaches 30.7%, AP50 reaches 51.0%, and APsmall reaches 21.6%. Compared to the baseline model, these represent improvements of 1.7%, 2.1%, and 1.7%, respectively. The results indicate that the improved modules have good complementarity and can synergistically improve the overall model performance.

[0059] exist Figure 5 The paper presents detection results for typical scenes selected on the VisDrone dataset, covering various challenging environments with different viewpoints, lighting conditions, and complex background interference. Compared with the baseline model RT-DETRv2, the proposed MESF-DETR shows significant advantages in detecting dense targets, severe occlusion, and small targets with low resolution. Visualization results demonstrate that MESF-DETR can more accurately locate the boundaries of small targets, effectively reduce the false negative rate, and maintain high confidence and detection accuracy even in complex backgrounds. Furthermore, even under extreme lighting conditions such as low light or strong reflection, the model can still stably identify targets, fully demonstrating the robustness and strong generalization ability of the proposed method in complex UAV-view scenarios.

[0060] Table 2 Ablation Experiment

[0061] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0062] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A small target detection method based on Mamba-enhanced spatial frequency domain fusion, characterized in that, Includes the following steps: A detection network is constructed, which consists of a backbone network, a Transformer-Mamba hybrid encoder, an IoU-aware query selection module, and a decoder. The backbone network adopts ResNet18, and the Transformer-Mamba hybrid encoder includes: the intra-scale interaction module AIFI, the parallel multi-kernel feature enhancement module PMFEM, and the Mamba-enhanced cross-scale feature fusion module MECSFF. The Mamba-enhanced cross-scale feature fusion module MECSFF includes: the Mamba-based semantic fusion module MSF and the dual-domain feature fusion module SFDDF; The decoder uses the WGIoU loss function, which is obtained by co-optimization of NWD and GIoU. Input the image to be detected, and extract multi-scale features through the backbone network to obtain feature maps S2, S3, S4, and S5; Feature map S5 is input to the intra-scale interaction module AIFI for intra-scale interaction, and feature maps S2, S3, and S4 are input to the parallel multi-kernel feature enhancement module PMFEM for parallel multi-kernel feature enhancement. The enhanced feature input is based on the Mamba-enhanced cross-scale feature fusion module MECSFF. Semantic fusion is performed through the Mamba-based semantic fusion module MSF, and spatial and frequency domain features are collaboratively fused through the dual-domain feature fusion module SFDDF to obtain fused features. The feature IoU perception query selection module filters query features, inputs them into the decoder, optimizes bounding box regression through the WGIoU loss function, and outputs small target detection results.

2. The small target detection method based on Mamba-enhanced spatial frequency domain fusion according to claim 1, characterized in that, The processing procedure of the parallel multi-core feature enhancement module PMFEM is as follows: The input feature map is first halved in channel dimension by a 1×1 convolution, and then input into three parallel branches; The first branch uses 1×1 convolution to linearly reorganize the channel dimensions; the second branch uses 3×3 windmill convolution PConv to extract features. The windmill convolution PConv uses 4 sets of asymmetric padding combined with horizontal and vertical convolution kernels to capture local features in four directions, and the receptive field decays in a Gaussian distribution; the third branch uses 5×5 depthwise separable convolution to extract features. After the output features of the three branches are concatenated, they are processed by a feature focusing structure activated by global average pooling (GAP) and sigmoid, which adaptively enhances key feature channels and suppresses redundant information. Finally, the output is connected via residuals.

3. The small target detection method based on Mamba-enhanced spatial frequency domain fusion according to claim 1, characterized in that, The processing procedure of the Mamba-based semantic fusion module MSF is as follows: The input feature X is preprocessed by mapping the original feature to the hidden dimension through a 1×1 convolutional layer and combining the SiLU activation function to enhance the nonlinear expression, thus obtaining the preprocessed feature X'. The preprocessed feature X' is subjected to 1×1 convolution, separable convolution with different sizes and depths, and GELU activation to enhance local features. It is then mapped back to the original channel dimension by 1×1 convolution to obtain global features X1' at multiple scales. Global features X1' at multiple scales are input into the SS2D module after layer normalization. The SS2D module expands the input image into a sequence by traversing four directions. Each sequence is processed by an independent S6 block, then scanned, merged and reconstructed into an image form, and outputs feature X2'. After feature X2' is normalized by the layer, it is expanded through a 1×1 convolution and split to obtain X. A X B ;X A Separable convolution with depth 3×3 and X A Residual connections, activated by GELU, yield X. A '; X A 'with X B Element-wise multiplication is performed, and the result is convolved with the X2' residual after 1×1 convolution to output the final fused feature Y.

4. The small target detection method based on Mamba-enhanced spatial frequency domain fusion according to claim 1, characterized in that, The processing procedure of the dual-domain feature fusion module SFDDF is as follows: The low-level feature map is downsampled by the Haar wavelet downsampling module (HWD) to align the high- and low-level features. The aligned features are then added together to obtain the fused feature X. merge ; Fusion Feature X merge The input spatial domain branch is processed by the Mamba-based semantic fusion module MSF to capture spatial domain information, and the output spatial domain information weight ω is generated. s ; Fusion Feature X merge The input frequency branch first undergoes a 1×1 convolution and GELU activation to introduce nonlinearity, and the absolute value is taken to highlight high-frequency information. Then, it undergoes two independent 1×1 convolutions, followed by Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT) respectively. Combined with learnable parameters α and β, the frequency domain feature X is obtained. f ''; The frequency branch is processed by a 1×1 depthwise separable convolution, ReLU activation, and a 1×1 convolution to output the frequency domain information weights ω. f High- and low-level features are respectively related to the spatial domain information weight ω s Frequency domain information weight ω f After multiplying and concatenating, we obtain feature X. concat Feature X concat After channel attention recalibration, 1×1 convolution, batch normalization, and ReLU activation, the final dual-domain fusion feature is output.

5. The small target detection method based on Mamba-enhanced spatial frequency domain fusion according to claim 4, characterized in that, The formula for the SFDDF (Single Domain Feature Fusion) module's processing procedure is as follows: Where, ω s Represents the spatial domain information weights. Represents the frequency domain information weights. This represents a dual mapping feature. This indicates a concatenation operation along the channel dimension. Indicates channel attention weights. Indicates the height of the feature map. Indicates the width of the feature map. An index representing the spatial height direction. Indicates the index in the direction of spatial width. Indicates the characteristics after fusion. Indicates the batch normalization layer. This represents a convolution operation with a kernel size of 1×1. This represents a 1D convolution with kernel k. It adapts to the number of input channels.

6. The small target detection method based on Mamba-enhanced spatial frequency domain fusion according to claim 1, characterized in that, The formula for calculating the WGIoU loss function is: Where α is the learnable weight parameter, NWD is the normalized Wasserstein distance, the bounding box is modeled as a Gaussian distribution and the Wasserstein distance between distributions is calculated, and GIoU is the intersection-union ratio loss function.

Images