Infrared small target detection method based on multi-dimensional frequency domain perception and feature refinement

Abstract

This invention discloses an infrared small target detection method based on multi-dimensional frequency domain perception and feature refinement, belonging to the field of computer vision target detection technology. The invention acquires and preprocesses multiple consecutive frames of infrared video images, then inputs them into a target detection neural network for detection. In the target detection neural network, a frequency domain perception feature extractor extracts anti-interference deep features through local contrast branches and global spectrum gating branches; a frequency domain guided alignment module guides deformable convolution for motion compensation and alignment by calculating inter-frame frequency domain correlation; a wavelet-Mamba feature refinement module decouples features into high- and low-frequency components, performs global modeling on the low-frequency components, performs local convolution on the high-frequency components, and reconstructs and outputs refined features through inverse transform. This invention effectively solves the problems of insufficient feature discriminative power and low cross-frame alignment accuracy in infrared small target detection, improving detection accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of computer vision target detection technology, specifically to an infrared small target detection method based on multi-dimensional frequency domain perception and feature refinement. Background Technology

[0002] Infrared small target detection has wide applications in civilian systems such as road traffic monitoring, industrial thermal imaging inspection, power equipment inspection, and forest fire early warning. Unlike conventional visible light targets, small targets in infrared images typically have the following characteristics: the target occupies an extremely small area in the image, often less than 0.1% of the total pixels; the target itself lacks identifiable texture and shape information; the contrast between the target and the background is low, resulting in a poor signal-to-noise ratio; and small targets are easily obscured by complex background interference such as clouds, ground features, and water ripples. These characteristics make it difficult for target detection methods based on single-frame static images to meet the requirements of high detection rate and low false alarm rate in practical applications.

[0003] To overcome the aforementioned problems, detection methods based on video sequences have been proposed. These methods utilize temporal motion information between consecutive frames, effectively improving the robustness of detecting small, dimly lit targets. However, analysis of existing technologies reveals that current video infrared small target detection methods still have the following shortcomings:

[0004] First, at the feature extraction level, most existing methods employ standard convolutional neural networks to extract features independently for each frame of the image. Standard convolution has a fixed local receptive field, which excels at capturing local texture information, but struggles to effectively model the spatial distribution of global background clutter while extracting local brightness contrast between the target and the background. This results in insufficient discriminative power of the extracted features for small infrared targets, and limited anti-interference capability when the target is in a complex background.

[0005] Secondly, at the cross-frame feature alignment level, due to the target's own motion and the jitter of the image acquisition device, there is significant spatial displacement between adjacent frames. Feature-level alignment is necessary to effectively integrate temporal information. Existing methods typically employ optical flow or standard deformable convolution, which mainly rely on image appearance similarity to predict motion offset fields. For small infrared targets with few pixels and blurred appearance features, it is difficult to accurately estimate their motion vectors based solely on appearance information, resulting in insufficient feature alignment accuracy. This, in turn, introduces misalignment noise during feature fusion, reducing detection performance.

[0006] Furthermore, at the feature refinement level, existing technologies often employ stacked standard convolutions or self-attention mechanisms to further refine the features after multi-frame alignment and fusion. Standard convolutions have a limited receptive field, making it difficult to capture the global contextual information crucial for target discrimination; while self-attention mechanisms, although capable of modeling long-distance dependencies, have computational complexity that increases quadratically with feature map size, making them unsuitable for high-resolution feature maps, and their ability to model local details in detail is also insufficient.

[0007] Therefore, providing a video infrared small target detection method that can improve the discriminative power of feature extraction, the accuracy of cross-frame alignment and the quality of feature refinement, while taking into account computational efficiency, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides an infrared small target detection method based on multi-dimensional frequency domain perception and feature refinement. This method introduces frequency domain technology from four dimensions: feature extraction, cross-frame alignment, feature refinement, and supervised training. By constructing a target detection neural network that includes a frequency domain-aware feature extractor, a frequency domain-guided alignment module, an adaptive feature fusion module, a wavelet-Mamba feature refinement module, and a prediction head, it achieves high-precision and robust detection of small targets in video infrared images.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] This invention proposes an infrared small target detection method based on multi-dimensional frequency domain sensing and feature refinement, comprising the following steps:

[0011] Acquire consecutive multi-frame video infrared images to be detected, preprocess the video infrared images to obtain multi-frame image tensors;

[0012] The multi-frame image tensor is input into a pre-trained target detection neural network. The target detection neural network performs target detection on the video infrared image and outputs the category label and bounding box coordinates of the detected target.

[0013] The target detection neural network includes a frequency domain-aware feature extractor, a frequency domain guided alignment module, an adaptive feature fusion module, a wavelet-Mamba feature refinement module, and a prediction head;

[0014] The frequency domain-aware feature extractor is used to extract deep features from image tensors;

[0015] The frequency domain guided alignment module is used to receive all frame deep features output by the frequency domain perceptual feature extractor, select one frame deep feature from all frame deep features as the reference frame feature, take the remaining frame deep features as the frame features to be aligned, calculate the frequency domain correlation between the reference frame feature and each frame feature to be aligned, and guide deformable convolution to perform motion compensation and alignment on the frame features to be aligned based on the frequency domain correlation to obtain the aligned features.

[0016] The adaptive feature fusion module is used to receive the reference frame features and all alignment features output by the frequency domain guided alignment module, and perform weighted fusion on the reference frame features and all alignment features to obtain the fused features.

[0017] The wavelet-Mamba feature refinement module is used to receive the fused features, decouple the fused features into high-frequency components and low-frequency components; perform global spatial selective scanning modeling on the low-frequency components to obtain low-frequency enhanced features; perform local convolution processing on the high-frequency components to obtain high-frequency enhanced features; and perform inverse transformation reconstruction on the low-frequency enhanced features and high-frequency enhanced features to output refined features.

[0018] The prediction head receives refined features and generates the category label and bounding box coordinates based on the refined features.

[0019] Furthermore, the frequency domain-aware feature extractor includes a frequency domain-aware feature block, which includes parallel local contrast branches and global spectrum gating branches;

[0020] The local contrast branch extracts local spatial contrast features from the input features using central difference convolution. The central difference convolution contains learnable difference weights to control the mixing ratio of standard convolution and difference convolution.

[0021] The global spectrum gating branch performs a two-dimensional Fast Fourier Transform (FFT) on the input features to obtain frequency domain features; the real and imaginary parts of the frequency domain features are separated, and the separated real and imaginary parts are modulated using channel-level complex weights to obtain modulated features; the modulated features are filtered through an adaptive spectrum gating network to obtain gated weighted features; a gain is applied to the high-frequency components in the frequency domain of the gated weighted features to obtain enhanced frequency domain features; the enhanced frequency domain features are then subjected to an inverse two-dimensional Fast Fourier Transform (IFFT) to obtain global features;

[0022] The local spatial contrast features are concatenated with the global features to obtain concatenated features; attention weighting is applied to the concatenated features to obtain the output of the frequency domain sensing feature block.

[0023] Furthermore, the frequency domain guided alignment module includes a frequency domain correlation unit and an enhanced deformable alignment unit;

[0024] The execution process of the frequency domain correlation unit includes: performing a two-dimensional fast Fourier transform on the reference frame features and the frame features to be aligned to obtain the frequency domain features of the reference frame and the frame features to be aligned; calculating the cross power spectrum of the frequency domain features of the reference frame and the frame features to be aligned; normalizing the cross power spectrum to obtain the frequency domain correlation between the reference frame features and the frame features to be aligned; performing a two-dimensional fast Fourier inverse transform on the frequency domain correlation to generate a correlation feature map; and normalizing the correlation feature map to obtain a frequency domain correlation attention map.

[0025] The execution process of the enhanced deformable alignment unit includes: splicing the reference frame features and the frame features to be aligned to obtain preliminary fused features; encoding the frequency domain related attention map to obtain an encoded attention map; splicing the encoded attention map with the preliminary fused features; first predicting the offset and mask weight of the deformable convolution; and then performing motion compensation and alignment on the frame features to be aligned through the deformable convolution to obtain the aligned features.

[0026] Furthermore, the wavelet-Mamba feature refinement module includes a wavelet-Mamba block, and the wavelet-Mamba block processes the input features as follows:

[0027] The input features are subjected to discrete wavelet transform, which decomposes the input features into one low-frequency component and three high-frequency components; the three high-frequency components are concatenated to obtain the concatenated high-frequency component.

[0028] Two-dimensional spatial selective scanning is performed on the low-frequency components to obtain low-frequency enhancement features;

[0029] The high-frequency components after splicing are processed by depthwise separable convolution to obtain high-frequency enhanced features;

[0030] After channel recombination of the low-frequency enhancement features and high-frequency enhancement features, the output features of the wavelet-Mamba block are reconstructed through inverse wavelet transform and upsampling.

[0031] Furthermore, the process of performing two-dimensional spatial selective scanning on the low-frequency components is as follows:

[0032] The low-frequency components are serialized and flattened along the horizontal positive direction, the horizontal negative direction, the vertical positive direction, and the vertical negative direction, respectively, to obtain four one-dimensional sequences;

[0033] Using the Mamba module, long-distance dependency modeling was performed on the four one-dimensional sequences to obtain four enhanced one-dimensional sequences;

[0034] The four enhanced one-dimensional sequences are respectively restored to two-dimensional spatial feature maps, and the four two-dimensional spatial feature maps are weighted and summed through learnable fusion weights to output the low-frequency enhanced features.

[0035] Furthermore, the discrete wavelet transform is implemented through grouped convolution, with the initial weights of the grouped convolution configured based on the Haar wavelet transform matrix; the initial weights of the inverse transform convolution are configured based on the Haar wavelet inverse transform matrix.

[0036] Furthermore, during the training phase of the target detection neural network, a focal frequency domain motion compensation loss function is used to supervise the frequency domain guided alignment module; the calculation process of the focal frequency domain motion compensation loss function is as follows:

[0037] Calculate the smooth L1 loss of alignment features and reference frame features in the spatial domain;

[0038] The focal frequency loss between the alignment feature and the reference frame feature in the frequency domain is calculated as follows: the logarithmic difference between the spectral amplitudes of the alignment feature and the reference frame feature is calculated; for each frequency component, a corresponding dynamic weight is assigned according to the magnitude of its logarithmic difference; the frequency domain mean square error is calculated based on the dynamic weights to obtain the focal frequency loss; the smoothing L1 loss and the focal frequency loss are weighted and summed to obtain the focal frequency domain motion compensation loss.

[0039] This invention also proposes an infrared small target detection system based on multi-dimensional frequency domain sensing and feature refinement, for implementing the above-mentioned infrared small target detection method, comprising:

[0040] The image acquisition module is used to acquire continuous multi-frame infrared images of the video to be detected;

[0041] The preprocessing module is used to preprocess the video infrared image to obtain an image tensor;

[0042] The detection processing module includes a pre-trained target detection neural network, which is used to perform target detection on the image tensor and output the category label and bounding box coordinates of the detected targets.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] (1) In this invention, a local contrast branch and a global spectrum gating branch are set up in parallel in the frequency domain sensing feature block. The local contrast branch adopts central difference convolution, and by introducing learnable differential weights, the difference between the center pixel and the neighboring pixels is explicitly calculated, directly modeling the brightness contrast relationship between the small target and the local background. The global spectrum gating branch transforms the features to the frequency domain through Fourier transform, and uses channel-level complex weights and spatial adaptive gating network to perform weighted modulation on different frequency components, and applies additional gain to the high-frequency components. Since the edge and detail information of infrared small targets are mainly distributed in the high-frequency region of the frequency domain, while the large-area slowly changing background clutter is concentrated in the low-frequency region, by enhancing and suppressing the high and low frequencies respectively, the network can effectively highlight the target signal and suppress background interference. After the outputs of the two branches are fused and weighted by attention, the extracted features have both local contrast discriminative ability and global background suppression ability, which significantly improves the detection rate of infrared small targets in complex backgrounds and reduces the false alarm rate.

[0045] (2) This invention introduces a frequency domain correlation unit in the frequency domain guided alignment module. The frequency domain correlation is obtained by calculating the normalized cross power spectrum between the features of the reference frame and the features of the frame to be aligned. Then, the correlation feature map is generated by inverse Fourier transform. After normalizing the correlation feature map, the frequency domain correlation attention map is obtained. The Fourier transform has translation invariance. The translational motion of two frames in the spatial domain is manifested as a phase difference in the frequency domain. This phase difference can be accurately extracted by the cross power spectrum, thereby obtaining an attention map that is highly correlated with the motion field. This attention map contains pixel-level motion displacement clues. After encoding, it is spliced ​​with the preliminary fusion features to jointly guide the deformable convolution to predict the offset and modulation mask. Since the frequency domain phase information is not sensitive to the intensity change and appearance blur of the target, even if a small target is almost invisible in a single frame, its motion information can still be accurately captured by the frequency domain phase difference. This solves the problem of cross-frame alignment of small targets with blurred appearance, makes the fusion of temporal information more accurate, and effectively avoids the introduction of misalignment noise.

[0046] (3) This invention losslessly decomposes the fused features into low-frequency structural components and three high-frequency detail components using learnable discrete wavelet transform. The low-frequency components correspond to the overall contour and large-scale structural information of the target, and their spatial distribution is global, requiring long-distance dependency modeling. This invention adopts a two-dimensional spatial selective scanning mechanism to flatten the low-frequency features into a one-dimensional sequence along four directions, and then restores it to a two-dimensional feature map after modeling using the Mamba module. Mamba is based on a state-space model, and its computational complexity is linearly related to the sequence length, which has a significant efficiency advantage compared to the quadratic complexity of self-attention in existing methods. Scanning in four directions can capture global dependencies from different directions, and then adaptively integrate them through learnable fusion weights, so that the low-frequency enhanced features have complete global structural information. The high-frequency components correspond to the edge, texture and other detail information of the target, and their spatial distribution is sparse and local, which can be efficiently extracted by using depthwise separable convolution. The low-frequency enhanced features and the high-frequency enhanced features are reconstructed through learnable inverse wavelet transform, so that the refined features have both accurate global structural localization ability and rich local detail expression ability, and the computational efficiency is significantly better than the self-attention mechanism.

[0047] (4) In the training phase, this invention employs a focal frequency domain motion compensation loss function, simultaneously calculating the spatial domain smoothing L1 loss and the frequency domain focal frequency loss of the alignment features and the reference frame features. The frequency domain focal frequency loss assigns higher dynamic weights to frequency components with larger differences by calculating the logarithmic difference in the spectral amplitudes of the two. Since the signal energy of infrared small targets is mainly distributed in the high-frequency region, while background clutter is concentrated in the low-frequency region, this dynamic weighting mechanism makes the network pay more attention to the alignment accuracy of target-related high-frequency components during training, while the penalty for low-frequency background deviation is relatively reduced. The focal weighting strategy further amplifies the gradient contribution of difficult-to-align frequency components, guiding the network to prioritize learning motion patterns that are difficult to align. Through joint supervision in the spatial and frequency domains, the alignment module can simultaneously maintain the accuracy of spatial position and the fidelity of frequency domain structure, further improving the alignment quality from the training optimization level. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the target detection neural network in an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the internal structure of the frequency domain sensing feature block in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the internal structure of the frequency domain guided alignment module in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the internal structure of the wavelet-Mamba block in an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Example

[0054] This embodiment proposes an infrared small target detection method based on multi-dimensional frequency domain sensing and feature refinement, including the following steps:

[0055] Step 1: Acquire continuous multi-frame infrared video images of the target device using an infrared image acquisition device. The infrared image acquisition device can be an infrared sensor mounted on a drone, a ground-based infrared sensor at a fixed location, or other types of acquisition devices.

[0056] In this embodiment, taking the current time t as the center, five frames of images at times t-2, t-1, t, t+1, and t+2 are taken as an input sample. The five frames of images together describe the short-term motion trajectory of the target.

[0057] The acquired infrared images are preprocessed, including size normalization and pixel value normalization. In this embodiment, the input image size is uniformly scaled to 544×544 pixels and normalized to conform to the input format of the target detection neural network, resulting in a multi-frame image tensor. The shape of the image tensor is (B, 3, N, H, W), where B is the batch size, N is the number of consecutive frames (in this embodiment, N=5), H and W are the height and width of the image, respectively, and H=W=544.

[0058] Step 2: Input the multi-frame image tensors obtained in Step 1 into the pre-trained target detection neural network. The target detection neural network performs multi-scale target detection on the video infrared image and outputs the category label and bounding box coordinates of the detected targets.

[0059] The target detection neural network includes a frequency domain-aware feature extractor, a frequency domain guided alignment module, an adaptive feature fusion module, a wavelet-Mamba feature refinement module, and a prediction head.

[0060] The frequency-domain aware feature extractor is used to extract deep features from image tensors. In this embodiment, the frequency-domain aware feature extractor consists of a basic convolutional layer and three frequency-domain aware feature blocks connected in series, with adjacent frequency-domain aware feature blocks connected by a 3×3 convolution with a stride of 2. The basic convolutional layer consists of a 3×3 convolution with a stride of 1 and a 3×3 convolution with a stride of 2 connected in series.

[0061] The process by which a frequency-domain-aware feature extractor extracts deep features from an image tensor is as follows:

[0062] The basic convolutional layer performs convolution operations on the image tensor and outputs basic features. In this embodiment, the first 3×3 convolution in the basic convolutional layer expands the number of channels of the image tensor from 3 to 48 while maintaining the spatial size of 544×544, outputting intermediate basic features; the second 3×3 convolution downsamples the spatial size of the intermediate basic features to 272×272 while maintaining the number of channels at 48, thus obtaining the basic features.

[0063] The basic features output by the basic convolutional layer are processed by the first frequency domain sensing feature block, and then downsampled by a 3×3 convolution with a stride of 2 to obtain the first intermediate feature; the spatial size of the first intermediate feature is 136×136 and the number of channels is 48.

[0064] The first intermediate feature is processed by the second frequency domain sensing feature block, and then downsampled by a 3×3 convolution with a stride of 2 to obtain the second intermediate feature; the spatial size of the second intermediate feature is 68×68 and the number of channels is 64.

[0065] The second intermediate feature is processed by the third frequency domain sensing feature block to obtain the deep feature; the spatial size of the deep feature is 68×68 and the number of channels is 64.

[0066] Each frequency-domain sensing feature block includes a parallel local contrast branch and a global spectrum-gated branch. The processing procedure for the input features is as follows:

[0067] After the local contrast branch performs 1×1 convolution on the input features, the first dimension-reduced features are obtained. The central difference convolution is then used to extract the local spatial contrast features from the first dimension-reduced features.

[0068] The central difference convolution introduces a difference convolution on top of the standard convolution. For any spatial location on the first dimension-reduced feature... The output of the central difference convolution is:

[0069]

[0070] in, The sampling grid for the convolution kernel. For the nth sampling point relative to The offset, for The corresponding convolution kernel weights, Offset position in the first dimensionality reduction feature Pixel value at that location, for Pixel value at that location, The learnable differential weights, constrained to the range (0,1) by the sigmoid function, are used to control the mixing ratio of standard convolution and differential convolution. In this embodiment, the differential weights... The network layers are initialized sequentially from shallowest to deepest as 0.0, 0.3, and 0.7.

[0071] The global spectrum gating branch performs a 1×1 convolution on the input features to obtain the second dimensionality-reduced features. A two-dimensional fast Fourier transform is then performed on the second dimensionality-reduced features to obtain the frequency domain features. The real and imaginary parts of the frequency domain features are separated, and the separated real and imaginary parts are modulated using channel-level complex weights to obtain modulated features. The modulated features are then filtered using an adaptive spectrum gating network to obtain gated weighted features. A gain is applied to the high-frequency components in the gated weighted features to obtain enhanced frequency domain features. Finally, a two-dimensional inverse fast Fourier transform is performed on the enhanced frequency domain features to obtain the global features.

[0072] The adaptive spectrum gating network generates a gating mask based on the spectrum amplitude, weights the real and imaginary parts of the spectrum respectively, and applies additional gain to the high-frequency components through a learnable high-frequency enhancement factor to enhance the response to the high-frequency components where infrared small targets are located.

[0073] The local spatial contrast features and global features are concatenated along the channel dimension to obtain the concatenated features. After performing a 1×1 fusion convolution on the concatenated features, attention weighting is applied to obtain the output of the frequency domain perceptual feature block. In this embodiment, the CBAM (Convolutional Block Attention Module) attention mechanism is used to sequentially apply channel attention weighting and spatial attention weighting to the concatenated features, focusing on the more discriminative feature channels and spatial locations.

[0074] The frequency domain guided alignment module includes a frequency domain correlation unit and an enhanced deformable alignment unit.

[0075] The frequency domain guided alignment module is used to receive all frame deep features output by the frequency domain aware feature extractor, select one frame deep feature from all frame deep features as the reference frame feature, and use the remaining frame deep features as the frame features to be aligned. The frequency domain correlation between the reference frame feature and each frame feature to be aligned is calculated by the frequency domain correlation unit. Based on the frequency domain correlation, the frame feature to be aligned is guided by the enhanced deformable alignment unit to perform motion compensation and alignment on the frame feature to be aligned, thereby obtaining the aligned feature.

[0076] In this embodiment, with 5 frames of input, the deep features corresponding to the image frame at time t are used as the reference frame features, and the deep features corresponding to the image frames at times t-2, t-1, t+1, and t+2 are used as the frame features to be aligned.

[0077] The execution process of the frequency domain correlation unit includes: performing a two-dimensional fast Fourier transform on the reference frame features and the current frame features to be aligned to obtain the frequency domain features of the reference frame and the frame features to be aligned; calculating the cross power spectrum of the frequency domain features of the reference frame and the frame features to be aligned; normalizing the cross power spectrum to obtain the frequency domain correlation between the reference frame features and the current frame features to be aligned; performing a two-dimensional inverse fast Fourier transform on the frequency domain correlation to generate a correlation feature map; and normalizing the correlation feature map to obtain a frequency domain correlation attention map.

[0078] The enhanced deformable alignment unit concatenates the features of the reference frame and the frame to be aligned along the channel dimension to obtain preliminary fused features; it encodes the frequency domain related attention map to obtain an encoded attention map; after concatenating the encoded attention map and the preliminary fused features along the channel dimension, it predicts the offset and mask weights of the deformable convolution through a convolutional layer; and then guides the deformable convolution to perform motion compensation and alignment on the current frame features to be aligned through the offset and mask weights to obtain the corresponding aligned features.

[0079] In this embodiment, when encoding the frequency domain related attention map, the mean of the frequency domain related attention map in the channel dimension is first calculated to obtain the channel mean map. The channel mean map is then encoded into a 64-channel encoded attention map through 7×7 convolution.

[0080] After encoding, the attention map and preliminary fusion features are concatenated and then predicted by a 3×3 convolutional layer to determine the offset and mask weights of the deformable convolution; the mask weights are constrained to the range of (0,1) by Sigmoid activation.

[0081] In this embodiment, the number of channels for offset is 144, corresponding to 8 deformable groups, 9 sampling points in each group, and 2 directional offsets per point; the number of channels for mask weight is 72, corresponding to 8 deformable groups, 9 sampling points in each group.

[0082] The deformable convolution guided by the predicted offset and mask weights performs irregular resampling and alignment of the features of the current frame to be aligned, resulting in resampled aligned features. The resampled aligned features are then activated by the ReLU function to obtain the aligned features.

[0083] The adaptive feature fusion module is used to receive the reference frame features and all alignment features output by the frequency domain guided alignment module, and to perform weighted fusion of the reference frame features and all alignment features to obtain the fused features.

[0084] In this embodiment, the frequency domain guided alignment module outputs one reference frame feature and four alignment features, and the adaptive feature fusion module performs weighted fusion on these five features to obtain the fused features.

[0085] The wavelet-Mamba feature refinement module is used to receive fused features, decouple the fused features into high-frequency components and low-frequency components; perform global spatial selective scanning modeling on the low-frequency components to obtain low-frequency enhanced features; perform local convolution processing on the high-frequency components to obtain high-frequency enhanced features; and perform inverse wavelet transform reconstruction on the low-frequency enhanced features and high-frequency enhanced features to output refined features.

[0086] In this embodiment, the wavelet-Mamba feature refinement module consists of two cascaded wavelet-Mamba blocks. The first wavelet-Mamba block processes the fused features to obtain intermediate refined features, and the second wavelet-Mamba block processes the intermediate refined features to obtain refined features. The processing procedure of the first wavelet-Mamba block for the fused features is the same as the processing procedure of the second wavelet-Mamba block for the intermediate refined features.

[0087] Each wavelet-Mamba block includes a discrete wavelet transform unit, a frequency domain processing unit, and a learnable inverse wavelet transform unit connected in sequence.

[0088] The process of feature processing for the first wavelet-Mamba block is as follows:

[0089] The fused features are decoupled using a discrete wavelet transform unit, decomposing them into one low-frequency component and three high-frequency components. The low-frequency component is the LL subband, and the three high-frequency components are the LH, HL, and HH subbands, respectively. In this embodiment, a learnable discrete wavelet transform is used, and frequency band separation is achieved through grouped convolution. The initial weights of the grouped convolutions are configured based on the Haar wavelet transform matrix, and the four output channels of each group of convolutions correspond to the LL, LH, HL, and HH subbands, respectively.

[0090] The frequency domain processing unit includes a low-frequency processing branch and a high-frequency processing branch. The frequency domain processing unit receives the output of the discrete wavelet transform unit and performs the following operations:

[0091] The low-frequency processing branch uses a state-space model to perform a two-dimensional spatial selective scan of the low-frequency components, obtaining low-frequency enhancement features. The specific process is as follows:

[0092] The low-frequency components are serialized and flattened along the horizontal positive direction, horizontal negative direction, vertical positive direction, and vertical negative direction respectively to obtain four one-dimensional sequences;

[0093] The Mamba module is used to model the long-distance dependency of four one-dimensional sequences, resulting in four enhanced one-dimensional sequences. These four enhanced one-dimensional sequences are then restored to two-dimensional spatial feature maps in reverse order of their flattened state. The four two-dimensional spatial feature maps are then weighted and summed using learnable fusion weights to output low-frequency enhanced features. In this embodiment, the state dimension of the Mamba module is 16, and the expansion factor is 2; the fusion weights are normalized using the Softmax function.

[0094] The high-frequency processing branch concatenates the three high-frequency components along the channel dimension to obtain the concatenated high-frequency components. Depthwise separable convolution is then used to process the concatenated high-frequency components to obtain high-frequency enhanced features. The specific process is as follows:

[0095] The high-frequency components after splicing are sequentially processed by 3×3 depthwise separable convolution and 1×1 convolution to obtain high-frequency enhancement features; the high-frequency enhancement features are then segmented along the channel dimension into enhanced LH subband, enhanced HL subband, and enhanced HH subband.

[0096] The learnable inverse wavelet transform unit reconstructs the low-frequency enhancement features with the enhanced LH, HL, and HH subbands through channel recombination, and then reconstructs the inverse transform features through inverse transform convolution and upsampling. The inverse transform features are then processed by 1×1 convolution and added element-wise to the input of the wavelet-Mamba block to obtain intermediate refined features. In this embodiment, the intermediate refined features have 64 channels and a spatial size of 68×68. The initial weights of the inverse transform convolution are configured based on the Haar wavelet inverse transform matrix, i.e., the transpose of the forward Haar wavelet transform matrix. The weights of the grouped convolutions are adaptively updated during network training via backpropagation.

[0097] The intermediate refined features of the first wavelet-Mamba block output are used as the input to the second wavelet-Mamba block.

[0098] The prediction head receives the refined features output by the wavelet-Mamba feature refinement module, and generates the category label and bounding box coordinates of the detected target based on the refined features.

[0099] The target detection neural network can be trained and validated using the publicly available IRDST (InfraRed Dim SmallTarget Dataset). During training, the input image size is set to 544×544 pixels, the batch size is 4, the SGD optimizer is used, the initial learning rate is 0.01, the momentum is 0.937, and the weight decay is 0.0005.

[0100] During the training phase of the target detection neural network, the frequency domain guided alignment module is supervised using a focal frequency domain motion compensation loss function.

[0101] The calculation process of the focal frequency domain motion compensation loss function is as follows: calculate the smoothing L1 loss of the alignment feature and the reference frame feature in the spatial domain, and the focal frequency loss in the frequency domain, respectively; wherein, the focal frequency loss in the frequency domain is calculated as follows: calculate the logarithmic difference between the spectral amplitudes of the alignment feature and the reference frame feature, and assign higher dynamic weights to the frequency components with larger differences, and calculate the frequency domain mean square error by weighting; the smoothing L1 loss and the focal frequency loss are weighted and summed to obtain the focal frequency domain motion compensation loss.

[0102] In this embodiment, for multiple frame features to be aligned, the focal frequency domain motion compensation loss corresponding to each frame feature is calculated, and then the focal frequency domain motion compensation losses of all frame features to be aligned are summed to obtain the final focal frequency domain motion compensation loss.

[0103] The final focal frequency domain motion compensation loss is weighted and summed with the classification loss, regression crossover ratio loss, and target confidence loss to form the total loss function of the target detection neural network.

[0104] When training an object detection neural network, the training objective is to minimize the total loss function.

[0105] To better illustrate the beneficial effects of the present invention, the target detection neural network (hereinafter referred to as the "invention model"), DTUM (Direction-Coded Temporal U-Shape Module), SSTNet (Sliced ​​Spatio-Temporal Network), DFAR (Deformable Feature Alignment and Refinement) model, improved DFAR model-1, improved DFAR model-2 and improved DFAR model-3 in the above embodiments were respectively verified on the publicly available IRDST.

[0106] The difference between the improved DFAR model-1 and the DFAR model is that in the feature extractor of the DFAR model, all 3×3 convolutions with a stride of 1 except the first one are replaced with frequency domain-aware feature blocks as described in the above embodiments.

[0107] The difference between the improved DFAR model-2 and the DFAR model is that the feature alignment module in the DFAR model is replaced with the frequency domain guided alignment module in the above embodiments.

[0108] The difference between the improved DFAR model-3 and the DFAR model is that the refinement module after the fusion module in the DFAR model is replaced with the wavelet-Mamba feature refinement module in the above embodiment.

[0109] Each model was validated under the same input size and training strategy. The evaluation metrics included the mean precision (mAP50:95), precision (Pr), recall (Re), and F1 score when the intersection-union ratio threshold was 0.5~0.95. The validation results are shown in Table 1.

[0110] Table 1 Performance comparison of each model on IRDST

[0111]

[0112] As can be seen from Table 1, in terms of detection accuracy, the mAP50:95 of the model of this invention is significantly better than that of the three existing models. At the same time, the mAP50:95 of the three improved models is better than that of the benchmark DFAR model, indicating that the frequency domain sensing feature block, frequency domain guided alignment module and wavelet-Mamba feature refinement module in this invention can effectively improve the detection accuracy.

[0113] In terms of false alarm suppression, the accuracy of the model in this invention is significantly better than that of three existing models. This is due to the frequency domain guided alignment module's improvement in cross-frame alignment accuracy and the wavelet-Mamba feature refinement module's effective suppression of background clutter.

[0114] In terms of missed detection control, the recall rate of the model in this invention is the best among all the comparison models. The recall rates of the improved DFAR model-2 and the improved DFAR model-3 are both better than the baseline DFAR model, which verifies the effectiveness of the frequency domain guided alignment module and the wavelet-Mamba feature refinement module in improving the target acquisition capability under low signal-to-noise ratio.

[0115] In terms of overall performance, the F1 score of the model in this invention is significantly better than that of the three existing models. This indicates that the model in this invention achieves a better balance between detection accuracy and robustness.

[0116] The above results demonstrate that the present invention effectively solves the problems of insufficient feature discriminative power, low cross-frame alignment accuracy, and severe background clutter interference in the prior art through the synergistic effect of three modules: frequency domain sensing feature extractor, frequency domain guided alignment, and wavelet-Mamba feature refinement, thereby achieving a significant improvement in detection performance.

[0117] The specific embodiments of the present invention are provided to enable those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention.

[0118] It should be understood that the present invention is not limited to the content already described above, and various modifications and changes can be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for detecting small infrared targets based on multi-dimensional frequency domain sensing and feature refinement, characterized in that, Includes the following steps: Acquire consecutive multi-frame video infrared images to be detected, preprocess the video infrared images to obtain multi-frame image tensors; The multi-frame image tensor is input into a pre-trained target detection neural network. The target detection neural network performs target detection on the video infrared image and outputs the category label and bounding box coordinates of the detected target. The target detection neural network includes a frequency domain-aware feature extractor, a frequency domain guided alignment module, an adaptive feature fusion module, a wavelet-Mamba feature refinement module, and a prediction head; The frequency domain-aware feature extractor is used to extract deep features from image tensors; The frequency domain guided alignment module is used to receive all frame deep features output by the frequency domain perceptual feature extractor, select one frame deep feature from all frame deep features as the reference frame feature, take the remaining frame deep features as the frame features to be aligned, calculate the frequency domain correlation between the reference frame feature and each frame feature to be aligned, and guide deformable convolution to perform motion compensation and alignment on the frame features to be aligned based on the frequency domain correlation to obtain the aligned features. The adaptive feature fusion module is used to receive the reference frame features and all alignment features output by the frequency domain guided alignment module, and perform weighted fusion on the reference frame features and all alignment features to obtain the fused features. The wavelet-Mamba feature refinement module is used to receive the fused features and decouple the fused features into high-frequency components and low-frequency components. Global spatial selective scanning modeling is performed on the low-frequency components to obtain low-frequency enhancement features; The high-frequency components are locally convolutionally processed to obtain high-frequency enhanced features; the low-frequency enhanced features and the high-frequency enhanced features are then inversely transformed and reconstructed to output refined features. The prediction head receives refined features and generates the category label and bounding box coordinates based on the refined features.

2. The infrared small target detection method based on multi-dimensional frequency domain sensing and feature refinement according to claim 1, characterized in that, The frequency domain sensing feature extractor includes a frequency domain sensing feature block, which includes a parallel local contrast branch and a global spectrum gating branch. The local contrast branch extracts local spatial contrast features from the input features using central difference convolution. The central difference convolution contains learnable difference weights to control the mixing ratio of standard convolution and difference convolution. The global spectrum gating branch performs a two-dimensional fast Fourier transform on the input features to obtain frequency domain features; the real and imaginary parts of the frequency domain features are separated, and the separated real and imaginary parts are modulated by channel-level complex weights to obtain modulated features; the modulated features are filtered by an adaptive spectrum gating network to obtain gated weighted features; and a gain is applied to the high-frequency components in the frequency domain of the gated weighted features to obtain enhanced frequency domain features. Perform a two-dimensional inverse fast Fourier transform on the enhanced frequency domain features to obtain global features; The local spatial contrast features are concatenated with the global features to obtain concatenated features; attention weighting is applied to the concatenated features to obtain the output of the frequency domain sensing feature block.

3. The infrared small target detection method based on multi-dimensional frequency domain sensing and feature refinement according to claim 1, characterized in that, The frequency domain guided alignment module includes a frequency domain correlation unit and an enhanced deformable alignment unit; The execution process of the frequency domain correlation unit includes: performing a two-dimensional fast Fourier transform on the reference frame features and the frame features to be aligned to obtain the frequency domain features of the reference frame and the frame features to be aligned; calculating the cross power spectrum of the frequency domain features of the reference frame and the frame features to be aligned; normalizing the cross power spectrum to obtain the frequency domain correlation between the reference frame features and the frame features to be aligned; performing a two-dimensional fast Fourier inverse transform on the frequency domain correlation to generate a correlation feature map; and normalizing the correlation feature map to obtain a frequency domain correlation attention map. The execution process of the enhanced deformable alignment unit includes: splicing the reference frame features and the frame features to be aligned to obtain preliminary fused features; encoding the frequency domain related attention map to obtain an encoded attention map; splicing the encoded attention map with the preliminary fused features; first predicting the offset and mask weight of the deformable convolution; and then performing motion compensation and alignment on the frame features to be aligned through the deformable convolution to obtain the aligned features.

4. The infrared small target detection method based on multi-dimensional frequency domain sensing and feature refinement according to claim 1, characterized in that, The wavelet-Mamba feature refinement module includes a wavelet-Mamba block, and the wavelet-Mamba block processes the input features as follows: The input features are subjected to discrete wavelet transform, which decomposes the input features into one low-frequency component and three high-frequency components; the three high-frequency components are concatenated to obtain the concatenated high-frequency component. Two-dimensional spatial selective scanning is performed on the low-frequency components to obtain low-frequency enhancement features; The high-frequency components after splicing are processed by depthwise separable convolution to obtain high-frequency enhanced features; After channel recombination of the low-frequency enhancement features and high-frequency enhancement features, the output features of the wavelet-Mamba block are reconstructed through inverse wavelet transform and upsampling.

5. The infrared small target detection method based on multi-dimensional frequency domain sensing and feature refinement according to claim 4, characterized in that, The process of performing two-dimensional spatial selective scanning on the low-frequency components is as follows: The low-frequency components are serialized and flattened along the horizontal positive direction, the horizontal negative direction, the vertical positive direction, and the vertical negative direction, respectively, to obtain four one-dimensional sequences; Using the Mamba module, long-distance dependency modeling was performed on the four one-dimensional sequences to obtain four enhanced one-dimensional sequences; The four enhanced one-dimensional sequences are respectively restored to two-dimensional spatial feature maps, and the four two-dimensional spatial feature maps are weighted and summed through learnable fusion weights to output the low-frequency enhanced features.

6. The infrared small target detection method based on multi-dimensional frequency domain sensing and feature refinement according to claim 4, characterized in that, The discrete wavelet transform is implemented through grouped convolution, with the initial weights of the grouped convolution configured based on the Haar wavelet transform matrix; the initial weights of the inverse transform convolution are configured based on the Haar wavelet inverse transform matrix.

7. The infrared small target detection method based on multi-dimensional frequency domain sensing and feature refinement according to claim 1, characterized in that, During the training phase of the target detection neural network, a focal frequency domain motion compensation loss function is used to supervise the frequency domain guided alignment module; the calculation process of the focal frequency domain motion compensation loss function is as follows: Calculate the smooth L1 loss of alignment features and reference frame features in the spatial domain; The focal frequency loss between the alignment feature and the reference frame feature in the frequency domain is calculated as follows: the logarithmic difference between the spectral amplitudes of the alignment feature and the reference frame feature is calculated, and for each frequency component, a corresponding dynamic weight is assigned according to the magnitude of its logarithmic difference. The mean square error in the frequency domain is calculated based on the dynamic weights to obtain the focal frequency loss. The focus frequency domain motion compensation loss is obtained by weighted summing of the smoothing L1 loss and the focus frequency loss.

8. An infrared small target detection system based on multi-dimensional frequency domain sensing and feature refinement, used to implement the infrared small target detection method according to any one of claims 1 to 7, characterized in that, include: The image acquisition module is used to acquire continuous multi-frame infrared images of the video to be detected; The preprocessing module is used to preprocess the video infrared image to obtain an image tensor; The detection processing module includes a pre-trained target detection neural network, which is used to perform target detection on the image tensor and output the category label and bounding box coordinates of the detected targets.

Images