A multi-dimensional frequency domain and deformable attention fusion salient object detection method

Abstract

The present application relates to the field of salient object detection, and specifically discloses a multi-dimensional frequency domain and deformable attention fusion salient object detection method, comprising: S1, inputting an infrared image to be detected; S2, multi-scale feature extraction and fusion to obtain low-level and high-level features; S3, phase spectrum analysis to obtain frequency domain primary perception features; S4, frequency domain feature fusion to obtain frequency domain salient features; S5, a deformable spatial attention module to extract spatial enhancement perception features; S6, fusion features to generate a prediction map and to be constrained by a loss function. The present application solves the problems of insufficient frequency domain utilization, weak global context and detail preservation, and poor complex deformation target detection of existing spatial domain methods, effectively improving the detection accuracy and robustness of multi-scale and deformation targets in complex scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of salient object detection, in particular to a multi-dimensional frequency domain and deformable attention fusion salient object detection method. BACKGROUND

[0002] In recent years, with the rapid development of deep learning technology, remote sensing image salient object detection (RSI-SOD) as a basic task in computer vision has made significant progress. The task aims to accurately identify and segment the most visually attractive objects from complex scenes, and plays an important role in many downstream applications such as target tracking, image segmentation and weakly supervised learning. However, traditional methods based on convolutional neural networks (CNN) still face many challenges when dealing with complex backgrounds, multi-scale targets and fine edge structures. In particular, in the actual scene of high-resolution remote sensing images (RSI), the model often shows insufficient global context awareness, low feature alignment accuracy and unsatisfactory detail information preservation.

[0003] Although the research on salient object detection is relatively mature, most existing methods still focus on local feature interaction or single-scale context modeling, and fail to fully realize the effective collaboration between global semantic information and local structural features. To improve the performance of the model, some researches introduce attention mechanisms, multi-path feature fusion and edge auxiliary supervision strategies. For example, the document “Boundary-Aware Salient Object Detection” constructs a coding-decoding model based on the U-Net architecture combined with residual optimization, which gradually refines the saliency map to improve the boundary details; the document “RecurrentResidual Refinement Network for Saliency Detection” proposes a recurrent residual optimization mechanism, which enhances the detail preservation and boundary accuracy through a multi-stage optimization process; the document “Salient Object Detection in Optical Remote Sensing Images Driven by Transformer” uses the Transformer architecture to extract global dependencies, extracts local details through spatial attention modules, and suppresses complex background interference; the document “Hybrid Feature Aligned Network for Salient Object Detection in Optical Remote Sensing Imagery” fuses the advantages of CNN and Transformer, combines the gating mechanism and neighborhood feature alignment strategy, to capture deeper context information.

[0004] However, although these airspace learning methods have made significant contributions, most models still only learn and interact features within the airspace, rarely introducing complementary information contained in the image frequency domain. This limitation weakens the model's ability to perceive global structural features. In fact, frequency domain features have been verified to be closely related to human visual attention mechanisms. Therefore, building a new perception framework that can synergistically utilize airspace and frequency domain features is of great significance for achieving high-precision and high-robustness salient object detection in complex scenes.

[0005] In summary, although current methods based on airspace learning have promoted the development of salient object detection, the utilization of frequency domain information is still insufficient. Frequency domain features can provide unique global priors and are highly related to visual saliency.

[0006] Therefore, there is an urgent need for a multi-dimensional frequency domain and deformable attention fusion saliency object detection method to achieve high-precision and high-robustness recognition ability in complex scenes, thereby overcoming the problems existing in the prior art. SUMMARY

[0007] The present application provides a multi-dimensional frequency domain and deformable attention fusion saliency object detection method to effectively solve the problems of lack of frequency domain information utilization, lack of global-local feature synergy, and poor performance in complex scene detection in existing methods, effectively improve the detection accuracy of deformable objects, and enhance the global structure perception ability and detail preservation effect.

[0008] To achieve the above-mentioned purpose, the present application provides a multi-dimensional frequency domain and deformable attention fusion saliency object detection method, comprising:

[0009] S1, inputting an infrared image to be detected;

[0010] S2, using a multi-scale feature extraction module to perform hierarchical feature coding on the input infrared image to generate multi-scale primary perception features, and then using a multi-scale feature fusion module to fuse the features of the first two layers as low-level features and the features of the last two layers as high-level features;

[0011] S3, based on a phase spectrum analysis module, performing multi-dimensional phase spectrum analysis on the low-level features obtained by fusing the features of the first two layers through the multi-scale feature fusion module to extract frequency domain primary perception features;

[0012] S4, using a frequency domain feature fusion module to enhance the frequency domain primary perception features to obtain frequency domain saliency features through multi-dimensional fusion;

[0013] S5, using a deformable spatial attention module, using a direction deformable convolution module with geometric self-adaptive function to model the deformation of the high-level features, and then performing grouping attention weighted fusion on the features to extract spatially enhanced perception features;

[0014] S6, input the frequency domain saliency feature and the spatial enhanced perception feature into a saliency prediction module to generate a prediction map, and constrain using a loss function.

[0015] Preferably, in S2, the input infrared image is hierarchically feature encoded by the multi-scale feature extraction module to generate multi-scale primary perception features, and then the last two layers of features are fused as high-level features by the multi-scale feature fusion module. The specific process is as follows:

[0016] S21, input the real infrared image extract four scale features through four-stage hierarchical Transformer encoder, the expression is as follows:

[0017] ;

[0018] In the formula, C is the number of image channels, W is the image width, H is the image height, represents the Transformer encoder, represents the four scale features, is the output channel number, is the output image width, is the output image height;

[0019] S22, the first two stage features obtained in S21 are up-sampled to the same resolution and then multi-scale feature fusion is performed to obtain low-level features containing detailed information, and the last two stage features obtained in S21 are up-sampled to the same resolution and then multi-scale feature fusion is performed to obtain high-level features containing semantic information, the calculation expression is as follows:

[0020] ;

[0021] In the formula, represents the multi-scale feature fusion operation, represents the splicing operation, represents the up-sampling, represents the low-level feature, represents the high-level feature.

[0022] Preferably, in S21, the four-scale features output by the four-stage layered Transformer encoder have output channel numbers, output image widths and output image heights satisfying: the output width and height of the feature of the latter stage are 1 / 2 of the feature of the former stage, and the output channel number is 2 times of the feature of the former stage; in S22, the upsampling operation adopts a bilinear interpolation method to make the features of the first two stages and the features of the last two stages reach the same resolution, and the multi-scale feature fusion operation is a concatenation operation followed by a 1x1 convolution to adjust the channel number.

[0023] Preferably, in S3, the multi-dimensional phase spectrum analysis of the low-level features by the phase spectrum analysis module includes the following steps:

[0024] S31, preprocessing the input low-level features:

[0025] ;

[0026] In the formula, indicates a median filter;

[0027] S32, using a preset directional filter kernel to extract multi-directional gradient features from the filtered image:

[0028] ;

[0029] In the formula, is a preset directional filter kernel, K0, K 45 , K 90 , K 135 indicates a directional gradient feature;

[0030] S33, normalizing and Gaussian smoothing the extracted directional gradient features:

[0031] ;

[0032] In the formula, indicates a Gaussian filter, indicates normalization, indicates the processed directional feature, ;

[0033] S34, constructing a four-channel feature map and performing Fourier transform to extract a phase spectrum:

[0034] ;

[0035] In the formula, indicates a stacking operation, indicates a Fourier transform, indicates a phase spectrum calculation, denotes a phase spectrum;

[0036] S35, an inverse Fourier transform is performed to obtain the saliency map:

[0037] ;

[0038] wherein, denotes an inverse Fourier transform, denotes a frequency domain primary perceptual feature output by the frequency domain module.

[0039] Preferably, in S4, the specific process of enhancing the frequency domain primary perceptual feature by using the frequency domain feature fusion module and fusing in multiple dimensions to obtain the frequency domain saliency feature is as follows:

[0040] S41, the frequency domain primary perceptual feature output by S3 is taken as the imaginary part, and the real part feature is the feature corresponding to the original input image, and the calculation expression is as follows:

[0041] ;

[0042] wherein, denotes a real part feature, denotes an imaginary part feature, denotes a phase spectrum analysis process;

[0043] S42, two-level convolutional coding processing is performed on the real part feature and the imaginary part feature respectively to enhance the feature expression capability, and the calculation expression is as follows:

[0044] ;

[0045] wherein, denotes a complex convolution, denotes a complex normalization, denotes a complex activation function, denotes an output coded complex feature;

[0046] S43, a channel and spatial attention mechanism is applied to selectively enhance the fused feature, and an output layer is used to generate the final attention weight map, and the calculation expression is as follows:

[0047] ;

[0048] wherein, denotes a convolution, denotes a channel and spatial attention mechanism, denotes a Sigmoid function, denotes an output frequency domain saliency feature.

[0049] Preferably, in S5, the process of using the deformable spatial attention module to model the deformation of high-level features by using the directionally deformable convolution module with geometric self-adaptive function includes the following sub-steps:

[0050] S511, the high-level features of the fused two-layer features input in S2 are extracted through a basic convolution layer to obtain basic features, and the calculation expression is as follows:

[0051] ;

[0052] In the formula, represents the basic convolution, represents the high-level feature, represents the basic feature;

[0053] S512, the basic features obtained in S511 are subjected to an offset convolution layer to generate spatial offsets, and the calculation expression is as follows:

[0054] ;

[0055] In the formula, represents the offset convolution, represents the hyperbolic tangent activation function, represents the spatial offset;

[0056] S513, based on the spatial offsets obtained in S512, a learnable sampling grid is constructed, and the calculation expression is as follows:

[0057] ;

[0058] In the formula, represents the standard sampling grid, represents the rearranged offset, represents the final learnable sampling grid;

[0059] S514, the basic features are divided into two groups and respectively applied to spatial transformation to obtain output results, and the calculation expression is as follows:

[0060] ;

[0061] In the formula, represents the grid sampling operation, respectively correspond to sampling grids in different directions, represents the channel splicing operation, represents the output directional feature.

[0062] Preferably, in S5, the specific process of grouping attention weighted fusion of features further includes:

[0063] S521, the direction features extracted in S514 are subjected to channel rearrangement operation to enhance feature diversity, and the expression is as follows:

[0064] ;

[0065] In the formula, indicates channel rearrangement into four groups, indicates the rearranged features;

[0066] S522, the rearranged feature tensor is uniformly divided into 4 sub-feature tensors according to the channel dimension, and the expression is as follows:

[0067] ;

[0068] In the formula, indicates division into four groups of features according to the number of channels, indicates the divided features;

[0069] S523, a spatial attention mechanism is applied to each sub-feature tensor to extract spatial attention weights, and the expression is as follows:

[0070] ;

[0071] In the formula, indicates the spatial attention mechanism, indicates the spatial attention weight.

[0072] Preferably, in S5, the specific process of extracting spatial enhancement perception features further comprises:

[0073] S531, to balance the importance of the four groups of spatial attention, initialize the learnable parameters to weight and fuse the spatial attention weights, and the learnable parameters are updated and learned through back propagation, and the expression is as follows:

[0074] ;

[0075] In the formula, indicates normalization of the i-th learnable weight parameter, indicates the weighted fused features;

[0076] S532, the fused attention weights are subjected to convolution processing and Sigmoid activation, the fused attention weights are multiplied by the original input features and added with residual connection to obtain spatial enhancement features, and the expression is as follows:

[0077] ;

[0078] In the formula, indicates the intermediate fused features, This indicates enhanced spatial perception features.

[0079] Preferably, in S6, the specific process of inputting the frequency domain saliency features and spatially enhanced sensing features into the saliency prediction module to generate the prediction map is as follows:

[0080] S611. Upsample the obtained spatial enhancement features to match the size of the frequency domain saliency features, and fuse the two features. The calculation formula is as follows:

[0081] ;

[0082] In the formula, This indicates a feature fusion operation. Indicates upsampling, Indicates the characteristics after fusion;

[0083] S612. The fused features are passed through the final convolutional layer to generate the original output of object detection. The output is upsampled to the original image size and Sigmoid activation is applied. The calculation expression is as follows:

[0084] ;

[0085] In the formula, This represents the final convolutional layer. This indicates upsampling to the original image size. This represents the Sigmoid function. This represents the generated prediction graph.

[0086] Preferably, in S6, the specific process of using the loss function for constraint is as follows: the optimization process of the prediction inference module adopts IOU loss and cross-entropy loss, and the calculation formula is as follows:

[0087] ;

[0088] In the formula, Represents the actual label pixels, Indicates the predicted probability. Indicates the total loss. express loss, This represents the cross-entropy loss.

[0089] Therefore, this invention proposes a saliency target detection method that integrates multi-dimensional frequency domain and deformable attention, with the following beneficial effects:

[0090] (1) Innovative frequency-spatial collaborative perception mechanism: The frequency domain structure information of low-level features is extracted by the phase spectrum analysis module, and complex features are constructed by the frequency domain feature fusion module to realize spatial-frequency fusion. This makes up for the shortcomings of existing methods that rely only on spatial learning and lack global structure perception, strengthens the capture of key structures such as target contours and textures, and improves the ability to distinguish between targets and backgrounds in complex scenes.

[0091] (2) Optimize the adaptability of deformable targets: By using the directional deformable convolution of the deformable spatial attention module, spatial offset and learnable sampling grid are dynamically generated to achieve accurate modeling of multi-scale and geometrically deformable targets; combined with grouped attention weighted fusion, the key areas of the target are further focused to solve the problems of insufficient expression of deformable target features and low positioning accuracy in traditional methods.

[0092] (3) Enhanced detail and semantic co-preservation: Features are extracted hierarchically by multi-scale Transformer encoder, the first two low-level features are fused to preserve edge details, and the last two high-level features are enhanced to strengthen semantic information; important features are selectively enhanced by attention mechanism to reduce background interference, effectively improve the detail integrity and semantic consistency of detection results, and improve detection robustness in complex backgrounds.

[0093] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0094] Figure 1 This is an overall flowchart of the salient target detection method that integrates multi-dimensional frequency domain and deformable attention according to the present invention;

[0095] Figure 2 This is a schematic diagram of the phase spectrum analysis module and the frequency domain feature fusion module of an embodiment of the multi-dimensional frequency domain and deformable attention fusion salient target detection method of the present invention;

[0096] Figure 3 This is a schematic diagram of a deformable spatial attention module, which is an embodiment of the saliency target detection method that integrates multi-dimensional frequency domain and deformable attention according to the present invention.

[0097] Figure 4 This is a schematic diagram of the multi-scale feature extraction results of a salient target detection method that integrates multi-dimensional frequency domain and deformable attention according to the present invention.

[0098] Figure 5 This is a schematic diagram of the frequency domain perception result of a salient target detection method that integrates multi-dimensional frequency domain and deformable attention according to the present invention.

[0099] Figure 6 This is a schematic diagram of the spatial enhancement feature results of a multi-dimensional frequency domain and deformable attention fusion saliency target detection method according to the present invention.

[0100] Figure 7 This is a schematic diagram of the detection results of the multi-dimensional frequency domain and deformable attention fusion salient target detection method of the present invention in different scenarios. Detailed Implementation

[0101] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below. The described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of this application.

[0102] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0103] like Figures 1-3 As shown, the present invention provides a multi-dimensional frequency domain and deformable attention fusion saliency target detection method, comprising:

[0104] S1. Input the infrared image to be detected;

[0105] S2. Multi-scale feature extraction and fusion: The multi-scale feature extraction module performs hierarchical feature encoding on the input infrared image to generate multi-scale primary perception features. Then, the multi-scale feature fusion module fuses the first two layers of features as low-level features and fuses the last two layers of features as high-level features.

[0106] S3. Frequency Domain Primary Sensing Feature Extraction: Based on the phase spectrum analysis module, multi-dimensional phase spectrum analysis is performed on the low-level features obtained by fusing the first two layers of features through the multi-scale feature fusion module to extract the frequency domain primary sensing features.

[0107] S4. Frequency Domain Feature Fusion Enhancement: The frequency domain feature fusion module is used to enhance the primary perceptual features in the frequency domain, and frequency domain saliency features are obtained by multi-dimensional fusion.

[0108] S5. Spatial Enhancement Perception Feature Extraction: Using a deformable spatial attention module, a geometrically adaptive directional deformable convolution module is used to model the deformation of high-level features. Then, the features are grouped and weighted by attention to extract spatial enhancement perception features.

[0109] S6. Saliency Prediction and Loss Constraint: Input the frequency domain saliency features and spatial augmented sensing features into the saliency prediction module to generate a prediction map, and use a loss function for constraint.

[0110] In S2, the multi-scale feature extraction module performs hierarchical feature encoding on the input infrared image to generate multi-scale primary sensing features. Then, the multi-scale feature fusion module fuses the first two layers of features as low-level features and the last two layers of features as high-level features. The specific process is as follows:

[0111] S21. Input the real infrared image The four-stage hierarchical Transformer encoder extracts features at four scales, as shown in the following expression:

[0112] ;

[0113] In the formula, C is the number of image channels, W is the image width, and H is the image height. Indicates Transformer encoder, Representing four scale features, Number of output channels To output image width, Output image height;

[0114] S22, The features of the first two stages obtained in S21 After upsampling to the same resolution, multi-scale feature fusion is performed to obtain low-level features containing detailed information. The features obtained in the last two stages of S21 are then combined. After upsampling to the same resolution, multi-scale feature fusion is performed to obtain high-level features containing semantic information. The calculation expression is as follows:

[0115] ;

[0116] In the formula, This indicates a multi-scale feature fusion operation. This indicates a splicing operation. Indicates upsampling, Indicates low-level features, Indicates characteristics of high-level personnel.

[0117] In S21, the four scale features output by the four-stage layered Transformer encoder satisfy the following conditions: the output width and height of the feature in the next stage are half of those in the previous stage, and the number of output channels is twice that of the previous stage. In S22, the upsampling operation uses bilinear interpolation to make the features in the first two stages and the features in the last two stages reach the same resolution. The multi-scale feature fusion operation is a stitching operation followed by a 1×1 convolution to adjust the number of channels.

[0118] Among them, the multi-scale feature extraction module, combined with the Transformer architecture, can directly model the long-range dependency between any two pixels in the image by leveraging its self-attention mechanism. This allows it to capture the structural contextual information of the target in a global view, while retaining local detail features through a layered fusion design, thus overcoming the limitation of the limited receptive field of traditional convolutional neural networks.

[0119] In S3, the specific process of extracting primary sensing features in the frequency domain by performing multi-dimensional phase spectrum analysis on low-level features based on the phase spectrum analysis module is as follows:

[0120] S31. First, preprocess the input low-level features:

[0121] ;

[0122] In the formula, Indicates a center-pass filter;

[0123] S32. Use a preset directional filter checker to perform multi-directional gradient feature extraction on the filtered image:

[0124] ;

[0125] In the formula, For the preset directional filter kernel, K0, K 45 K 90 K 135 Indicates directional gradient characteristics;

[0126] S33. Normalize and Gaussian smooth the extracted directional gradient features:

[0127] ;

[0128] In the formula, Indicates Gaussian filtering. Indicates normalization, Indicates the directional features after processing. ;

[0129] S34. Construct a four-channel feature map and perform Fourier transform to extract the phase spectrum:

[0130] ;

[0131] In the formula, Indicates a stacking operation. Indicates Fourier transform, This indicates the calculation of the phase spectrum. Represents the phase spectrum;

[0132] S35. The saliency map is obtained by inverse Fourier transform:

[0133] ;

[0134] In the formula, Indicates the inverse Fourier transform. This represents the primary sensing features in the frequency domain output by the frequency domain module.

[0135] The phase spectrum analysis module performs phase spectrum analysis on low-level features, significantly enhancing the perception of structural features such as edges, textures, and contours. Leveraging the inherent invariance of phase information to changes in illumination intensity and contrast, it effectively suppresses background noise and illumination interference, improving the robustness of feature representation. This module, through multi-directional filtering and Fourier transform, provides complementary structural representations of spatial features from the frequency domain dimension, forming a synergistic feature enhancement mechanism across the space and frequency domains.

[0136] In S4, the frequency domain feature fusion module is used to enhance the primary frequency domain perceptual features, and the specific process of obtaining the frequency domain saliency features through multi-dimensional fusion is as follows:

[0137] S41. Taking the frequency domain primary perceptual features output from S3 as the imaginary part and the real part features as the features corresponding to the original input image, the calculation expression is as follows:

[0138] ;

[0139] In the formula, Indicates the real part feature. Indicates the characteristics of the imaginary part. This indicates the phase spectrum analysis process;

[0140] S42. Perform two-stage convolutional coding on the real and imaginary features respectively to enhance feature representation capabilities. The calculation expression is as follows:

[0141] ;

[0142] In the formula, Represents complex convolution. To indicate the normalization of complex numbers, Represents the complex activation function. This represents the encoded complex features of the output;

[0143] S43. Apply channel and spatial attention mechanisms to selectively enhance the fused features, and generate the final attention weight map through the output layer. The calculation expression is as follows:

[0144] ;

[0145] In the formula, Represents convolution. This represents the channel and spatial attention mechanism. This represents the Sigmoid function. This indicates the frequency domain saliency characteristics of the output.

[0146] Among them, the frequency domain feature fusion module combines the real part features of the spatial domain with the imaginary part features of the frequency domain by constructing a complex representation. It adopts complex convolution and attention mechanism to achieve deep fusion and adaptive enhancement of dual-domain features, which effectively improves the model's ability to discriminate salient targets and suppress complex backgrounds. At the same time, the serial combination of channel attention and spatial attention can finely calibrate the features in the channel dimension and spatial dimension respectively, which can highlight the feature responses of important channels and focus on key spatial regions.

[0147] In S5, the process of modeling high-level features using a deformable spatial attention module and a geometrically adaptive oriented deformable convolution module includes the following sub-steps:

[0148] S511. The high-level features of the fused two-layer features from input S2 are passed through a basic convolutional layer to extract basic features. The calculation expression is as follows:

[0149] ;

[0150] In the formula, Represents the basic convolution. Indicates high-level characteristics, Indicates basic features;

[0151] S512. The basic features obtained in S511 are passed through an offset convolutional layer to generate spatial offsets. The calculation expression is as follows:

[0152] ;

[0153] In the formula, This represents offset convolution. This represents the hyperbolic tangent activation function. Indicates spatial offset;

[0154] S513. Construct a learnable sampling mesh based on the spatial offset obtained in S512. The calculation expression is as follows:

[0155] ;

[0156] In the formula, Indicates the standard sampling grid. This represents the offset after rearrangement. This represents the final learnable sampling grid;

[0157] S514. Divide the basic features into two groups and apply spatial transformation to each group to obtain the output results. The calculation expression is as follows:

[0158] ;

[0159] In the formula, Indicates grid sampling operation, Each corresponds to a sampling grid in a different direction. This indicates a channel splicing operation. This indicates the directional characteristics of the output.

[0160] In S5, the specific process of grouping and weighting features for attention fusion also includes:

[0161] S521. Perform channel rearrangement on the directional features extracted in S514 to enhance feature diversity. The calculation expression is as follows:

[0162] ;

[0163] In the formula, This indicates that the channels have been rearranged into four groups. Indicates the features after rearrangement;

[0164] S522. Divide the rearranged feature tensor into 4 sub-feature tensors evenly according to the channel dimension. The calculation expression is as follows:

[0165] ;

[0166] In the formula, This indicates that the features are divided into four groups based on the number of channels. Represents the features after segmentation;

[0167] S523. Apply spatial attention mechanism to each sub-feature tensor to extract spatial attention weights. The calculation expression is as follows:

[0168] ;

[0169] In the formula, This represents the spatial attention mechanism. This represents the spatial attention weights.

[0170] In S5, the specific process of extracting spatial augmentation perception features also includes:

[0171] S531. To balance the importance of the four sets of spatial attention, the learnable parameters are initialized and the spatial attention weights are weighted and fused. The learnable parameters are updated and learned through backpropagation, and the calculation expression is as follows:

[0172] ;

[0173] In the formula, This indicates that the i-th learnable weight parameter is normalized. This indicates the characteristics of weighted fusion;

[0174] S532. Perform convolution and sigmoid activation on the fused attention weights, multiply the fused attention weights by the original input features and add residual connections to obtain spatially enhanced features. The calculation expression is as follows:

[0175] ;

[0176] In the formula, Indicates intermediate fusion features, This indicates enhanced spatial perception features.

[0177] In S6, the specific process of inputting frequency domain saliency features and spatial enhanced sensing features into the saliency prediction module to generate a prediction map is as follows:

[0178] S611. Upsample the obtained spatial enhancement features to match the size of the frequency domain saliency features, and fuse the two features. The calculation formula is as follows:

[0179] ;

[0180] In the formula, This indicates a feature fusion operation. Indicates upsampling, Indicates the characteristics after fusion;

[0181] S612. The fused features are passed through the final convolutional layer to generate the original output of object detection. The output is upsampled to the original image size and Sigmoid activation is applied. The calculation expression is as follows:

[0182] ;

[0183] In the formula, This represents the final convolutional layer. This indicates upsampling to the original image size. This represents the Sigmoid function. This represents the generated prediction graph.

[0184] In S6, the specific process of using loss functions for constraints is as follows: The optimization process of the prediction inference module adopts IOU loss and cross-entropy loss, and the calculation formula is as follows:

[0185] ;

[0186] In the formula, Represents the actual label pixels, Indicates the predicted probability. Indicates the total loss. express loss, This represents the cross-entropy loss.

[0187] Example 1

[0188] This invention provides a saliency target detection method that integrates multi-dimensional frequency domain and deformable attention fusion, the technical route of which is as follows: Figure 1 As shown, it includes the following steps:

[0189] S1. Input the infrared image to be detected; To verify the effectiveness of this salient target detection method for infrared weak target detection and early warning, the public dataset NUDT-SIRST is selected for testing. NUDT-SIRST is a dataset specifically designed for infrared weak target detection tasks, which contains 1327 images with a resolution of 256x256, covering five main background scenes: city, field, highlight, ocean and cloud.

[0190] In S2, a multi-scale feature extraction module performs hierarchical feature encoding on the input infrared image to generate multi-scale primary sensing features. Then, a multi-scale feature fusion module fuses the first two layers of features as low-level features and the last two layers as high-level features. In this embodiment, the low-level and high-level features extracted from some images are as follows: Figure 4 As shown, the specific process is as follows:

[0191] S21. Input the real infrared image The four-stage hierarchical Transformer encoder extracts features at four scales, as shown in the following expression:

[0192] ;

[0193] In the formula, C is the number of image channels, W is the image width, and H is the image height. Indicates Transformer encoder, Representing four scale features, Number of output channels To output image width, Output image height;

[0194] S22, The features of the first two stages obtained in S21 After upsampling to the same resolution, multi-scale feature fusion is performed to obtain low-level features containing detailed information. The features obtained in the last two stages of S21 are then combined. After upsampling to the same resolution, multi-scale feature fusion is performed to obtain high-level features containing semantic information. The calculation expression is as follows:

[0195] ;

[0196] In the formula, This indicates a multi-scale feature fusion operation. This indicates a splicing operation. Indicates upsampling, Indicates low-level features, Indicates characteristics of high-level personnel.

[0197] In S21, the four scale features output by the four-stage layered Transformer encoder satisfy the following conditions: the output width and height of the feature in the next stage are half of those in the previous stage, and the number of output channels is twice that of the previous stage. In S22, the upsampling operation uses bilinear interpolation to make the features in the first two stages and the features in the last two stages reach the same resolution. The multi-scale feature fusion operation is a stitching operation followed by a 1×1 convolution to adjust the number of channels.

[0198] In S3, the specific process of extracting primary sensing features in the frequency domain by performing multi-dimensional phase spectrum analysis on low-level features based on the phase spectrum analysis module is as follows:

[0199] S31. First, preprocess the input low-level features:

[0200] ;

[0201] In the formula, Indicates a center-pass filter;

[0202] S32. Use a preset directional filter checker to perform multi-directional gradient feature extraction on the filtered image:

[0203] ;

[0204] In the formula, For the preset directional filter kernel, K0, K 45 K 90 K 135 Indicates directional gradient characteristics;

[0205] S33. Normalize and Gaussian smooth the extracted directional gradient features:

[0206] ;

[0207] In the formula, Indicates Gaussian filtering. Indicates normalization, Indicates the directional features after processing. ;

[0208] S34. Construct a four-channel feature map and perform Fourier transform to extract the phase spectrum:

[0209] ;

[0210] In the formula, Indicates a stacking operation. Indicates Fourier transform, This indicates the calculation of the phase spectrum. Represents the phase spectrum;

[0211] S35. The saliency map is obtained by inverse Fourier transform:

[0212] ;

[0213] In the formula, Indicates the inverse Fourier transform. This represents the primary sensing features in the frequency domain output by the frequency domain module.

[0214] The phase spectrum analysis module performs phase spectrum analysis on low-level features, significantly enhancing the perception of structural features such as edges, textures, and contours. Leveraging the inherent invariance of phase information to changes in illumination intensity and contrast, it effectively suppresses background noise and illumination interference, improving the robustness of feature representation. This module, through multi-directional filtering and Fourier transform, provides complementary structural representations of spatial features from the frequency domain dimension, forming a spatial-frequency dual-domain collaborative feature enhancement mechanism. In this embodiment, the phase spectrum obtained in S34 and the primary frequency domain perceptual features obtained in S35 are as follows: Figure 5 As shown.

[0215] In S4, the frequency domain feature fusion module is used to enhance the primary frequency domain perceptual features, and the specific process of obtaining the frequency domain saliency features through multi-dimensional fusion is as follows:

[0216] S41. Taking the frequency domain primary perceptual features output from S3 as the imaginary part and the real part features as the features corresponding to the original input image, the calculation expression is as follows:

[0217] ;

[0218] In the formula, Indicates the real part feature. Indicates the characteristics of the imaginary part. This indicates the phase spectrum analysis process;

[0219] S42. Perform two-stage convolutional coding on the real and imaginary features respectively to enhance feature representation capabilities. The calculation expression is as follows:

[0220] ;

[0221] In the formula, Represents complex convolution. To represent the normalization of complex numbers, Represents the complex activation function. This represents the encoded complex features of the output;

[0222] S43. Apply channel and spatial attention mechanisms to selectively enhance the fused features, and generate the final attention weight map through the output layer. The calculation expression is as follows:

[0223] ;

[0224] In the formula, Represents convolution. This represents the channel and spatial attention mechanism. This represents the Sigmoid function. This indicates the frequency domain saliency characteristics of the output.

[0225] Among them, the frequency domain feature fusion module combines the real part features of the spatial domain with the imaginary part features of the frequency domain by constructing a complex representation. It adopts complex convolution and attention mechanism to achieve deep fusion and adaptive enhancement of dual-domain features, which effectively improves the model's ability to discriminate salient targets and suppress complex backgrounds. At the same time, the serial combination of channel attention and spatial attention can finely calibrate the features in the channel dimension and spatial dimension respectively, which can highlight the feature responses of important channels and focus on key spatial regions.

[0226] In S5, the process of modeling high-level features using a deformable spatial attention module and a geometrically adaptive oriented deformable convolution module includes the following sub-steps:

[0227] S511. The high-level features of the fused two-layer features from input S2 are passed through a basic convolutional layer to extract basic features. The calculation expression is as follows:

[0228] ;

[0229] In the formula, Represents the basic convolution. Indicates high-level characteristics, Indicates basic features;

[0230] S512. The basic features obtained in S511 are passed through an offset convolutional layer to generate spatial offsets. The calculation expression is as follows:

[0231] ;

[0232] In the formula, This represents offset convolution. This represents the hyperbolic tangent activation function. Indicates spatial offset;

[0233] S513. Construct a learnable sampling mesh based on the spatial offset obtained in S512. The calculation expression is as follows:

[0234] ;

[0235] In the formula, Indicates the standard sampling grid. This represents the offset after rearrangement. This represents the final learnable sampling grid;

[0236] S514. Divide the basic features into two groups and apply spatial transformation to each group to obtain the output results. The calculation expression is as follows:

[0237] ;

[0238] In the formula, Indicates grid sampling operation, Each corresponds to a sampling grid in a different direction. This indicates a channel splicing operation. This indicates the directional characteristics of the output.

[0239] In S5, the specific process of grouping and weighting features for attention fusion also includes:

[0240] S521. Perform channel rearrangement on the directional features extracted in S514 to enhance feature diversity. The calculation expression is as follows:

[0241] ;

[0242] In the formula, This indicates that the channels have been rearranged into four groups. Indicates the features after rearrangement;

[0243] S522. Divide the rearranged feature tensor into 4 sub-feature tensors evenly according to the channel dimension. The calculation expression is as follows:

[0244] ;

[0245] In the formula, This indicates that the features are divided into four groups based on the number of channels. Represents the features after segmentation;

[0246] S523. Apply spatial attention mechanism to each sub-feature tensor to extract spatial attention weights. The calculation expression is as follows:

[0247] ;

[0248] In the formula, This represents the spatial attention mechanism. This represents the spatial attention weights.

[0249] In S5, the specific process of extracting spatial augmentation perception features also includes:

[0250] S531. To balance the importance of the four sets of spatial attention, the learnable parameters are initialized and the spatial attention weights are weighted and fused. The learnable parameters are updated and learned through backpropagation, and the calculation expression is as follows:

[0251] ;

[0252] In the formula, This indicates that the i-th learnable weight parameter is normalized. This indicates the characteristics of weighted fusion;

[0253] S532. Perform convolution and sigmoid activation on the fused attention weights, multiply the fused attention weights by the original input features and add residual connections to obtain spatially enhanced features. The calculation expression is as follows:

[0254] ;

[0255] In the formula, Indicates intermediate fusion features, This indicates enhanced spatial perception features.

[0256] In this embodiment, the spatial enhancement feature obtained in S532 is as follows: Figure 6 As shown.

[0257] In S6, the specific process of inputting frequency domain saliency features and spatial enhanced sensing features into the saliency prediction module to generate a prediction map is as follows:

[0258] S611. Upsample the obtained spatial enhancement features to match the size of the frequency domain saliency features, and fuse the two features. The calculation formula is as follows:

[0259] ;

[0260] In the formula, This indicates a feature fusion operation. Indicates upsampling, Indicates the characteristics after fusion;

[0261] S612. The fused features are passed through the final convolutional layer to generate the original output of object detection. The output is upsampled to the original image size and Sigmoid activation is applied. The calculation expression is as follows:

[0262] ;

[0263] In the formula, This represents the final convolutional layer. This indicates upsampling to the original image size. This represents the Sigmoid function. This represents the generated prediction graph.

[0264] In S6, the specific process of using loss functions for constraints is as follows: The optimization process of the prediction inference module adopts IOU loss and cross-entropy loss, and the calculation formula is as follows:

[0265] ;

[0266] In the formula, Represents the actual label pixels, Indicates the predicted probability. Indicates the total loss. express loss, This represents the cross-entropy loss.

[0267] In this embodiment, to compare the advantages of the multi-dimensional frequency domain and deformable attention fusion salient target detection method model provided by the present invention in different scenarios, such as... Figure 7 As shown, the final saliency detection visualization results are presented.

[0268] Intuitively, the multi-dimensional frequency domain and deformable attention fusion saliency target detection method provided by this invention ensures the integrity and accuracy of the detection results.

[0269] The results show that the saliency detection advantages of the multi-dimensional frequency domain and deformable attention fusion saliency target detection method provided by this invention in different scenarios are reflected in the following aspects:

[0270] (a) Small target objects: For small targets in the image, there are problems such as easy interference and the target being too small. The multi-dimensional frequency domain and deformable attention fusion salient target detection method provided by the present invention can solve this problem, accurately predict the target position and contour, and show significant advantages.

[0271] (b) Objects in complex outdoor environments: Outdoor environments often contain complex background interference such as dense vegetation, messy rocks, and varied shadows, which can easily cause false detections. The multi-dimensional frequency domain and deformable attention fusion salient target detection method provided by this invention has shown excellent anti-interference ability in such scenarios, effectively locating and detecting various objects in the scene.

[0272] (c) Objects in a sea surface interference environment: The main challenges of sea surface scenes come from dynamic texture interference such as waves, solar flares, and water surface reflections. These interferences are visually very similar to real targets. However, the multi-dimensional frequency domain and deformable attention fusion salient target detection method provided by this invention can clearly, cleanly and accurately locate UAV targets.

[0273] (d) Low visibility objects: Under adverse weather conditions such as fog, haze, smoke, rain, and snow, the image contrast is low and the target features are blurred, making it easy to encounter situations where detection is difficult or incomplete. However, the multi-dimensional frequency domain and deformable attention fusion salient target detection method provided by this invention successfully recovers the complete outline of all key targets.

[0274] (e) Objects in bright light environments: Bright light environments can lead to unclear targets and severe interference from background shadows. The multi-dimensional frequency domain and deformable attention fusion salient target detection method provided by this invention achieves effective detection of targets under bright light interference by perceiving frequency domain information.

[0275] The above results demonstrate the reliability of the multi-dimensional frequency domain and deformable attention fusion salient target detection method provided by this invention in the application of infrared weak target early warning.

[0276] Therefore, this invention provides a multi-dimensional frequency domain and deformable attention fusion salient target detection method. First, an infrared image to be detected is input. A multi-scale feature extraction module performs hierarchical encoding and fusion to obtain low-level and high-level features. Then, the low-level features are used to extract primary frequency domain features through phase spectrum analysis, which are enhanced by a frequency domain fusion module to obtain frequency domain salient features. For the high-level features, a deformable spatial attention module is used to extract spatially enhanced features. Finally, the two types of features are fused to generate a prediction map, constrained by a joint loss of IOU and cross-entropy. This effectively solves the problems of existing methods relying only on the spatial domain, weak global context awareness, and poor adaptation to deformable targets, significantly improving the accuracy and robustness of multi-scale target detection in complex scenes. This invention can provide a methodological reference for infrared weak target early warning.

[0277] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A saliency target detection method that integrates multi-dimensional frequency domain and deformable attention, characterized in that, include: S1. Input the infrared image to be detected; S2. Use the multi-scale feature extraction module to perform hierarchical feature encoding on the input infrared image to generate multi-scale primary sensing features. Then, use the multi-scale feature fusion module to fuse the first two layers of features as low-level features and fuse the last two layers of features as high-level features. S3. Based on the phase spectrum analysis module, perform multi-dimensional phase spectrum analysis on the low-level features obtained by fusing the first two layers of features through the multi-scale feature fusion module to extract the primary sensing features in the frequency domain. S4. Enhance the primary perceptual features in the frequency domain using the frequency domain feature fusion module, and obtain salient features in the frequency domain through multi-dimensional fusion; S5. Using a deformable spatial attention module, a geometrically adaptive directional deformable convolution module is used to model the deformation of high-level features, and then the features are grouped and weighted by attention to extract spatially enhanced perception features. S6. Input the frequency domain saliency features and spatial augmented sensing features into the saliency prediction module to generate a prediction map, and use a loss function for constraint.

2. The saliency target detection method fused with multi-dimensional frequency domain and deformable attention according to claim 1, characterized in that, In S2, the process of using a multi-scale feature extraction module to perform hierarchical feature encoding on the input infrared image to generate multi-scale primary sensing features, and then using a multi-scale feature fusion module to fuse the first two layers of features as low-level features and the last two layers of features as high-level features is as follows: S21. Input the real infrared image The four-stage hierarchical Transformer encoder extracts features at four scales, as shown in the following expression: ； In the formula, C is the number of image channels, W is the image width, and H is the image height. Indicates Transformer encoder, Representing four scale features, Number of output channels To output image width, Output image height; S22, The features of the first two stages obtained in S21 After upsampling to the same resolution, multi-scale feature fusion is performed to obtain low-level features containing detailed information. The features obtained in the last two stages of S21 are then combined. After upsampling to the same resolution, multi-scale feature fusion is performed to obtain high-level features containing semantic information. The calculation expression is as follows: ； In the formula, This indicates a multi-scale feature fusion operation. This indicates a splicing operation. Indicates upsampling, Indicates low-level features, Indicates characteristics of high-level personnel.

3. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method according to claim 2, characterized in that, In S21, the four scale features output by the four-stage layered Transformer encoder satisfy the following conditions: the output width and height of the feature in the next stage are half of those in the previous stage, and the number of output channels is twice that of the previous stage. In S22, the upsampling operation uses bilinear interpolation to make the features in the first two stages and the features in the last two stages reach the same resolution. The multi-scale feature fusion operation is a stitching operation followed by a 1×1 convolution to adjust the number of channels.

4. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method according to claim 1, characterized in that, In S3, the specific process of performing multi-dimensional phase spectrum analysis on low-level features based on the phase spectrum analysis module to extract primary sensing features in the frequency domain is as follows: S31. First, preprocess the input low-level features: ； In the formula, Indicates a center-pass filter; S32. Use a preset directional filter checker to perform multi-directional gradient feature extraction on the filtered image: ； In the formula, For the preset directional filter kernel, Represents directional gradient features; S33. Normalize and Gaussian smooth the extracted directional gradient features: ； In the formula, Indicates Gaussian filtering. Indicates normalization, Indicates the directional features after processing. ; S34. Construct a four-channel feature map and perform Fourier transform to extract the phase spectrum: ； In the formula, Indicates a stacking operation. Indicates Fourier transform, This indicates the calculation of the phase spectrum. Represents the phase spectrum; S35. The saliency map is obtained by inverse Fourier transform: ； In the formula, Indicates the inverse Fourier transform. This represents the primary sensing features in the frequency domain output by the frequency domain module.

5. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method according to claim 4, characterized in that, In S4, the specific process of enhancing the primary frequency domain sensing features using the frequency domain feature fusion module and obtaining the frequency domain saliency features through multi-dimensional fusion is as follows: S41. Taking the frequency domain primary perceptual features output from S3 as the imaginary part and the real part features as the features corresponding to the original input image, the calculation expression is as follows: ； In the formula, Indicates the real part feature. Indicates the characteristics of the imaginary part. This indicates the phase spectrum analysis process; S42. Perform two-stage convolutional coding on the real and imaginary features respectively to enhance feature representation capabilities. The calculation expression is as follows: ； In the formula, Represents complex convolution. To represent the normalization of complex numbers, Represents the complex activation function. This represents the encoded complex features of the output; S43. Apply channel and spatial attention mechanisms to selectively enhance the fused features, and generate the final attention weight map through the output layer. The calculation expression is as follows: ； In the formula, Represents convolution. This represents the channel and spatial attention mechanism. This represents the Sigmoid function. This indicates the frequency domain saliency characteristics of the output.

6. The saliency target detection method fused with multi-dimensional frequency domain and deformable attention according to claim 1, characterized in that, In S5, the process of using a deformable spatial attention module and an oriented deformable convolution module with geometric adaptation to perform deformation modeling of high-level features includes the following sub-steps: S511. The high-level features of the fused two-layer features from the input S2 are passed through a basic convolutional layer to extract basic features. The calculation expression is as follows: ； In the formula, Represents the basic convolution. Indicates high-level characteristics, Indicates basic features; S512. The basic features obtained in S511 are processed through an offset convolutional layer to generate spatial offsets. The calculation expression is as follows: ； In the formula, This represents offset convolution. This represents the hyperbolic tangent activation function. Indicates spatial offset; S513. Construct a learnable sampling mesh based on the spatial offset obtained in S512. The calculation expression is as follows: ； In the formula, Indicates the standard sampling grid. This represents the offset after rearrangement. This represents the final learnable sampling grid; S514. Divide the basic features into two groups and apply spatial transformation to each group to obtain the output results. The calculation expression is as follows: ； In the formula, Indicates grid sampling operation, Each corresponds to a sampling grid in a different direction. This indicates a channel splicing operation. This indicates the directional characteristics of the output.

7. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method according to claim 6, characterized in that, In S5, the specific process of grouping and weighting the attention of features further includes: S521. Perform channel rearrangement on the directional features extracted in S514 to enhance feature diversity. The calculation expression is as follows: ； In the formula, This indicates that the channels have been rearranged into four groups. Indicates the features after rearrangement; S522. Divide the rearranged feature tensor into 4 sub-feature tensors evenly according to the channel dimension. The calculation expression is as follows: ； In the formula, This indicates that the features are divided into four groups based on the number of channels. Represents the features after segmentation; S523. Apply spatial attention mechanism to each sub-feature tensor to extract spatial attention weights. The calculation expression is as follows: ； In the formula, This represents the spatial attention mechanism. This represents the spatial attention weights.

8. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method according to claim 7, characterized in that, In S5, the specific process of extracting spatially enhanced perception features also includes: S531. To balance the importance of the four sets of spatial attention, the learnable parameters are initialized and the spatial attention weights are weighted and fused. The learnable parameters are updated and learned through backpropagation, and the calculation expression is as follows: ； In the formula, This indicates that the i-th learnable weight parameter is normalized. This indicates the characteristics of weighted fusion; S532. Perform convolution and sigmoid activation on the fused attention weights, multiply the fused attention weights by the original input features and add residual connections to obtain spatially enhanced features. The calculation expression is as follows: ； In the formula, Indicates intermediate fusion features, This indicates enhanced spatial perception features.

9. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method according to claim 1, characterized in that, In S6, the specific process of inputting the frequency domain saliency features and spatially enhanced sensing features into the saliency prediction module to generate the prediction map is as follows: S611. Upsample the obtained spatial enhancement features to match the size of the frequency domain saliency features, and fuse the two features. The calculation formula is as follows: ； In the formula, This indicates a feature fusion operation. Indicates upsampling, Indicates the characteristics after fusion; S612. The fused features are passed through the final convolutional layer to generate the original output of object detection. The output is upsampled to the original image size and Sigmoid activation is applied. The calculation expression is as follows: ； In the formula, This represents the final convolutional layer. This indicates upsampling to the original image size. This represents the Sigmoid function. This represents the generated prediction graph.

10. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method according to claim 9, characterized in that, In S6, the specific process of using loss functions for constraints is as follows: the optimization process of the prediction inference module adopts IOU loss and cross-entropy loss, and the calculation formula is as follows: ； In the formula, Represents the actual label pixels, Indicates the predicted probability. Indicates the total loss. express loss, This represents the cross-entropy loss.

Images