Multi-dimensional frequency domain and deformable attention fusion saliency target detection method

Abstract

The invention relates to the field of saliency target detection, and particularly discloses a multi-dimensional frequency domain and deformable attention fusion saliency target detection method, which comprises the steps of S1, inputting an infrared image to be detected; s2, performing multi-scale feature extraction and fusion to obtain low-level and high-level features; s3, phase spectrum analysis is carried out to extract frequency domain primary perception features; s4, fusing the frequency domain features to obtain frequency domain saliency features; s5, a deformable space attention module extracts space enhanced perception features; and S6, fusing the features to generate a prediction map and constraining the prediction map by a loss function. According to the method, the problems of insufficient frequency domain utilization, weak global context and detail retention and poor complex deformation target detection of an existing spatial domain method are solved, and the detection precision and robustness of a multi-scale and deformation target in a complex scene are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of salient object detection, in particular to a multi-dimensional frequency domain and deformable attention fusion salient object detection method. BACKGROUND

[0002] In recent years, with the rapid development of deep learning technology, remote sensing image salient object detection (RSI-SOD) as a basic task in computer vision has made significant progress. The task aims to accurately identify and segment the most visually attractive objects from complex scenes, and plays an important role in many downstream applications such as target tracking, image segmentation and weakly supervised learning. However, traditional methods based on convolutional neural networks (CNN) still face many challenges when dealing with complex backgrounds, multi-scale targets and fine edge structures. In particular, in the actual scene of high-resolution remote sensing images (RSI), the model often shows insufficient global context awareness, low feature alignment accuracy and unsatisfactory detail information preservation.

[0003] Although the research on salient object detection is relatively mature, most existing methods still focus on local feature interaction or single-scale context modeling, and fail to fully realize the effective collaboration between global semantic information and local structural features. To improve the performance of the model, some researches introduce attention mechanisms, multi-path feature fusion and edge auxiliary supervision strategies. For example, the document “Boundary-Aware Salient Object Detection” constructs a coding-decoding model based on the U-Net architecture combined with residual optimization, which gradually refines the saliency map to improve the boundary details; the document “RecurrentResidual Refinement Network for Saliency Detection” proposes a recurrent residual optimization mechanism, which enhances the detail preservation and boundary accuracy through a multi-stage optimization process; the document “Salient Object Detection in Optical Remote Sensing Images Driven by Transformer” uses the Transformer architecture to extract global dependencies, extracts local details through spatial attention modules, and suppresses complex background interference; the document “Hybrid Feature Aligned Network for Salient Object Detection in Optical Remote Sensing Imagery” combines the advantages of CNN and Transformer, and combines the gating mechanism and neighborhood feature alignment strategy to capture deeper context information.

[0004] However, although these airspace learning methods have made significant contributions, most models still only learn and interact features within the airspace, rarely introducing complementary information contained in the image frequency domain. This limitation weakens the model's ability to perceive global structural features. In fact, frequency domain features have been verified to be closely related to human visual attention mechanisms. Therefore, building a new perception framework that can synergistically utilize airspace and frequency domain features is of great significance for achieving high-precision and high-robustness salient object detection in complex scenes.

[0005] In summary, although current methods based on airspace learning have promoted the development of salient object detection, the utilization of frequency domain information is still insufficient. Frequency domain features can provide unique global priors and are highly related to visual saliency.

[0006] Therefore, there is an urgent need for a multi-dimensional frequency domain and deformable attention fusion salient object detection method to achieve high-precision and high-robustness recognition ability in complex scenes, thereby overcoming the problems existing in the prior art. SUMMARY

[0007] The present application provides a multi-dimensional frequency domain and deformable attention fusion salient object detection method to effectively solve the problems of lack of frequency domain information utilization, insufficient global-local feature synergy, and poor performance in complex scene detection in existing methods, effectively improve the detection accuracy of deformable objects, and enhance the global structure perception ability and detail preservation effect.

[0008] To achieve the above-mentioned purpose, the present application provides a multi-dimensional frequency domain and deformable attention fusion salient object detection method, comprising: S1, inputting an infrared image to be detected; S2, using a multi-scale feature extraction module to perform hierarchical feature coding on the input infrared image to generate multi-scale primary perception features, and then using a multi-scale feature fusion module to fuse the features of the first two layers as low-level features and the features of the last two layers as high-level features; S3, based on a phase spectrum analysis module, performing multi-dimensional phase spectrum analysis on the low-level features obtained by fusing the features of the first two layers through the multi-scale feature fusion module to extract frequency domain primary perception features; S4, using a frequency domain feature fusion module to enhance the frequency domain primary perception features to obtain frequency domain saliency features through multi-dimensional fusion; S5, using a deformable spatial attention module, using a direction deformable convolution module with geometric self-adaptive function to model the deformation of the high-level features, and then performing grouping attention weighted fusion on the features to extract spatially enhanced perception features; S6, inputting the frequency domain perception features and the spatially enhanced perception features into a saliency prediction module to generate a prediction map, and using a loss function to constrain.

[0009] Preferably, in S2, the multi-scale feature extraction module encodes the input infrared image in a hierarchical manner to generate multi-scale primary perception features, and then the multi-scale feature fusion module fuses the features of the first two layers as low-level features and the features of the last two layers as high-level features. The specific process is as follows: S21, the input real infrared image is encoded by a four-stage hierarchical Transformer encoder to extract four scale features, and the expression is as follows: ; In the formula, C is the number of image channels, W is the image width, H is the image height, represents the Transformer encoder, represents the four scale features, is the output channel number, is the output image width, is the output image height; S22, the first two stage features obtained in S21 are upsampled to the same resolution and then fused to obtain low-level features containing detailed information, and the last two stage features obtained in S21 are upsampled to the same resolution and then fused to obtain high-level features containing semantic information, and the calculation expression is as follows: ; In the formula, represents the multi-scale feature fusion operation, represents the splicing operation, represents the up-sampling, represents the low-level feature, represents the high-level feature.

[0010] Preferably, in S21, the four scale features output by the four-stage hierarchical Transformer encoder have the following characteristics: the output width and height of the feature of the next stage are 1 / 2 of the feature of the previous stage, and the output channel number is twice that of the previous stage; in S22, the up-sampling operation adopts a bilinear interpolation method to make the first two stage features and the last two stage features reach the same resolution, and the multi-scale feature fusion operation is a splicing operation followed by a 1x1 convolution to adjust the channel number.

[0011] Preferably, in S3, the multi-dimensional phase spectrum analysis module performs multi-dimensional phase spectrum analysis on the low-level features to extract frequency domain primary perception features, and the specific process is as follows: S31, the input low-level features are preprocessed: ​​​ ; In the formula, indicates a median filter; S32, using a preset directional filter kernel to perform multi-directional gradient feature extraction on the filtered image: ; In the formula, is a preset directional filter kernel, indicates a directional gradient feature; S33, normalizing and Gaussian smoothing the extracted directional gradient features: ; In the formula, indicates a Gaussian filter, indicates normalization, indicates the processed directional feature, ; S34, constructing a four-channel feature map and performing Fourier transform to extract a phase spectrum: ; In the formula, indicates a stacking operation, indicates a Fourier transform, indicates a phase spectrum calculation, indicates a phase spectrum; S35, inverse Fourier transform to obtain a saliency map: ; In the formula, indicates an inverse Fourier transform, indicates a frequency domain primary perceptual feature output by a frequency domain module.

[0012] Preferably, in S4, the specific process of enhancing the frequency domain primary perceptual feature by using a frequency domain feature fusion module and obtaining a frequency domain saliency feature through multi-dimensional fusion is as follows: S41, taking the frequency domain primary perceptual feature output by S3 as an imaginary part, and taking a feature corresponding to the original input image as a real part feature, and calculating the expression as follows: ; In the formula, indicates a real part feature, indicates an imaginary part feature, indicates a phase spectrum analysis process; S42, performing two-level convolutional coding processing on the real part feature and the imaginary part feature respectively to enhance the feature expression capability, and calculating the expression as follows: ; In the formula, indicates a complex number convolution, To indicate the normalization of complex numbers, Represents the complex activation function. This represents the encoded complex features of the output; S43. Apply channel and spatial attention mechanisms to selectively enhance the fused features, and generate the final attention weight map through the output layer. The calculation expression is as follows: ; In the formula, Represents convolution. This represents the channel and spatial attention mechanism. This represents the Sigmoid function. This indicates the frequency domain saliency characteristics of the output.

[0013] Preferably, in S5, the process of using a deformable spatial attention module and an oriented deformable convolution module with geometric adaptive function to perform deformation modeling of high-level features includes the following sub-steps: S511. The high-level features of the fused two-layer features from input S2 are passed through a basic convolutional layer to extract basic features. The calculation expression is as follows: ; In the formula, Represents the basic convolution. Indicates high-level characteristics, Indicates basic features; S512. The basic features obtained in S511 are passed through an offset convolutional layer to generate spatial offsets. The calculation expression is as follows: ; In the formula, This represents offset convolution. This represents the hyperbolic tangent activation function. Indicates spatial offset; S513. Construct a learnable sampling mesh based on the spatial offset obtained in S512. The calculation expression is as follows: ; In the formula, Indicates the standard sampling grid. This represents the offset after rearrangement. This represents the final learnable sampling grid; S514. Divide the basic features into two groups and apply spatial transformation to each group to obtain the output results. The calculation expression is as follows: ; In the formula, Indicates grid sampling operation, Each corresponds to a sampling grid in a different direction. represents a channel concatenation operation, represents the direction feature of the output.

[0014] Preferably, in S5, the specific process of grouping and attention-weighted fusion of features further comprises: S521, the direction features extracted in S514 are subjected to a channel rearrangement operation to enhance feature diversity, and the calculation expression is as follows: ; In the formula, represents channel rearrangement into four groups, represents the rearranged features; S522, the rearranged feature tensor is uniformly divided into 4 sub-feature tensors according to the channel dimension, and the calculation expression is as follows: ; In the formula, represents dividing into four groups of features according to the number of channels, represents the divided features; S523, a spatial attention mechanism is applied to each sub-feature tensor to extract spatial attention weights, and the calculation expression is as follows: ; In the formula, represents the spatial attention mechanism, represents the spatial attention weight.

[0015] Preferably, in S5, the specific process of extracting spatial enhancement perception features further comprises: S531, to balance the importance of the four groups of spatial attention, initialize a learnable parameter to weight and fuse the spatial attention weights, and the learnable parameter is updated and learned through back propagation, and the calculation expression is as follows: ; In the formula, represents normalizing the i-th learnable weight parameter, represents the weighted fused features; S532, the fused attention weights are subjected to convolution processing and Sigmoid activation, the fused attention weights are multiplied by the original input features and added with a residual connection to obtain spatial enhancement features, and the calculation expression is as follows: ; In the formula, represents the intermediate fused features, represents the enhanced spatial perception features.

[0016] Preferably, in S6, the specific process of inputting the frequency domain perception feature and the spatial enhancement perception feature into the saliency prediction module to generate a prediction map is as follows: S611, upsample the obtained spatial enhancement feature to match the size of the frequency domain perception feature, fuse the two features, and the calculation formula is as follows: In the formula, denotes a feature fusion operation, denotes upsampling, denotes the fused feature; S612, generate a target detection original output by passing the fused feature through a final convolution layer, upsample the output result to the original image size and apply Sigmoid activation, and the calculation expression is as follows: In the formula, denotes a final convolution layer, denotes upsampling to the original image size, denotes a Sigmoid function, denotes the generated prediction map.

[0017] Preferably, in S6, the specific process of using a loss function for constraint is as follows: the optimization process of the prediction inference module adopts an IOU loss and a cross-entropy loss, and the calculation formula is as follows: In the formula, denotes a true label pixel, denotes a prediction probability, denotes a total loss, denotes loss, denotes a cross-entropy loss.

[0018] Therefore, the present application proposes a multi-dimensional frequency domain and deformable attention fusion saliency target detection method, which has the following beneficial effects: (1) Innovative frequency-space collaborative perception mechanism: the phase spectrum analysis module extracts the frequency domain structure information of low-level features, and the frequency domain feature fusion module constructs complex features to realize space-frequency fusion, which makes up for the defects of existing methods that only rely on space domain learning and lack of global structure perception, strengthens the capture of key structures such as target contour and texture, and improves the ability to distinguish targets and backgrounds in complex scenes.

[0019] ​​​(2) Optimize the deformation target adaptation ability: with the help of the direction deformable convolution of the deformable spatial attention module, the spatial offset and the learnable sampling grid are dynamically generated to realize the accurate modeling of multi-scale and geometric deformation targets; combined with the grouping attention weighted fusion, the key areas of the target are further focused, and the problems of insufficient feature expression and low positioning accuracy of traditional methods for deformation targets are solved.

[0020] (3) Enhance the details and semantic collaborative reservation: through the hierarchical extraction of features by the multi-scale Transformer encoder, the low-level features of the first two layers are fused to retain edge details, and the high-level features of the last two layers are fused to strengthen semantic information; combined with the attention mechanism, important features are selectively enhanced to reduce background interference, effectively improving the detail integrity and semantic consistency of the detection results, and improving the detection robustness in complex backgrounds.

[0021] The technical solutions of the present application will be further described in detail below through the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is the overall flowchart of a multi-dimensional frequency domain and deformable attention fusion saliency target detection method of the present application; Figure 2 is the phase spectrum analysis module and frequency domain feature fusion module schematic diagram of an embodiment of a multi-dimensional frequency domain and deformable attention fusion saliency target detection method of the present application; Figure 3 is the deformable spatial attention module schematic diagram of an embodiment of a multi-dimensional frequency domain and deformable attention fusion saliency target detection method of the present application.

[0023] Figure 4 is the multi-scale feature extraction result schematic diagram of a multi-dimensional frequency domain and deformable attention fusion saliency target detection method of the present application.

[0024] Figure 5 is the frequency domain perception result schematic diagram of a multi-dimensional frequency domain and deformable attention fusion saliency target detection method of the present application.

[0025] Figure 6 is the spatial domain enhanced feature result schematic diagram of a multi-dimensional frequency domain and deformable attention fusion saliency target detection method of the present application.

[0026] Figure 7 is the detection result schematic diagram of a multi-dimensional frequency domain and deformable attention fusion saliency target detection method in different scenes. DETAILED DESCRIPTION

[0027] In order to make the technical solutions, advantages and objectives of the present application clearer, the technical solutions of the embodiments of the present application will be clearly and completely described below. The described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0028] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the usual meaning understood by those of ordinary skill in the art to which the present application belongs.

[0029] As shown in Figures 1-3 The present application provides a multi-dimensional frequency domain and deformable attention fusion saliency target detection method, which comprises: S1, inputting an infrared image to be detected; S2, multi-scale feature extraction and fusion: using a multi-scale feature extraction module to perform hierarchical feature coding on the input infrared image to generate multi-scale primary perception features, and then using a multi-scale feature fusion module to fuse the features of the first two layers as low-level features and the features of the last two layers as high-level features; S3, frequency domain primary perception feature extraction: based on a phase spectrum analysis module, performing multi-dimensional phase spectrum analysis on the low-level features obtained by fusing the features of the first two layers through the multi-scale feature fusion module to extract frequency domain primary perception features; S4, frequency domain feature fusion enhancement: using a frequency domain feature fusion module to enhance the frequency domain primary perception features to obtain frequency domain saliency features through multi-dimensional fusion; S5, spatial enhanced perception feature extraction: using a deformable spatial attention module, using a direction deformable convolution module with geometric self-adaptive function to model the deformation of the high-level features, and then performing grouping attention weighted fusion on the features to extract spatial enhanced perception features; S6, saliency prediction and loss constraint: inputting the frequency domain perception features and the spatial enhanced perception features into a saliency prediction module to generate a prediction map, and using a loss function to constrain.

[0030] In S2, the multi-scale feature extraction module is used to perform hierarchical feature coding on the input infrared image to generate multi-scale primary perception features, and then the multi-scale feature fusion module is used to fuse the features of the first two layers as low-level features and the features of the last two layers as high-level features. The specific process is as follows: S21, inputting a real infrared image extracting four scale features through a four-stage hierarchical Transformer encoder, and the expression is as follows: ; In the formula, C is the number of image channels, W is the image width, and H is the image height. Indicates Transformer encoder, Representing four scale features, Number of output channels To output image width, Output image height; S22, The features of the first two stages obtained in S21 After upsampling to the same resolution, multi-scale feature fusion is performed to obtain low-level features containing detailed information. The features obtained in the last two stages of S21 are then combined. After upsampling to the same resolution, multi-scale feature fusion is performed to obtain high-level features containing semantic information. The calculation expression is as follows: ; In the formula, This indicates a multi-scale feature fusion operation. This indicates a splicing operation. Indicates upsampling, Indicates low-level features, Indicates characteristics of high-level personnel.

[0031] In S21, the four scale features output by the four-stage layered Transformer encoder satisfy the following conditions: the output width and height of the feature in the next stage are half of those in the previous stage, and the number of output channels is twice that of the previous stage. In S22, the upsampling operation uses bilinear interpolation to make the features in the first two stages and the features in the last two stages reach the same resolution. The multi-scale feature fusion operation is a stitching operation followed by a 1×1 convolution to adjust the number of channels.

[0032] Among them, the multi-scale feature extraction module, combined with the Transformer architecture, can directly model the long-range dependency between any two pixels in the image by leveraging its self-attention mechanism. This allows it to capture the structural contextual information of the target in a global view, while retaining local detail features through a layered fusion design, thus overcoming the limitation of the limited receptive field of traditional convolutional neural networks.

[0033] In S3, the specific process of extracting primary sensing features in the frequency domain by performing multi-dimensional phase spectrum analysis on low-level features based on the phase spectrum analysis module is as follows: S31. First, preprocess the input low-level features: ; In the formula, Indicates a center-pass filter; S32, multi-direction gradient feature extraction is performed on the filtered image using a preset direction filter kernel; ; In the formula, is a preset direction filter kernel, indicates a direction gradient feature; S33, normalization and Gaussian smoothing processing are performed on the extracted direction gradient feature: ; In the formula, indicates Gaussian filtering, indicates normalization, indicates a processed direction feature, ; S34, a four-channel feature map is constructed and a Fourier transform is performed to extract a phase spectrum: ; In the formula, indicates a stacking operation, indicates a Fourier transform, indicates phase spectrum calculation, indicates a phase spectrum; S35, an inverse Fourier transform is performed to obtain a saliency map: ; In the formula, indicates an inverse Fourier transform, indicates a frequency domain primary perception feature output by a frequency domain module.

[0034] The phase spectrum analysis module performs phase spectrum analysis on low-level features, which can significantly enhance the perception ability of structural features such as edges, textures and contours, and effectively suppress background noise and light interference by virtue of the natural invariance of phase information to light intensity and contrast changes, thereby improving the robustness of feature expression. The module provides complementary structural representation for spatial domain features from the frequency domain dimension through multi-direction filtering and Fourier transform, forming a feature enhancement mechanism of spatial-frequency dual-domain cooperation.

[0035] In S4, the frequency domain primary perception feature output by S3 is used as an imaginary part, and a real part feature is a feature corresponding to an original input image, and a calculation expression is as follows: S41, the frequency domain primary perception feature output by S3 is used as an imaginary part, and a real part feature is a feature corresponding to an original input image, and a calculation expression is as follows: ; In the formula, indicates a real part feature, indicates an imaginary part feature, indicates a phase spectrum analysis processing process; S42, the real part feature and the imaginary part feature are respectively processed by two-level convolutional coding to enhance the feature expression ability, and the calculation expression is as follows: ; In the formula, represents complex convolution, represents complex normalization, represents a complex activation function, represents the output coded complex feature; S43, the channel and spatial attention mechanism is applied to selectively enhance the fused feature, and the final attention weight map is generated through the output layer, and the calculation expression is as follows: ; In the formula, represents convolution, represents channel and spatial attention mechanism, represents a Sigmoid function, represents the output frequency domain saliency feature.

[0036] Among them, the frequency domain feature fusion module combines the spatial real part feature with the frequency domain imaginary part feature by constructing a complex representation, realizes the deep fusion and adaptive enhancement of the double domain features by using complex convolution and attention mechanism, effectively improves the discrimination ability of the model to the saliency target and the suppression ability to the complex background, and the serial combination of channel attention and spatial attention can fine-tune the features in channel dimension and spatial dimension respectively, which can highlight the feature response of important channels and focus on key spatial regions.

[0037] In S5, the deformable spatial attention module is used, and the direction deformable convolution module with geometric adaptive function is used to model the deformation of high-level features, including the following sub-steps: S511, the high-level features of the fused two-layer features input by S2 are extracted through the basic convolution layer to obtain the basic features, and the calculation expression is as follows: ; In the formula, represents basic convolution, represents high-level features, represents basic features; S512, the basic features obtained by S511 are processed through the offset convolution layer to generate spatial offsets, and the calculation expression is as follows: ; In the formula, represents offset convolution, represents a hyperbolic tangent activation function, represents spatial offsets; S513, constructing the learnable sampling grid based on the spatial offset obtained in S512, the calculation expression is as follows: ; In the formula, denotes the standard sampling grid, denotes the rearranged offset, denotes the final learnable sampling grid; S514, dividing the basic feature into two groups and applying spatial transformation respectively to obtain the output result, the calculation expression is as follows: ; In the formula, denotes the grid sampling operation, correspond to sampling grids in different directions respectively, denotes the channel splicing operation, denotes the output directional feature.

[0038] In S5, the specific process of grouping attention weighted fusion of the feature also includes: S521, performing channel rearrangement operation on the directional feature extracted in S514 to enhance the diversity of the feature, the calculation expression is as follows: ; In the formula, denotes the channel rearrangement into four groups, denotes the rearranged feature; S522, dividing the rearranged feature tensor into 4 sub-feature tensors uniformly according to the channel dimension, the calculation expression is as follows: ; In the formula, denotes the division into four groups of features according to the number of channels, denotes the divided feature; S523, applying spatial attention mechanism to each sub-feature tensor respectively to extract spatial attention weight, the calculation expression is as follows: ; In the formula, denotes the spatial attention mechanism, denotes the spatial attention weight.

[0039] In S5, the specific process of extracting spatial enhancement perception feature also includes: S531, to balance the importance of the four groups of spatial attention, initializing the learnable parameter to weight and fuse the spatial attention weight, the learnable parameter is updated and learned through back propagation, the calculation expression is as follows: ; wherein, denotes normalizing the i-th learnable weight parameter, denotes the weighted fused feature; S532, the fused attention weight is convoluted and Sigmoid activated, the fused attention weight is multiplied by the original input feature with a residual connection added, and a spatial enhanced feature is obtained, and the calculation expression is as follows: ; wherein, denotes the intermediate fused feature, denotes the enhanced spatial perception feature.

[0040] In S6, the specific process of inputting the frequency domain perception feature and the spatial enhanced perception feature into the saliency prediction module to generate a prediction map is as follows: S611, the obtained spatial enhanced feature is upsampled to match the size of the frequency domain perception feature, and the two features are fused, and the calculation formula is as follows: ; wherein, denotes the feature fusion operation, denotes up-sampling, denotes the fused feature; S612, the fused feature is input into the final convolutional layer to generate the original output of the target detection, the output result is upsampled to the size of the original image and Sigmoid activated, and the calculation expression is as follows: ; wherein, denotes the final convolutional layer, denotes up-sampling to the size of the original image, denotes the Sigmoid function, denotes the generated prediction map.

[0041] In S6, the specific process of using a loss function for constraint is as follows: the optimization process of the prediction inference module uses IOU loss and cross-entropy loss, and the calculation formula is as follows: ; wherein, denotes the real label pixel, denotes the prediction probability, denotes the total loss, denotes loss, denotes the cross-entropy loss.

[0042] Embodiment one The application provides a multi-dimensional frequency domain and deformable attention fusion salient object detection method, and a technical route thereof is as shown in the figure Figure 1 , and comprises the following steps: S1, inputting an infrared image to be detected; in order to verify the effectiveness of the salient object detection method on infrared dim small target detection and early warning, a public data set NUDT-SIRST is selected for testing, the NUDT-SIRST is a data set specially designed for the infrared dim small target detection task, and the data set contains 1327 images with a resolution of 256x256, covering five main background scenes of city, field, highlight, ocean and cloud.

[0043] In S2, a multi-scale feature extraction module is used to perform hierarchical feature coding on the input infrared image to generate multi-scale primary perception features, and then a multi-scale feature fusion module is used to fuse the features of the first two layers as low-level features and the features of the last two layers as high-level features, as shown in the figure Figure 4 , and the specific process is as follows: S21, the input real infrared image is extracted through a four-stage hierarchical Transformer encoder, and the expression is as follows: ; In the formula, C represents the number of image channels, W represents the width of the image, H represents the height of the image, represents the Transformer encoder, represents the four-scale features, is the output channel number, is the output image width, is the output image height; S22, the first two-stage features obtained in S21 are up-sampled to the same resolution and then multi-scale feature fusion is performed to obtain low-level features containing detailed information, and the last two-stage features obtained in S21 are up-sampled to the same resolution and then multi-scale feature fusion is performed to obtain high-level features containing semantic information, and the calculation expression is as follows: ; In the formula, represents the multi-scale feature fusion operation, represents the splicing operation, represents the up-sampling, represents the low-level features, represents the high-level features.

[0044] In S21, four scale features output by the four-stage hierarchical Transformer encoder satisfy: the output width and height of the feature of the next stage is 1 / 2 of the feature of the previous stage, and the output channel number of the feature of the next stage is 2 times of the feature of the previous stage; in S22, the bilinear interpolation is used for the upsampling operation, so that the features of the first two stages and the features of the last two stages reach the same resolution, and the multi-scale feature fusion operation is a concatenation operation followed by a 1x1 convolution to adjust the channel number.

[0045] In S3, the specific process of extracting the frequency domain primary perception feature based on the phase spectrum analysis module for multi-dimensional phase spectrum analysis of the low-level feature is as follows: S31, the input low-level feature is preprocessed first: ; In the formula, indicates a mean filter; S32, a preset directional filter kernel is used to extract multi-directional gradient features from the filtered image: ; In the formula, is a preset directional filter kernel, indicates a directional gradient feature; S33, the extracted directional gradient features are normalized and Gaussian smoothed: ; In the formula, indicates a Gaussian filter, indicates normalization, indicates the processed directional feature, ; S34, a four-channel feature map is constructed and Fourier transform is performed to extract the phase spectrum: ; In the formula, indicates a stacking operation, indicates Fourier transform, indicates phase spectrum calculation, indicates a phase spectrum; S35, an inverse Fourier transform is performed to obtain a saliency map: ; In the formula, indicates an inverse Fourier transform, indicates a frequency domain primary perception feature output by the frequency domain module.

[0046] The phase spectrum analysis module performs phase spectrum analysis on the low-level features, which can significantly enhance the perception ability of structural features such as edges, textures and contours, and effectively suppress background noise and illumination interference by virtue of the natural invariance of phase information to illumination intensity and contrast changes, thereby improving the robustness of feature expression. The module provides complementary structural representation for spatial domain features from the frequency domain dimension through multi-directional filtering and Fourier transform, forming a feature enhancement mechanism with spatial-frequency dual-domain cooperation. In this embodiment, the phase spectrum obtained by S34 and the frequency domain primary perception feature obtained by S35 are as shown in FIG. 8. Figure 5

[0047] In S4, the frequency domain primary perception feature is enhanced by using a frequency domain feature fusion module, and the specific process of obtaining the frequency domain saliency feature through multi-dimensional fusion is as follows. S41, the frequency domain primary perception feature output by S3 is taken as the imaginary part, and the real part feature is the feature corresponding to the original input image, and the calculation expression is as follows: In the formula, represents the real part feature, represents the imaginary part feature, represents the phase spectrum analysis processing process; S42, the real part feature and the imaginary part feature are respectively processed by two-level convolutional coding to enhance the feature expression ability, and the calculation expression is as follows: In the formula, represents complex convolution, represents complex normalization, represents a complex activation function, represents the output coded complex feature; S43, a channel and spatial attention mechanism is applied to selectively enhance the fused feature, and the final attention weight map is generated through the output layer, and the calculation expression is as follows: In the formula, represents convolution, represents a channel and spatial attention mechanism, represents a Sigmoid function, represents the output frequency domain saliency feature.

[0048] ​​​​The frequency domain feature fusion module combines the spatial real part feature and the frequency domain imaginary part feature by constructing a complex representation, realizes deep fusion and adaptive enhancement of the double-domain features by using complex convolution and attention mechanism, effectively improves the discrimination ability of the model to the salient target and the suppression ability to the complex background, and the serial combination of the channel attention and the spatial attention can finely calibrate the features in the channel dimension and the spatial dimension respectively, which can highlight the feature response of important channels and focus on key spatial regions.

[0049] In S5, the process of deforming the high-level feature by using the direction deformable convolution module with geometric adaptive function includes the following sub-steps: S511, the high-level feature of the fused two-layer feature input in S2 is extracted into a basic feature by a basic convolution layer, and the calculation expression is as follows: ; In the formula, represents the basic convolution, represents the high-level feature, represents the basic feature; S512, the basic feature obtained in S511 is subjected to an offset convolution layer to generate a spatial offset, and the calculation expression is as follows: ; In the formula, represents the offset convolution, represents the hyperbolic tangent activation function, represents the spatial offset; S513, a learnable sampling grid is constructed based on the spatial offset obtained in S512, and the calculation expression is as follows: ; In the formula, represents the standard sampling grid, represents the rearranged offset, represents the final learnable sampling grid; S514, the basic feature is divided into two groups and respectively applied to the spatial transformation to obtain an output result, and the calculation expression is as follows: ; In the formula, represents the grid sampling operation, respectively correspond to the sampling grids in different directions, represents the channel splicing operation, represents the output direction feature.

[0050] In S5, the specific process of grouping attention weighted fusion of the feature also includes: S521, the direction features extracted in S514 are subjected to channel rearrangement operation to enhance feature diversity, and the expression is as follows: ; In the formula, indicates channel rearrangement into four groups, indicates the rearranged features; S522, the rearranged feature tensor is uniformly divided into 4 sub-feature tensors according to the channel dimension, and the expression is as follows: ; In the formula, indicates that the features are divided into four groups according to the number of channels, indicates the divided features; S523, a spatial attention mechanism is applied to each sub-feature tensor to extract spatial attention weights, and the expression is as follows: ; In the formula, indicates the spatial attention mechanism, indicates the spatial attention weight.

[0051] In S5, the specific process of extracting spatial enhanced perception features further includes: S531, to balance the importance of the four groups of spatial attention, initialize the learnable parameters to weight and fuse the spatial attention weights, and the learnable parameters are updated and learned through back propagation, and the expression is as follows: ; In the formula, indicates that the i-th learnable weight parameter is normalized, indicates the weighted fused features; S532, the fused attention weights are subjected to convolution processing and Sigmoid activation, the fused attention weights are multiplied by the original input features and added with residual connection to obtain spatial enhanced features, and the expression is as follows: ; In the formula, indicates the intermediate fused features, indicates the enhanced spatial perception features.

[0052] In this embodiment, the spatial enhanced features obtained by S532 are as shown in Figure 6 .

[0053] In S6, the specific process of inputting the frequency domain perception features and the spatial enhanced perception features into the saliency prediction module to generate a prediction map is as follows: S611, up-sample the obtained spatial enhanced features to match the frequency domain perception feature size, fuse the two features, and the calculation formula is as follows: ; In the formula, denotes a feature fusion operation, denotes up-sampling, denotes a fused feature; S612, generate a target detection original output through a final convolution layer, up-sample the output result to the original image size and apply Sigmoid activation, and the calculation expression is as follows: ; In the formula, denotes a final convolution layer, denotes up-sampling to the original image size, denotes a Sigmoid function, denotes a generated prediction map.

[0054] In S6, the specific process of using a loss function for constraint is as follows: the optimization process of the prediction inference module adopts an IOU loss and a cross-entropy loss, and the calculation formula is as follows: ; In the formula, denotes a real label pixel, denotes a prediction probability, denotes a total loss, denotes loss, denotes a cross-entropy loss.

[0055] In this embodiment, in order to compare the advantages of the multi-dimensional frequency domain and deformable attention fusion saliency target detection method model provided by the application in different scenes, as shown in Figure 7 , the final saliency detection visualization result is shown.

[0056] Intuitively, the multi-dimensional frequency domain and deformable attention fusion saliency target detection method provided by the application guarantees the integrity and accuracy of the detection result.

[0057] The results show that the saliency detection advantages of the multi-dimensional frequency domain and deformable attention fusion saliency target detection method provided by the application in different scenes are embodied in the following aspects: (a) Weak and small target objects: For weak and small targets existing in the image, there are problems such as being easily disturbed and being too small, and the multi-dimensional frequency domain and deformable attention fusion saliency target detection method provided by the application can solve this problem, accurately predict the target position and contour, and exhibit significant advantages.

[0058] (b) Objects in complex outdoor environments: Outdoor environments often contain dense vegetation, cluttered rocks, and variable shadows, which can easily cause false positives. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method provided by the present application exhibits excellent anti-interference ability in such scenes, effectively locating and detecting various objects in the scene.

[0059] (c) Objects in sea surface interference environment: The main challenge of sea surface scenes comes from dynamic texture interference such as waves, solar flares, and water surface reflections, which are very similar to real targets in visual features. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method provided by the present application can clearly, cleanly, and accurately locate the unmanned aerial vehicle target.

[0060] (d) Low-visibility objects: In fog, haze, smoke, rain, snow, and other adverse weather conditions, the image contrast is low, the target features are blurred, and it is difficult to detect or detect incompletely. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method provided by the present application successfully restores the complete contour of all key targets.

[0061] (e) Objects in high light environment: High light environment can cause unclear targets and serious background shadow interference. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method provided by the present application realizes effective detection of targets under high light interference through perception of frequency domain information.

[0062] The above results show that the multi-dimensional frequency domain and deformable attention fusion saliency target detection method provided by the present application has reliability in the application of infrared dim target early warning.

[0063] Therefore, the present application provides a multi-dimensional frequency domain and deformable attention fusion saliency target detection method, which first inputs the infrared image to be detected, encodes and fuses the low-level and high-level features through the multi-scale feature extraction module, then extracts the frequency domain primary features from the low-level features through phase spectrum analysis, combines the frequency domain fusion module to enhance the frequency domain saliency features, uses the deformable spatial attention module to extract spatial enhancement features from the high-level features, and finally fuses the two types of features to generate a prediction map and is constrained by the IOU and cross-entropy joint loss, effectively solving the problem that the existing method only relies on spatial domain, global context perception is weak, and deformation target adaptation is poor, significantly improving the multi-scale target detection accuracy and robustness in complex scenes. The present application can provide a method reference for infrared dim target early warning work.

[0064] It should be pointed out finally that the above examples are only used to illustrate the technical solutions of the present application but not to limit it, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can still be modified or replaced equivalently, and these modifications or equivalent replacements should not make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A method for multi-dimensional frequency domain and deformable attention fusion salient object detection, characterized in that, include: S1. Input the infrared image to be detected; S2. Use the multi-scale feature extraction module to perform hierarchical feature encoding on the input infrared image to generate multi-scale primary perception features. Then, use the multi-scale feature fusion module to fuse the first two layers of features as low-level features and fuse the last two layers of features as high-level features. S3. Based on the phase spectrum analysis module, perform multi-dimensional phase spectrum analysis on the low-level features obtained by fusing the first two layers of features through the multi-scale feature fusion module to extract the primary sensing features in the frequency domain. S4. Enhance the primary perceptual features in the frequency domain using the frequency domain feature fusion module, and obtain salient features in the frequency domain through multi-dimensional fusion; S5. Using a deformable spatial attention module, a geometrically adaptive directional deformable convolution module is used to model the deformation of high-level features, and then the features are grouped and weighted by attention to extract spatially enhanced perception features. S6. Input the frequency domain sensing features and spatial augmented sensing features into the saliency prediction module to generate a prediction map, and use a loss function for constraint. 2.The multi-dimensional frequency domain and deformable attention fusion saliency object detection method of claim 1, wherein, In S2, the process of using a multi-scale feature extraction module to perform hierarchical feature encoding on the input infrared image to generate multi-scale primary sensing features, and then using a multi-scale feature fusion module to fuse the first two layers of features as low-level features and the last two layers of features as high-level features is as follows: S21, input the real infrared image Four scale features are extracted by four-stage hierarchical Transformer encoder, and the expression is as follows: ； where C is the number of image channels, W is the image width, H is the image height, denotes a Transformer encoder, denotes four scale features, is the number of output channels, is the output image width, is the output image height; S22, the first two stage features obtained in S21 After upsampling to the same resolution, multi-scale feature fusion is performed to obtain low-level features containing detailed information, and the last two stage features obtained in S21 After upsampling to the same resolution, multi-scale feature fusion is performed to obtain high-level features containing semantic information, and the calculation expression is as follows: ； wherein denotes a multi-scale feature fusion operation, denotes a concatenation operation, denotes an up-sampling, denotes a low-level feature, denotes a high-level feature.

3. The multi-dimensional frequency domain and deformable attention fusion saliency object detection method according to claim 2, characterized in that, In S21, the four scale features output by the four-stage layered Transformer encoder satisfy the following conditions: the output width and height of the feature in the next stage are half of those in the previous stage, and the number of output channels is twice that of the previous stage. In S22, the upsampling operation uses bilinear interpolation to make the features in the first two stages and the features in the last two stages reach the same resolution. The multi-scale feature fusion operation is a stitching operation followed by a 1×1 convolution to adjust the number of channels.

4. The multi-dimensional frequency domain and deformable attention fusion salient object detection method according to claim 1, characterized in that, In S3, the specific process of performing multi-dimensional phase spectrum analysis on low-level features based on the phase spectrum analysis module to extract primary sensing features in the frequency domain is as follows: S31. First, preprocess the input low-level features: ； In the formula, denotes a median filter; S32. Use a preset directional filter checker to perform multi-directional gradient feature extraction on the filtered image: ； In the formula, is a preset directional filter kernel, represents a directional gradient feature; S33. Normalize and Gaussian smooth the extracted directional gradient features: ； wherein denotes a Gaussian filter, denotes a normalization, denotes a processed orientation feature, ; S34. Construct a four-channel feature map and perform Fourier transform to extract the phase spectrum: ； wherein denotes a stacking operation, denotes a Fourier transform, denotes a phase spectrum calculation, denotes a phase spectrum; S35. The saliency map is obtained by inverse Fourier transform: ； wherein denotes an inverse Fourier transform, denotes a frequency domain primary perceptual feature output by the frequency domain module.

5. The multi-dimensional frequency domain and deformable attention fusion salient object detection method according to claim 4, characterized in that, In S4, the specific process of enhancing the primary frequency domain sensing features using the frequency domain feature fusion module and obtaining the frequency domain saliency features through multi-dimensional fusion is as follows: S41. Taking the frequency domain primary perceptual features output from S3 as the imaginary part and the real part features as the features corresponding to the original input image, the calculation expression is as follows: ； wherein represents the real part characteristic, represents the imaginary part characteristic, represents the phase spectrum analysis process; S42. Perform two-stage convolutional coding on the real and imaginary features respectively to enhance feature representation capabilities. The calculation expression is as follows: ； wherein, denotes a complex convolution, denotes a complex normalization, denotes a complex activation function, denotes an encoded complex feature of the output; S43. Apply channel and spatial attention mechanisms to selectively enhance the fused features, and generate the final attention weight map through the output layer. The calculation expression is as follows: ； wherein, denotes a convolution, denotes a channel and spatial attention mechanism, denotes a Sigmoid function, denotes the output frequency domain saliency features.

6. The multi-dimensional frequency domain and deformable attention fusion salient object detection method according to claim 1, characterized in that, In S5, the process of using a deformable spatial attention module and an oriented deformable convolution module with geometric adaptation to perform deformation modeling of high-level features includes the following sub-steps: S511. The high-level features of the fused two-layer features from the input S2 are passed through a basic convolutional layer to extract basic features. The calculation expression is as follows: ； wherein denotes a base convolution, denotes a high-level feature, denotes a base feature; S512. The basic features obtained in S511 are processed through an offset convolutional layer to generate spatial offsets. The calculation expression is as follows: ； wherein denotes a shift convolution, denotes a hyperbolic tangent activation function, denotes a spatial shift; S513. Construct a learnable sampling mesh based on the spatial offset obtained in S512. The calculation expression is as follows: ； In the formula, denotes a standard sampling grid, denotes a rearranged offset, denotes a final learnable sampling grid; S514. Divide the basic features into two groups and apply spatial transformation to each group to obtain the output results. The calculation expression is as follows: ； In the formula, represents a grid sampling operation, respectively correspond to sampling grids in different directions, represents a channel concatenation operation, represents the output directional feature.

7. The multi-dimensional frequency domain and deformable attention fusion salient object detection method according to claim 6, characterized in that, In S5, the specific process of grouping and weighting the attention of features further includes: S521. Perform channel rearrangement on the directional features extracted in S514 to enhance feature diversity. The calculation expression is as follows: ； wherein represents rearranging into four groups of channels, represents the features after rearrangement; S522. Divide the rearranged feature tensor into 4 sub-feature tensors evenly according to the channel dimension. The calculation expression is as follows: ； In the formula, denotes the division into four groups of features divided by the number of channels, denotes the divided features; S523. Apply spatial attention mechanism to each sub-feature tensor to extract spatial attention weights. The calculation expression is as follows: ； In the formula, represents the spatial attention mechanism, represents the spatial attention weight.

8. The multi-dimensional frequency domain and deformable attention fusion salient object detection method according to claim 7, characterized in that, In S5, the specific process of extracting spatially enhanced perception features also includes: S531. To balance the importance of the four sets of spatial attention, the learnable parameters are initialized and the spatial attention weights are weighted and fused. The learnable parameters are updated and learned through backpropagation, and the calculation expression is as follows: ； wherein denotes normalizing the i-th learnable weight parameter, denotes the weighted fused feature; S532. Perform convolution and sigmoid activation on the fused attention weights, multiply the fused attention weights by the original input features and add residual connections to obtain spatially enhanced features. The calculation expression is as follows: ； In the formula, Indicates intermediate fusion features, This indicates enhanced spatial perception features.

9. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method according to claim 1, characterized in that, In S6, the specific process of inputting frequency domain sensing features and spatially enhanced sensing features into the saliency prediction module to generate a prediction map is as follows: S611. Upsample the obtained spatial enhancement features to match the size of the frequency domain sensing features, and fuse the two features. The calculation formula is as follows: ； In the formula, This indicates a feature fusion operation. Indicates upsampling, Indicates the characteristics after fusion; S612. The fused features are passed through the final convolutional layer to generate the original output of object detection. The output is upsampled to the original image size and Sigmoid activation is applied. The calculation expression is as follows: ； In the formula, This represents the final convolutional layer. This indicates upsampling to the original image size. This represents the Sigmoid function. This represents the generated prediction graph.

10. The multi-dimensional frequency domain and deformable attention fusion saliency target detection method according to claim 9, characterized in that, In S6, the specific process of using loss functions for constraints is as follows: the optimization process of the prediction inference module adopts IOU loss and cross-entropy loss, and the calculation formula is as follows: ； In the formula, Represents the actual label pixels, Indicates the predicted probability. Indicates the total loss. express loss, This represents the cross-entropy loss.

Images