Method for detecting visible-thermal salient target based on bidirectional alternating fusion strategy
By employing a bidirectional alternating fusion strategy and utilizing feature erasure and global context enhancement modules, the problem of insufficient modality information fusion in existing methods is solved, achieving more efficient salient target detection results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI UNIV
- Filing Date
- 2024-03-19
- Publication Date
- 2026-06-02
Smart Images
Figure CN118038228B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to computer image processing technology, specifically to a visible light-thermal infrared salient target detection method based on a bidirectional alternating fusion strategy. Background Technology
[0002] Visible-thermal-infrared salient target detection aims to simulate human visual mechanisms by extracting salient targets shared by two modalities from visible light and thermal infrared images. For visual research tasks, it involves techniques such as target detection, image segmentation, and feature extraction. In real-world applications, visible-thermal-infrared salient target detection has a wide range of uses, covering fields such as security monitoring and intelligent transportation, and is of great significance for improving system performance and reliability.
[0003] Early researchers proposed effective solutions for salient target detection in visible light. However, in low-light scenarios, the performance of salient target detection methods relying solely on visible light images degrades significantly. Thermal infrared imaging technology locates targets by detecting the thermal radiation of objects, making it highly adaptable to environments such as nighttime, low light, and smoke, and capable of acquiring information invisible to the human eye. Researchers have proposed a salient target detection method based on visible light-thermal infrared by utilizing information from both visible light and thermal infrared images, achieving remarkable performance surpassing salient target detection methods based solely on the visible light modality.
[0004] Current research on salient target detection in the visible-thermal-infrared spectrum can be divided into two categories: one based on traditional learning methods and the other based on deep learning methods. Traditional learning methods typically require manual feature design and classifier selection. They can be tailored to specific tasks, resulting in relatively simple model structures and fast training speeds. However, their poor generalization ability prevents them from handling complex scene data. Deep learning methods, on the other hand, can learn feature representations end-to-end and adapt to more complex data distributions. They typically utilize techniques such as multimodal fusion and transfer learning to improve detection performance. The widespread application of deep learning has greatly advanced salient target detection in the visible-thermal-infrared spectrum, but challenges remain, such as low contrast between salient targets and background, low-quality modal information, and missing modal information.
[0005] Most existing methods fuse features extracted by the encoder, failing to enable the model to uncover modality-specific information. Furthermore, existing methods typically focus on effectively fusing information from two modalities, with the main fusion methods falling into three categories: front-end fusion, mid-end fusion, and back-end fusion. These methods, based on the fact that unidirectional or bidirectional parallel fusion of the two modalities cannot fully capture the interaction between them.
[0006] Therefore, how to enable the network to mine more modal information and how to effectively fuse information from two modalities remain the core of improving the performance of visible light-thermal infrared salient target detection. Summary of the Invention
[0007] Purpose of the invention: The purpose of this invention is to overcome the shortcomings of the existing technology and provide a visible light-thermal infrared salient target detection method based on a bidirectional alternating fusion strategy.
[0008] Technical solution: The present invention provides a visible light-thermal infrared salient target detection method based on a bidirectional alternating fusion strategy, comprising the following steps:
[0009] Step S1: Obtain the dataset, which includes visible light images and thermal infrared images. Divide the dataset into training set and test set according to the proportions.
[0010] For any pair of visible light images and thermal infrared images, a feature extractor with shared pre-trained weights is used to extract the corresponding multi-layer features of these two modalities: the corresponding visible light multi-layer features. and thermal infrared multilayer features Chinese f r 1 f r 2 and f r 3 For low-level features, f r 4 Characterized by high-level features Chinese f t 1 f t 2 and f t 3 For low-level features, f t 4 Characterized by high-level features;
[0011] Step S2: First, for the visible light multilayer features obtained in step S1... and thermal infrared multilayer features The common modal information of the two is obtained, and then the feature erasure module is used to erase the visible light multilayer features respectively. and thermal infrared multilayer features The central modal information is used to obtain specific information about the visible light multilayer features. Specific information of thermal infrared multilayer features
[0012] Step S3: Transfer the specific information E generated in S2 r 4 and E t 4 and high-level characteristics fr 4 and f t 4 are input into the global context enhancement module together to obtain the enhanced global context information G respectively r 4 and G t 4 ;
[0013] Step S4: The specific information E generated in S2 r 4 and E t 4 , the two global context information G obtained in step S3 r 4 and G t 4 as well as the low-level features and are input into the multi-modal interaction module in the bidirectional alternating decoder together to generate new decoded features and Add the decoded features and to generate the final significant target prediction map P.
[0014] Steps 2 to 4 of the present invention are equivalent to embedding the feature erasure module into the global context enhancement module and the bidirectional alternating decoder respectively to make full use of the specific information in the two-modal image features; embedding the global context enhancement module and the multi-modal interaction module in parallel into two branches to perform alternating fusion and continuously decode to generate new decoded features, and adding the decoded features of the two modalities to obtain the significant target prediction map.
[0015] Furthermore, the detailed method for calculating the specific information in step S2 is as follows:
[0016] First, calculate the correlation between and to obtain the correlation matrix R (d,k) , that is, obtain the common activation region of the two modalities; the calculation process of R (d,k) is as follows:
[0017] R (d,k) =(f r (d,k) ) T (w T f t (d,k) );
[0018] where 1 < d < H, 1 < k < w, (d, k) represents the pixel coordinate position, w represents the mapped feature space, and (H, W) represents the picture resolution size;
[0019] Secondly, regarding the correlation matrix R (d,k) Binarization is performed to generate a binary mask, denoted as B, to identify the area to be erased;
[0020] Next, the softmax activation function is used to select the regions to be erased to generate a new matrix G. The formula for calculating G is: G = R·B; '·' represents element-wise multiplication.
[0021] Finally, based on the binarization mask B and matrix G, generation... Erasure features Will As specific information of visible light multilayer features, it is generated based on the binarized mask B and matrix G. Erasure features Will As specific information of thermal infrared multilayer features.
[0022] Furthermore, step S3 obtains global context information G. r 4 The detailed method is as follows:
[0023] High-level feature f r 4 High-level characteristics f t 4 and E r 4 Expanded into a two-dimensional vector P r R∈(HW,C) P t R∈(HW,C) and P e R∈(HW,C) and P e With P t Adding them together generates a new two-dimensional vector P. et And use it as a query, P r The information, along with the key, is fed into the multi-head attention mechanism to obtain global information g shared by both modalities. rt 4 Then introduce deformable convolution and f r 4 The adaptive positional information of the salient target is obtained as input and unfolded into a two-dimensional vector P. dr After being added to the global information, the result is normalized and fed into a feedforward neural network. Residual connections are then used to prevent gradient vanishing. Finally, normalization is performed to obtain the enhanced global context information, denoted as G. r 4 ;
[0024] The calculation formula is expressed as follows:
[0025] P r=flatten(f r 4 ), P t =flatten(f t 4 ), P e =flatten(E r 4 ), P et =P e +P t ;
[0026] Where flatten is a built-in function in the PyTorch library, '+' is an element-wise addition operation, H, W, and C correspond to the resolution size and the number of channels, respectively;
[0027] P dr =DConv(f r 4 );
[0028]
[0029] Where MSA represents multi-head attention, and DConv(*) represents... Deformable convolution, LN is a normalization layer, and FFN is a feedforward neural network.
[0030] Further, step S3 calculates the global context information G. t 4 The detailed method is as follows:
[0031] High-level feature f r 4 High-level characteristics f t 4 and E t 4 Expanded into a two-dimensional vector P r R∈(HW,C) P t R∈(HW,C) and P e R∈(HW,C) and P e With P t Adding them together generates a new two-dimensional vector P. et And use it as a query, P r The information, along with the key, is fed into the multi-head attention mechanism to obtain global information g shared by both modalities. rt 4 Then introduce deformable convolution and f r 4 The adaptive positional information of the salient target is obtained as input and unfolded into a two-dimensional vector P. drAfter being added to the global information, the result is normalized and fed into a feedforward neural network. Residual connections are then used to prevent gradient vanishing. Finally, normalization is performed to obtain the enhanced global context information, denoted as G. t 4 ;
[0032] G t 4 The calculation process is as follows:
[0033] P r =flatten(f r 4 ), P t =flatten(f t 4 ), P er =P e +P r ;
[0034] Where flatten is a built-in function in the PyTorch library, '+' is an element-wise addition operation, H, W, and C correspond to the resolution size and the number of channels, respectively;
[0035] P dt =DConv(f t 4 );
[0036]
[0037] Where MSA represents multi-head attention, and DConv(*) represents... Deformable convolution, LN is a normalization layer, and FFN is a feedforward neural network.
[0038] Furthermore, the detailed generation process of the salient target prediction map P in step S4 is as follows:
[0039] With G r 4 f r 3 and f t 3 As the initial input to the multimodal interaction module of this invention;
[0040] First, f r 3 and f t 3 The feature erasure module in input S2 generates specific information, denoted as E. r 3 and with f r 3 Adding them together generates rich visible light modal features, which are then combined with fr 3 Element-wise multiplication yields shared feature information, denoted as f. re 3 The calculation process is as follows:
[0041] f re 3 =(f r 3 +E r 3 )·f r 3 ;
[0042] Where '+' represents element-wise addition, and '·' represents element-wise multiplication;
[0043] Because high-level features possess rich global contextual information, they can guide low-level features to locate salient targets. This high-level information G... r 4 The features are divided according to channel dimension and downsampled by a factor of two to ensure that the channel dimension and resolution of high-level features are consistent with those of low-level features. The resulting features are denoted as G. r1 4 G r2 4 To fully utilize the high-level features of different channel dimensions, G r1 4 with f re 3 Guide G through attention mechanisms. r2 4 After concatenating and fusing the channels, the resulting feature, denoted as G, is obtained through convolution. hr 4 Furthermore, it maintains consistency with the channel dimension of high-level features, G hr 4 The calculation process is as follows:
[0044]
[0045]
[0046] Where split(*) represents the channel-dimension splitting operation, up(*) represents the ×2 upsampling operation, Conv(*) represents a convolution consisting of a 3-kernel convolution, an LN normalized layer, and a ReLU activation function, and softmax represents the activation function from the PyTorch library. [*,*] represents matrix multiplication, and [*,*] represents concatenation along the channel dimension.
[0047] Furthermore, in order to fully utilize global context information to G r4 After feeding into the convolution, perform twice the downsampling operation to ensure that the number of channels and resolution size are consistent with G. hr 4 Maintain consistency, and then with G hr 4 After adding and fusing the features, the result is fed into a convolutional layer to obtain a new fused decoded feature, denoted as M. 3 The generated features capture both the global contextual information of high-level features across different channel dimensions and the detailed information of low-level features; the calculation formula is as follows:
[0048]
[0049] Where up(*) represents a ×2 upsampling operation, Conv(*) represents a convolution consisting of a kernel of size 3, an LN normalized layer, and a ReLU activation function, and '+' represents an element-wise addition operation;
[0050] By continuously alternating and fully utilizing the features of each layer, new decoding features are generated. Finally, the decoded features of the two modalities are added together to generate an accurate salient prediction map, denoted as P.
[0051] Further, step S4 yields a salient target prediction map P, and the ground truth map is denoted as G. The total loss function L during the training phase is then:
[0052] L = L bce +L smooth +L dice ;
[0053] Among them, L bce L is a binary cross-entropy loss function used to calculate the pixel difference between P and G; smooth As a smoothing loss function, constraining the model to produce a smooth prediction map helps reduce noise and discontinuous information, thereby improving model robustness and helping the model better adapt to the shape and boundary features of the target, thus improving the clarity of the prediction map boundaries and ensuring the consistency of salient regions; L dice The dice coefficient function measures the overlap between the predicted and ground truth maps to improve model robustness. The closer the dice coefficient is to 1, the greater the overlap between the predicted and ground truth maps. The calculation formulas for the above three functions are as follows:
[0054]
[0055] Where N is the total number of pixels, G i Let P be the ground truth label for the i-th pixel, where 0 represents a non-salient target region and 1 represents a salient target region. i This represents the predicted probability of the i-th pixel.
[0056]
[0057] Where P (i,j) To predict the pixel prediction values of an image, and P (i+1,j) and P (i,j+1) These correspond to the predicted values of adjacent pixels in the prediction map.
[0058]
[0059] Where |P| represents the number of pixels in the salient target region in the prediction map, |G| represents the number of pixels in the salient target detection region in the true value label map, and |P∩G| represents the number of pixels in the intersection region between P and G.
[0060] Beneficial effects: Compared with the prior art, the present invention has the following advantages:
[0061] (1) Based on a bidirectional alternating fusion strategy, the present invention proposes a new fusion strategy and achieves the best performance compared with the existing technical solutions.
[0062] (2) The feature erasure module of the present invention can mine the specific information of the mode itself.
[0063] (3) The global context enhancement module of the present invention fully integrates the global context information of high-level features in different channel dimensions with the detailed information of low-level features, thereby enhancing the representation capability of high-level features.
[0064] (4) The bidirectional alternating decoder of the present invention uses an embedded multimodal interaction module to alternately fuse high-level features of one modality with low-level features of another modality, thereby fully capturing the interaction between the two modalities. Attached Figure Description
[0065] Figure 1 This is a schematic diagram of the overall process of the present invention;
[0066] Figure 2 This is a schematic diagram of the feature erasure module of the present invention;
[0067] Figure 3 This is a schematic diagram of the global context enhancement module of the present invention;
[0068] Figure 4 This is a schematic diagram of the multimodal interaction module of the present invention;
[0069] Figure 5 This is a visual comparison diagram between the present invention and existing methods;
[0070] Figure 6 This is a schematic diagram comparing the PR curves of the present invention and existing methods. Detailed Implementation
[0071] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.
[0072] like Figure 1 As shown, the present invention provides a visible light-thermal infrared salient target detection method based on a bidirectional alternating fusion strategy, comprising the following steps:
[0073] Step S1: Obtain the dataset, which includes visible light images and thermal infrared images. Divide the dataset into training set and test set according to the proportions.
[0074] For any pair of visible light images and thermal infrared images, a feature extractor with shared pre-trained weights is used to extract the corresponding multi-layer features of these two modalities: the corresponding visible light multi-layer features. and thermal infrared multilayer features Chinese f r 1 f r 2 and f r 3 For low-level features, f r 4 Characterized by high-level features Chinese f t 1 f t 2 and f t 3 For low-level features, f t 4 Characterized by high-level features;
[0075] Step S2: First, for the visible light multilayer features obtained in step S1... and thermal infrared multilayer features The common modal information of the two is obtained, and then the feature erasure module is used to erase the visible light multilayer features respectively. and thermal infrared multilayer features The central modal information is used to obtain specific information about the visible light multilayer features. Specific information of thermal infrared multilayer features
[0076] The detailed method for calculating specificity information in step S2 is as follows:
[0077] First, calculate and The correlation is used to obtain the correlation matrix R. (d,k) This yields the activation region shared by both modes; R (d,k) The calculation process is as follows:
[0078] R (d,k) = (f r (d,k) ) T (w T f t (d,k) )
[0079] where 1 < d < H, 1 < k < w, (d, k) represents the pixel coordinate position, w represents the mapped feature space, and (H, W) represents the picture resolution size;
[0080] Secondly, perform binarization on the correlation matrix R (d,k) to generate a binarization mask B to identify the area to be erased;
[0081] Then, filter the area to be erased through the soft max activation function to generate a new matrix G. The calculation formula of G is: G = R · B; · is element-wise multiplication;
[0082] Finally, generate the erased feature Take as the specific information of the visible light multi-layer feature, and generate the erased feature Take as the specific information of the thermal infrared multi-layer feature;
[0083] Step S3. Input the specific information E r 4 and E t 4 as well as the high-level feature f r 4 and f t 4 into the global context enhancement module together to obtain the enhanced global context information G r 4 and G t 4 ;
[0084] Expand the high-level feature f r 4 and the high-level feature f t 4 and the specific information E r 4 into two-dimensional vectors and denote them as P r R∈(HW,C) , P t R∈(HW,C) and P e R∈(HW,C) , and take P eR∈(HW,C) With P t R∈(HW,C) Adding them together generates a new two-dimensional vector P. et And use it as a query, P r R∈(HW,C) The information is fed together as key and value into the multi-head attention mechanism to obtain global information shared by both modalities. Then introduce deformable convolution and combine the high-level features f r 4 The adaptive positional information of the salient target is obtained as input and unfolded into a two-dimensional vector P. dr This enhanced global context information is then added to the global information and normalized to obtain the final enhanced global context information, denoted as G. r 4 ;
[0085] High-level feature f r 4 High-level characteristics f t 4 and specific information E t 4 Expanded into two-dimensional vectors, denoted as P r R∈(HW,C) P t R∈(HW,C) and P e R∈(HW,C) and P e R∈(HW,C) With P t R∈(HW,C) Adding them together generates a new two-dimensional vector P. er And use it as a query, P t R∈(HW,C) The keys and values are fed together into a multi-head attention mechanism and then processed by deformable convolution to obtain f. t 4 The location information of salient targets in the middle is used to obtain the enhanced global context information, denoted as G. t 4 ;
[0086] Step S4: Transfer the specific information E generated in S2 r 4 and E t 4 The two types of global context information G obtained in step S3 r 4 and G t 4 and low-level features and The multimodal interaction module in the bidirectional alternating decoder is input together, and new decoding and fusion features are continuously generated alternately. and Finally, an accurate salient target prediction map P is generated.
[0087] First, f r 3 and f t 3 The specific information generated by the input feature erasure module is denoted as E. r 3 and with f r 3 Adding them together generates rich visible light modal features, which are then combined with f r 3 Element-wise multiplication yields shared feature information, denoted as f. re 3 The calculation process is as follows:
[0088]
[0089] Where '+' represents element-wise addition, and '·' represents element-wise multiplication;
[0090] Then, the high-level information G r 4 The features are divided according to channel dimension and downsampled by a factor of two to ensure that the channel dimension and resolution of high-level features are consistent with those of low-level features. The resulting features are denoted as G. r1 4 G r2 4 ; G r1 4 with f re 3 Guide G through attention mechanisms. r2 4 Channel-level concatenation and fusion are performed, and the fused features are then convolved to obtain feature G. hr 4 Furthermore, it maintains consistency with the channel dimension of high-level features, G hr 4 The calculation process is as follows:
[0091]
[0092]
[0093] Where split(*) represents the channel-dimension splitting operation, up(*) represents the ×2 upsampling operation, Conv(*) represents a convolution consisting of a 3-kernel convolution, an LN normalized layer, and a ReLU activation function, and softmax represents the activation function from the PyTorch library. [*,*] represents matrix multiplication, and [*,*] represents concatenation along the channel dimension.
[0094] Next, G r 4 After feeding into the convolution, perform twice the downsampling operation, and then combine it with G. hr 4 After adding and fusing the features, the result is fed into a convolutional layer to obtain a new fused decoded feature, denoted as M. 3 ;
[0095] The calculation process is as follows:
[0096]
[0097] Where up(*) represents a ×2 upsampling operation, Conv(*) represents a convolution consisting of a kernel of size 3, an LN normalized layer, and a ReLU activation function, and '+' represents an element-wise addition operation;
[0098] Finally, by continuously alternating and fully utilizing the features of each layer, new decoding features are generated.
[0099] Decoding features The generation process is as follows:
[0100]
[0101]
[0102]
[0103] Example 1:
[0104] This embodiment uses 2500 pairs of images from the VT5000-Train dataset of the existing visible light-thermal infrared salient target detection dataset VT5000 as the training set, and three existing public datasets, VT5000-Test, VT1000, and VT821, as the test set to verify the effectiveness of the network.
[0105] like Figure 1 The image shows the multimodal features obtained after the feature extractor. and High-level feature f r 4 and f t 4 The enhanced global feature G is obtained by feeding it into the global context enhancement module. r 4 and G t 4 The remaining low-level features and Along with the global features, they are fed into a bidirectional alternating decoder. Through top-down alternating decoding, new fused features are generated, making full use of and interacting with the information of the two modalities to generate a salient prediction map P.
[0106] The network in this embodiment is based on the PyTorch framework and trained for 100 epochs on a GeForce RTX 3090 GPU with a batch size of 4. The learning rate is set to 1e-5 and then to 1e-4 after 40 epochs. The input image pairs have a resolution of 384*384.
[0107] This embodiment employs four quantitative metrics widely used in visible-thermal infrared salient target detection: weighted F-metric (wF), mean absolute error (MAE), S-metric (Sm), and E-metric (Em) to quantitatively evaluate the network's performance. The technical solution of this invention is compared with 12 other existing technical solutions based on these four quantitative metrics, including ADF, MIDD, CSRNet, CGFNet, SwinNet, OSRNet, TNet, DCNet, LSNet, HRTransNet, MCFNet, and CAVER.
[0108] Quantitative comparison:
[0109] The specific comparative experimental results of this embodiment with 12 existing technical solutions are shown in Table 1. The red text indicates the best performance, while the blue text indicates the second best performance. As can be seen from Table 1, compared with the second best method, the performance of this invention on four indicators based on three public datasets is improved by an average of 0.8%, 0.6%, 0.33%, and 26.5%, respectively.
[0110]
[0111] Table 1 Performance Comparison Experiment Results
[0112] Visual comparison:
[0113] The visual comparison results of this embodiment with the saliency prediction maps generated by 12 existing technical solutions are as follows: Figure 5As shown in the figure. This embodiment selected eight challenging samples, including those with low contrast between the target and background, small salient targets, and complex salient targets. From rows 1, 4, and 7 of Figure 5, it can be seen that the salient target is similar to the background. Existing methods segment the background as well or only segment a portion of the salient target, while our method can completely and accurately segment the salient target, proving its effectiveness. For rows 2, 5, and 8 of the figure, it can be seen that complex salient targets have higher requirements for detail. Existing methods, even if they can accurately locate the salient target, struggle to extract its boundary contours. Our method, through alternating decoding, continuously guides the low-level detail information of another modality through high-level fusion features of one modality, achieving accurate target location while supplementing detailed contour information. Furthermore, as... Figure 6 The visualization of the PR curve results shows that the present invention has a higher confidence level compared to other existing technical solutions. Therefore, the present invention can achieve more accurate salient target detection.
Claims
1. A visible light-thermal infrared salient target detection method based on a bidirectional alternating fusion strategy, characterized in that, Includes the following steps: Step S1: Obtain the dataset, which includes visible light images and thermal infrared images. Divide the dataset into training set and test set according to the proportions. For any pair of visible light images and thermal infrared images, a feature extractor with shared pre-trained weights is used to extract the corresponding multi-layer features of these two modalities: the corresponding visible light multi-layer features. and thermal infrared multilayer features ; middle , and Low-level features Characterized by high-level features middle , and Low-level features Characterized by high-level features; Step S2: First, for the visible light multilayer features obtained in step S1... and thermal infrared multilayer features The common modal information of the two is obtained, and then the feature erasure module is used to erase the visible light multilayer features respectively. and thermal infrared multilayer features The central modal information is used to obtain specific information about the visible light multilayer features. Specific information of thermal infrared multilayer features ; Step S3: Utilize the specific information generated in S2. and and high-level characteristics and Inputting them together into the global context enhancement module yields the enhanced visible light global context information. and thermal infrared global context information ; Step S4: Utilize the specific information generated in S2 and The two types of global context information obtained in step S3 and and low-level features and The multimodal interaction module in the bidirectional alternating decoder is input together, and new visible light decoding and fusion features are continuously generated alternately. Fusion features with thermal infrared decoding This ultimately generates an accurate salient target prediction map P; The detailed method for calculating specificity information in step S2 is as follows: First, calculate and The correlation is used to obtain the correlation matrix. This means obtaining the activation region shared by both modes; The calculation process is as follows: ； in , , Indicates pixel coordinate position, The feature space representing the mapping, Indicates the image resolution; Secondly, regarding the correlation matrix Perform binarization to generate a binarized mask. To identify the area to be erased; Next, through The activation function selects the regions to be erased and generates a new matrix. , The calculation formula is: ; Element-wise multiplication; Finally, based on the binarization mask sum matrix generate Erasure features ,Will As specific information of visible light multilayer features, based on binarization masks sum matrix generate Erasure features ,Will As specific information of thermal infrared multilayer features.
2. The visible light-thermal infrared salient target detection method based on a bidirectional alternating fusion strategy according to claim 1, characterized in that, Step S3 obtains global context information. The detailed method is as follows: High-level characteristics High-level characteristics and specific information Expanded into two-dimensional vectors, they are denoted as follows: , and and will and Addition generates a new two-dimensional vector And use it as a query, The information is fed together as key and value into the multi-head attention mechanism to obtain global information shared by both modalities. Then, deformable convolution is introduced and high-level features are... The adaptive positional information of the salient target is obtained as input and unfolded into a two-dimensional vector. This enhanced global context information is then added to the global information and normalized to obtain the final enhanced global context information, denoted as . ; The calculation formula is expressed as follows: ， ， ， ； in These are functions included in the PyTorch library. For element-wise addition operations, H, W, and C correspond to the resolution size and the number of channels, respectively. ； ； ； in For the attention of the bulls, for Deformable convolution, For normalization layer, It is a feedforward neural network.
3. The visible light-thermal infrared salient target detection method based on a bidirectional alternating fusion strategy according to claim 1, characterized in that, Step S3 calculates global context information. The detailed method is as follows: High-level characteristics High-level characteristics and specific information Expanded into two-dimensional vectors, they are denoted as follows: , and and will and Addition generates a new two-dimensional vector And use it as a query, The keys and values are fed together into a multi-head attention mechanism and then processed by deformable convolution to obtain... The location information of salient targets in the middle is used to obtain the enhanced global context information, denoted as . The calculation formula is expressed as follows: ， ， ， ； ； ； 。 4. The visible light-thermal infrared salient target detection method based on a bidirectional alternating fusion strategy according to claim 1, characterized in that, Decoding features in step S4 The generation process is as follows: First, and The specific information generated by the input feature erasure module is denoted as: and Adding them together generates rich visible light modal features, which are then combined with... Element-wise multiplication yields shared feature information, denoted as The calculation process is as follows: ； in This is an element-wise addition operation. Represented as element-wise multiplication; Then, high-level information The features are divided according to channel dimension and downsampled by a factor of two to ensure that the channel dimension and resolution of high-level features are consistent with those of low-level features. The resulting features are denoted as follows: , ;Will and Guide through attention mechanisms and then... Channel-level concatenation and fusion are performed, and the fused features are then convolved to obtain the final features. And it is consistent with the channel dimension of the high-level features. The calculation process is as follows: ； ； in This indicates the operation of dividing the channel dimension. Represented as Upsampling operation, This can be represented as a convolution with a kernel size of 3. Normalized layers and Activation function composition, This is represented as an activation function in the PyTorch library. This represents matrix multiplication. This indicates a splicing operation at the channel level; Next, After feeding into the convolution, perform a double downsampling operation, and then... After adding and fusing the features, the result is fed into a convolutional layer to obtain a new fused decoded feature, denoted as . ; The calculation process is as follows: ； in Represented as Upsampling operation, This can be represented as a convolution with a kernel size of 3. It consists of a normalization layer and a ReLU activation function. This is an element-wise addition operation; Finally, by continuously alternating and fully utilizing the features of each layer, new decoding features are generated. .
5. The visible light-thermal infrared salient target detection method based on a bidirectional alternating fusion strategy according to claim 1, characterized in that, Decoding features in step S4 The generation process is as follows: ； ； 。 6. The visible light-thermal infrared salient target detection method based on a bidirectional alternating fusion strategy according to claim 1, characterized in that, After obtaining the salient target prediction map P in step S4, the ground truth map is denoted as G, and the total loss function during the training phase is... : ； The The binary cross-entropy loss function is used to calculate the pixel difference between P and G. ； in Total number of pixels Represented as the first The ground truth labels for each pixel, where 0 indicates a non-salient target region and 1 indicates a salient target region. Indicates the first The predicted probability of each pixel; The To smooth the loss function, thereby improving the clarity of the prediction map boundaries and ensuring the consistency of salient regions; ; To predict the pixel prediction values of an image, and and These correspond to the predicted values of adjacent pixels in the prediction image. The The dice coefficient function is used to measure the overlap area between the prediction plot and the ground truth plot. The calculation formula is as follows: ； This indicates the number of pixels in the salient target region of the prediction image. This indicates the number of pixels in the salient target detection region of the truth label image. This represents the number of pixels in the intersection region between P and G.