Asymmetric illumination-aware progressive decoding RGB-T salient object detection method

Abstract

The present invention discloses an asymmetric illumination-aware progressive decoding RGB-T salient object detection method. The method comprises the following steps: obtaining an RGB-T salient object detection dataset and inputting it into a trained RGB-T salient object detection network; employing two Swin Transformer backbone networks to obtain multi-level features; utilizing an asymmetric illumination-aware interaction module to achieve effective interaction between RGB features and T features in an asymmetric manner, thereby reducing interference information; implementing cross-modal feature fusion in both channel and spatial dimensions through a channel-space feature fusion module; employing a staged progressive decoding strategy to decode the fused features and output a final saliency map. The method explores cross-modal feature interaction and fusion methods as well as a multi-scale information aggregation strategy, thereby improving the accuracy of RGB-T salient object detection.

Description

Technical Field

[0001] The present invention relates to an asymmetric illumination perception progressive decoding RGB-T salient target detection method, belonging to the technical field of computer vision. Background Art

[0002] Salient object detection is a key technology in computer vision. It focuses on accurately identifying the most striking areas or objects in an image or video and clearly defining their boundaries. As a preprocessing step for various computer vision tasks, salient object detection plays a vital role in areas such as image segmentation, object tracking, and object recognition. Furthermore, salient object detection technology has a wide range of applications, demonstrating its importance and practicality in fields such as industrial automation, intelligent healthcare, and quality inspection.

[0003] Because thermal infrared cameras can generate high-quality thermal images by capturing the radiant heat of objects even in extreme environments such as low light or inclement weather, a growing number of researchers are using RGB images and thermal maps as inputs for neural networks to address challenging scenarios for salient object detection, such as low light or dark scenes. However, RGB-T-based salient object detection still faces several challenges, such as effectively integrating and fusing cross-modal features and aggregating multi-scale information. Summary of the Invention

[0004] The purpose of the present invention is to solve the above problems and propose an asymmetric illumination perception progressive decoding RGB-T salient object detection method.

[0005] To achieve the above object, the technical solution of the present invention is:

[0006] A method for detecting salient objects using RGB-T with asymmetric illumination perception and progressive decoding, comprising the following steps:

[0007] S1: Obtain the RGB-T salient object detection dataset, preprocess it, and then input it into the trained RGB-T salient object detection network;

[0008] S2: Two Swin Transformer backbone network branches are used to extract features from RGB images and thermal infrared images respectively to obtain RGB features and T characteristics Among them, r represents RGB image, t represents thermal infrared image, and i represents the level of feature;

[0009] S3: Using the asymmetric illumination perception interaction module, the effective interaction between RGB features and T features is achieved in an asymmetric manner to reduce interference information;

[0010] S4: Through the channel-space feature fusion module, cross-modal feature fusion is achieved in both channel and spatial dimensions;

[0011] S5: A phased progressive decoding strategy is used to decode the fused features and output the final saliency map.

[0012] A further technical solution is that the asymmetric illumination perception interaction module obtains the common information of the two features through element-wise multiplication operation. Then the RGB feature difference information is obtained by element subtraction operation and T feature differential information The specific calculation formula is as follows:

[0013]

[0014] Furthermore, the feature interaction process is divided into two parts: T→R and R→T; for the T→R part, the differential information of the T modality is and The concatenation is passed through the BConv layer, followed by the residual connection and BConv layer to ensure that the original RGB information is preserved during the feature interaction process; next, the residual connection operation is introduced again to further preserve and strengthen the semantic information of the RGB features; BConv(·) represents the Conv layer, BN layer and Relu activation function, and Cat[·] represents the concatenation operation. The specific calculation formula is as follows:

[0015]

[0016] Furthermore, for the R→T part, the differential information of the RGB mode is converted to and Splice them together, then pass through the BConv layer, residual connection and BConv layer in sequence, and combine the results with The generated spatial attention maps are multiplied to highlight the areas in the T modality that should be paid special attention to; SA(·) represents spatial attention, and its specific calculation formula is as follows:

[0017]

[0018] A further technical solution is that the channel-space feature fusion module fuses cross-modal features in both channel and spatial dimensions; for the channel dimension, global average pooling and global maximum pooling are used to obtain the global information of the features on each channel, and then the results of the global average pooling and global maximum pooling operations are spliced separately and input into the fully connected layer to learn the importance weights W of different channels. A and W M Next, the calculated weight vector is element-wise added and passed through a Sigmoid layer to generate the key statistics for fusing the two modalities. and Finally, adaptive channel feature selection is achieved through element-by-element multiplication operations; GAP(·) and GMP(·) represent global average pooling and global maximum pooling, and FC(·) represents the fully connected layer. The specific calculation formula is as follows:

[0019]

[0020] Furthermore, for the spatial dimension, a set of dilated convolutions with dilation rates of 1, 2, and 4 are used to obtain and The multi-scale context information is then stitched together, and the features on the channel dimension are aggregated through the BConv layer to obtain the spatial attention map of the two modalities. and Next, the two modal spatial attention maps are spliced together and passed through the BConv layer and the Sigmoid layer to obtain the RGB image relative weight map. Thermal image relative weight map Subtracting the matrix E Finally, the cross-modal features are fused using element-wise multiplication and element-wise addition operations; DConv r (·) represents a dilated convolution block with a dilation rate of r, and E represents an all-1 matrix. The specific calculation formula is as follows:

[0021]

[0022] A further technical solution is that the phased progressive decoding strategy divides the multi-layer cross-modal features into and The feature decoding process is divided into two stages. In the first stage, the feature set T1 is input to the first decoder block, and then the initial saliency map S1 is generated through the Conv layer. Then, S1 is used as a guide to perform element-wise multiplication with the features in T2 for further improvement and refinement to suppress irrelevant background interference. In the second stage, the refined feature set T2 is input to the second decoder block, and then the final saliency map S2 is generated through the Conv layer. DB i Represents the i-th decoder block, and its specific calculation formula is as follows:

[0023] S1=Conv(DB1(T1)),

[0024]

[0025] Furthermore, the decoder block uses pyramid multiplication and concatenation operations to aggregate adjacent features layer by layer; specifically, each layer of features Multiply with all higher-level features to obtain updated features Subsequently, the updated features are continuously passed through upsampling and BConv layers, and the output saliency map S is generated through a progressive splicing strategy. j ; where i∈{1,2,3}, k max =3 or i∈{3,4,5}, k max =5; k=3, j=1 or k=1, j=2; UP(·) represents the upsampling operation, and its specific calculation formula is as follows:

[0026]

[0027] A further technical solution is that the training steps of the trained RGB-T salient object detection network include:

[0028] Build an RGB-T salient object detection network;

[0029] Constructing a training set, wherein the training set is an RGB image and its corresponding T image and saliency map;

[0030] Input the training set into the RGB-T salient object detection network for training;

[0031] The RGB-T salient object detection network outputs the detection results;

[0032] Calculate the loss by combining the detection result and the saliency map corresponding to the input RGB image;

[0033] When the loss value reaches the minimum, the model converges and the training stops, and the trained RGB-T salient object detection network is obtained.

[0034] The beneficial effects of the above technical solution are as follows: the present invention provides an asymmetric illumination perception interaction module to achieve effective interaction between RGB and T features; designs a channel-space feature fusion module to achieve cross-modal complementary information fusion; and develops a phased progressive decoding strategy to effectively integrate multi-scale information. These three modules, integrated into a network, significantly improve the accuracy of RGB-T salient object detection, demonstrating the advantages of the proposed technical solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0036] Figure 1 This is a diagram of the overall network architecture of an embodiment of the present invention;

[0037] Figure 2 This is a structural diagram of the asymmetric illumination perception interaction module in an embodiment of the present invention;

[0038] Figure 3 This is a structural diagram of a channel space feature fusion module in an embodiment of the present invention;

[0039] Figure 4 This is a result diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] The present invention provides an asymmetric illumination perception progressive decoding RGB-T salient target detection method, such as Figure 1 As shown, the following steps are included:

[0042] S1: Two Swin Transformer backbone network branches are used to extract features from RGB images and thermal infrared images respectively to obtain RGB features and T characteristics Among them, r represents RGB image, t represents thermal infrared image, and i represents the level of feature;

[0043] S2: Construct an asymmetric illumination perception interaction module, such as Figure 2 As shown;

[0044] S2-1: Asymmetric illumination perception interaction module obtains the common information of two features through element-wise multiplication Then the RGB feature difference information is obtained by element subtraction operation and T feature differential information The specific calculation formula is as follows:

[0045]

[0046] S2-2: The asymmetric illumination perception interaction module divides the feature interaction process into two parts: T→R and R→T; for the T→R part, the differential information of the T modality is converted to and The concatenation is passed through the BConv layer, followed by the residual connection and BConv layer to ensure that the original RGB information is preserved during the feature interaction process; next, the residual connection operation is introduced again to further preserve and strengthen the semantic information of the RGB features; BConv(·) represents the Conv layer, BN layer and Relu activation function, and Cat[·] represents the concatenation operation. The specific calculation formula is as follows:

[0047]

[0048] S2-3: For the R→T part, the differential information of the RGB mode is converted and Splice them together, then pass through the BConv layer, residual connection and BConv layer in sequence, and combine the results with The generated spatial attention maps are multiplied to highlight the areas in the T modality that should be paid special attention to; SA(·) represents spatial attention, and its specific calculation formula is as follows:

[0049]

[0050] S3: Construct channel space feature fusion module, such as Figure 3 As shown;

[0051] S3-1: In the channel dimension, global average pooling and global maximum pooling are used to obtain the global information of the feature in each channel. The results of the global average pooling and global maximum pooling operations are then concatenated and input into the fully connected layer to learn the importance weights W of different channels. A and W M Next, the calculated weight vector is element-wise added and passed through a Sigmoid layer to generate the key statistics for fusing the two modalities. and Finally, adaptive channel feature selection is achieved through element-by-element multiplication operations; GAP(·) and GMP(·) represent global average pooling and global maximum pooling, and FC(·) represents the fully connected layer. The specific calculation formula is as follows:

[0052]

[0053]

[0054]

[0055] S3-2: In the spatial dimension, a set of dilated convolutions with dilation rates of 1, 2, and 4 are used to obtain and The multi-scale context information is then stitched together, and the features on the channel dimension are aggregated through the BConv layer to obtain the spatial attention map of the two modalities. and Next, the two modal spatial attention maps are spliced together and passed through the BConv layer and the Sigmoid layer to obtain the RGB image relative weight map. Thermal image relative weight map Subtracting the matrix E Finally, the cross-modal features are fused using element-wise multiplication and element-wise addition operations; DConv r(·) represents a dilated convolution block with a dilation rate of r, and E represents an all-1 matrix. The specific calculation formula is as follows:

[0056]

[0057]

[0058]

[0059] S4: Develop a staged progressive decoding strategy;

[0060] S4-1: Divide multi-layer cross-modal features into and The feature decoding process is divided into two stages. In the first stage, the feature set T1 is input to the first decoder block, and then the initial saliency map S1 is generated through the Conv layer. Then, S1 is used as a guide to perform element-wise multiplication with the features in T2 for further improvement and refinement to suppress irrelevant background interference. In the second stage, the refined feature set T2 is input to the second decoder block, and then the final saliency map S2 is generated through the Conv layer. DB i Represents the i-th decoder block, and its specific calculation formula is as follows:

[0061] S1=Conv(DB1(T1)),

[0062]

[0063] S4-2: The decoder block uses pyramid multiplication and concatenation operations to aggregate adjacent features layer by layer; specifically, each layer of features Multiply with all higher-level features to obtain updated features Subsequently, the updated features are continuously passed through upsampling and BConv layers, and the output saliency map S is generated through a progressive splicing strategy. j ; where i∈{1,2,3}, k max =3 or i∈{3,4,5}, k max =5; k=3, j=1 or k=1, j=2; UP(·) represents the upsampling operation, and its specific calculation formula is as follows:

[0064]

[0065]

[0066] S5: Build an RGB-T salient object detection network and train it;

[0067] S5-1: Construct a training set, which includes an RGB image and its corresponding T image and saliency map.

[0068] S5-2: Input the training set into the RGB-T salient object detection network to train the network. The input image resolution is adjusted to 382×382, and random horizontal flipping is used for data augmentation. The SGD algorithm is used to train the network with an initial learning rate of 0.001.

[0069] S5-3: RGB-T salient object detection network outputs detection results.

[0070] S5-4: Calculate the loss of the detection results and the saliency map corresponding to the input RGB image. Cross entropy loss and smoothness loss are used as loss functions. S1 and S2 are the two output saliency maps, G is the saliency label, and L is the saliency label. bce and L s Represent the cross entropy loss and smoothness loss respectively, then the final loss function is expressed as follows:

[0071]

[0072] S5-5: When the loss value reaches the minimum, the model converges, the training stops, the parameters are saved, and the trained RGB-T salient object detection network is obtained.

[0073] S6: Input the RGB image to be detected and the corresponding T image into the trained RGB-T salient object detection model, thereby outputting the final salient prediction map of the RGB image to be detected.

[0074] In order to verify the effectiveness of the above examples, the performance of the proposed method is compared with other advanced methods on three data sets: VT821, VT1000 and VT5000, and four commonly used indicators are selected: E m (E-measure), max F β (max F-measure), S m (S-measure) and MAE (Mean Absolute Error). Among these four indicators, except MAE, E m 、maxF β and S m The larger the value, the better the performance. The experimental results are shown in Table 1.

[0075] Table 1 Comparison of detection accuracy on three datasets

[0076]

[0077] Figure 4The following is a comparison of the results of the proposed method. The first column is the RGB image, the second column is the T image, the third column is the ground truth image, and the fourth column is the result of the proposed method. This comparison shows that the proposed solution can accurately locate and segment salient objects.

[0078] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It is apparent to those skilled in the art that various changes, modifications, substitutions, and variations to these embodiments may be made without departing from the principles and spirit of the present invention, and these changes and modifications still fall within the scope of protection of the present invention.

Claims

1. A method for detecting salient objects using RGB-T with asymmetric illumination-aware progressive decoding, characterized in that The following steps are involved: S1: Obtain the RGB-T salient object detection dataset, preprocess it, and then input it into the trained RGB-T salient object detection network; S2: Two Swin Transformer backbone network branches are used to extract features from RGB images and thermal infrared images respectively to obtain RGB features and T characteristics , where r represents the RGB image, t represents the thermal infrared image, and i represents the level of the feature; S3: Using the asymmetric illumination perception interaction module, the effective interaction between RGB features and T features is achieved in an asymmetric manner to reduce interference information; The asymmetric illumination perception interaction module obtains the common information of the two features through element-wise multiplication. , and then obtain the RGB feature difference information through element subtraction operation and T feature differential information ; Next, the feature interaction process is divided into two parts: and ;for Part, the differential information of T mode and The features are concatenated and passed through the BConv layer, and then through the residual connection and BConv layer to ensure that the original RGB information is preserved during the feature interaction process; next, the residual connection operation is introduced again to further preserve and enhance the semantic information of the RGB features; Part, the differential information of RGB mode and Splice them together, then pass through the BConv layer, residual connection and BConv layer in sequence, and combine the results with The generated spatial attention maps are multiplied to highlight the areas that should be paid special attention to in the T modality; Represents Conv layer, BN layer and Relu activation function, Represents the splicing operation, Represents spatial attention, and its specific calculation formula is as follows: ， ， ， ， ； S4: Through the channel-space feature fusion module, cross-modal feature fusion is achieved in both channel and spatial dimensions; The channel-space feature fusion module fuses cross-modal features in both channel and spatial dimensions. For the channel dimension, global average pooling and global maximum pooling are used to obtain the global information of the features on each channel. The results of the global average pooling and global maximum pooling operations are then concatenated and input into the fully connected layer to learn the importance weights of different channels. and Next, the calculated weight vector is element-wise added and passed through a Sigmoid layer to generate the key statistics for fusing the two modalities. and ;Finally, adaptive channel feature selection is achieved through element-by-element multiplication operation; and represents global average pooling and global maximum pooling, Represents the fully connected layer, and its specific calculation formula is as follows: ， ， ， ， ； For the spatial dimension, a set of dilated convolutions with dilation rates of 1, 2, and 4 are used to obtain and The multi-scale context information is then stitched together, and the features on the channel dimension are aggregated through the BConv layer to obtain the spatial attention map of the two modalities. and ; Next, the two modal spatial attention maps are spliced together and passed through the BConv layer and the Sigmoid layer to obtain the RGB image relative weight map. , thermal image relative weight map Subtracting the matrix E Finally, the cross-modal features are fused using element-wise multiplication and element-wise addition operations; represents the dilated convolution block with a dilation rate of r, E represents the all-1 matrix, and its specific calculation formula is as follows: ， ， ， ， ； S5: Using a phased progressive decoding strategy, the fused features are decoded and output as the final saliency map; The staged progressive decoding strategy divides the multi-layer cross-modal features into and The feature decoding process is divided into two stages; in the first stage, the feature set It is input into the first decoder block and then passes through the Conv layer to generate the initial saliency map ; Then, As a guide, The features in the are further improved and refined by performing element-wise multiplication operations to suppress irrelevant background interference; in the second stage, the refined feature set is input to the second decoder block and then passes through the Conv layer to generate the final saliency map ; Represents the i-th decoder block, and its specific calculation formula is as follows: ， ， 。 2. The asymmetric illumination-aware progressive decoding RGB-T salient object detection method according to claim 1, characterized in that: The decoder block uses pyramid multiplication and concatenation operations to aggregate adjacent features layer by layer; specifically, each layer of features Multiply with all higher-level features to obtain updated features ; Then, the updated features are continuously upsampled and BConv layers, and the output saliency map is generated through a progressive splicing strategy ;in, or ; or ; Represents the upsampling operation, and its specific calculation formula is as follows: ， 。 3. The asymmetric illumination-aware progressive decoding RGB-T salient object detection method according to claim 1, characterized in that: The training steps of the trained RGB-T salient object detection network include: Build an RGB-T salient object detection network; Constructing a training set, wherein the training set is an RGB image and its corresponding T image and saliency map; Input the training set into the RGB-T salient object detection network for training; The RGB-T salient object detection network outputs the detection results; Calculate the loss by combining the detection result and the saliency map corresponding to the input RGB image; When the loss value reaches the minimum, the model converges and the training stops, and the trained RGB-T salient object detection network is obtained.

Images