An RGB-thermal semantic segmentation method based on semantic perception correction

Abstract

The application discloses an RGB-Thermal semantic segmentation method based on semantic perception correction, and belongs to the technical field of computer vision and image processing, which comprises the following steps: synchronously acquiring an RGB image and a thermal image of the same scene, normalizing and data-enhancing the acquired images, and constructing a data set; constructing an RGB-Thermal semantic segmentation network model based on semantic perception correction, which comprises a double-branch encoder, a semantic perception correction module, a modal reconstruction module and a decoder; training the RGB-Thermal semantic segmentation model, saving the best network parameters on the validation set during the training process, and obtaining an optimal model; and applying the optimal model obtained through the training to a test set for forward inference, so as to obtain a final semantic segmentation result. The application can realize accurate fusion of multi-modal features, improve the precision and robustness of the segmentation model in a complex environment, and overcome the shortcomings of the prior art in terms of insufficient thermal modal semantics, cross-modal mismatch and noise interference.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to an RGB-Thermal semantic segmentation method based on semantic awareness correction. Background Technology

[0002] Semantic segmentation, a core task in computer vision, aims to assign a semantic category label to each pixel in an input image. This technology is widely used in fields such as autonomous driving, intelligent transportation, security monitoring, robot navigation, and medical image analysis. In recent years, with the development of deep learning, semantic segmentation methods based on RGB images have made significant progress. However, in complex environments such as nighttime, low light, and rainy / foggy weather, the imaging quality of a single RGB modality deteriorates severely, leading to a significant degradation in segmentation performance. To overcome this deficiency, researchers have proposed an RGB-T multimodal fusion semantic segmentation method. Thermal imaging modalities can stably image under low light conditions, providing supplementary information for segmentation; therefore, RGB-T semantic segmentation has gradually become a research hotspot. Existing technologies mainly fall into two categories:

[0003] (1) Fusion method based on element-wise operation

[0004] Early methods (such as FuseSeg, RTFNet, MFFENet, etc.) fused RGB and thermal modal features by element-wise addition or multiplication. These methods have a simple structure and can improve segmentation performance to some extent, but due to the lack of modeling for the differences between different modal features, the problem of insufficient thermal modal semantics is often not solved. In addition, redundant information and invalid features are introduced during the fusion process, so they are not well adapted to complex scenarios.

[0005] (2) Fusion methods based on attention mechanism

[0006] With the advancement of deep learning, researchers have introduced mechanisms such as channel attention, spatial attention, and cross attention to enhance intermodal dependencies. Representative methods include CCFFNet, MFTNet, MMSMCNet, and C... 4 Net et al. These methods improve fusion results by capturing the correlation between modalities. However, their shortcomings are that they are not specifically designed to address the problem of insufficient semantics in hot modalities, and category confusion may still occur; they lack accurate category alignment at the channel level, which can easily introduce erroneous features; and modal differences and noise are not effectively suppressed.

[0007] In summary, although existing technologies have made some progress in RGB-T semantic segmentation, they still have significant shortcomings in addressing issues such as insufficient semantic representation of thermal modalities and cross-modal channel mismatch. Therefore, a new network architecture is urgently needed that can simultaneously enhance the semantic representation capabilities of thermal modalities, achieve accurate alignment of cross-modal category information, and suppress noise and differences during the fusion process, thereby further improving the accuracy and robustness of RGB-T semantic segmentation. Summary of the Invention

[0008] To address the issues of insufficient thermal modal semantics and cross-modal channel mismatch in existing RGB-Thermal fusion processes, this invention proposes an RGB-Thermal semantic segmentation method based on semantic awareness correction. This method is rationally designed, overcomes the shortcomings of existing technologies, and achieves good results.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] An RGB-Thermal semantic segmentation method based on semantic-aware correction includes the following steps:

[0011] Step 1: Simultaneously acquire RGB and thermal images of the same scene, standardize and augment the acquired images to form dataset D, and divide dataset D into training set and training set. Validation set test set ;

[0012] Step 2: Construct an RGB-Thermal semantic segmentation network based on semantic-aware correction The network consists of four parts: a dual-branch encoder, a semantic awareness correction module (SPCM), a modality reconstruction module (MRM), and a decoder. The dual-branch encoder is used to extract multi-level features from RGB and thermal images respectively. The SPCM is used to correct the semantic information of the thermal modality and align the category information of the two modalities at the channel level. It is divided into shallow texture correction SPCM-S and deep semantic correction SPCM-D according to different network layers. The MRM is used to achieve accurate alignment and fusion of cross-modal features. Finally, the decoder restores the pixel-level segmentation results.

[0013] Step 3: Design the loss function and training strategy;

[0014] Step 4: [Regarding...] During training, the validation set is saved. The optimal model is obtained by selecting the network parameters that best perform. Selection of evaluation indicators The highest mIoU standard on;

[0015] Step 5: Develop the optimal model obtained through training. Applied to the test set Forward inference is performed to obtain the final semantic segmentation result.

[0016] Furthermore, in step 1, standardization selects channel normalization, and data augmentation uses color dithering, random flipping, scaling, and cropping to improve robustness.

[0017] Furthermore, in step 2, the working process of the dual-branch encoder is specifically as follows:

[0018] The input RGB image and thermal image are fed into a dual-branch encoder to obtain four layers of RGB features. With thermal features , The first two layers are shallow features, rich in texture and edge information, while the last two layers are deep features, rich in semantic category information.

[0019] Furthermore, in step 2, the working process of the SPCM module is as follows:

[0020] The shallow RGB features and thermal features are input into the SPCM-S module, which first processes them separately. and , By applying 3×3 convolution, normalization, and activation operations, texture information is extracted from both modalities, resulting in... and The expression is:

[0021] (1);

[0022] in, It contains an LN layer and a GELU activation function; Indicates the convolution operation; and All Convolution kernel set, ;

[0023] Reconstruct the channel relationship matrix between RGB and thermal features. The expression is:

[0024] (2);

[0025] in, For matrix multiplication, Indicates transpose. Indicates the flattening operation;

[0026] Then utilize thermal modal features Channel mapping and correction are performed to output the corrected shallow thermal modal features. , The expression is;

[0027] (3);

[0028] in, Indicates the reverse flattening operation. This represents minimum-maximum normalization;

[0029] The deep RGB features and thermal features are input into the SPCM-D module, which first processes them separately. and , The shape obtained by flattening the space is , , The height and width of the image are given, respectively; subsequently, the cross-modal relation matrix is ​​calculated. The expression is:

[0030] (4);

[0031] based on Weighted correction is applied to the deep thermal modal channels to output semantically corrected deep thermal modal features. , The expression is:

[0032] (5);

[0033] also,

[0034] (6).

[0035] Furthermore, in step 2, the working process of the MRM module is as follows:

[0036] The SPCM-corrected RGB and thermal features of each layer are input into the MRM module, which consists of two parts: a feature enhancement branch and a fusion branch. In the feature enhancement branch, the input features are first processed separately. and , Perform 5×5 average pooling A nonlinear compression operation is applied to map the features to a low-dimensional space to highlight the region of interest, resulting in soft binarized interest masks. and The expression is:

[0037] (7);

[0038] in, This represents the tanh activation function;

[0039] Then, and Multiplication yields characteristics of common interest areas Modality-specific features are obtained through... respectively with and The feature is obtained by performing element-wise subtraction and then introducing the Gumbel sampling method to finally obtain the modality-specific region features. and The expressions are as follows:

[0040] (8);

[0041] in, Represents Gumbel sampling, Represents the ReLU activation function;

[0042] Finally, and , The concatenation is performed on the channels, and then subjected to depthwise separable convolution. The final enhanced features are obtained by mapping with the Sigmoid algorithm. The expression is:

[0043] (9);

[0044] in, This indicates a splicing operation. Represents the Sigmoid function;

[0045] In the fusion branch, the extracted regions of interest are first removed from the original corrected features to obtain the remaining features. and The expression is

[0046] (10);

[0047] in, This indicates element-wise subtraction;

[0048] Subsequently, and Instance Normalization is applied separately to obtain class features beyond the modality distribution. Residual features are then calculated to extract feature representations related to the modality distribution. Distribution features obtained from different modalities are interacted using element-wise multiplication to obtain common distribution features among the modalities. The expression is:

[0049] (11);

[0050] in, This indicates the Instance Normalization operation;

[0051] The common distribution features are added element-wise to the category features of the two modalities to reconstruct the modal distribution features, resulting in the fused representation. The expression is:

[0052] (12);

[0053] Finally, the enhanced features obtained from the enhanced branches are... As a weight, for reconstructed features Perform weighted fusion and output the final fusion feature of this layer. The expression is:

[0054] (13).

[0055] Furthermore, in step 2, the decoder's operation process is as follows:

[0056] The obtained fused features are input into the decoder for layer-by-layer decoding, generating fused features from deep to shallow at each layer. sequentially with the output features of the layer above the decoder Element-wise addition is performed, followed by upsampling through transpose convolution to obtain the decoder output of the current layer. The expression is:

[0057] (14);

[0058] in, This represents a standard transposed convolutional layer, followed by LayerNorm and GELU activation functions. At that time, the upper layer decoder The output is empty, only the fused features are used. As input;

[0059] Finally, at the first layer output of the decoder A 1×1 convolution and softmax mapping are applied to obtain pixel-level semantic segmentation prediction results.

[0060] Furthermore, in step 3, during the training phase, the loss function uses weighted cross-entropy. With Dice loss The weighted combination is expressed as:

[0061] (15);

[0062] in, , These are the weighting coefficients. This indicates the Ground Truth tag. Total loss;

[0063] To improve training performance, a pre-trained SegFormer-B3 network was used as the backbone network of the encoder, and the optimizer was either Ranger or AdamW. The learning rate scheduling strategy adopted a poly learning rate decay strategy, with the initial learning rate set to 1e-4. The batch size was adjusted according to GPU resources.

[0064] The beneficial technical effects of this invention are as follows:

[0065] This invention proposes a semantic-aware correction network for RGB-Thermal semantic segmentation. Compared with existing technologies, this method establishes an inter-modal relationship matrix based on shallow texture and deep semantics through a semantic awareness and correction module, enhancing the semantic attributes of thermal modalities and aligning category information along the channel dimension, thereby reducing the risk of irrelevant category features being fused. Simultaneously, a modal reconstruction module is proposed, which extracts modal commonalities and selectively optimizes modality-specific components to suppress noise and reduce modality distribution differences, thus improving the final fusion quality. Attached Figure Description

[0066] Figure 1 This is a flowchart of an RGB-Thermal semantic segmentation method based on semantic awareness correction in this invention;

[0067] Figure 2 This is a block diagram of the RGB-Thermal semantic segmentation network model in this invention;

[0068] Figure 3 This is a schematic diagram of the semantic perception correction module in this invention;

[0069] Figure 4 This is a schematic diagram of the modal reconstruction module in this invention; Detailed Implementation

[0070] The specific embodiments of the present invention will be further described below with reference to specific examples:

[0071] An RGB-Thermal semantic segmentation method based on semantic-aware correction, such as Figure 1 As shown, it includes the following steps:

[0072] Step 1: Simultaneously acquire RGB and thermal images of the same scene. Standardize and augment the acquired images. Standardization uses channel normalization. Data augmentation uses color dithering, random flipping, scaling, and cropping to improve robustness, forming dataset D. Divide dataset D into training set and training set. Validation set test set ;

[0073] Step 2: Construct an RGB-Thermal semantic segmentation network model based on semantic-aware correction. ,like Figure 2 As shown, the network consists of four parts: a dual-branch encoder, a semantic awareness correction module (SPCM), a modality reconstruction module (MRM), and a decoder. The dual-branch encoder is used to extract multi-level features from RGB and thermal images respectively. The SPCM is used to correct the semantic information of the thermal modality and align the category information of the two modalities at the channel level. It is divided into shallow texture correction SPCM-S and deep semantic correction SPCM-D according to different network layers. The MRM is used to achieve accurate alignment and fusion of cross-modal features. Finally, the decoder restores the pixel-level segmentation results.

[0074] The working process of the semantic awareness correction network is as follows:

[0075] The input RGB image and thermal image are fed into a dual-branch encoder to obtain four layers of RGB features. With thermal features , The first two layers are shallow features ( The first two layers are rich in texture and edge information, while the last two layers are deep features. (), rich in semantic category information.

[0076] Input the shallow RGB features and thermal features into the SPCM-S module, such as... Figure 3 As shown, this module first performs the following operations respectively: and , By applying 3×3 convolution, normalization, and activation operations, texture information is extracted from both modalities, resulting in... and The expression is:

[0077] (1);

[0078] in, It includes an LN (Layer Normalization) layer and a GELU (Gaussian Error Linear Unit) activation function; Indicates the convolution operation; and All Convolution kernel set, ;

[0079] Reconstruct the channel relationship matrix between RGB and thermal features. The expression is:

[0080] (2);

[0081] in, For matrix multiplication, Indicates transpose. Indicates a flattening operation, Convert to , Number of channels;

[0082] Then utilize thermal modal features Channel mapping and correction are performed to output the corrected shallow thermal modal features. , The expression is;

[0083] (3);

[0084] in, This indicates a reverse flattening operation, which will Convert to , This represents minimum-maximum normalization;

[0085] Input the deep RGB features and thermal features into the SPCM-D module, such as... Figure 3 As shown, this module first performs the following operations respectively: and , The shape obtained by flattening the space is , , The height and width of the image are given, respectively; subsequently, the cross-modal relation matrix is ​​calculated. The expression is:

[0086] (4);

[0087] based on Weighted correction is applied to the deep thermal modal channels to output semantically corrected deep thermal modal features. , To make its channel semantic distribution more consistent with the corresponding RGB channels, the expression is:

[0088] (5);

[0089] also,

[0090] (6).

[0091] The SPCM-corrected RGB and thermal features of each layer are input into the MRM module, such as... Figure 4 As shown, this module consists of two parts: a feature enhancement branch and a fusion branch. In the feature enhancement branch, the input features are first processed separately. and , Perform 5×5 average pooling A nonlinear compression operation is applied to map the features to a low-dimensional space to highlight the region of interest, resulting in soft binarized interest masks. and The expression is:

[0092] (7);

[0093] in, This represents the tanh activation function;

[0094] Then, and Multiplication yields characteristics of common interest areas Modality-specific features are obtained through... respectively with and The feature is obtained by performing element-wise subtraction. To further suppress redundant information, the Gumbel sampling method is introduced in this process to identify and mask irrelevant channels, ultimately obtaining mode-specific regional features. and The expressions are as follows:

[0095] (8);

[0096] in, Represents Gumbel sampling, This represents the ReLU (Rectified Linear Unit) activation function;

[0097] Finally, and , The concatenation is performed on the channels, and then subjected to depthwise separable convolution. The final enhanced features are obtained by mapping with the Sigmoid algorithm. The expression is:

[0098] (9);

[0099] in, This indicates a splicing operation. Represents the Sigmoid function;

[0100] In the fusion branch, the extracted regions of interest are first removed from the original corrected features to retain those less significant but richly detailed regions, thus obtaining the remaining features. and The expression is:

[0101] (10);

[0102] in, This indicates element-wise subtraction;

[0103] Subsequently, and Instance Normalization is applied separately to obtain class features beyond the modality distribution. Residual features are then calculated to extract feature representations related to the modality distribution. Distribution features obtained from different modalities are interacted using element-wise multiplication to obtain common distribution features among the modalities. The expression is:

[0104] (11);

[0105] in, This indicates the Instance Normalization operation;

[0106] The common distribution features are added element-wise to the category features of the two modalities to reconstruct the modal distribution features, resulting in the fused representation. The expression is:

[0107] (12);

[0108] Finally, the enhanced features obtained from the enhanced branches are... As a weight, for reconstructed features Perform weighted fusion and output the final fusion feature of this layer. The expression is:

[0109] (13).

[0110] The obtained fused features are input into the decoder for layer-by-layer decoding, generating fused features from deep to shallow at each layer. sequentially with the output features of the layer above the decoder Element-wise addition is performed, followed by upsampling through transpose convolution to obtain the decoder output of the current layer. This layer-by-layer decoding and feature fusion approach effectively achieves the complementarity between high-level semantic information and low-level detailed information, as expressed in the following expression:

[0111] (14);

[0112] in, This represents a standard transposed convolutional layer, followed by LayerNorm and GELU activation functions. At that time, the upper layer decoder The output is empty, only the fused features are used. As input;

[0113] Finally, at the first layer output of the decoder A 1×1 convolution and softmax mapping are applied to obtain pixel-level semantic segmentation prediction results.

[0114] Step 3: Design the loss function and training strategy;

[0115] During the training phase, the loss function used is weighted cross-entropy. With Dice loss The weighted combination, which simultaneously addresses class imbalance and boundary prediction accuracy, is expressed as:

[0116] (15);

[0117] in, , These are weighting coefficients, which can be set separately based on category imbalance and task requirements. This indicates the Ground Truth tag. This represents the total loss.

[0118] To improve training performance, a pre-trained SegFormer-B3 network was used as the encoder backbone, and either Ranger or AdamW was selected as the optimizer. A poly learning rate decay strategy was employed, with an initial learning rate set to 1e-4. Data augmentation included color dithering, random flipping, random scaling, and cropping to enhance the model's robustness. The batch size was adjusted based on GPU resources.

[0119] Step 4: Use the training set right Training is performed to obtain the predicted output. During training, the weighted cross-entropy loss and Dice loss of the predicted output are calculated separately, and then weighted and combined to obtain the total loss. Minimize the loss value and update the network parameters on the validation set. The mIoU index is calculated. If the mIoU value is better than the historical best value, the corresponding network parameters are saved to obtain the optimal model. ;

[0120] Step 5: Develop the optimal model obtained through training. Applied to the test set Forward inference is performed to obtain the final semantic segmentation result.

[0121] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. An RGB-Thermal semantic segmentation method based on semantic-aware correction, characterized in that, Includes the following steps: Step 1: Simultaneously acquire RGB and thermal images of the same scene, standardize and augment the acquired images to form dataset D, and divide dataset D into training set and training set. Validation set test set ; Step 2: Construct an RGB-Thermal semantic segmentation network based on semantic-aware correction The network consists of four parts: a dual-branch encoder, a semantic awareness correction module (SPCM), a modality reconstruction module (MRM), and a decoder. The dual-branch encoder is used to extract multi-level features from RGB and thermal images, respectively. SPCM is used to correct the semantic information of the thermal modality and align the category information of the two modalities at the channel level. It is divided into shallow texture correction SPCM-S and deep semantic correction SPCM-D according to different network layers; MRM is used to achieve accurate alignment and fusion of cross-modal features; finally, the pixel-level segmentation results are recovered through the decoder. Step 3: Design the loss function and training strategy; Step 4: [Regarding...] During training, the validation set is saved. The optimal model is obtained by selecting the network parameters that best perform. Selection of evaluation indicators The highest mIoU standard on; Step 5: Develop the optimal model obtained through training. Applied to the test set Forward inference is performed to obtain the final semantic segmentation result; In step 2, the working process of the SPCM module is as follows: The shallow RGB features and thermal features are input into the SPCM-S module, which first processes them separately. and , By applying 3×3 convolution, normalization, and activation operations, texture information is extracted from both modalities, resulting in... and The expression is: (1)； in, It contains an LN layer and a GELU activation function; Indicates the convolution operation; and All Convolution kernel set, ; Reconstruct the channel relationship matrix between RGB and thermal features. The expression is: (2)； in, For matrix multiplication, Indicates transpose. Indicates the flattening operation; Then utilize thermal modal features Channel mapping and correction are performed to output the corrected shallow thermal modal features. , The expression is; (3)； in, Indicates the reverse flattening operation. This represents minimum-maximum normalization; The deep RGB features and thermal features are input into the SPCM-D module, which first processes them separately. and , The shape obtained by flattening the space is , , The height and width of the image are given, respectively; subsequently, the cross-modal relation matrix is ​​calculated. The expression is: (4)； based on Weighted correction is applied to the deep thermal modal channels to output semantically corrected deep thermal modal features. , The expression is: (5)； also, (6)； The specific working process of the MRM module is as follows: The SPCM-corrected RGB and thermal features of each layer are input into the MRM module, which consists of two parts: a feature enhancement branch and a fusion branch. In the feature enhancement branch, the input features are first processed separately. and , Perform 5×5 average pooling A nonlinear compression operation is applied to map the features to a low-dimensional space to highlight the region of interest, resulting in soft binarized interest masks. and The expression is: (7)； in, This represents the tanh activation function; Then, and Multiplication yields characteristics of common interest areas Modality-specific features are obtained through... respectively with and The feature is obtained by performing element-wise subtraction and then introducing the Gumbel sampling method to finally obtain the modality-specific region features. and The expressions are as follows: (8)； in, Represents Gumbel sampling, Represents the ReLU activation function; Finally, and , The concatenation is performed on the channels, and then subjected to depthwise separable convolution. The final enhanced features are obtained by mapping with the Sigmoid algorithm. The expression is: (9)； in, This indicates a splicing operation. Represents the Sigmoid function; In the fusion branch, the extracted regions of interest are first removed from the original corrected features to obtain the remaining features. and The expression is (10)； in, This indicates element-wise subtraction; Subsequently, and Instance Normalization is applied separately to obtain class features beyond the modality distribution. Residual features are then calculated to extract feature representations related to the modality distribution. Distribution features obtained from different modalities are interacted using element-wise multiplication to obtain common distribution features among the modalities. The expression is: (11)； in, This indicates the Instance Normalization operation; The common distribution features are added element-wise to the category features of the two modalities to reconstruct the modal distribution features, resulting in the fused representation. The expression is: (12)； Finally, the enhanced features obtained from the enhanced branches are... As a weight, for reconstructed features Perform weighted fusion and output the final fusion feature of this layer. The expression is: (13)。 2. The RGB-Thermal semantic segmentation method based on semantic awareness correction according to claim 1, characterized in that, In step 1, standardization selects channel normalization, and data augmentation uses color dithering, random flipping, scaling and cropping to improve robustness.

3. The RGB-Thermal semantic segmentation method based on semantic awareness correction according to claim 1, characterized in that, In step 2, the working process of the dual-branch encoder is as follows: The input RGB image and thermal image are fed into a dual-branch encoder to obtain four layers of RGB features. With thermal features , The first two layers are shallow features, rich in texture and edge information, while the last two layers are deep features, rich in semantic category information.

4. The RGB-Thermal semantic segmentation method based on semantic awareness correction according to claim 1, characterized in that, In step 2, the decoder operates as follows: The obtained fused features are input into the decoder for layer-by-layer decoding, generating fused features from deep to shallow at each layer. sequentially with the output features of the layer above the decoder Element-wise addition is performed, followed by upsampling through transpose convolution to obtain the decoder output of the current layer. The expression is: (14)； in, This represents a standard transposed convolutional layer, followed by LayerNorm and GELU activation functions. At that time, the upper layer decoder The output is empty, only the fused features are used. As input; Finally, at the first layer output of the decoder A 1×1 convolution and softmax mapping are applied to obtain pixel-level semantic segmentation prediction results.

5. The RGB-Thermal semantic segmentation method based on semantic awareness correction according to claim 1, characterized in that, In step 3, during the training phase, the loss function is weighted cross-entropy. With Dice loss The weighted combination is expressed as: (15)； in, , These are the weighting coefficients. This indicates the Ground Truth tag. Total loss; To improve training performance, a pre-trained SegFormer-B3 network was used as the backbone network of the encoder, and the optimizer was either Ranger or AdamW. The learning rate scheduling strategy adopted a poly learning rate decay strategy, with the initial learning rate set to 1e-4. The batch size was adjusted according to GPU resources.

Images