A semantic segmentation method based on RGB-T image knowledge interaction

Abstract

The application relates to an RGB-T image semantic segmentation method named KIMNet, which is used for realizing high-precision image segmentation in weak light and complex scenes. The method comprises the following steps: constructing a multi-modal input network architecture of RGB and thermal infrared (TIR) images; introducing a knowledge interaction module (KIMM) based on a Mamba architecture, performing multi-level and context-related semantic enhancement and modal interaction interaction on RGB and TIR modal features; designing a content-guided interaction module (CGIM), and through channel and spatial attention fusion, the channel feature discrimination is improved in a top-down semantic guidance mode, and efficient integration and semantic alignment of multi-level features are realized; finally, a segmentation result is generated through feature fusion decoding and a multi-level supervision mechanism; through the multi-modal interaction and feature guidance mechanism, the segmentation performance of the model in the weak light environment is effectively improved, and the method has high practical value and popularization prospect.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and multimodal image processing technology, specifically relating to a semantic segmentation method based on RGB-T image knowledge interaction. Background Technology

[0002] Semantic segmentation is an important research direction in computer vision, aiming to perform semantic classification on each pixel in an image to achieve accurate understanding of the image content. This technology has been widely applied in various practical scenarios such as autonomous driving, robotics, and medical image analysis. In recent years, although deep learning-based RGB semantic segmentation methods have made significant progress, performance degradation still exists in complex environments (such as low light, long distance, and inclement weather). This is mainly because RGB images are sensitive to changes in lighting and struggle to provide stable visual information. In contrast, thermal infrared images, by sensing the thermal radiation of objects, can provide clear contours and semantic information even in low-light conditions, exhibiting good robustness. Therefore, the RGB-T semantic segmentation method, which fuses RGB images with thermal infrared images, has become a research hotspot in recent years.

[0003] Existing RGB-T semantic segmentation methods mainly fall into two categories: direct fusion and feedback fusion. Direct fusion extracts features from RGB and thermal infrared images using two independent backbone neural networks, then fuses these features using a specific module. While simple, this method ignores the differences between features at different levels and lacks effective inter-level information interaction, resulting in the fused features failing to fully leverage the advantages of both modalities. Feedback fusion feeds the fused features back into the backbone network, enabling deeper feature interactions at multiple levels. However, information bottlenecks can easily form in the shallow layers of the neural network during feedback, preventing shallow features from being effectively transmitted to deeper layers and failing to fully explore the complementary relationship between RGB and thermal infrared images. Furthermore, current supervision strategies primarily focus on the network's endpoints, lacking explicit guidance for multi-level feature interactions, resulting in unclear fused feature representations that can lead to information loss or error accumulation, impacting segmentation performance. Therefore, efficiently modeling the complementary relationship between RGB and thermal infrared features and achieving effective interaction between multi-level features remains a key technical challenge in RGB-T semantic segmentation. Summary of the Invention

[0004] The present invention addresses the shortcomings of the existing technology by proposing a semantic segmentation method based on RGB-T image knowledge interaction, which aims to effectively explore the complementary relationships between different modalities and realize knowledge interaction between different levels.

[0005] A semantic segmentation method based on RGB-T image knowledge interaction, characterized by the following steps:

[0006] Step 1: Obtain visible light RGB modal images and corresponding thermal infrared images, perform spatial registration, and construct an RGB-T image pair dataset;

[0007] Step 2: Input the RGB modal image and the thermal infrared modal image into a dual encoder composed of two ResNet50 networks respectively, and extract multi-level features from the two modal images respectively. The RGB modal features are denoted as Rs and the thermal infrared modal features are denoted as Ts, where s∈{1,2,3,4,5} represents the feature level.

[0008] Step 3: Input the multi-level features Rs and Ts of the two modalities into the constructed Knowledge Interaction Module (KIMM). Through feature concatenation, adaptive pooling, weighted fusion, and Mamba context modeling, cross-modal semantic enhancement and knowledge interaction are achieved. The core processing formula is as follows:

[0009]

[0010] Wherein, F1 is the initial splicing feature, F2 is the weighted fusion feature, and F(KIMM) is the module output fusion feature; the thermal infrared feature serves as guiding information to dynamically adjust the RGB feature response area;

[0011] Step 4: Input the high-level features (R5, T5) and KIMM output features into the Content-Guided Interaction Module (CGIM). Generate a joint attention map of channels and space through global pooling, and achieve content-oriented multi-scale semantic alignment and feature enhancement through channel shuffling and group convolution operations; thereby improving the network's ability to perceive complex structures and key boundaries.

[0012] Step 5: Construct a decoding module using the ASPP module or the Lawin module, which has multi-scale context modeling capabilities. Input the interactive features of different levels into the decoding module, first perform channel unification and scale alignment, then achieve feature fusion through multi-scale feature concatenation and 1×1 convolution compression, and finally use a pixel-by-pixel classification layer to generate category predictions for each pixel, and output the final semantic segmentation result.

[0013] Step 6: During the training phase, a multi-level joint supervision mechanism is adopted, including semantic segmentation loss and edge detection loss. Supervision branches are set in different decoding output layers, and the model is guided throughout the process by weighted combination loss function to achieve synergistic optimization of semantic consistency and boundary accuracy.

[0014] Beneficial effects:

[0015] 1. The knowledge interaction module proposed in this invention introduces a context modeling mechanism based on the Mamba structure, which can effectively model the contextual correlation between RGB images and thermal infrared images in mid-level features, realize deep fusion of inter-modal information, and improve the accuracy and robustness of semantic segmentation.

[0016] 2. This invention, by designing a content-guided interactive module and combining channel attention and spatial attention mechanisms, guides the model to focus on important regions of different channels during the fusion stage, thereby strengthening the fusion of shallow edge information and deep semantic information and effectively alleviating the information transmission bottleneck problem in existing methods.

[0017] 3. This invention further proposes a multi-level loss function design, which guides features at different levels in stages during network training, improving the clarity and discriminativeness of feature representation and avoiding the problem of semantic information gradually weakening or being lost in deep networks.

[0018] 4. This invention fully utilizes the stability of thermal infrared images in nighttime and low-light scenarios, and combines the rich texture semantics of RGB images to achieve high-precision perception of dynamic targets (such as pedestrians and vehicles) in complex scenes, and has good promotion and practical value. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the semantic segmentation method provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the flowchart of the semantic segmentation method provided in the embodiments of the present invention.

[0021] Figure 3 This is a schematic diagram of the knowledge interaction module proposed in this invention.

[0022] Figure 4 This is a schematic diagram of the content-guided interaction module proposed in this invention. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] This embodiment provides a scene semantic segmentation method based on cross-modal fusion of RGB images and thermal infrared images, such as... Figure 1 As shown, the RGB-T semantic segmentation network mainly includes: 1. a dual-branch feature extraction module (RGB branch and Thermal branch); 2. a feature complementarity enhancement module; 3. a content-guided interaction module; 4. a Lawin ASPP decoding module; and 5. a multi-decoder supervision module.

[0025] Step 1: Image Preprocessing and Modal Input. This invention first preprocesses the input image. Let the input be a visible light image. With thermal infrared images Since thermal infrared images are single-channel, they are copied to three channels to adapt to the neural network input structure. .

[0026] Step 2: Dual-branch multi-level feature extraction, and The inputs are respectively fed into a dual-branch encoder composed of two ResNet50 networks, and five layers of semantic features are extracted from each. These are denoted as... and ,in , This module provides hierarchical and fusionable multi-scale feature representations for subsequent modal interactions.

[0027] Step 3: The knowledge interaction module performs modal semantic enhancement. To improve the semantic complementarity between RGB and thermal infrared modes, a knowledge interaction module based on the Mamba structure is constructed for cross-modal semantic alignment of intermediate layer features. (The last sentence appears to be incomplete and possibly refers to a different step.) Taking layer features as an example, RGB modal features are: The thermal modal characteristics are The two are combined into a fused feature at the channel dimension. Attention mechanism via the channel Generate weights and combine them with spatial attention graphs. Generate weighted features After inputting this feature into the Mamba context modeling structure, the output is obtained. Enhanced output is formed through residual connection. This process enables intermodal referencing and semantic enhancement.

[0028] Step 4: The Content-Guided Interaction Module (CGIM) implements high-level semantic fusion. In the high-level semantic alignment stage, the Content-Guided Interaction Module (CGIM) processes the fifth-layer features. and First, let's look at the splicing features. Global average pooling and max pooling are performed to obtain the channel guiding vectors, respectively. Spatial response diagram The two together form a joint attention diagram. Note that the weighted output is generated by multiplying the graph and the fused features element-wise. This enhances high-level semantic collaboration between modalities and focuses attention on salient regions.

[0029] Step 5: The feature alignment and decoding module generates segmentation predictions, and combines the multi-level fused features output by the KIMM and CGIM modules. The unified channel number is C, obtained through linear transformation. Then, use bilinear interpolation or deconvolution to upsample to a uniform spatial size. ,get All upsampled features are then concatenated and fused into a single channel. The final fused features are generated through convolutional compression. The semantic segmentation results are output through a pixel-by-pixel classifier. Each pixel location contains a predicted probability distribution for each category.

[0030] Step 6: Multi-level supervised loss function design and edge-guided optimization. To improve segmentation accuracy and edge quality, a multi-level supervision mechanism was designed. During the training phase, the model's total loss function consists of semantic segmentation loss and edge detection loss, expressed as:

[0031]

[0032] in For semantic segmentation loss based on cross-entropy, For binary classification loss in edge regions, and This represents the loss balancing coefficient. Considering the model has multiple decoding branches and edge prediction layers, loss supervision is further applied to each branch, and the overall loss is expanded to:

[0033]

[0034] The multi-level supervision is set on at least three decoding output layers with different resolutions, including scales of 1 / 4, 1 / 8 and 1 / 16 of the original input resolution. The output at each scale corresponds to the learning tasks of global semantic information and local boundary information, respectively. λi and μj represent the loss weight coefficients of each branch, ensuring the collaborative optimization of information at different levels during the training process.

[0035] To verify the effectiveness of the method of this invention, two industry-standard RGB-T semantic segmentation public datasets, MFNet and PST900, were used for experimental verification, corresponding to urban autonomous driving scenarios and industrial safety monitoring scenarios, respectively. The specific implementation process is as follows.

[0036] Experimental setup

[0037] A) Datasets: This invention uses two publicly available datasets. The first is the PST900 dataset for industrial safety scenarios, containing 894 pairs of high-resolution (1280×720) RGB-T image pairs, labeled with 5 categories including background and fire extinguishers, divided into 597 training pairs and 297 test pairs. The second is the MFNet dataset for autonomous driving urban scenarios, containing 1569 pairs of day and night RGB-T image pairs (480×640 resolution), labeled with 9 categories including cars and pedestrians, divided into 784 training pairs and 393 test pairs, to fully verify the robustness of the model under different lighting conditions.

[0038] B) Comparison Method: To verify the effectiveness of the method of this invention, nine representative mainstream deep learning models in the current semantic segmentation field were selected as comparison methods to ensure the fairness and authority of the performance comparison. These models cover the mainstream structures and technical routes in RGB-T fusion or high-performance semantic segmentation.

[0039] This invention better adapts to the fusion requirements of RGB and thermal infrared image information. Final experimental results show that the proposed KIMNet model achieves an mIoU of 57.9% on the MFNet dataset, outperforming all compared methods and demonstrating a significant performance improvement. This result fully verifies the effectiveness and advancement of the proposed method for semantic segmentation by fusing multimodal information in complex scenes.

[0040] C) Evaluation Metrics: In the comparative experiments, to comprehensively evaluate the performance of the semantic segmentation model proposed in this invention, the mean Intersection over Union (mIoU) was used as the main evaluation metric. mIoU is the most commonly used and authoritative metric in semantic segmentation tasks, used to measure the degree of overlap between the predicted regions and the ground truth labeled regions in each category. A higher mIoU value indicates higher segmentation accuracy.

[0041] Experimental results

[0042] The differences in parameter count and computational cost (FLOPS) among the various models listed in Table 1 were systematically compared, and the results are shown in Table 1. Table 1 shows that the proposed model achieves a good balance between model performance, parameter size, and computational overhead.

[0043] Table 1. Comparison of parameter counts for different methods

[0044]

[0045] As can be seen from the table, there is a clear trade-off between computational complexity, parameter size, and segmentation performance among the various methods. While our proposed method, with an input size of 640×480, achieves a relatively high computational cost of 303.91G FLOPs and 142.83M parameters, it achieves the best performance of 57.9% in the key metric mIoU, outperforming methods such as MTANet (56.1%) and SFGNet (57.6%). Simultaneously, its mAcc reaches 73.5%, placing it in the upper-middle range. In contrast, some lightweight models (such as ABMDRNet and EGFNet) have advantages in parameter size and computational cost, but their segmentation accuracy is slightly lower; while traditional deep models (such as the RTFNet series) suffer from parameter redundancy and limited performance improvement. Overall, our proposed method effectively improves segmentation accuracy while maintaining strong feature representation capabilities, demonstrating superior performance and making it more suitable for applications requiring high segmentation accuracy.

Claims

1. A semantic segmentation method based on RGB-T image knowledge interaction, characterized in that, Includes the following steps: Step S1: Obtain the visible light RGB modal image and the corresponding thermal infrared modal image, and perform spatial registration to construct an RGB-T image pair dataset; Step S2: Input the RGB modal image and the thermal infrared modal image into a dual encoder composed of two ResNet50 networks respectively, and extract multi-level features from the two modal images respectively. The RGB modal features are denoted as Rs and the thermal infrared modal features are denoted as Ts, where s∈{1,2,3,4,5} represents the feature level. Step S3: Input the multi-level features Rs and Ts of the two modalities into the constructed Knowledge Interaction Module (KIMM). Through feature concatenation, adaptive pooling, weighted fusion, and Mamba context modeling, cross-modal semantic enhancement and knowledge interaction are achieved. The core processing formula is as follows:

2. Among them, F1 is the initial splicing feature, F2 is the weighted fusion feature, and F(KIMM) is the module output fusion feature; Step S4: Input the high-level features (R5, T5) and KIMM output features F (KIMM) into the content-guided interaction module (CGIM). Generate a joint attention map of channels and space through global pooling, and achieve content-oriented multi-scale semantic alignment and feature enhancement through channel shuffling and group convolution operations, thereby improving the network's ability to perceive complex structures and key boundaries. Step S5: Construct a decoding module using the ASPP module or Lawin module with multi-scale context modeling capabilities. Input the interactive features of different levels into the decoding module, first perform channel unification and scale alignment, then achieve feature fusion through multi-scale feature splicing and 1×1 convolution compression, and finally use the classification layer to generate category prediction for each pixel and output the final semantic segmentation result. Step S6: During the model training phase, a multi-level joint supervision mechanism is adopted, including semantic segmentation loss and edge detection loss. Supervision branches are set in different decoding output layers, and the model is guided throughout the process by weighted combination loss function to achieve synergistic optimization of semantic consistency and boundary accuracy.

3. The method according to claim 1, characterized in that, In step S1, spatial registration is performed using a method based on geometric alignment or optical flow estimation to ensure that the RGB image and the thermal infrared image are consistent at the pixel level, thereby guaranteeing the comparability of cross-modal features at the same spatial location.

4. The method according to claim 1, characterized in that, In step S3, the knowledge interaction module KIMM controls the RGB feature response region through a gating mechanism and combines the Mamba residual structure and weighted fusion strategy to achieve cross-modal global semantic modeling, thereby enhancing the expression of semantic information.

5. The method according to claim 1, characterized in that, In the content-guided interaction module CGIM described in step S4, the channel attention weights are generated by fusing global average pooling and global max pooling on the input features, and the spatial attention weights are generated by compressing the feature map in the channel dimension and then performing convolution operations. The channel shuffling is used to rearrange different grouped feature channels, and the group convolution is used to perform group convolution operations on the rearranged features to reduce computational complexity and enhance feature interaction.

6. The method according to claim 1, characterized in that, In the decoding module described in step S5, features at different levels are upsampled before multi-scale feature splicing to unify spatial resolution. The channel compression is achieved through 1×1 convolution to reduce feature dimension and computational overhead.

7. The method according to claim 1, characterized in that, The multi-level supervision in step S6 is set on at least three decoding output layers with different resolutions. The different resolutions include scales of 1 / 4, 1 / 8 and 1 / 16 of the original input resolution. The output at each scale corresponds to the learning tasks of global semantic information and local boundary information, respectively.

8. A semantic segmentation system based on RGB-T image knowledge interaction, characterized in that, include: The data storage module is used to store RGB-T image datasets; A dual-branch encoder module is used to extract multi-level features of RGB and thermal infrared. The knowledge interaction module is used to achieve cross-modal semantic enhancement; the content-guided interaction module is used for multi-scale semantic alignment and feature enhancement. The feature fusion and decoding module is used to generate the final semantic segmentation map; the multi-level supervision module is used to provide joint supervision to optimize model training and boundary accuracy.

9. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs that, when executed by the processor, cause the processor to implement the RGB-T image semantic segmentation method according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the RGB-T image semantic segmentation method according to any one of claims 1-6.

Images