A scene semantic segmentation method based on thermal image and visible light image feature fusion

Abstract

This invention discloses a scene semantic segmentation method based on feature fusion of thermal and visible light images. The method inputs pairs of visible light and thermal images of the same scene into a trained semantic segmentation model to obtain segmentation results of objects in the images. This segmentation model includes: a dual-branch backbone feature extraction network (Segformer) for extracting global features from the input data; an auxiliary feature selection module for complementing the hierarchical features extracted by the backbone network between the two modalities; a cross-modal feature fusion module for further fusing features between modalities to obtain a rich semantic information feature set; a hierarchical feature fusion decoder module for fine-grained upsampling of the decoder; and a multi-loss supervision module for supervising the model's learning. This invention effectively utilizes the characteristics of visible light and thermal images to mine their complementary features, effectively improving the segmentation accuracy and generalization ability of the scene semantic segmentation model while maintaining a relatively small number of model parameters.

Description

Technical Field

[0001] This invention relates to the field of deep learning-based semantic segmentation technology, and in particular to a scene semantic segmentation method based on the fusion of features from thermal images and visible light images. Background Technology

[0002] With the development of technologies such as computer vision and robotics, unmanned systems, represented by robots and autonomous vehicles, have been widely applied in various fields. Environmental perception is a crucial element for achieving autonomous navigation in unmanned systems, playing a vital role in the robot's understanding and interaction with its external environment. Environmental perception primarily employs two methods: object detection and semantic segmentation. Compared to object detection, semantic segmentation achieves pixel-level segmentation, providing more semantic information and better assisting unmanned systems in recognizing and understanding targets in their surroundings. However, existing semantic segmentation methods often fail to achieve satisfactory results with current RGB image datasets, exhibiting poor robustness and performing poorly in environments with object occlusion, poor lighting conditions, or unfavorable weather.

[0003] To improve the robustness of existing scene semantic segmentation methods, many researchers have incorporated thermal imaging into semantic segmentation. By utilizing the different imaging mechanisms of infrared thermal imaging cameras compared to visible light cameras, infrared information can be obtained by capturing the heat radiated by objects. This can enhance robustness to changes in light and weather. Thermal infrared information is very effective in addressing recognition ambiguity caused by poor lighting conditions. Therefore, researchers have shifted their focus to the field of multimodal semantic segmentation, using visible light with its rich texture and color information and stable thermal image features to improve the robustness and accuracy of semantic segmentation.

[0004] Because the fusion of features from visible light and thermal images can introduce unpredictable noise, simply utilizing features from both modalities results in lower segmentation accuracy compared to a single modality. In 2017, Ha et al. proposed the MFNet network and the first RGB-T semantic segmentation dataset for urban street scenes. This network employed two encoders to extract features from RGB and thermal images respectively, along with a decoder structure. The encoders fused the information from both modalities before partial upsampling. In 2019, Sun et al. designed RTFNet, using ResNet as its backbone. The encoder connected the feature maps of corresponding stages from both modalities through addition, and the decoder designed two modules to progressively complete feature extraction and resolution restoration. In 2020, Shivakumar et al. designed a dual-path neural network structure that effectively fused RGB and RGB-T information. They also proposed a method for RGB-T dataset correction, using depth information to correct the alignment of RGB and RGB-T information, and applying the mapping relationship from RGB-T to RGB images for correction. In 2021, Zhou et al. proposed a multi-level feature multi-label learning network, which designed corresponding modules for feature map processing based on the features extracted from the encoder, and introduced three labels to supervise the network. Following this, Liu et al. proposed the CMX model, which calibrates the features of the current modality in space and dimension by combining features from other modalities.

[0005] Current research methods still fall short of satisfactory segmentation accuracy, failing to balance precision and model parameter count. A significant challenge lies in effectively utilizing the characteristics of visible light and thermal images to uncover their complementary features and reduce noise introduced by different imaging mechanisms, thereby improving the model's generalization ability. Summary of the Invention

[0006] To address the shortcomings of existing methods, this invention proposes a scene semantic segmentation method based on the fusion of features from thermal images and visible light images. The method aims to selectively complement features from the two modalities, thereby fully utilizing the feature advantages of both modalities to achieve feature interaction between them. It combines a cross-modal feature fusion method and a step-by-step feature fusion decoder method, and uses multi-loss supervision to locate, segment, and refine target edges to improve the semantic segmentation accuracy in complex scenes.

[0007] Based on the above ideas, the technical solution adopted to realize the present invention is as follows:

[0008] A scene semantic segmentation method based on feature fusion of thermal images and visible light images, the segmentation model including:

[0009] The backbone feature extraction network is used to extract features at different levels from the input visible light image and thermal image;

[0010] The auxiliary feature selection module is used to complement the feature information of the multi-level features extracted by the backbone feature extraction network between the two modalities, and then input them into the backbone network for further feature extraction.

[0011] The cross-modal feature fusion module fuses the features obtained from the complementary feature selection module to obtain a feature rich in semantic information from both modalities.

[0012] The stepwise feature fusion decoder module is used to fuse the feature maps and upsampled feature maps in the cross-modal feature dense fusion module to achieve fine upsampling of the decoder;

[0013] The multi-loss supervision module provides foreground segmentation prediction, semantic segmentation prediction, and contour segmentation prediction, which are used to locate, segment, and refine the target edges.

[0014] Optionally, the backbone feature extraction network is a Segformer network. The encoder of the Segformer network is a position-free hierarchical Transformer encoder, which can adapt to any test resolution without affecting the segmentation performance. At the same time, the hierarchical Transformer decoder can generate high-resolution fine features and low-resolution coarse features, generating multi-scale features.

[0015] Optionally, the auxiliary feature selection module includes a splicing module, a channel attention module, a spatial attention module, and an auxiliary feature fusion module; the splicing module is used to combine the input feature P main and another input feature P assist The features are concatenated along the channel dimension to obtain feature P1; feature P1 is then input in parallel into the channel attention module and the spatial attention module to obtain two feature outputs P1 respectively. C and P S ; feature P main P C and P S The input is fed into the auxiliary feature fusion module to obtain the final output feature F. out F out It will be input into the backbone network and the cross-modal feature fusion module.

[0016] Optionally, the cross-modal feature fusion module includes a 1×1 convolution module, a grouped convolution module, a densely concatenated semantic information module, and residual connections. The 1×1 convolution module is used to process the first input feature... and the second input feature Features after splicing Learn the feature channel information to obtain features Features are obtained after passing through a grouped convolutional module and a densely concatenated semantic information module. and Features are connected via residual connections and The output feature obtained by fusing the features by adding the features is:

[0017] Optionally, the grouped convolutional module takes as input a feature with a scale of 2C×H×W. feature The feature learning process involves a 1×1 convolution module for channel transformation, followed by 3×3 grouped convolutions, non-linear learning using the ReLU activation function, and then another 1×1 convolution. At this point, the feature size is C×H×W. Finally, regularization is applied to obtain the final feature size.

[0018] Optionally, the densely concatenated semantic information module is a dense feature connection structure. The convolutional kernel is 3×3, with padding and dilation rates of 3, 5, and 7, respectively. This module constitutes dense intra-feature connections, and the input features are... Output features are

[0019] Optionally, the step-by-step feature fusion decoder module includes a 1×1 convolution operation to perform semantic information convolution on one channel. This convolution is set for fusing the feature maps in the corresponding cross-modal feature fusion module later, at which point feature F is obtained. CBR1×1 Feature F CBR1×1 The first refined upsampling feature is obtained after 3×3 convolution and transposed convolution. Feature F CBR1×1 The second refined upsampling feature is obtained after transposing the convolution. feature and characteristics The output features are obtained by fusing the features by adding the features together.

[0020] Optionally, the multi-loss supervision module obtains foreground segmentation prediction, semantic segmentation prediction, and contour segmentation prediction. At the same time, it uses three cross-entropy loss functions to calculate the corresponding loss for the three segmentation predictions, and adds the three losses to obtain the loss output of the entire network.

[0021] The present invention has the following advantages:

[0022] 1. This invention significantly improves the segmentation accuracy of targets in complex scenes, while the model has a small number of parameters, making it easy to deploy on embedded devices.

[0023] 2. This invention can effectively integrate features from two modalities, explore the effectiveness and complementarity of features between the two modalities, and the model has strong generalization ability. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the network structure of a scene semantic segmentation method based on the fusion of features from thermal images and visible light images according to the present invention.

[0025] Figure 2 This is a schematic diagram of the auxiliary feature selection module proposed in this invention.

[0026] Figure 3 This is a schematic diagram of the cross-modal feature fusion module proposed in this invention.

[0027] Figure 4 This is a schematic diagram of the densely cascaded semantic information module proposed in this invention.

[0028] Figure 5 This is a schematic diagram of the step-by-step feature fusion decoder module proposed in this invention.

[0029] Figure 6 This is a schematic diagram of the multi-loss supervision module proposed in this invention. Detailed Implementation

[0030] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the invention. The invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0031] Typical visible light images are RGB images, while typical thermal images can be acquired by infrared thermal imagers and are denoted as images T. The training network is implemented using the PyTorch framework based on a deep learning model. The input RGB and T images of the same scene are of the same size and are denoted as (B, C, H, W), where B is the batch size, C is the number of channels, H is the image height, and W is the image width. This embodiment provides a scene semantic segmentation method based on the feature fusion of thermal and visible light images, such as... Figure 1 As shown, the semantic segmentation model includes:

[0032] The backbone feature extraction network is used to extract features at different levels from the input visible light image and thermal image;

[0033] The auxiliary feature selection module is used to supplement the hierarchical features extracted by the backbone feature extraction network with complementary feature information between the two modalities.

[0034] The cross-modal feature fusion module fuses the features obtained from the complementary feature selection module to obtain a feature rich in semantic information from both modalities.

[0035] The step-by-step feature fusion decoder module is used to step-by-step fuse the output features in the cross-modal feature fusion module and the output features of the upper-level decoder to achieve fine upsampling of the decoder;

[0036] The multi-loss supervision module provides foreground segmentation prediction, semantic segmentation prediction, and contour segmentation prediction, which are used to locate, segment, and refine the target edges.

[0037] A backbone feature extraction network is used to implement multi-level feature extraction, and it can adopt existing backbone feature extraction networks as needed. As an optional implementation, such as... Figure 1 As shown, this embodiment employs a dual-branch backbone feature network. Backbone feature extraction network A and backbone feature extraction network B both utilize Segformer networks to extract features at different levels from the input visible light image and thermal image, respectively. The Segformer network's encoder is a position-free hierarchical Transformer encoder, adaptable to arbitrary test resolutions without affecting segmentation performance. Simultaneously, the hierarchical Transformer encoder can generate high-resolution fine features and low-resolution coarse features, generating multi-scale features. Both backbone feature extraction networks A and B contain four feature extraction encoder layers, denoted as LayerAi (i = 1, 2, 3, 4) and LayerBi (i = 1, 2, 3, 4), corresponding to four levels of extracted features, denoted as... and

[0038] The Auxiliary Feature Selection Module (AFSM) is used to complement the feature information between the two modalities of the hierarchical features extracted by the backbone feature extraction network. There are four Auxiliary Feature Selection Modules in the branch containing backbone network A, denoted as... There are 4 auxiliary feature selection modules in the branch where backbone network B is located, denoted as All auxiliary feature selection modules have the same structure, such as Figure 2 As shown, the input is P. main and P assist In a two-branch backbone network, the features extracted by backbone network A in the RGB image branch... For auxiliary feature selection module Feature P in main Input, features extracted by backbone network B For auxiliary feature selection module Feature P in assist Input; features extracted from the backbone network B on the T image branch. For auxiliary feature selection module Feature P in main Input, features extracted by backbone network A For auxiliary feature selection module Feature P in assist Input. The first auxiliary feature selection module for the RGB image branch. To illustrate the internal data processing of this auxiliary feature selection module, we take the following example: features for Input features P main ,feature for Another input feature P assist , take two input features P main and P assist The features P1 are concatenated along the channel dimension. Features P1 are then subjected to adaptive max pooling and adaptive average pooling to obtain two channel feature maps of size (B, 2C, 1, 1). These two channel feature maps are concatenated along the channel dimension and then passed through a multilayer perceptron (MLP) to obtain a channel weight C of dimension (B, C, 1, 1). weight1 Feature P1 is processed by a multilayer perceptron (MLP) to obtain a spatial weight S with dimension (B, 1, H, W). weight1 Finally, the output features on the RGB branch are Similarly, in the T-image branch, the first auxiliary feature selection module... To illustrate the internal data processing of this auxiliary feature selection module, we take the following example: features for Input features P main ,feature for Another input feature P assist , take two input features P main and P assist The features P1 are concatenated along the channel dimension. Features P1 are then subjected to adaptive max pooling and adaptive average pooling to obtain two channel feature maps of size (B, 2C, 1, 1). These two channel feature maps are concatenated along the channel dimension and then passed through a multilayer perceptron (MLP) to obtain a channel weight C of dimension (B, C, 1, 1). weight2 Feature P1 is processed by a multilayer perceptron (MLP) to obtain a spatial weight S with dimension (B, 1, H, W). weight2 Finally, the output features on the T-image branch are The output obtained from the feature assistance module is then fed into the subsequent backbone network and cross-modal feature fusion module.

[0039] The Cross-module Feature Fusion Module (CMFFM) is used to fuse the output features of the complementary feature selection modules in the two backbone networks, obtaining fused features between the two modalities, which are then input into the model's step-by-step feature fusion decoder. There are a total of four cross-module feature fusion modules, denoted as CMFFM. i (i = 1, 2, 3, 4), all cross-modal feature fusion modules have the same internal structure, such as Figure 3 As shown, its input is and After passing through the auxiliary feature selection module on backbone network A Output characteristics As input features in the cross-modal feature fusion module After passing through the auxiliary feature selection module on backbone network B Output characteristics Features in the cross-modal feature fusion module Input. The outputs of the four cross-modal feature fusion modules are respectively Taking the first cross-modal feature fusion module, CMFFM1, as an example, the internal data processing process of this cross-modal feature fusion module is explained. First, a 1×1 convolution module is used to process the features. feature Features after splicing on the channel Learning the feature channel information yields the first feature of size (B, C, H, W). After undergoing channel dimension transformation in a 1×1 convolution module, the data is then input into a 3×3 grouped convolution module, an activation function, a 1×1 convolution, and a regularization layer to obtain features with an output size of (B, C, H, W). feature The features obtained after densely concatenated semantic information modules are: The cross-modal feature fusion module uses residual connections to fuse feature elements by addition to obtain the final output.

[0040] The Stepwise Feature Fusion DecoerModule (SFFDM) is used to progressively fuse the output features from the cross-modal feature fusion module and the output features from the previous level decoder, achieving refined upsampling of the decoder. Figure 5 As shown, Figure (a) represents the fourth decoder, and Figure (b) represents a schematic diagram of the first three decoders. There are a total of four decoder layers, denoted as SFFDM.i (i = 1, 2, 3, 4), the fourth decoder differs from the previous three in its input; its SFFDM4 input is... The output is SFFDM i The input for (i = 1, 2, 3) is and Feature addition and fusion, output is The following uses the fourth stepwise feature fusion decoder module SFFDM4 and the third stepwise feature fusion decoder module SFFDM3 as examples to illustrate the internal data processing process of this stepwise feature fusion decoder module. The fourth decoder inputs features... By directly inputting a 1×1 convolutional module, we obtain the feature F. CBR1×1 Next feature F CBR1×1 The input is divided into two paths. First, one branch inputs into a 3×3 convolutional module for feature learning. Then, it inputs into a transposed convolutional module, resulting in features upsampled to the feature map size. Another branch feature F CBR1×1 The feature map is upsampled after transposing the convolution to obtain the feature map. The final module output is The input to the third decoder is The output features of the fourth decoder The features are added and fused, then input into a 1×1 convolutional module to obtain feature F. CBR1×1 Next feature F CBR1×1 The input is divided into two paths. First, one branch inputs into a 3×3 convolutional module for feature learning, and then inputs into a transposed convolutional module. This yields features upsampled to the feature map size. Another branch feature F CBR1×1 The feature map is upsampled after transposing the convolution to obtain the feature map. The final module output is Output features obtained by the step-by-step feature fusion decoder module The input will be fed into the multi-loss supervision module.

[0041] The Multi-Loss Monitoring Module is used to obtain foreground segmentation prediction, semantic segmentation prediction, and contour segmentation prediction to locate, segment, and refine target edges, such as... Figure 6 As shown, the output features obtained in the step-by-step feature fusion decoder module feature The input is fed into a 1×1 convolutional module, a regularization function, and an activation function, and then into a 1×1 convolutional module with 2 output channels to obtain feature F'. Feature F' is then interpolated and upsampled to obtain a foreground segmentation prediction result F of the same size as the original image. bin Convolutional operations on feature F' yield a single-channel feature map, which is then processed using a sigmoid activation function to obtain the weights of feature F'. The weight coefficients of feature F' are multiplied together and then fed into a 1×1 convolutional module, followed by a regularization function and an activation function to obtain feature F'. This feature F' is then fed into a 9-channel 1×1 convolutional module for interpolation and upsampling to obtain a semantic segmentation prediction result F of the same size as the original image. sem ; Features The feature F” is concatenated across channels to obtain a feature map of dimension (B, 2C, H, W). This feature map is then fed into a 1×1 convolutional module, followed by a regularization function and an activation function to obtain a feature map of size (B, 2C, H, W). Finally, this feature map is fed into a 1×1 convolutional module with 2 channels and subjected to interpolation and upsampling to obtain a contour segmentation prediction result F of the same size as the original image. boud Using the cross-entropy loss function l bin l sem l boud Its three prediction results F bin F sem F boud Supervised training is conducted, where l bin l boud It is the binary classification cross-entropy loss function; sem It is a multi-class cross-entropy loss function, and the total loss during model training is S, where S = l bin +l sem +l boud In this model structure, foreground segmentation prediction F bin Used to determine the location of targets in an image, semantic segmentation prediction F sem For each target segmented in the image, the contour segmentation prediction F boud Refine edge features to achieve image target localization, segmentation of various targets in an image, and refinement of edge features.

[0042] To implement this invention, the publicly available multimodal semantic segmentation datasets MFNet and PST900 are used as validation data, and the specific implementation process is illustrated using these as examples.

[0043] Comparative Experimental Results Analysis: The test set was input into the trained network model, and mAcc and mIou were used to measure the detection accuracy of the test model. mAcc represents the average accuracy across all categories, and mIou represents the average intersection-union ratio. Table 1 shows the comparison results of this invention with other state-of-the-art algorithms on the RTF-Net dataset. The RTF-Net dataset has nine classes: car, person, bicycle, curve, station, guardrail, warning line, protrusion, and unlabeled. The comparison results show that the method of this invention brings significant performance improvement, achieving detection accuracies of 75.4% in mAcc and 59.1% in mIou. Table 2 shows the segmentation results for daytime and nighttime, and its accuracy metrics are also superior to other mainstream semantic segmentation models. It is worth noting that the method of this invention does not increase the depth or width of the backbone network.

[0044] Table 1: Experimental results comparing the present invention with other algorithms on the RTF-Net dataset.

[0045]

[0046] Table 2: Experimental results of this invention on the RTF-NET dataset comparing day and night conditions.

[0047]

[0048] Comparison of generalization experiment results: Table 3 shows the comparison results of our method with other state-of-the-art algorithms on the PST900 dataset, which has five classes: fire extinguisher, backpack, hand drill, survivor, and unlabeled. It can be seen that the semantic segmentation network model proposed in this invention also exhibits good performance on other multimodal datasets, verifying the generalization ability of the proposed model;

[0049] Table 3: Experimental results comparing the method of the present invention with other algorithms on PST900

[0050]

[0051] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A scene semantic segmentation method based on feature fusion of thermal images and visible light images, characterized in that, Thermal images and visible light images are input into a trained semantic segmentation model to obtain segmentation results for each type of target in the image. The street scene semantic segmentation method based on feature fusion of thermal images and visible light images includes: The backbone feature extraction network A employs a Segformer network to extract features at different levels from the input visible light image. The Segformer network is a hierarchical Transformer encoder containing four feature extraction encoder layers, denoted as Layer. The extracted features have four levels, denoted as... ; The backbone feature extraction network B uses a Segformer network to extract features at different levels from the input thermal image, denoted as Layer. The extracted features have four levels, denoted as... ; The auxiliary feature selection module is placed after each layer of the feature extraction encoder in the backbone feature extraction network A, and is denoted as... It is placed after each layer of the feature extraction encoder in the backbone feature extraction network B, and is denoted as . ; The input is and The output is ; The input is and The output is The auxiliary feature selection module includes a splicing module, a channel attention module, a spatial attention module, and an auxiliary feature fusion module; the splicing module is used to integrate the input feature P. main and another input feature P assist The features are concatenated along the channel dimension to obtain feature P1; feature P1 is then input in parallel into the channel attention module and the spatial attention module to obtain two feature outputs P1 respectively. C and P S ; feature P main P C and P S The input is fed into the auxiliary feature fusion module to obtain the final output features. ; There are four cross-modal feature fusion modules in total, denoted as: Its input is and The output is ; The step-by-step feature fusion decoder module has a total of four decoder layers, denoted as... ; The input is The output is ; The input is and The output is ; The multi-loss supervision module supervises foreground segmentation prediction, semantic segmentation prediction, and contour segmentation prediction to achieve localization, segmentation, and refinement of edge targets.

2. The scene semantic segmentation method based on the fusion of thermal image and visible light image features according to claim 1, characterized in that, The cross-modal feature fusion module includes a 1×1 convolution module, a grouped convolution module, a densely concatenated semantic information module, and residual connections; the 1×1 convolution module is used to process the first input feature... Features after splicing Learn the feature channel information to obtain features ; Features are obtained after passing through a grouped convolutional module and a densely concatenated semantic information module. and Features are connected via residual connections. , , , and The output feature is obtained by fusing the features by adding the feature elements. .

3. The scene semantic segmentation method based on the fusion of thermal image and visible light image features according to claim 1, characterized in that, The step-by-step feature fusion decoder module includes a 1×1 convolution module, a 3×3 convolution module, and a transposed convolution module; the 1×1 convolution module performs semantic information convolution on the input features in one channel to obtain the features. ;feature The first refined upsampling feature is obtained after 3×3 convolution and transposed convolution. ;feature The second refined upsampling feature is obtained after transposing the convolution. ;feature and characteristics The output features are obtained by fusing the features by adding the features together. .

4. The scene semantic segmentation method based on the fusion of thermal image and visible light image features according to claim 1, characterized in that, The multi-loss supervision module evaluates the errors between foreground segmentation prediction, semantic segmentation prediction, contour segmentation prediction, and their corresponding three ground truth labels, aiding the network model in learning. It uses the cross-entropy loss function. , , Its three segmentation prediction outputs , , Conduct training supervision: in, , Both use the binary classification cross-entropy loss function, defined as: （1） In the formula, N is the sample size. The label represents sample i, with 1 for positive and 0 for negative. This represents the probability that sample i is predicted to be of the positive class; It is the multi-class cross-entropy loss function, defined as: （2） In the formula, M represents the number of categories; Let c be the sign function, and when the true class of sample i is equal to c, ,otherwise ; Let i be the predicted probability that observed sample i belongs to category c; The total training loss of the model is S: 。

Images