A semantic segmentation method based on RGB-T image single-stream cross-modal fusion

Abstract

The application discloses a kind of based on RGB-T image single stream cross-modal fusion's semantic segmentation method, the visible light image and thermal image obtained in pairs are input to the semantic segmentation model trained to obtain the segmentation result of the image object, the segmentation model is composed of image fusion module, main feature extraction network, receptive field expansion module, local-context reverse perception module and multi-head parallel decoder.The application realizes that two kinds of modal images of RGB and thermal image are handled using single stream cross-modal fusion network, with small parameter quantity, low computational complexity, high precision and real-time inference performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning-based semantic segmentation, and in particular to a semantic segmentation method based on RGB-T image single-stream cross-modal fusion. BACKGROUND

[0002] Image semantic segmentation, as an important task in the field of computer vision, is significant in extracting semantic information contained in images to help real-world automated devices better understand the surrounding environment, such as autonomous vehicles, unmanned systems, virtual reality, etc. In the past decade, semantic segmentation methods for RGB images have made great progress in speed and accuracy. However, due to the characteristics of RGB image sensors, it is difficult to obtain high-quality RGB images in some specific environments such as night, poor visibility, rainy days, etc., which seriously affects the performance of these methods. In recent years, the characteristics of infrared thermal imaging cameras, which are not affected by light and weather, can provide compensation information for RGB images in low-visibility and poor lighting conditions, attracting great attention from researchers. Therefore, exploring segmentation networks for RGB-T images has its practical significance.

[0003] Currently, the main form of networks for RGB-T image segmentation adopts a double-stream structure, which inputs RGB and thermal images into independent backbone networks to extract features. In the process of extracting features at each level, the extracted RGB and thermal features are fused and interacted, and a feature map containing rich information of both RGB and thermal images is obtained at the end of the network. Finally, dense prediction is performed on the feature map. The main difference between such methods lies in the design of the feature fusion strategy. With the development of convolutional neural networks, the introduction of attention mechanisms, the improvement of fusion strategies, and the update of training strategies, RGB-T image semantic segmentation has made great progress in accuracy. However, the repeated extraction of two backbone networks causes a lot of information redundancy and low network efficiency. Compared with a single backbone network extracting features, deploying two backbone networks inevitably increases the number of network parameters, and the limited memory, Flash, and other storage resources on embedded computing platforms cannot meet the needs of existing model deployment. If the RGB-T image semantic segmentation model is deployed to practical application scenarios such as autonomous vehicles, unmanned systems, and virtual reality, it still faces many challenges.

[0004] How to effectively utilize the complementary information of visible light image and thermal image features, improve the network's ability to process multi-modal information, fully exploit the features between the two modalities, and require the model to have small parameter quantity and low computational complexity to facilitate the deployment of embedded platforms. SUMMARY

[0005] The present application aims at the shortcomings of the existing methods, and invents a semantic segmentation method based on RGB-T image single-stream cross-modal fusion, which aims to fuse RGB images and thermal images through a single-stream framework, combines a feature hierarchical processing module and a multi-head parallel decoder supervision network, to solve the problems of large parameter quantity inherent in RGB-T double-stream networks, and maintain good segmentation accuracy.

[0006] According to the above idea, the technical scheme adopted by the present application is:

[0007] A semantic segmentation method based on RGB-T image single-stream cross-modal fusion, the model comprises:

[0008] Image fusion is used for pixel-level fusion of input visible light images and thermal images, and then the fused images are sent to a backbone feature extraction network;

[0009] The backbone feature extraction network extracts multi-level features from the fused images, each time deeper, to obtain feature maps of different levels;

[0010] The receptive field expansion module is used to expand the receptive field of the feature, and to obtain the maximum receptive field feature f max-recept under the premise of retaining as much original feature information as possible;

[0011] The local-context inversion perception module is used to expand the feature distance between adjacent levels, to realize comprehensive interaction at different levels, different aspects and tasks, and to obtain features f reversal rich in semantic information of two modalities and f soft ;

[0012] The multi-head parallel decoder is used to supervise the learning tasks of different aspects of the network, and to perceive image information more comprehensively.

[0013] The image fusion is to input the traditional image fusion method as a priori knowledge into the convolutional network, to fuse RGB and thermal images, and to eliminate noise and overlap that will affect image quality;

[0014] The backbone feature extraction network extracts and processes the fused images sent by the image fusion through a single-stream network, which contains four layers of feature extraction encoders and can generate multi-scale features to obtain feature information of different receptive fields;

[0015] The receptive field expansion module includes convolution operation, pooling operation and splicing operation; the module performs down-sampling on the input feature F4 to obtain feature to complete receptive field expansion; secondly, the down-sampled feature is up-sampled to restore the resolution to obtain the restored feature Then combine feature F4 with feature By splicing the data, we can obtain the features. Finally, a convolutional layer is used to learn the features. Obtain the maximum receptive field feature f max-recept ;

[0016] The local-context inversion awareness module includes pixel-level addition and subtraction operations, concatenation operations, and upsampling operations on features. The module takes features F3 and F4 as input, upsampling the features to obtain features with the same dimension as F3, and convolving the difference between F3 and Up(F4) to obtain the inverted feature f. reversal Using learnable parameter β to apply f reversal Assigning certain weights, and then adding them to feature F3 at the pixel level, yields semantic information feature f with rich details. soft ;

[0017] The multi-head parallel decoder is used to supervise different aspects of the network's learning tasks. The Main Head is used for the final dense prediction and exists throughout the training and testing processes. The remaining three decoders are secondary decoder heads, existing only during training. The Aux Head generates a reference prediction map to adjust the network's final prediction; the Incoherent Head makes the feature map as close to the edges as possible; and the Front Head increases the network's attention to foreground pixels. During training, four cross-entropy loss functions are used to supervise the corresponding feature maps, yielding corresponding losses. The overall network loss output is a weighted sum of the four cross-entropy loss functions.

[0018] The present invention has the following advantages:

[0019] 1. The single-stream cross-modal fusion segmentation framework based on RGB-T images proposed in this invention has the characteristics of small parameter quantity and low computational complexity, which facilitates deployment on embedded devices.

[0020] 2. It has high accuracy and real-time inference performance, and the model has strong generalization ability. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the network structure of a semantic segmentation method based on RGB-T image single-stream cross-modal fusion according to the present invention.

[0022] Figure 2 This is a schematic diagram of the receptive field expansion module proposed in this invention.

[0023] Figure 3 This is a schematic diagram of the local-context inversion sensing module proposed in this invention. Detailed Implementation

[0024] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the invention. The invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0025] This embodiment provides a scene semantic segmentation method based on the fusion of features from thermal images and visible light images, such as... Figure 1 As shown, the semantic segmentation model includes:

[0026] Image fusion involves performing pixel-level fusion on the input visible light image and thermal image to obtain the fused image.

[0027] The backbone feature extraction network performs multi-level feature extraction on the fused image, extracting features at a deeper level each time to obtain feature maps at different levels.

[0028] The receptive field expansion module is used to expand the receptive field of the fused features, obtaining the feature with the largest receptive field f while preserving as much of the original feature information as possible. max-recept ;

[0029] The local-context inversion awareness module is used to expand the feature distance between adjacent layers, enabling comprehensive interaction across different levels, aspects, and tasks, and obtaining features f that pay more attention to edge details. reversal and semantic information features with rich details f soft ;

[0030] Multi-head parallel decoders are used to supervise different aspects of the network's learning tasks, enabling a more comprehensive perception of image information.

[0031] The image fusion process involves pixel-level fusion of RGB and thermal images, removing noise and overlaps that could affect image quality. The backbone feature extraction network is used to perform multi-level feature extraction, specifically targeting the fused image for feature extraction. Figure 1 As shown, the fused image is denoted as I. A Segformer network containing a four-layer feature extraction encoder is used to extract features from image I. Figure 1 The diagram shows Layer 1, Layer 2, Layer 3, and Layer 4. The encoder of the Segformer network is a position-free hierarchical Transformer encoder that can adapt to any test resolution without affecting segmentation performance. At the same time, the hierarchical Transformer encoder can generate high-resolution fine features and low-resolution coarse features, generating multi-scale features. Each feature extraction encoder will gradually extract features with more detailed semantic information, which are denoted as features F1, F2, F3, and F4 respectively.

[0032] The receptive field expansion module (REF) is used to expand the receptive field of a feature, obtaining the feature with the largest receptive field while preserving as much of the original information as possible. This module is as follows... Figure 2 As shown, the input is feature F4; firstly, the input feature F4 is downsampled to fully understand the high-level features of the feature information, thus obtaining the feature... First, expand the receptive field; second, process the downsampled features. Then, upsampling is performed to restore the resolution, resulting in the restored features. Then combine feature F4 with feature By splicing the data, we can obtain the features. Finally, a convolutional layer is used to learn the features. Obtain the maximum receptive field feature f max-recept Maximum receptive field feature f max-recept The input is fed into the Main Head detector to perform the final dense prediction task and predict the results.

[0033] The Local Context Aware Block (LIA) module is used to expand the feature distance between adjacent layers, enabling comprehensive interaction across different levels, aspects, and tasks. This module, as shown... Figure 3 As shown, its inputs are features F3 and F4. Feature F4 is upsampled to obtain a feature Up(F4) with the same dimension as F3. Then, the difference between F3 and Up(F4) is convolved to obtain a feature f that pays more attention to edge details. reversal Use learnable parameter β to apply f reversal Assigning certain weights, and then adding them to feature F3 at the pixel level, yields semantic information feature f with rich details. soft ; will f soft with f reversal These are the two outputs of the local-context inversion awareness module, where f soft The output is sent to the Main Head detector to perform the final dense prediction task, and the results are predicted; reversal Then the output is sent to the Incoherent Head to avoid the loss of edge information.

[0034] The multi-head parallel decoder is used to supervise different aspects of the network's learning tasks, enabling a more comprehensive perception of image information. The multi-head parallel decoder contains four parallel decoding heads for supervising different aspects of the network's learning tasks, such as... Figure 1 As shown, they are denoted as Main Head, Aux Head, Incoherent Head, and Front Head, respectively. The input to Main Head is features F1, F2, and f...max-recept soft , for final dense prediction. As the main head, Main Head exists in the whole process of training and testing. Main Head decodes the input features using the Upernet structure composed of FPN feature pyramid network and PPM pyramid pooling module. Aux Head, Incoherent Head and Front Head are auxiliary decoding heads, which only exist in the training process and are responsible for completing specific tasks: the input of Aux Head is feature F3, which is used to generate a reference prediction map to adjust the final prediction of the network; the input of Incoherent Head is feature f reversal , which is used to make the feature map as close to the edge as possible; the input of Front Head is feature F3, which is used to increase the attention of the network to the foreground pixels. The three auxiliary decoding heads Aux Head, Incoherent Head and Front Head all use FCN structure for decoding. The training is supervised by binary cross-entropy loss functions l main , l aux , l incoherent and l front , and the total loss of model training is S, S = a l main + e l aux + g l incoherent + d l front , where the learnable parameters a, e, g and d represent the weights of each loss function. According to the experimental results, the weights of the loss functions in the present application are a = 1, e = 0.4, g = 1 and d = 1.

[0035] The present application is implemented, and the disclosed multi-modal semantic segmentation dataset MFNet and PST900 are used as verification data to illustrate the specific implementation process.

[0036] Comparative experimental result analysis: the test set is input into the trained network model, and mIoU is used to measure the detection accuracy of the test model, where mIoU represents the average intersection over union. Table 1 shows the performance comparison of the present application and 11 mainstream deep learning semantic segmentation networks on the MFNet dataset, where SSNet (Segfomer-b5) represents the method proposed in the present application. The dataset has 9 classes: car, person, bicycle, curve, station, guardrail, warning line, convex and unlabeled label. The comparative results show that the present application method brings significant performance improvement, with a detection accuracy of 59.9% in mIoU.

[0037] Table 1: mIoU value comparison of the present method and other algorithms on MFNet (%)

[0038] ​

[0039] Table 1

[0040] The model complexity of the present application and other algorithms is given in Table 2, wherein SSNet (Segformer-b5) represents the method proposed in the present application. FLOPs represents the number of floating-point operations per second in actual network operation, which can be used to measure model complexity, and the size of the model is represented by the number of parameters. The present application can maintain high accuracy with a small number of parameters. The inference speed of the method of the present application is 22 frames / s on an NVIDIA GeForce RTX 30803, which can realize real-time inference.

[0041] Table 2: Comparison of model complexity of the present application and other algorithms

[0042]

[0043] Table 2

[0044] Comparison of generalization experiment results: Table 3 gives the comparison results of the present method and other advanced algorithms on the PST900 dataset, wherein SSNet (Segformer-b5) represents the method proposed in the present application. The dataset has 5 classes: fire extinguisher, backpack, hand drill, survivor and unlabelled label. It can be seen that the semantic segmentation network model proposed in the present application also has good performance on other multi-modal datasets, verifying the generalization of the proposed model;

[0045] Table 3: Comparison of experimental results of the method of the present application and other algorithms on PST900

[0046]

[0047] Table 3

[0048] The above describes the preferred embodiments of the present application in detail. It should be understood that those skilled in the art can make many modifications and changes to the present application without creative labor according to the concept of the present application. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiment on the basis of the prior art according to the concept of the present application shall be within the protection scope determined by the claims.

Claims

1. A semantic segmentation method based on RGB-T image single-stream cross-modal fusion, characterized in that, The thermal image and the visible light image are input into the trained semantic segmentation model to obtain a segmentation result of each target class in the image, and the semantic segmentation method based on the single-stream cross-modal RGB-T image comprises the following steps: Image fusion: performing pixel-level fusion on the input visible light image and the thermal image to obtain a fused image I; The backbone feature extraction network extracts features of the fused image I. The backbone feature extraction network includes four layers of feature extraction encoders, denoted as Layeri(i=1, 2, 3, 4). The features extracted by each layer of feature extraction encoders are denoted as feature F i (i=1, 2, 3, 4). The receptive field expansion module is placed after the fourth layer feature extraction encoder Layer4 of the backbone feature extraction network, and the input is the feature F4, and the output is the maximum receptive field feature f max-recept ; Local-context inversion perception module, input features are F3 and F4, and output features are features f that pay more attention to edge detail information reversal and features f rich in two-modal semantic information soft ; The multi-head parallel decoder includes four parallel decoding heads for supervising learning tasks of different aspects of the network, which are respectively denoted as MainHead, Aux Head, Incoherent Head and Front Head; the input of MainHead is features F1, F2, f max-recept , f soft for final dense prediction; the input of Aux Head is feature F3 for generating a reference prediction map to adjust the final prediction of the network; the input of Incoherent Head is feature f reversal for making the feature map as close to the edge as possible; and the input of Front Head is feature F3 for increasing the attention of the network to foreground pixels.

2. The method of claim 1, wherein the RGB-T image based single stream cross- modality fusion semantic segmentation method is characterized by, The receptive field expansion module comprises a convolution operation, a pooling operation and a splicing operation; the receptive field expansion module performs down-sampling on the input feature F4 to obtain a feature After the receptive field expansion is completed, the down-sampled feature is up-sampled to restore the resolution, to obtain a restored feature The feature F4 is spliced with the feature to obtain a feature Finally, a layer of convolution is used to learn the feature to obtain a maximum receptive field feature f max-recept .

3. The method of claim 1, wherein the RGB-T image based single stream cross- modality fusion semantic segmentation method is characterized by, The local-context inversion perception module includes pixel-level addition and subtraction operations, splicing operations and up-sampling operations on the features; the input of the local-context inversion perception module is the feature F3 and the feature F4, the feature F4 is up-sampled to obtain a feature Up(F4) consistent with the dimension of F3, and then the difference between F3 and Up(F4) is convolved to obtain an inversion feature f reversal The expression is: f reversal = Conv(F3 - Up(F4)) Using the learnable parameter β to f reversal Assign a certain weight, and then add pixel-level with feature F3 to get semantic information feature f with rich detail information soft The expression is: f soft = F3 + β x f reversal。 4. The method of claim 1, wherein the RGB-T image based single stream cross- modality fusion semantic segmentation method is characterized by, The multi-head parallel decoder is used for learning tasks of different aspects of the network; the Main Head is a main decoding head and exists in the whole training and testing process; the Aux Head, the Incoherent Head and the Front Head are all auxiliary decoding heads and only exist in the training process; four loss functions are included, which are respectively a loss l main for the main decoding head aux , a loss l incoherent for the Aux Head front , a loss l main for the Incoherent Head aux and a loss l front for the Front Head incoherent ; and l main , l aux , l incoherent , l front , l main , l aux and l front are all binary cross-entropy loss functions and are defined as: where N is the number of samples, y i represents the label of sample i, 1 for positive class and 0 for negative class, p i represents the probability that sample i is predicted as positive class. The weights of the loss functions are represented by learnable parameters a, e, g and d; The total loss of the model training is S: S = αl main +εl aux +γl incoherent +δl front。

Images