An RGB-T bimodal feature fusion semantic segmentation method

Abstract

The present application belongs to the technical field of semantic segmentation, and in particular to a kind of RGB-T bimodal feature fusion's semantic segmentation method, comprising the following steps: step 1, construct network model: the whole network is mainly composed of two parts of double-branch encoder and decoder, attention fusion module;Step 2, prepare data set: the whole convolutional network is first trained using semantic segmentation data set;Step 3, train network model: the data set prepared in step 2 is input into the network model constructed in step 1 for training.The present application, when extracting infrared feature and visible light image feature information, layer by layer, the output feature map is transmitted into the attention mechanism bimodal fusion module, the input model is given, the spatial and channel features are enriched through the attention mechanism, then more shallow detail information is provided for the decoder network through skip connection, redundant feature information is removed, excellent semantic segmentation performance is obtained, and the segmentation effect and precision are better.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of semantic segmentation, in particular to an RGB-T bimodal feature fusion semantic segmentation method. BACKGROUND

[0002] Image semantic segmentation is to give a semantic category value of each pixel point in an image, semantic segmentation of the image can obtain semantic information of the image, and the high-level content and represented information of the image are better understood. However, the complexity of the image semantic segmentation task leads to huge spatial complexity and calculation complexity of the network. In order to maximize the complementary advantages of multiple data groups and reduce the difficulty of data fusion, a feature fusion method based on deep learning is proposed. At present, good results have been achieved in the RGB image segmentation based on deep learning. However, in a complex scene, due to many object types, large light changes, easy occlusion and other factors, the RGB image semantic segmentation task has problems such as serious boundary blur, intra-class misrecognition and small target object loss.

[0003] A Chinese patent with the publication number CN107564025B and the name of a power equipment infrared image semantic segmentation method based on a deep neural network is disclosed. The method first inputs an infrared image. Then, the features of the input image are extracted through a convolution module operation, and multi-scale feature extraction is performed through a spatial pyramid pooling module. Then, the feature recovery is performed by using an up-sampling module. Finally, a fully connected conditional random field is selected for post-processing, the final semantic segmentation result image is smoothed and the edge is optimized to output the semantic segmentation result. The semantic segmentation image obtained by the method has low segmentation accuracy, the model parameters are complex, and the image feature information is not rich enough. SUMMARY

[0004] (I) Technical problems to be solved

[0005] In view of the defects of the prior art, the application provides an RGB-T bimodal feature fusion semantic segmentation method, which solves the problems of insufficient image feature information, low segmentation accuracy and complex model parameters.

[0006] (II) Technical scheme

[0007] In order to achieve the above purpose, the application specifically adopts the following technical scheme:

[0008] An RGB-T bimodal feature fusion semantic segmentation method comprises the following steps:

[0009] Step 1, constructing a network model: the whole network mainly comprises a double-branch encoder and a decoder, and an attention fusion module;

[0010] Step 2, preparing the data set: the entire convolutional network is first trained with the semantic segmentation data set;

[0011] Step 3, training the network model: input the data set prepared in step 2 into the network model constructed in step 1 for training;

[0012] Step 4, selecting the best loss function and optimal evaluation index: by minimizing the loss function of the network output image and the label, until the training times reach the set threshold or the value of the loss function reaches the set range, it can be considered that the model parameters have been pre-trained, and the model parameters are saved; The optimal evaluation index selects the pixel accuracy, the average intersection over union, the parameter size and the inference speed, which can evaluate the accuracy and real-time performance of the algorithm and measure the effect of network semantic segmentation;

[0013] Step 5, fine-tuning the model: train and fine-tune the model with infrared and visible light images to obtain stable and usable model parameters, and finally further improve the visible light and infrared image semantic segmentation capability of the model; Finally, the image segmentation effect of the model is more accurate;

[0014] Step 6, save the model: solidify the final determined model parameters, and then when visible light and infrared image semantic segmentation is needed, directly input the image into the network to obtain the final semantic segmentation image.

[0015] Further, the linear activation function is used in the convolution block one activation function in the visible light image encoder in step 1, the S-shaped function is used in the convolution block two, convolution block three, convolution block four and convolution block five activation function, the linear rectifier function is used in the convolution block six activation function in the infrared image encoder, the S-shaped function is used in the convolution block seven, convolution block eight, convolution block nine and convolution block ten activation function, and the linear activation function is used in the convolution block eleven, convolution block twelve, convolution block thirteen, convolution block fourteen, convolution block fifteen and convolution block sixteen activation function; The S-shaped function is used in the attention fusion module activation function, the size of the convolution kernel in all convolution blocks is uniform n x n; The size of all feature maps is consistent with the size of the input image.

[0016] Further, the double-branch encoder in step 1 is a visible light image encoder and an infrared image encoder, and the attention fusion module is divided into two parts of an attention mechanism and a fusion module; the convolution block one encodes the input visible light picture to extract primary features, and the convolution block two, the convolution block three, the convolution block four and the convolution block five extract high-level features in the visible light image; the convolution block six encodes the input infrared picture to extract primary features, and the convolution block seven, the convolution block eight, the convolution block nine and the convolution block ten extract high-level features in the infrared image; the double-modal feature information is input into the spatial pyramid pooling layer one and the spatial pyramid pooling layer two respectively for multi-scale information processing, and is output to the convolution block eleven for decoding operation; and the visible light feature information of the convolution block two, the convolution block three, the convolution block four and the convolution block five and the infrared image feature information of the convolution block seven, the convolution block eight, the convolution block nine and the convolution block ten are input into the attention fusion module for complementary fusion of feature information, and are output to the convolution block twelve, the convolution block thirteen, the convolution block fourteen and the convolution block fifteen respectively for jump connection with the decoder, and finally the convolution block sixteen is used for classification and segmentation to obtain a final semantic segmentation result picture.

[0017] Further, in step 3, the MFNet dataset is used in the pre-training process; the four-channel pictures in the dataset are separated to obtain visible light color images and infrared images; images with complex scenes, various details and complete categories are selected as training samples, and the remaining images are used as test set samples; the visible light images and the infrared images are input into the network respectively for training.

[0018] Further, in step 4, the cross-entropy loss function is selected in the training process; the selection of the loss function affects the quality of the model, can truly reflect the difference between the predicted value and the true value, and can correctly feedback the quality of the model.

[0019] Further, in step 5, the SODA dataset is used in the process of fine-tuning the model parameters.

[0020] (Three) beneficial effects

[0021] Compared with the prior art, the present application provides an RGB-T double-modal feature fusion semantic segmentation method, which has the following beneficial effects:

[0022] In the present application, when extracting infrared feature and visible light image feature information, the output feature map is transmitted into the attention mechanism double-modal fusion module layer by layer, the spatial and channel features are enriched through the given input model through the attention mechanism, then the shallow detail information is provided for the decoder network through the jump connection, the redundant feature information is removed, and excellent semantic segmentation performance is obtained, and the segmentation effect and accuracy are better.

[0023] The application uses dilated convolution of convolutional blocks two to five and convolutional blocks seven to ten in the encoder backbone network, so that the receptive field becomes larger, the network parameters are reduced, the depth of the network becomes shallower, and finally the entire network realizes simple structure, improves segmentation efficiency, and the spatial pyramid pooling module is designed for processing object segmentation problems in multiple scales.

[0024] The application uses dual-modal feature fusion, visible light and infrared dual-flow feature information extraction, richer feature information, and attention fusion modules to make the feature information complementary and obtain deeper, more accurate and rich semantic information. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The application is a semantic segmentation method flowchart;

[0026] Figure 2 The application is a semantic segmentation method network structure diagram;

[0027] Figure 3 The application is a specific composition diagram of the attention module;

[0028] Figure 4 The application is a specific composition diagram of the attention fusion module;

[0029] Figure 5 The application is a specific composition diagram of the decoding unit. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0031] EMBODIMENT

[0032] Embodiment one: as shown in the figure, an RGB-T dual-modal feature fusion semantic segmentation method is proposed in one embodiment of the application, including the following steps: Figures 1-5

[0033] Step 1, constructing a network model: the entire network mainly consists of a dual-branch encoder and a decoder, and an attention fusion module;

[0034] ​The dual-branch encoder is a visible light image encoder and an infrared image encoder, the attention fusion module is divided into two parts of an attention mechanism and a fusion module, the first convolutional block extracts primary features from the input visible light image, the second convolutional block, the third convolutional block, the fourth convolutional block and the fifth convolutional block extract high-level features from the visible light image, the sixth convolutional block extracts primary features from the input infrared image, the seventh convolutional block, the eighth convolutional block, the ninth convolutional block and the tenth convolutional block extract high-level features from the infrared image, the dual-modal feature information is input into the first spatial pyramid pooling layer and the second spatial pyramid pooling layer for multi-scale information processing, and is output to the eleventh convolutional block for decoding operation, and the visible light feature information of the second convolutional block, the third convolutional block, the fourth convolutional block and the fifth convolutional block and the infrared image feature information of the seventh convolutional block, the eighth convolutional block, the ninth convolutional block and the tenth convolutional block are input into the attention fusion module for complementary fusion of feature information, and are output to the twelfth convolutional block, the thirteenth convolutional block, the fourteenth convolutional block and the fifteenth convolutional block for jump connection with the decoder, and finally the sixteenth convolutional block is used for classification and segmentation to obtain a final semantic segmentation result image.

[0035] In the step 1, the activation function of the first convolutional block in the visible light image encoder uses a linear activation function, the activation functions of the second convolutional block, the third convolutional block, the fourth convolutional block and the fifth convolutional block use an S-shaped function, the activation function of the sixth convolutional block in the infrared image encoder uses a linear rectifier function, the activation functions of the seventh convolutional block, the eighth convolutional block, the ninth convolutional block and the tenth convolutional block use an S-shaped function, the activation functions of the eleventh convolutional block, the twelfth convolutional block, the thirteenth convolutional block, the fourteenth convolutional block, the fifteenth convolutional block and the sixteenth convolutional block use linear activation functions, the activation function of the attention fusion module uses an S-shaped function, and the size of the convolution kernel in all convolutional blocks is uniform as n*n; the size of all feature maps is consistent with the size of the input image.

[0036] In the step 2, the data set is prepared: the entire convolutional network is trained using a semantic segmentation data set;

[0037] In the pre-training process, the semantic segmentation data set uses the MFNet data set; the visible light color image and the infrared image are obtained by separating the four-channel image channels of the data set, the images with complex scenes, various details and complete categories are selected as training samples, the remaining images are used as test set samples, and the visible light image and the infrared image are input into the network for training;

[0038] In the step 3, the network model is trained: the data set prepared in the step 2 is input into the network model constructed in the step 1 for training;

[0039] Step 4, select the optimal loss function and optimal evaluation index: by minimizing the loss function of the network output image and the label, until the training times reach the set threshold or the value of the loss function reaches the set range, it can be considered that the model parameters have been pre-trained, and the model parameters are saved; At the same time, the optimal evaluation index is selected to measure the accuracy of the algorithm and evaluate the performance of the system; The selection of the loss function affects the quality of the model, which can truly reflect the difference between the predicted value and the true value, and correctly feedback the quality of the model; The optimal evaluation index selects the pixel accuracy, average intersection over union, parameter size and inference speed, which can evaluate the accuracy and real-time performance of the algorithm and measure the effect of network semantic segmentation;

[0040] Step 5, fine-tune the model: train and fine-tune the model with infrared and visible light images to make the model better for fusion; In the process of fine-tuning the model parameters, use the SODA dataset to get stable and usable model parameters; Finally, the model parameters are further improved to improve the visible light and infrared image semantic segmentation ability of the model; Finally, the model can more accurately segment the image;

[0041] Step 6, save the model: solidify the finally determined model parameters, and then when visible light and infrared image semantic segmentation is needed, directly input the image into the network to get the final semantic segmentation image.

[0042] As shown in Figure 1 In some embodiments, the linear activation function is used in the convolution block one activation function in the visible light image encoder in step 1, the S-shaped function is used in the convolution block two, convolution block three, convolution block four and convolution block five activation function, the linear rectifier function is used in the convolution block six activation function in the infrared image encoder, the S-shaped function is used in the convolution block seven, convolution block eight, convolution block nine and convolution block ten activation function, and the linear activation function is used in the convolution block eleven, convolution block twelve, convolution block thirteen, convolution block fourteen, convolution block fifteen and convolution block sixteen activation function; The S-shaped function is used in the attention fusion module activation function, the size of the convolution kernel in all convolution blocks is uniform n x n; The size of all feature maps is consistent with the size of the input image.

[0043] As shown in Figure 1 and 2As shown, in some embodiments, the dual-branch encoder in step 1 is a visible light image encoder and an infrared image encoder. The attention fusion module consists of an attention mechanism and a fusion module. Convolutional block one of the visible light image encoder performs primary feature extraction on the input visible light image, while convolutional blocks two, three, four, and five extract high-level features from the visible light image. Convolutional block six of the infrared image encoder performs primary feature extraction on the input infrared image, while convolutional blocks seven, eight, nine, and ten extract high-level features from the infrared image, providing dual-modal features. The information is processed in spatial pyramid pool layer 1 and spatial pyramid pool layer 2 for multi-scale information processing, and then output to convolutional block 11 for decoding. The visible light features from convolutional blocks 2, 3, 4, and 5, and the infrared features from convolutional blocks 7, 8, 9, and 10 are input to the attention fusion module for complementary feature fusion, and then output to convolutional blocks 12, 13, 14, and 15 for skip connections with the decoder. Finally, the semantic segmentation result is obtained by classification and segmentation through convolutional block 16.

[0044] like Figure 1 and 3 As shown, in some embodiments, the semantic segmentation dataset used in step 3 during the pre-training process is the MFNet dataset; visible light color images and infrared images are obtained by separating the four-channel images of the dataset; images with complex scenes, diverse details, and complete categories are selected as training samples, and the remaining images are used as test set samples; visible light images and infrared images are used as input networks for training.

[0045] like Figure 1 and 4 As shown, in some embodiments, the cross-entropy loss function is selected as the loss function in step 4 during the training process; the choice of loss function affects the quality of the model, can truly reflect the difference between the predicted value and the true value, and can correctly reflect the quality of the model.

[0046] like Figure 1 and 5 As shown, in some embodiments, the SODA dataset is used in step 5 during the fine-tuning of model parameters.

[0047] Example 2: The network model structure in step 1 is as follows Figure 2As shown, the network model includes a total of 16 convolutional blocks, an attention fusion module, a spatial pyramid pooling layer, convolutional block one and convolutional block six are composed of one layer of convolution and linear rectifier function, the convolution kernel size is 3x3, the step and padding are both 1, the compositions of convolutional block two, convolutional block three, convolutional block four, convolutional block five, convolutional block seven, convolutional block eight, convolutional block nine and convolutional block ten are the same, which contain two layers of convolution and two times of activation, the convolution kernel size is 3x3, the step and padding are both 1, the expansion rates of convolutional block two, convolutional block three, convolutional block four and convolutional block five are 6, 12, 18 and 24 respectively, the expansion rates of convolutional block seven, convolutional block eight, convolutional block nine and convolutional block ten are 6, 12, 18 and 24 respectively, the activation function selects a Sigmoid function, because the Sigmoid function can limit the output of the feature map to [0, 1], which will make the edge and content reconstruction ability of the network stronger, and fuse higher quality images, convolutional block one and convolutional block six only have one layer of convolution, the convolution kernel size is 3x3, the step and padding are both 1, and the linear rectifier function and the S-shaped function are defined as shown:

[0048]

[0049]

[0050] The attention mechanism module is as shown in (a) of Figure 3 The spatial attention module adopts average pooling layer one and maximum pooling layer one to pool the input feature map to ensure that the feature extraction is in a sufficient receptive field while retaining the background semantic information of the image, then the combined results are fused according to the same dimension to obtain the spatial attention weight, then the spatial attention feature map is obtained through convolution layer one and sigmoid normalization operation, the convolution kernel is 7x7, the channel attention module structure is as shown in (b) of Figure 3 The channel attention module adopts average pooling layer two and maximum pooling layer two to process the image in parallel, then increases the network depth through convolution layer two and convolution layer three, the convolution kernel is 1x1, then the feature information is processed through average pooling layer three and maximum pooling layer three, the feature information is spliced and fused, the attention weight on the channel is obtained through the activation function sigmoid, and finally the spatial attention feature information and the channel attention feature information are summed to obtain the final attention module weight.

[0051] The attention feature fusion module is as shown in Figure 4As shown in the figure, using the cross-multiplication mode, the visible light image feature information processed by the attention module is multiplied by the infrared image modal feature not processed by the attention module, the infrared image feature information processed by the attention module two is multiplied by the visible light image modal feature not processed by the attention module, and the complementarity of the feature information between the two modalities is enhanced, and finally the result of multiplying the two modal information is fused by addition and subtraction.

[0052] The decoder is composed of six identical decoder units, as shown in the figure. Figure 5 The convolution block of each decoder unit is composed of a convolution layer and an upsampling operation; the convolution layer includes convolution+BN layer and S-type activation function, the decoding unit first passes through the convolution layer one, the convolution kernel is 1*1; then through the convolution layer two, the convolution kernel is 3*1; then through the convolution layer three, the convolution kernel is 1*3; and the feature information before the convolution layer one is connected with the processing result of the convolution layer three by jump connection and addition operation, and finally the up-sampling is performed, the up-sampling adopts the bilinear interpolation method, and the image resolution is restored to the input size after the up-sampling operation to obtain the semantic segmentation image.

[0053] In step 2, the image data set uses MFNet data set; the data set contains 1569 pairs of RGB and infrared images, of which 820 are taken in daytime and 749 are taken at night, there are 9 manually labeled semantic classes, including the background class not labeled in the real background, and the image resolution of the data set is 480*640; the training set is composed of 50% of daytime images and 50% of nighttime images, the verification set is composed of 25% of daytime images and 25% of nighttime images, and the other images are used for testing.

[0054] In step 3, the image enhancement is performed on the pictures in the data set, the random diffraction transformation is performed on the same picture, and the picture is cropped to the size of the input picture as the input of the whole network, and the pictures labeled in the data set are used as labels, wherein the random size and position can be realized by software algorithm, and the pictures labeled in the data set are used as labels to enable the network to learn better feature extraction capability and finally achieve better segmentation effect.

[0055] In step 4, the output of the network and the label are used to calculate the loss function, and the better fusion effect is achieved by minimizing the loss function, and the cross-entropy loss function is selected as the loss function, and the calculation formula of the cross-entropy loss function is as follows:

[0056]

[0057] Wherein, C represents the cost, x represents the sample, y represents the actual value, a represents the output value, n represents the total number of samples, the pixel accuracy refers to the proportion of the predicted correct pixels in the total pixels, and the average intersection over union refers to the ratio of the intersection and the union of two sets, the two sets being the real value and the predicted value, and the pixel accuracy and the average intersection over union are calculated according to the following formula:

[0058]

[0059]

[0060] Wherein, k represents the number of pixel classes, p ii represents the number of correct pixels, p ij and p ji represent the number of false positives and false negatives.

[0061] The training number is set to 150, the number of pictures input to the network each time is about 8-16, and the upper limit of the number of pictures input to the network each time is mainly determined by the performance of the computer graphics processor. Generally, the larger the number of pictures input to the network each time, the better, so that the network is more stable. The learning rate of the training process is set to 0.0001, which can ensure that the network is quickly fitted without causing overfitting of the network. The network parameter optimizer selects the Adam optimizer, which has the advantages of simple implementation, high efficiency, low memory requirement, and stable parameter update unaffected by gradient scaling and transformation, so that the parameters are relatively stable. The loss function threshold is set to about 0.0005, and less than 0.0005 can be considered as the completion of the training of the entire network.

[0062] The step 5 uses the image data set to fine-tune the entire network model parameters, wherein the data set is the soda data set.

[0063] After the network training is completed in the step 6, all parameters in the network need to be saved, and then the infrared and visible light images to be segmented are input into the network to obtain the segmented images. The network has no size requirement for the two input images, and any size is acceptable, but the sizes of the two images must be consistent.

[0064] Wherein, the implementation of convolution, activation function, splicing operation, dilated convolution and spatial pyramid pooling is an algorithm known to those skilled in the art, and the specific process and method can be found in corresponding textbooks or technical documents.

[0065] The present application can generate a segmentation image directly by constructing an RGB-T bimodal feature fusion semantic segmentation network, without going through other steps, avoiding manual design of related segmentation rules. Under the same conditions, by calculating the relevant indicators of the image obtained by the existing method, the feasibility and superiority of the method are further verified. The relevant indicators of the prior art and the method proposed in the present application are compared as shown in Table 1.

[0066] Table 1 Comparison of relevant indicators of prior art and method proposed in the present application

[0067]

[0068] As can be seen from the table, the method proposed in the present application has higher image pixel accuracy and average intersection over union than the existing method, which further indicates that the method proposed in the present application has better segmentation image accuracy.

[0069] The parameters and running time of the prior art and the method proposed in the present application are shown in Table 2.

[0070] Table 2 Comparison of parameters and running time of prior art and method proposed in the present application

[0071]

[0072] As can be seen from the table, the running time of the method proposed in the present application is shortened by 26 milliseconds compared with the prior art, and the training parameters are less than the existing method, which further indicates that the method proposed in the present application has higher segmentation efficiency.

[0073] Finally, it should be noted that: the above only describes the preferred embodiments of the present application and does not limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1.A semantic segmentation method of RGB-T bimodal feature fusion, characterized in that: Comprising the following steps: Step 1, constructing a network model: the whole network mainly consists of a double-branch encoder and a decoder, and an attention fusion module; The double-branch encoder in step 1 is a visible light image encoder and an infrared image encoder, and the attention fusion module consists of an attention mechanism and a fusion module; Convolution block one of the visible light image encoder performs primary feature extraction on the input visible light picture, and convolution block two, convolution block three, convolution block four and convolution block five perform advanced feature extraction on the primary feature of the visible light image to obtain the advanced feature of the visible light image; Convolution block six of the infrared image encoder performs primary feature extraction on the input infrared picture, and convolution block seven, convolution block eight, convolution block nine and convolution block ten perform advanced feature extraction on the primary feature of the infrared image to obtain the advanced feature of the infrared image, the double-modal advanced features enter spatial pyramid pooling layer one and spatial pyramid pooling layer two respectively for multi-scale information processing, and are output to convolution block eleven for decoding operation, and the visible light feature information of convolution block two, convolution block three, convolution block four and convolution block five and the infrared image feature information of convolution block seven, convolution block eight, convolution block nine and convolution block ten are respectively input to the attention fusion module for complementary fusion of feature information, and are respectively output to convolution block twelve, convolution block thirteen, convolution block fourteen and convolution block fifteen for jump connection with the decoder, and finally the classification and segmentation are performed through convolution block sixteen to obtain the final semantic segmentation result image; Step 2, preparing the data set: the whole convolutional network is first trained with a semantic segmentation data set; Step 3, training the network model: input the data set prepared in step 2 into the network model constructed in step 1 for training; Step 4, selecting the best loss function and optimal evaluation index: by minimizing the loss function of the output image and the label, until the training times reach the set threshold or the value of the loss function reaches the set range, the model parameters are considered to have been pre-trained, and the model parameters are saved; The optimal evaluation index selects the pixel accuracy, average intersection over union, parameter size and inference speed, which can evaluate the accuracy and real-time performance of the algorithm and measure the effect of network semantic segmentation; Step 5, fine-tuning the model: training and fine-tuning the model with infrared and visible light images to obtain stable and usable model parameters, and finally further improving the visible light and infrared image semantic segmentation capability of the model; Finally, the image segmentation effect of the model is more accurate; Step 6, saving the model: solidify the final determined model parameters, and then when visible light and infrared image semantic segmentation is needed, directly input the image into the network to obtain the final semantic segmentation image. 2.The RGB-T dual-modal feature fusion semantic segmentation method of claim 1, characterized in that: The linear activation function is used in the convolution block one of the visible light image encoder in the step 1, the S-shaped function is used in the convolution block two, the convolution block three, the convolution block four and the convolution block five, the linear rectifier function is used in the convolution block six of the infrared image encoder, the S-shaped function is used in the convolution block seven, the convolution block eight, the convolution block nine and the convolution block ten, the linear activation function is used in the convolution block eleven, the convolution block twelve, the convolution block thirteen, the convolution block fourteen, the convolution block fifteen and the convolution block sixteen; the S-shaped function is used in the attention fusion module activation function, the size of the convolution kernel in all convolution blocks is unified to ; the size of all feature maps is consistent with the size of the input image. 3.The RGB-T dual-modal feature fusion semantic segmentation method of claim 1, characterized in that: In the pre-training process in step 3, the semantic segmentation data set uses the MFNet data set; The visible light color image and the infrared image are obtained by separating the four-channel pictures in the data set, the images with complex scene, various details and complete categories are selected as training samples, the remaining images are used as test set samples, and the visible light image and the infrared image are input into the network for training respectively. 4.The RGB-T dual-modal feature fusion semantic segmentation method of claim 1, characterized in that: In the step 4, the loss function selected in the training process is a cross-entropy loss function; the selection of the loss function affects the quality of the model, can truly reflect the difference between the predicted value and the true value, and can correctly feedback the quality of the model. 5.The RGB-T dual-modal feature fusion semantic segmentation method of claim 1, characterized in that: In the step 5, the SODA dataset is used in the process of fine-tuning the model parameters.

Images