Road scene analysis method based on color visible light and infrared image fusion

Abstract

The invention relates to the technical field of scene analysis, and discloses a road scene analysis method based on color visible light and infrared image fusion, which comprises the following steps of: constructing a training set comprising a plurality of pairs of color visible light images and infrared images corresponding to the color visible light images, and constructing a neural network; the neural network is mainly composed of a color information and infrared information interaction and high-level information guiding module, a color information and infrared information merging module, a high-level low-resolution characteristic dynamic amplification module and a high-performance segmentation decoder. Inputting the plurality of pairs of color visible light images in the training set and the infrared images corresponding to the color visible light images into a neural network for multiple rounds of network training, and obtaining a neural network training model after the network training is finished; predicting the test image pair by using the neural network training model, and predicting to obtain a scene analysis image of the test image pair; the method has the advantage that the accuracy of scene analysis can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of scene analysis, and in particular, relates to a road scene analysis method based on the fusion of color visible light and infrared images. Background Art

[0002] In autonomous driving and intelligent transportation systems, accurate recognition and analysis of road scenes is one of the important technologies for achieving safe driving. Traditional scene analysis methods mainly rely on single visible light images, but under complex lighting conditions (such as nighttime, strong light, etc.), the quality of visible light images is often limited, and information loss or reduced analysis accuracy may occur. In order to address these limitations, multimodal data fusion technology has gradually become a research hotspot, especially multimodal methods that combine color visible light and infrared images, which have shown significant advantages.

[0003] In the current multimodal scene analysis technology, it is first necessary to collect multimodal image data. Using dual-modal imaging devices, such as the InfRec R500 camera, color visible light and infrared image data can be obtained simultaneously. This camera can clearly capture the thermal radiation signals of objects at night or in bad weather conditions, making up for the lack of information in visible light images in specific environments such as insufficient light. By fusing color visible light images and infrared images, the complementary characteristics of the two types of data can be fully utilized, significantly improving the ability to recognize key objects such as vehicles, pedestrians, and obstacles in scene analysis.

[0004] However, achieving efficient multimodal image fusion still faces challenges. How to maintain the complementarity and consistency of different spectral information during the fusion process has become a core issue in multimodal scene analysis. Existing fusion methods are often limited to simple feature superposition or single-layer information interaction, and cannot fully explore the cross-layer information correlation of multimodal data, resulting in limited analysis results and difficulty in meeting high-precision requirements in complex scenes.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0007] A road scene analysis method based on the fusion of color visible light and infrared images includes the following steps:

[0008] Step S1: firstly, all data are divided into a training set and a test set, and then all training set images are input into the neural network constructed by the present invention, the output result of the network is compared with the real label, and the output result of the neural network is continuously optimized according to the loss function;

[0009] Step S2: After the training is completed, the model with the best performance is used to test the test images, and the performance of the obtained model on the test set is evaluated;

[0010] Step S3: The constructed neural network is mainly composed of a feature extraction module, a color information and infrared information interaction and high-level information guidance module, a color information and infrared information merging module, a high-level low-resolution feature dynamic amplification module and a high-performance segmentation decoder, wherein:

[0011] The feature extraction module includes two Swin Transformer backbone networks; the input end of the first layer of the first Swin Transformer backbone network receives a color visible light image of size H×W×3, and the feature map output by the output end of the first layer of the first Swin Transformer backbone network is denoted as FR 1 , the input of the second layer of the first SwinTransformer backbone network receives FR 1 , the feature map output from the output end of the second layer of the first Swin Transformer backbone network is denoted as FR 2 , the input of the third layer of the first Swin Transformer backbone network receives FR 2 , the feature map output from the output end of the third layer of the first Swin Transformer backbone network is denoted as FR 3 , the input of the fourth layer of the first Swin Transformer backbone network receives FR 3 , the feature map output from the output end of the fourth layer of the first Swin Transformer backbone network is denoted as FR 4 ; The input end of the first layer of the second Swin Transformer backbone network receives an infrared image of size H×W×3, and the feature map output by the output end of the first layer of the second Swin Transformer backbone network is denoted as FT 1 , the second layer of the second Swin Transformer backbone network receives FT 1 , the feature map output from the output end of the second layer of the second Swin Transformer backbone network is denoted as FT 2 , the input of the third layer of the second SwinTransformer backbone network receives FT 2 , the feature map output from the output end of the third layer of the second Swin Transformer backbone network is denoted as FT 3 , the input of the fourth layer of the second Swin Transformer backbone network receives FT 3, the feature map output from the output end of the fourth layer of the second Swin Transformer backbone network is denoted as FT 4 ;

[0012] Step S4: The Swin Transformer backbone network has four layers, FR 1 and FT 1 The size is FR 2 and FT 2 The size is FR 3 and FT 3 The size is FR 4 and FT 4 The size is In this embodiment, H×W is taken as 384×384;

[0013] The color information and infrared information interaction and high-level information guidance modules are four modules with the same structure, hereinafter referred to as interaction and guidance modules; the first input terminal of the first interaction and guidance module receives FR 1 , the second input terminal receives FT 1 The feature map output from the first output terminal of the first interaction and guidance module is denoted as FRA 1 The feature map output from the second output terminal is denoted as FTA 1 ; The first input terminal of the second interaction and guidance module receives FR 2 , the second input terminal receives FT 2 , the feature map output from the first output end of the second interaction and guidance module is denoted as FRA 2 The feature map output from the second output terminal is denoted as FTA 2 ; The first input terminal of the third interaction and guidance module receives FR 3 , the second input terminal receives FT 3 The feature map output from the first output of the third interaction and guidance module is denoted as FRA 3 The feature map output from the second output terminal is denoted as FTA 3 ; The first input terminal of the fourth interaction and guidance module receives FR 4 , the second input terminal receives FT 4 , the feature map output from the first output of the fourth interaction and guidance module is denoted as FRA 4 The feature map output from the second output terminal is denoted as FTA 4 Among them, FRA 1 With FTA 1 The size is FRA 2 With FTA 2The size is FRA 3 With FTA 3 The size is FRA 4 With FTA 4 The size is

[0014] Step S5: The color information and infrared information merging module comprises four modules with the same structure, hereinafter referred to as merging modules; the first input terminal of the first merging module receives FRA 1 , the second input terminal receives FTA 1 , the feature map output at the output of the first merging module is denoted as FM 1 ; The first input of the second merging module receives FRA 2 , the second input terminal receives FTA 2 , the feature map output at the output of the second merging module is denoted as FM 2 ; The first input of the third merging module receives FRA 3 , the second input terminal receives FTA 3 , the feature map output at the output of the third merging module is denoted as FM 3 ; The first input of the fourth merging module receives FRA 4 , the second input terminal receives FTA 4 , the feature map output at the output of the fourth merging module is denoted as FM 4 ; Among them, FM 1 The size is FM 2 The size is FM 3 The size is FM 4 The size is

[0015] Step S6: The high-level low-resolution feature dynamic amplification module is composed of three modules with the same structure, hereinafter referred to as amplification modules; the input end of the first amplification module receives FM 2 , the output of the first amplifier module outputs The input of the second amplifier module receives FM 3 , the output of the second amplifier module outputs The input of the third amplifier module receives FM 4 , the output of the fourth amplifier module outputs in The size is The size is The size is

[0016] The high-performance segmentation decoder is an existing Mask2Former segmentation decoder, and the input end of the high-performance segmentation decoder accepts FM 1 ,FM 2 ,FM 3 and FM 4 , the predicted scene parsing target image S output at the output end, where the size of S is

[0017] As a preferred embodiment of the present invention, the process of constructing a training set is: select at least 700 pairs of original color visible light images and their corresponding original infrared images; then downsample each original color visible light image and its corresponding original infrared image to downsample the image size to H×W; then all the color visible light images and their corresponding infrared images of size H×W constitute a training set; wherein, in this embodiment, 784 pairs of original color visible light images and their corresponding original infrared images are selected, and H=W=384.

[0018] As a preferred embodiment of the present invention, the process of obtaining the neural network training model is as follows: each pair of color visible light images and the corresponding infrared images in the training set are input into the neural network for network training, and the loss function L is calculated before each round of network training to optimize the neural network, and the neural network training model is obtained after a total of 200 rounds of network training; wherein the loss function L is divided into classification loss L cls and the mask loss L mask The classification loss can be expressed as L cls =L ce (S,G), the mask loss can be expressed as: L mask =5L focal (S,G)+5L dice (S,G), the total loss is L = 2L cls +L mask , L ce () represents the cross entropy (CE) loss, L focal () indicates focal loss, L dice () represents the region similarity (dice) loss.

[0019] As a preferred embodiment of the present invention, a neural network training model is used to predict a test image pair, and the process of predicting a scene resolution image of the test image pair is as follows: arbitrarily select a pair of original color visible light images and the corresponding original infrared images; then downsample the pair of original color visible light images and the original infrared images, downsample the image size to H×W, and use them as a test image pair; then input the test image pair into the neural network training model to predict a scene resolution image of the test image pair; wherein, H=W=384.

[0020] As a preferred embodiment of the present invention, the interaction and guidance module is mainly composed of the first channel-level feature learning module to the second channel-level feature learning module, the first spatial-level feature learning module to the second spatial-level feature learning module, and the seventh convolutional layer to the sixth convolutional layer; the channel-level feature learning module is composed of the first average pooling layer, the first maximum pooling layer, the first fully connected layer to the fourth fully connected layer, the first GELU activation layer to the second GELU activation layer, the first Dropout layer to the fourth Dropout layer, and the first Sigmoid activation layer; the spatial-level feature learning module is composed of the first convolutional layer to the sixth convolutional layer and the first ReLU activation layer.

[0021] As a preferred embodiment of the present invention, the merging module is mainly composed of the first flattening layer to the second flattening layer, the first linear layer to the second linear layer, the first attention mechanism layer to the second attention mechanism layer, the ninth convolution layer to the twelfth convolution layer, and the first batch normalization layer; for the i-th merging module, the input end of the first flattening layer receives FRA as the first input end of the i-th merging module i The input of the first linear layer receives the feature vector output by the output of the first flattening layer and outputs three vectors The input of the second flattening layer receives the FTA as the second input of the i-th merging module i The input of the second linear layer receives the feature vector output by the output of the second flattening layer and outputs three vectors The first attention layer receives as input The second attention layer receives as input

[0022] As a preferred embodiment of the present invention, the high-level low-resolution feature dynamic amplification module is mainly composed of the 1st deconvolution layer, the 13th convolution layer, the 3rd ReLU activation layer, and the 14th convolution layer; for the i-th high-level low-resolution feature dynamic amplification module, the input end of the 1st deconvolution layer receives the output of the i+1th merging module, the input end of the 13th convolution layer receives the output of the output end of the 1st deconvolution layer, the input end of the 3rd ReLU activation layer receives the output of the output end of the 13th convolution layer, and the input end of the 14th convolution layer receives the output of the output end of the 3rd ReLU activation layer. The output of the output end of the 14th convolution layer is recorded as The 13th to 14th convolutional layers are all 2D convolutional layers. In the first high-level low-resolution feature dynamic amplification module: the input size of the first deconvolution layer is The output size is The convolution kernel size is 2×2, the sliding step is 2, the automatic padding is 0, and the input size of the 13th convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the 14th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0; in the second high-level low-resolution feature dynamic amplification module: the input size of the first deconvolution layer is The output size is The convolution kernel size is 2×2, the sliding step is 2, the automatic padding is 0, and the input size of the 13th convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the 14th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0; in the third high-level low-resolution feature dynamic amplification module: the input size of the first deconvolution layer is The output size is The convolution kernel size is 2×2, the sliding step is 2, the automatic padding is 0, and the input size of the 13th convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the 14th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The present invention uses the Swin Transformer backbone network to extract four levels of features from color visible light images and infrared images. Feature fusion starts from the highest layer, and each layer of fusion is divided into two stages: the first stage realizes the information exchange of the two modal features; the second stage merges and integrates the features into the next layer of fusion to realize cross-layer information sharing. Finally, the four-layer fusion features are integrated through a high-performance decoder to significantly improve the segmentation accuracy of scene parsing.

[0025] In order to solve the problem of large differences in features between color visible light and infrared images, the present invention designs an information exchange mechanism based on channel and space levels in the first stage of feature fusion, which effectively reduces feature conflicts and the generation of noise features. At the same time, by introducing high-level features to guide information exchange, the bimodal features are more focused on the segmentation target area during the fusion process, thereby improving the quality of feature fusion.

[0026] In the second stage of fusion, the present invention builds a semantic relationship model between features to associate each position feature in the feature space with local and global features, thereby enhancing the model's ability to understand the target area. Finally, the two-way features are deeply fused and passed down layer by layer, realizing a complete cross-layer fusion network that can cope with the challenges brought by complex scenarios.

[0027] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In the attached picture:

[0029] Figure 1 It is the overall implementation framework diagram of the method of the present invention;

[0030] Figure 2 A schematic diagram of the scene analysis network structure for the fusion of color visible light and infrared images of the present invention;

[0031] Figure 3 It is a structural schematic diagram of the color information and infrared information interaction and high-level information guidance module of the present invention;

[0032] Figure 4 It is a structural schematic diagram of the color information and infrared information merging module of the present invention;

[0033] Figure 5 It is a schematic diagram of the structure of the high-level low-resolution feature dynamic amplification module of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The following embodiments are used to illustrate the present invention.

[0035] Example:

[0036] A road scene analysis method based on the fusion of color visible light and infrared images. The overall implementation framework is shown in the figure Figure 1 As shown in the figure, the method first divides all data into a training set and a test set, then inputs all training set images into the neural network constructed by the present invention, compares the network output with the true label, and continuously optimizes the output of the neural network according to the loss function. After the training, the model with the best performance is used to test the test images, and the performance of the obtained model on the test set is evaluated. Figure 2 As shown in the figure, the constructed neural network is mainly composed of feature extraction module, color information and infrared information interaction and high-level information guidance module, color information and infrared information merging module, high-level low-resolution feature dynamic amplification module and high-performance segmentation decoder, among which:

[0037] The feature extraction module includes two Swin Transformer backbone networks; the input end of the first layer of the first Swin Transformer backbone network receives a color visible light image of size H×W×3, and the feature map output by the output end of the first layer of the first Swin Transformer backbone network is denoted as FR 1 , the input of the second layer of the first SwinTransformer backbone network receives FR 1 , the feature map output from the output end of the second layer of the first Swin Transformer backbone network is denoted as FR 2 , the input of the third layer of the first Swin Transformer backbone network receives FR 2 , the feature map output from the output end of the third layer of the first Swin Transformer backbone network is denoted as FR 3 , the input of the fourth layer of the first Swin Transformer backbone network receives FR 3 , the feature map output from the output end of the fourth layer of the first Swin Transformer backbone network is denoted as FR 4 ; The input end of the first layer of the second Swin Transformer backbone network receives an infrared image of size H×W×3, and the feature map output by the output end of the first layer of the second Swin Transformer backbone network is denoted as FT 1 , the second layer of the second Swin Transformer backbone network receives FT 1, the feature map output from the output end of the second layer of the second Swin Transformer backbone network is denoted as FT 2 , the input of the third layer of the second SwinTransformer backbone network receives FT 2 , the feature map output from the output end of the third layer of the second Swin Transformer backbone network is denoted as FT 3 , the input of the fourth layer of the second Swin Transformer backbone network receives FT 3 , the feature map output from the output end of the fourth layer of the second Swin Transformer backbone network is denoted as FT 4 ; Among them, the Swin Transformer backbone network is an existing structural framework, and its network structure has been made public, as recorded in the literature Z.Liu, Y.Lin, Y.Cao, H.Hu, Y.Wei, Z.Zhang, S.Lin, and B.Guo, "Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows," in Proceedings of the IEEE / CVFInternational Conference on Computer Vision (ICCV), 2021, pp.10012–10022. ("Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows"), the Swin Transformer backbone network has four layers, FR 1 and FT 1 The size is FR 2 and FT 2 The size is FR 3 and FT 3 The size is FR 4 and FT 4 The size is In this embodiment, H×W is set to 384×384;

[0038] The color information and infrared information interaction and high-level information guidance modules are four modules with the same structure, hereinafter referred to as interaction and guidance modules; the first input terminal of the first interaction and guidance module receives FR 1 , the second input terminal receives FT 1 The feature map output from the first output terminal of the first interaction and guidance module is denoted as FRA 1The feature map output from the second output terminal is denoted as FTA 1 ; The first input terminal of the second interaction and guidance module receives FR 2 , the second input terminal receives FT 2 , the feature map output from the first output end of the second interaction and guidance module is denoted as FRA 2 The feature map output from the second output terminal is denoted as FTA 2 ; The first input terminal of the third interaction and guidance module receives FR 3 , the second input terminal receives FT 3 The feature map output from the first output of the third interaction and guidance module is denoted as FRA 3 The feature map output from the second output terminal is denoted as FTA 3 ; The first input terminal of the fourth interaction and guidance module receives FR 4 , the second input terminal receives FT 4 , the feature map output from the first output of the fourth interaction and guidance module is denoted as FRA 4 The feature map output from the second output terminal is denoted as FTA 4 Among them, FRA 1 With FTA 1 The size is FRA 2 With FTA 2 The size is FRA 3 With FTA 3 The size is FRA 4 With FTA 4 The size is

[0039] The color information and infrared information merging module comprises four modules with the same structure, hereinafter referred to as merging modules; the first input end of the first merging module receives FRA 1 , the second input terminal receives FTA 1 , the feature map output at the output of the first merging module is denoted as FM 1 ; The first input of the second merging module receives FRA 2 , the second input terminal receives FTA 2 , the feature map output at the output of the second merging module is denoted as FM 2 ; The first input of the third merging module receives FRA 3 , the second input terminal receives FTA 3 , the feature map output at the output of the third merging module is denoted as FM 3 ; The first input of the fourth merging module receives FRA 4 , the second input terminal receives FTA4 , the feature map output at the output of the fourth merging module is denoted as FM 4 ; Among them, FM 1 The size is FM 2 The size is FM 3 The size is FM 4 The size is

[0040] The high-level low-resolution feature dynamic amplification module is composed of three modules with the same structure, hereinafter referred to as amplification modules; the input end of the first amplification module receives FM 2 , the output of the first amplifier module outputs The input of the second amplifier module receives FM 3 , the output of the second amplifier module outputs The input of the third amplifier module receives FM 4 , the output of the fourth amplifier module outputs in The size is The size is The size is

[0041] The high-performance segmentation decoder is an existing Mask2Former segmentation decoder, whose specific structure has been made public, as described in the literature B. Cheng, I. Misra, AGSchwing, A. Kirillov, and R. Girdhar, "Masked-AttentionMask Transformer for Universal Image Segmentation," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 1290–1299. (Masked-Attention Mask Transformer for Universal Image Segmentation). The input end of the high-performance segmentation decoder accepts FM 1 ,FM 2 ,FM 3 and FM 4 , the predicted scene parsing target image S output at the output end, where the size of S is

[0042] Furthermore, in a specific embodiment, the process of constructing the training set is: select at least 700 pairs of original color visible light images and their corresponding original infrared images; then downsample each original color visible light image and its corresponding original infrared image to downsample the image size to H×W; then all color visible light images of size H×W and their corresponding infrared images constitute a training set; wherein, in this embodiment, 784 pairs of original color visible light images and their corresponding original infrared images are selected, and H=W=384.

[0043] Furthermore, in a specific embodiment, the process of obtaining the neural network training model is as follows: each pair of color visible light images and the corresponding infrared images in the training set are input into the neural network for network training, and the loss function L is calculated before each round of network training to optimize the neural network, and the neural network training model is obtained after a total of 200 rounds of network training; wherein the loss function L is divided into classification loss L cls and the mask loss L mask The classification loss can be expressed as L cls =L ce (S,G), the mask loss can be expressed as L mask =5L focal (S,G)+5L dice (S,G), the total loss is L = 2L cls +L mask , L ce () represents the cross entropy (CE) loss, L focal () indicates focal loss, L dice () represents the region similarity (dice) loss.

[0044] Furthermore, in a specific embodiment, a neural network training model is used to predict a test image pair, and the process of predicting a scene resolution image of the test image pair is as follows: arbitrarily select a pair of original color visible light images and the corresponding original infrared images; then downsample the pair of original color visible light images and the original infrared images, downsample the image size to H×W, and use them as a test image pair; then input the test image pair into the neural network training model to predict a scene resolution image of the test image pair; wherein, H=W=384.

[0045] Furthermore, the structures of the four interaction and guidance modules are the same, except that the inputs and outputs are different. In a specific embodiment, Figure 3As shown, the interaction and guidance module is mainly composed of the first channel-level feature learning module to the second channel-level feature learning module, the first spatial-level feature learning module to the second spatial-level feature learning module, and the seventh convolutional layer to the sixth convolutional layer; the channel-level feature learning module is composed of the first average pooling layer, the first maximum pooling layer, the first fully connected layer to the fourth fully connected layer, the first GELU activation layer to the second GELU activation layer, the first Dropout layer to the fourth Dropout layer, and the first Sigmoid activation layer; the spatial-level feature learning module is composed of the first convolutional layer to the sixth convolutional layer and the first ReLU activation layer. The two spatial-level feature learning modules have the same structure as the two channel-level feature learning modules, except that the input and output are different. For the i-th interaction and guidance module, the two channel-level feature learning modules respectively receive the two input FRs of the i-th interaction and guidance module. i and FT i , the first average pooling layer receives FR i and FT i , the first fully connected layer receives the feature vector output by the first average pooling layer, the first GELU activation layer receives the feature vector output by the first fully connected layer, the first Dropout layer receives the feature vector output by the first GELU activation layer, the second fully connected layer receives the feature vector output by the first Dropout layer, and the second Dropout layer receives the feature vector output by the second fully connected layer. The feature vector output by the second Dropout layer is denoted as FRV i and FTV i , the first maximum pooling layer receives FR i and FT i , the third fully connected layer receives the feature vector output by the first maximum pooling layer, the second GELU activation layer receives the feature vector output by the three fully connected layers, the third Dropout layer receives the feature vector output by the second GELU activation layer, the fourth fully connected layer receives the feature vector output by the third Dropout layer, and the fourth Dropout layer receives the feature vector output by the fourth fully connected layer. The feature vector output by the second Dropout layer is and and and They are added element by element, and the first Sigmoid activation layer receives the result of the addition. The feature vectors output by the first Sigmoid activation layer are FRV i and FTV i , the eigenvector FRV iThe three inputs of the interaction and guidance modules are weighted respectively, and the weighted results are concatenated. The seventh convolutional layer receives the concatenated feature map, and the feature vector FTV i The three inputs of the interaction and guidance module are weighted by channels respectively, and the weighted results are concatenated by channels. The eighth convolutional layer receives the concatenated feature map, and the two spatial feature learning modules receive the two input FRs of the i-th interaction and guidance module respectively. i and FT i , the first convolution layer, the second convolution layer, the third convolution layer, and the fourth convolution layer receive FR i and FT i , the outputs of the four convolutional layers are added element by element, and the feature map obtained after the addition is input into the fifth convolutional layer. The first ReLU activation layer receives the output of the fifth convolutional layer, and the sixth convolutional layer receives the output of the first ReLU activation layer. The output feature maps of the sixth convolutional layer are FRS i and FTS i , for FR i , FTS i , the output of the 7th convolutional layer is added element by element, and the resulting feature map is denoted as FRA i , for FT i 、FRS i , the output of the 8th convolutional layer is added element by element, and the resulting feature map is denoted as FTA i , FRA i and FTA i are the first and second outputs of the interaction and merging modules respectively; where 1≤i≤4, FR i Represents the feature map output by the output of the i-th layer of the first Swin Transformer backbone network, FT i Represents the feature map of the output of the i-th layer of the second SwinTransformer backbone network; the first to the eighth convolutional layers are all 2D convolutional layers. In the first interaction and guidance module: the input size of the first convolutional layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the second convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the third convolution layer is The output size is The convolution kernel size is 5×5, the sliding step is 1, the automatic padding is 2, and the input size of the fourth convolution layer is The output size is The convolution kernel size is 7×7, the sliding step is 1, the automatic padding is 3, and the input size of the 5th and 6th convolutional layers is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 7th and 8th convolutional layers is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0; in the second interaction and guidance module: the input size of the first convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the second convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the third convolution layer is The output size is The convolution kernel size is 5×5, the sliding step is 1, the automatic padding is 2, and the input size of the fourth convolution layer is The output size is The convolution kernel size is 7×7, the sliding step is 1, the automatic padding is 3, and the input size of the 5th and 6th convolutional layers is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 7th and 8th convolutional layers is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0; in the third interaction and guidance module: the input size of the first convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the second convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the third convolution layer is The output size is The convolution kernel size is 5×5, the sliding step is 1, the automatic padding is 2, and the input size of the fourth convolution layer is The output size is The convolution kernel size is 7×7, the sliding step is 1, the automatic padding is 3, and the input size of the 5th and 6th convolutional layers is The output size is The convolution kernel size is 1×1, the sliding step is 0, the automatic padding is 0, and the input size of the 7th and 8th convolutional layers is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0; in the fourth interaction and guidance module: the input size of the first convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the second convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the third convolution layer is The output size is The convolution kernel size is 5×5, the sliding step is 1, the automatic padding is 2, and the input size of the fourth convolution layer is The output size is The convolution kernel size is 7×7, the sliding step is 1, the automatic padding is 3, and the input size of the 5th and 6th convolutional layers is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 7th and 8th convolutional layers is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0;

[0046] Here, channel weighting operation, element-by-element addition operation, channel connection operation (Concatenation), Dropout, full connection, GELU activation, ReLU activation, Sigmoid activation, maximum pooling, and average pooling are all common operations in neural networks;

[0047] Furthermore, the four merge modules have the same structure, but different inputs and outputs. In a specific embodiment, Figure 4 As shown in Figure 1, the merging module mainly consists of the 1st flattening layer to the 2nd flattening layer, the 1st linear layer to the 2nd linear layer, the 1st attention mechanism layer to the 2nd attention mechanism layer, the 9th convolution layer to the 12th convolution layer, and the 1st batch normalization layer; for the i-th merging module, the input end of the first flattening layer receives FRA as the first input end of the i-th merging module i The input of the first linear layer receives the feature vector output by the output of the first flattening layer and outputs three vectors The input of the second flattening layer receives the FTA as the second input of the i-th merging module i The input of the second linear layer receives the feature vector output by the output of the second flattening layer and outputs three vectors The first attention layer receives as input The second attention layer receives as input The attention mechanism layer is a common operation in neural networks, and its operation method has been made public, such as A. Vaswani, S. Shardlow, N. Parmar, J. Uszkoreit, L. Jones, ANGomez, L. Kaiser, S. Kattner, NK, I. Polosukhin, W. Wang, Y. Wu, N. Yang, and Y. Zhang, “Attention is All You Need,” in Advances in Neural Information Processing. Systems (NeurIPS), 2017, pp.5998–6008. (“Attention is all you need”); the output of the first flattening layer and the output of the first attention mechanism layer are concatenated, the input of the ninth convolutional layer receives the concatenation result, and the output of the first flattening layer and the output of the ninth convolutional layer are element-by-element added; the output of the second flattening layer and the output of the second attention mechanism layer are concatenated, the input of the tenth convolutional layer receives the concatenation result, and the output of the second flattening layer and the output of the tenth convolutional layer are element-by-element added; the result of the element-by-element addition of the output of the first flattening layer and the output of the ninth convolutional layer is concatenated with the result of the element-by-element addition of the output of the second flattening layer and the output of the tenth convolutional layer, and the concatenation result is denoted as FA. i , the input of the 11th convolutional layer receives FA i , the input of the second ReLU receives the output of the output of the 11th convolutional layer, the input of the 12th convolutional layer receives the output of the second ReLU output, and the input of the first batch normalization layer receives the output of the 12th convolutional layer output. i Perform an element-by-element addition operation on the output of the first batch normalization layer, and record the result of the addition as FM i ; The 9th to 12th convolutional layers are all 2D convolutional layers. In the first merge module: the input size of the 9th convolutional layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 10th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 11th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 12th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic padding is 0; in the second merge module: the input size of the 9th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 10th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 11th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 12th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic padding is 0; in the third merge module: the input size of the 9th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 10th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 11th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 12th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic padding is 0; in the 4th merging module: the input size of the 9th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 10th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 11th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, the automatic padding is 0, and the input size of the 12th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0;

[0048] Here, flattening layer, channel concatenation, element-by-element addition operation, normalization layer, and ReLU activation are all common operations in neural networks;

[0049] Furthermore, the structures of the three high-level low-resolution feature dynamic amplification modules are the same, except for the input and output. In a specific embodiment, Figure 5 As shown in FIG. 1 , the high-level low-resolution feature dynamic amplification module is mainly composed of the 1st deconvolution layer, the 13th convolution layer, the 3rd ReLU activation layer, and the 14th convolution layer. For the i-th high-level low-resolution feature dynamic amplification module, the input end of the 1st deconvolution layer receives the output of the i+1th merging module, the input end of the 13th convolution layer receives the output of the 1st deconvolution layer output end, the input end of the 3rd ReLU activation layer receives the output of the 13th convolution layer output end, and the input end of the 14th convolution layer receives the output of the 3rd ReLU activation layer output end. The output of the 14th convolution layer output end is recorded as The 13th to 14th convolutional layers are all 2D convolutional layers. In the first high-level low-resolution feature dynamic amplification module: the input size of the first deconvolution layer is The output size is The convolution kernel size is 2×2, the sliding step is 2, the automatic padding is 0, and the input size of the 13th convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the 14th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0; in the second high-level low-resolution feature dynamic amplification module: the input size of the first deconvolution layer is The output size is The convolution kernel size is 2×2, the sliding step is 2, the automatic padding is 0, and the input size of the 13th convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the 14th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0; in the third high-level low-resolution feature dynamic amplification module: the input size of the first deconvolution layer is The output size is The convolution kernel size is 2×2, the sliding step is 2, the automatic padding is 0, and the input size of the 13th convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the 14th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0.

[0050] Experimental example:

[0051] In order to further illustrate the feasibility and effectiveness of the method of the present invention, the method of the present invention was tested.

[0052] In this embodiment, the method of the present invention is used to test the color visible light and infrared image scene parsing dataset (MFNet) established by Qishen Ha et al. The training set in MFNet includes 784 pairs of color visible light images and infrared images, and the test set in MFNet has a total of 393 pairs of color visible light images and infrared images, that is, 393 pairs of test image pairs.

[0053] In this embodiment, four commonly used objective parameters are selected to evaluate the performance of the method of the present invention, namely, Acc, IoU, mAc, and mIoU. Table 1 shows the correlation between the scene parsing image and the label image obtained by the method of the present invention on the MFNet dataset.

[0054] Table 1 Acc, IoU, mAcc, and mIoU between scene parsing images and label images obtained using the method of the present invention on the MFNet dataset

[0055]

[0056] From the results given in Table 1, it can be found that the method of the present invention has achieved higher Acc, IoU, mAcc, and mIoU on the existing color visible light and infrared image scene parsing datasets, which means that the scene parsing image obtained by the method of the present invention is relatively close to the label image, and the method of the present invention can effectively complete the scene parsing of color visible light and infrared images.

Claims

1. A road scene analysis method based on the fusion of color visible light and infrared images, characterized in that: The steps include: Step S1: firstly, all data are divided into a training set and a test set, and then all training set images are input into the neural network constructed by the present invention, the output result of the network is compared with the real label, and the output result of the neural network is continuously optimized according to the loss function; Step S2: After the training is completed, the model with the best performance is used to test the test images, and the performance of the obtained model on the test set is evaluated; Step S3: The constructed neural network is mainly composed of a feature extraction module, a color information and infrared information interaction and high-level information guidance module, a color information and infrared information merging module, a high-level low-resolution feature dynamic amplification module and a high-performance segmentation decoder, wherein: The feature extraction module includes two Swin Transformer backbone networks; the input end of the first layer of the first Swin Transformer backbone network receives a color visible light image of size H×W×3, the feature map output by the output end of the first layer of the first Swin Transformer backbone network is denoted as FR1, the input end of the second layer of the first Swin Transformer backbone network receives FR1, the feature map output by the output end of the second layer of the first Swin Transformer backbone network is denoted as FR2, the input end of the third layer of the first Swin Transformer backbone network receives FR2, the feature map output by the output end of the third layer of the first Swin Transformer backbone network is denoted as FR3, the input end of the fourth layer of the first Swin Transformer backbone network receives FR3, and the feature map output by the output end of the fourth layer of the first Swin Transformer backbone network is denoted as FR4; the input end of the first layer of the second Swin Transformer backbone network receives an infrared image of size H×W×3, the second Swin The feature map outputted from the output of the first layer of the Transformer backbone network is denoted as FT1, the input of the second layer of the second Swin Transformer backbone network receives FT1, the feature map outputted from the output of the second layer of the second Swin Transformer backbone network is denoted as FT2, the input of the third layer of the second Swin Transformer backbone network receives FT2, the feature map outputted from the output of the third layer of the second Swin Transformer backbone network is denoted as FT3, the input of the fourth layer of the second Swin Transformer backbone network receives FT3, and the feature map outputted from the output of the fourth layer of the second Swin Transformer backbone network is denoted as FT4; wherein, the Swin Transformer backbone network has four layers, and the sizes of FR1 and FT1 are The size of FR2 and FT2 is The size of FR3 and FT3 is The size of FR4 and FT4 is In this embodiment, H×W is taken as 384×384; Step S4: The color information and infrared information interaction and high-level information guidance modules are four modules with the same structure, hereinafter referred to as interaction and guidance modules; the first input end of the first interaction and guidance module receives FR1, the second input end receives FT1, the feature map output by the first output end of the first interaction and guidance module is denoted as FRA1, and the feature map output by the second output end is denoted as FTA1; the first input end of the second interaction and guidance module receives FR2, the second input end receives FT2, and the feature map output by the first output end of the second interaction and guidance module is denoted as FRA2 , the feature map output by the second output end is recorded as FTA2; the first input end of the third interaction and guidance module receives FR3, the second input end receives FT3, the feature map output by the first output end of the third interaction and guidance module is recorded as FRA3, and the feature map output by the second output end is recorded as FTA3; the first input end of the fourth interaction and guidance module receives FR4, the second input end receives FT4, the feature map output by the first output end of the fourth interaction and guidance module is recorded as FRA4, and the feature map output by the second output end is recorded as FTA4; wherein, the size of FRA1 and FTA1 is The sizes of FRA2 and FTA2 are The sizes of FRA3 and FTA3 are The sizes of FRA4 and FTA4 are Step S5: The color information and infrared information merging module includes four modules with the same structure, hereinafter referred to as merging modules; the first input end of the first merging module receives FRA1, the second input end receives FTA1, and the feature map output by the output end of the first merging module is recorded as FM1; the first input end of the second merging module receives FRA2, the second input end receives FTA2, and the feature map output by the output end of the second merging module is recorded as FM2; the first input end of the third merging module receives FRA3, the second input end receives FTA3, and the feature map output by the output end of the third merging module is recorded as FM3; the first input end of the fourth merging module receives FRA4, the second input end receives FTA4, and the feature map output by the output end of the fourth merging module is recorded as FM4; wherein the size of FM1 is The size of FM2 is The size of FM3 is The size of FM4 is Step S6: The high-level low-resolution feature dynamic amplification modules are three modules with the same structure, hereinafter referred to as amplification modules; the input end of the first amplification module receives FM2, and the output end of the first amplification module outputs The input of the second amplifier module receives FM3, and the output of the second amplifier module outputs The input of the third amplifier module receives FM4, and the output of the fourth amplifier module outputs in, The size is The size is The size is The high-performance segmentation decoder is an existing Mask2Former segmentation decoder, which accepts FM1, FM2, FM3 and FM4 at its input and outputs a predicted scene parsing target image S, where the size of S is 2. The road scene analysis method based on color visible light and infrared image fusion according to claim 1 is characterized in that: The training set construction process is as follows: at least 700 pairs of original color visible light images and their corresponding original infrared images are selected; then each original color visible light image and its corresponding original infrared image are downsampled to a size of H×W; and then all color visible light images and their corresponding infrared images of size H×W are used to form a training set; wherein, in this embodiment, 784 pairs of original color visible light images and their corresponding original infrared images are selected, and H=W=384.

3. The road scene analysis method based on the fusion of color visible light and infrared images according to claim 2 is characterized in that: The process of obtaining the neural network training model is as follows: each pair of color visible light images and their corresponding infrared images in the training set are input into the neural network for network training. Before the end of each round of network training, the loss function L is calculated to optimize the neural network. After a total of 200 rounds of network training, the neural network training model is obtained. The loss function L is divided into classification loss L cls and the mask loss L mask The classification loss can be expressed as L cls =L ce (S,G), the mask loss can be expressed as: Lmask = 5Lfocal(S,G) + 5Ldice(S,G), the total loss is L = 2Lcls + Lmask, L ce () represents the cross entropy (CE) loss, L focal () indicates focal loss, L dice () represents the region similarity (dice) loss.

4. The road scene analysis method based on color visible light and infrared image fusion according to claim 1 is characterized in that: The neural network training model is used to predict the test image pair, and the process of predicting the scene resolution image of the test image pair is as follows: arbitrarily select a pair of original color visible light images and the corresponding original infrared images; then downsample the pair of original color visible light images and the original infrared images, downsample the image size to H×W, and use them as the test image pair; then input the test image pair into the neural network training model, and predict the scene resolution image of the test image pair; wherein H=W=384.

5. The road scene analysis method based on color visible light and infrared image fusion according to claim 1 is characterized in that: The interaction and guidance module is mainly composed of the first channel-level feature learning module to the second channel-level feature learning module, the first spatial-level feature learning module to the second spatial-level feature learning module, and the seventh convolutional layer to the sixth convolutional layer; the channel-level feature learning module is composed of the first average pooling layer, the first maximum pooling layer, the first fully connected layer to the fourth fully connected layer, the first GELU activation layer to the second GELU activation layer, the first Dropout layer to the fourth Dropout layer, and the first Sigmoid activation layer; the spatial-level feature learning module is composed of the first convolutional layer to the sixth convolutional layer and the first ReLU activation layer.

6. The road scene analysis method based on color visible light and infrared image fusion according to claim 1 is characterized in that: The merging module is mainly composed of the 1st flattening layer to the 2nd flattening layer, the 1st linear layer to the 2nd linear layer, the 1st attention mechanism layer to the 2nd attention mechanism layer, the 9th convolution layer to the 12th convolution layer, and the 1st batch normalization layer; for the i-th merging module, the input end of the first flattening layer receives FRA as the first input end of the i-th merging module i The input of the first linear layer receives the feature vector output by the output of the first flattening layer and outputs three vectors The input of the second flattening layer receives the FTA as the second input of the i-th merging module i The input of the second linear layer receives the feature vector output by the output of the second flattening layer and outputs three vectors The first attention layer receives as input The second attention layer receives as input 7. The road scene analysis method based on color visible light and infrared image fusion according to claim 1 is characterized in that: The high-level low-resolution feature dynamic amplification module is mainly composed of the 1st deconvolution layer, the 13th convolution layer, the 3rd ReLU activation layer, and the 14th convolution layer. For the i-th high-level low-resolution feature dynamic amplification module, the input end of the 1st deconvolution layer receives the output of the i+1th merging module, the input end of the 13th convolution layer receives the output of the 1st deconvolution layer output end, the input end of the 3rd ReLU activation layer receives the output of the 13th convolution layer output end, and the input end of the 14th convolution layer receives the output of the 3rd ReLU activation layer output end. The output of the 14th convolution layer output end is recorded as The 13th to 14th convolutional layers are all 2D convolutional layers. In the first high-level low-resolution feature dynamic amplification module: the input size of the first deconvolution layer is The output size is The convolution kernel size is 2×2, the sliding step is 2, the automatic padding is 0, and the input size of the 13th convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the 14th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0; in the second high-level low-resolution feature dynamic amplification module: the input size of the first deconvolution layer is The output size is The convolution kernel size is 2×2, the sliding step is 2, the automatic padding is 0, and the input size of the 13th convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the 14th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0; in the third high-level low-resolution feature dynamic amplification module: the input size of the first deconvolution layer is The output size is The convolution kernel size is 2×2, the sliding step is 2, the automatic padding is 0, and the input size of the 13th convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, the automatic padding is 1, and the input size of the 14th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 0.

Images