Urban scene analysis prediction method of multi-mode urban scene analysis network based on three-mode contrast learning distillation

Through three-modal comparison learning distillation technology, combined with soft label response relationship and multi-scale graph relationship distillation, the multi-modal urban market scene analysis network is optimized, solving the problems of urban market scene analysis accuracy and resource consumption in the existing technology, and achieving high efficiency , accurate analysis of urban market scenarios.

CN119068334APending Publication Date: 2024-12-03ENJOYOR COMPANY LIMITED
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411134439.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing multimodal urban scene analysis method lacks accuracy in strong and low-light environments, and lacks discriminability in the analysis of similar objects, resulting in a decrease in the accuracy of scene analysis and requires a large number of parameters and computing resources, which cannot be limited in resources. efficient operation on the equipment.

Method used

Using a multimodal urban market scenario analysis network based on three-modal contrast learning distillation, through the cross-training of teacher transformers and student transformers, combined with soft label response relationship distillation and multi-scale graph relationship distillation, optimize feature extraction and fusion to achieve efficient city Scene analysis.

Benefits of technology

It improves the accuracy and efficiency of urban scene analysis, optimizes multimodal information processing, enhances the semantic understanding ability and discernibility of the model, and is suitable for high-precision and efficient urban scene analysis on resource-constrained devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068334A_ABST
    Figure CN119068334A_ABST
Patent Text Reader

Abstract

The invention discloses an urban scene analysis prediction method of a multi-modal urban scene analysis network based on three-modal contrast learning distillation, and relates to the technical field of deep learning. Comprising the following steps: acquiring an original city scene image, a thermal infrared image corresponding to the original city image, and a real city scene analysis image, converting the real city scene analysis image into a thermal coding image as a thermodynamic diagram data set, constructing a multispectral image to enhance and generate a color diagram data set, dividing the thermodynamic diagram data set and the color diagram data set into a training set and a test set according to a certain proportion, constructing a teacher transformer network and a student transformer network, respectively inputting the training sets into the teacher transformer network and the student network, constructing a knowledge distillation model, and calculating the knowledge distillation model; and the trained student transformer network learns the trained teacher transformer network by using the knowledge distillation model to obtain the learned student transformer network and test the learned student transformer network to obtain an urban scene prediction analysis graph. According to the invention, a high-precision and high-efficiency city scene analysis task can be executed on resource-limited equipment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to an urban scene parsing prediction method of a multimodal urban scene parsing network based on trimodal contrastive learning distillation. Background Art

[0002] In the intelligent transportation system, the accuracy and efficiency of scene analysis are crucial to improving road safety and traffic efficiency. Traditional visual methods usually use RGB images for scene analysis. With the advancement of technology, traditional visual sensing methods have gradually failed to meet the needs of complex traffic environments, especially strong and weak light environments. In recent years, multimodal perception technologies, such as red, green, blue-thermal infrared image fusion and red, green, blue-depth image fusion, especially multimodal image fusion, are considered to be an effective way to improve the accuracy of urban scene analysis. Multimodal urban scene analysis combines visible light images and thermal infrared images. Since thermal infrared images can provide good complementary information in strong and weak light environments based on the principle of thermal radiation, and visible light images can provide rich details and texture information, the complementary information of the two is used to enhance the ability to understand scenes under various weather and lighting conditions. In addition, depth images contain rich spatial geometric information, which can well distinguish different parsed objects that are confused in thermal infrared images due to the same thermal radiation. Therefore, depth images can well complement the shortcomings of thermal infrared images in complex environments.

[0003] As a powerful feature learning method, contrastive learning has demonstrated its advantages in single-modality learning. By maximizing the consistency between similar samples and enhancing the distinction between samples, it effectively improves the feature expression ability of learning. In multimodal tasks, the application of contrastive learning can not only extract the unique features of the two modalities separately, but also enhance the model's ability to capture the complex correlation between the two modal data through cross-modal comparison, thereby better performing information fusion.

[0004] Knowledge distillation is a technique to improve model accuracy and efficiency by optimizing the learning process by transferring knowledge from a complex model (teacher model) to a simplified model (student model). In multimodal vision tasks, knowledge distillation can be used to fuse high-level features of multiple modalities in the teacher model, and learn these fused features through the student model to achieve a lightweight model that can run efficiently on devices with limited computing resources. In addition, knowledge distillation can also help improve the student model's ability to adapt to multiple dynamic objects and environmental changes in complex urban scenes.

[0005] Existing multimodal urban scene parsing usually adopts a two-branch independent encoder-decoder architecture, where one branch encodes the RGB image and the other branch encodes the thermal image, and then their encoded features are fused and then the decoder is used to implement urban scene parsing. However, firstly, they only explore the full complementary fusion of these two modalities when the RGB and thermal modalities are input, but do not explore the complementary fusion of generated depth information. Secondly, they lack the distinguishability in parsing similar objects, resulting in confusion between intra-class and inter-class object parsing, which reduces the accuracy of scene parsing. Finally, they usually require a large number of parameters and huge computational effort to achieve high-performance urban scene parsing, which makes them unusable on resource-constrained devices.

[0006] Therefore, it is an urgent problem for those skilled in the art to propose an urban scene parsing prediction method based on a multimodal urban scene parsing network based on trimodal contrastive learning distillation to solve the problems existing in the prior art. Summary of the invention

[0007] In view of this, the present invention provides an urban scene parsing prediction method based on a multimodal urban scene parsing network with trimodal contrastive learning distillation, which can perform high-precision and efficient urban scene parsing tasks on resource-constrained devices.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A method for urban scene parsing prediction based on a multimodal urban scene parsing network with trimodal contrastive learning distillation, including two processes: a training phase and a testing phase;

[0010] The training phase consists of the following steps:

[0011] S1-1, obtaining the original urban scene image and the thermal infrared image corresponding to the original urban image, and the real urban scene parsed image, and generating an initial data set;

[0012] S1-2, convert the real city scene parsed images in the initial data set into one-hot encoded images as a heat map data set, construct a multispectral image from the original city scene images in the initial data set and the corresponding thermal infrared images, enhance the multispectral image data to generate a color map data set, and divide the heat map data set and the color map data set into a training set and a test set according to a certain ratio;

[0013] S1-3, build a teacher transformer network;

[0014] S1-4. After the training set data is enhanced, it is input into the teacher transformer network and the semantic segmentation prediction image is output. And binary segmentation prediction image

[0015] S1-5. Use the cross entropy loss function to evaluate the one-hot encoded image and the semantic segmentation prediction image in S2 The error loss value is , and the hybrid contrast learning similarity loss function is used to evaluate the unique hot encoding image and the binary segmentation prediction image in S2 Predicting images with semantic segmentation And binary segmentation prediction image The intra-class and inter-class similarity gap loss value between them is calculated, and the error loss value and the intra-class and inter-class similarity gap loss value are added together to form the total loss value;

[0016] S1-6, repeat steps S1-4-S1-5 for V times until the transformer network converges and the value of the loss function drops to the lowest point to obtain the trained teacher transformer network, and set the weights and biases corresponding to the lowest loss value as the optimal weight vector W t _best and the optimal bias b t_best ;

[0017] S1-7, build the student transformer network, repeat steps S1-4-S1-5 for V times until the transformer network converges and the value of the loss function drops to the lowest point to obtain the trained student transformer network, and set the weights and biases corresponding to the lowest loss value as the optimal weight vector W s_best and the optimal bias b s_best ;

[0018] S1-8. Construct knowledge distillation including soft label response relation distillation and multi-scale graph relation distillation. Soft label response relation distillation uses the true label to correct the teacher transformer network output, and obtains the correlation loss function established between the corrected teacher output and the student output. Multi-scale graph relation distillation is used to construct a multi-scale graph, and multi-scale graph convolution is used to obtain multi-scale enhanced teacher features and multi-scale enhanced student features and establish a correlation loss function.

[0019] S1-9, repeat steps S1-4, S1-5 and S1-8 until the transformer network converges stably and the value of the loss function drops to the lowest point to obtain a well-learned student transformer network, and set the weights and biases corresponding to the lowest loss value as the optimal weight vector W kd_best and the optimal bias b kd_best ;

[0020] The testing phase includes the following steps:

[0021] S2-1, select the Pth group of urban scene images to be tested in the test set and mark them as I p , where 1≤p≤P;

[0022] S2-2, input various channel components of the Pth group of urban scene images to be detected into the learned student transformer network, and obtain the urban scene analysis prediction map corresponding to the original urban scene image, denoted as

[0023] In the above method, optionally, in S1-3, the teacher transformer network includes feature extraction and feature fusion;

[0024] Feature extraction includes RGB image input layer, thermal image input layer, eight transformer modules, and an efficient context alignment module;

[0025] Feature fusion includes four tri-modal fusion modules.

[0026] In the above method, optionally, the RGB image passes through a first transformer module, a second transformer module, a third transformer module and a fourth transformer module in sequence;

[0027] The thermal image passes through the fifth transformer module, the sixth transformer module, the seventh transformer module and the eighth transformer module in sequence;

[0028] The RGB image generates a depth map through a depth map generation algorithm;

[0029] The first transformer module, the second transformer module, the third transformer module and the fourth transformer module correspond vertically to the fifth transformer module, the sixth transformer module, the seventh transformer module and the eighth transformer module, and the vertically corresponding transformer module outputs and the depth map are respectively input into the first trimodal fusion module, the second trimodal fusion module, the third trimodal fusion module and the fourth trimodal fusion module;

[0030] The fourth trimodal fusion module is connected to the efficient context module, the first decoder module, the second decoder module, the third decoder module and the fourth decoder module in sequence;

[0031] The fourth trimodal fusion module, the third trimodal fusion module, the second trimodal fusion module, and the first trimodal fusion module are jump-connected to the first decoder module, the second decoder module, the third decoder module, and the fourth decoder module, respectively.

[0032] In the above method, optionally, the transformer module includes a plurality of groups of overlapping patch encoding units, transformer blocks and random path discarding operation units connected in sequence;

[0033] The overlapping patch encoding unit uses a 3×3 convolution kernel, a stride of 1, and a padding of 1 to segment the image into overlapping patch embeddings;

[0034] The transformer block includes layer normalization, an 8-head attention mechanism, and a 2-layer multilayer perceptron to focus on and process key features;

[0035] Random path dropout units are used for the entire embedding to help regularize and avoid overfitting.

[0036] In the above method, optionally, in the tri-modal fusion module, the RGB image features are passed through the ninth transformer block to obtain output seven;

[0037] The heat map features are sequentially passed through the second denoising block and the eleventh transformer block to obtain output nine;

[0038] The depth map features and the heat map features are multiplied after the first denoising block and the second denoising block respectively to obtain a multiplication result, which is added element-by-element with output seven to obtain output ten. Output ten is respectively subjected to matrix multiplication and fully connected layer with output seven and output nine. The output data of the fully connected layer is multiplied element-by-element and then input into the activation function. The result of the activation function is multiplied with the RGB map features and then added to obtain the output of the three-modal fusion module.

[0039] In the above method, optionally, the efficient context alignment module includes four pooling layers, four upsampling layers, a transformer module and an activation function;

[0040] The pooling layer is divided into two adaptive maximum pooling layers and two adaptive average pooling layers. Each pooling layer is connected to the upsampling layer to obtain contextual features in different directions of the input image.

[0041] The upsampling layer is used to restore the input image size, and the obtained image features are concatenated according to the channel dimension and input into the transformer module;

[0042] The transformer module output is added to the input image element-by-element and input to the activation function to obtain the output of the efficient context alignment module.

[0043] The above method, optionally, the decoder module has two adaptive upsampling layers, two convolutional layers, two batch normalization layers and three activation functions;

[0044] The output of the trimodal fusion module passes through the first adaptive sampling layer, the first convolution layer, the first batch normalization layer and the first ReLu layer in sequence; the output of the efficient context alignment module passes through the second convolution layer, the second batch normalization layer and the second ReLu layer in sequence; the output of the first ReLu layer is added to the output of the second ReLu layer and passes through the second adaptive upsampling layer and the activation function in sequence to obtain the decoder output.

[0045] In the above method, optionally, constructing knowledge distillation in S1-8 includes inputting the heat map dataset and the color map dataset into the teacher transformer network and the student transformer network respectively, and the heat map dataset and the color map dataset are sequentially distilled through the first teacher fusion feature, the second teacher fusion feature, the third teacher fusion feature, the fourth teacher fusion feature, the teacher response output and the soft label response relationship in the teacher transformer network;

[0046] The heat map dataset and the color map dataset are sequentially distilled through the first student fusion feature, the second student fusion feature, the third student fusion feature, the fourth student fusion feature, the student response output, and the soft label response relationship in the student transformer network;

[0047] The first teacher fusion feature, the second teacher fusion feature, the third teacher fusion feature, the fourth teacher fusion feature and the teacher response output correspond vertically to the first student fusion feature, the second student fusion feature, the third student fusion feature, the fourth student fusion feature and the student response output;

[0048] The vertically corresponding fused features are input into the multi-scale graph relation distillation, and the teacher response output and the student response output are input into the hybrid contrastive learning block with the hard labels respectively.

[0049] It can be seen from the above technical solution that, compared with the prior art, the present invention provides an urban scene parsing prediction method of a multimodal urban scene parsing network based on trimodal contrastive learning distillation, which has the following beneficial effects:

[0050] 1) The novel trimodal fusion module proposed in this paper optimizes the processing method of multimodal information. It realizes the deep integration of data from each modality by simulating the two-stage strategy of common alignment and difference complementation of brain-like thinking. In the first stage, by aligning common features, the model promotes the synchronization of basic information between different modalities. In the second stage, by complementing differences, it accurately captures the unique details of each modality, thereby enriching the semantic understanding ability of the model;

[0051] 2) The hybrid contrastive learning strategy proposed in this invention effectively optimizes the identifiability of the network from the perspective of binary and semantics. By promoting the reduction of intra-class gaps and the increase of inter-class differences, hybrid contrastive learning ensures that the network can distinguish different categories more accurately. This method not only improves the classification accuracy, but also enhances the model's ability to understand and process complex data features. Therefore, the hybrid contrastive learning strategy significantly improves the performance of the network in difficult recognition tasks, making it more robust and effective in dealing with diverse and complex scenarios;

[0052] 3) The soft label response relationship distillation and multi-scale graph relationship distillation strategies proposed in the present invention have significant advantages. First, the soft label response relationship distillation strategy can refine and optimize the output of the student network to make it closer to the actual label distribution, thereby improving the accuracy of scene parsing. Secondly, the multi-scale graph relationship distillation strategy can capture richer and more detailed feature information at different scales, enhancing the student network's ability to recognize objects in complex scenes. Therefore, it is particularly suitable for performing high-precision and efficient urban scene parsing tasks on resource-constrained devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0054] Figure 1 A flow chart of a method for urban scene parsing prediction based on a multimodal urban scene parsing network using trimodal contrastive learning distillation disclosed in the present invention;

[0055] Figure 2 A schematic diagram of a teacher / student transformer network disclosed in the present invention;

[0056] Figure 3 This is a schematic diagram of the tri-modal fusion module disclosed in the present invention;

[0057] Figure 4 This is a schematic diagram of the efficient context alignment module disclosed in the present invention;

[0058] Figure 5 It is a schematic diagram of a decoder module disclosed in the present invention;

[0059] Figure 6 This is a diagram of the knowledge distillation architecture disclosed in the present invention;

[0060] Figure 7a This is the first original daytime city scene parsed image disclosed in the present invention;

[0061] Figure 7b The city scene analysis prediction map obtained by the prediction disclosed by the present invention;

[0062] Figure 8a The second original daytime city scene parsed image disclosed in the present invention;

[0063] Figure 8b The city scene analysis prediction map obtained by the prediction disclosed by the present invention;

[0064] Figure 9a The third original nighttime city scene parsed image disclosed in the present invention;

[0065] Figure 9b The city scene analysis prediction map obtained by the prediction disclosed by the present invention;

[0066] Fig.10a The fourth original nighttime city scene parsed image disclosed in the present invention;

[0067] Fig.10b This is the city scene analysis prediction map obtained by the prediction disclosed by the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0070] Reference Figure 1 As shown, the present invention discloses an urban scene parsing prediction method of a multimodal urban scene parsing network based on trimodal contrastive learning distillation, including two processes: a training phase and a testing phase;

[0071] The training phase consists of the following steps:

[0072] S1-1, obtaining the original urban scene image and the thermal infrared image corresponding to the original urban image, and the real urban scene parsed image, and generating an initial data set;

[0073] S1-2, convert the real city scene parsed images in the initial data set into one-hot encoded images as a heat map data set, construct a multispectral image from the original city scene images in the initial data set and the corresponding thermal infrared images, enhance the multispectral image data to generate a color map data set, and divide the heat map data set and the color map data set into a training set and a test set according to a certain ratio;

[0074] S1-3, build a teacher transformer network;

[0075] S1-4. After the training set data is enhanced, it is input into the teacher transformer network and the semantic segmentation prediction image is output. And binary segmentation prediction image

[0076] S1-5. Use the cross entropy loss function to evaluate the one-hot encoded image and the semantic segmentation prediction image in S2 The error loss value is , and the hybrid contrast learning similarity loss function is used to evaluate the unique hot encoding image and the binary segmentation prediction image in S2 Predicting images with semantic segmentation And binary segmentation prediction image The intra-class and inter-class similarity gap loss value between them is calculated, and the error loss value and the intra-class and inter-class similarity gap loss value are added together to form the total loss value;

[0077] S1-6, repeat steps S1-4-S1-5 for V times until the transformer network converges and the value of the loss function drops to the lowest point to obtain the trained teacher transformer network, and set the weights and biases corresponding to the lowest loss value as the optimal weight vector W t _best and the optimal bias b t_best ;

[0078] S1-7, build the student transformer network, repeat steps S1-4-S1-5 for V times until the transformer network converges and the value of the loss function drops to the lowest point to obtain the trained student transformer network, and set the weights and biases corresponding to the lowest loss value as the optimal weight vector W s_best and the optimal bias b s_best ;

[0079] S1-8. Construct knowledge distillation including soft label response relation distillation and multi-scale graph relation distillation. Soft label response relation distillation uses the true label to correct the teacher transformer network output, and obtains the correlation loss function established between the corrected teacher output and the student output. Multi-scale graph relation distillation is used to construct a multi-scale graph, and multi-scale graph convolution is used to obtain multi-scale enhanced teacher features and multi-scale enhanced student features and establish a correlation loss function.

[0080] S1-9, repeat steps S1-4, S1-5 and S1-8 until the transformer network converges stably and the value of the loss function drops to the lowest point to obtain a well-learned student transformer network, and set the weights and biases corresponding to the lowest loss value as the optimal weight vector W kd _ best and the optimal bias b kd _ best ;

[0081] The testing phase includes the following steps:

[0082] S2-1, select the Pth group of urban scene images to be tested in the test set and mark them as I p , where 1≤p≤P;

[0083] S2-2, input various channel components of the Pth group of urban scene images to be detected into the learned student transformer network, and obtain the urban scene analysis prediction map corresponding to the original urban scene image, denoted as

[0084] Furthermore, in S1-1, Q original city scene images are selected, each image is equipped with a corresponding thermal infrared image and a corresponding real city scene analysis image, and Q is a positive integer. Each original city scene image is defined as {I q (i,j)}, and the corresponding real city scene parsed image is defined as Then, using the existing unique hot encoding technology, each Converted into 9 one-hot encoded images, the set of these one-hot encoded images is recorded as Among them, if Q=784, q is a positive integer, 1≤q≤Q, the original urban scene image is composed of RGB color images, which contain the spectral information of the three bands of red, green and blue. At the same time, each image also includes infrared multispectral images, which provide additional thermal infrared band information, and each band is equivalent to a channel component. In this way, each original image not only includes the R, G, and B channel components of the RGB image, but also contains additional channel components of thermal infrared. This comprehensive data format allows for more comprehensive analysis and processing of urban scenes under different spectral conditions to enhance the understanding and prediction capabilities of the model.

[0085] Furthermore, in S1-2, a variety of data enhancement techniques are implemented for each urban scene image and its corresponding multispectral image, including adjusting brightness, contrast and hue, as well as performing random flipping and cropping operations.

[0086] Further, see Figure 2 As shown, in S1-3, the teacher transformer network includes feature extraction and feature fusion;

[0087] Feature extraction includes RGB image input layer, thermal image input layer, eight transformer modules, and an efficient context alignment module;

[0088] Feature fusion includes four tri-modal fusion modules.

[0089] Further, the RGB image passes through the first transformer module, the second transformer module, the third transformer module and the fourth transformer module in sequence;

[0090] The thermal image passes through the fifth transformer module, the sixth transformer module, the seventh transformer module and the eighth transformer module in sequence;

[0091] The RGB image generates a depth map through a depth map generation algorithm;

[0092] The first transformer module, the second transformer module, the third transformer module and the fourth transformer module correspond vertically to the fifth transformer module, the sixth transformer module, the seventh transformer module and the eighth transformer module, and the vertically corresponding transformer module outputs and the depth map are respectively input into the first trimodal fusion module, the second trimodal fusion module, the third trimodal fusion module and the fourth trimodal fusion module;

[0093] The fourth trimodal fusion module is connected to the efficient context module, the first decoder module, the second decoder module, the third decoder module and the fourth decoder module in sequence;

[0094] The fourth trimodal fusion module, the third trimodal fusion module, the second trimodal fusion module, and the first trimodal fusion module are jump-connected to the first decoder module, the second decoder module, the third decoder module, and the fourth decoder module, respectively.

[0095] Further, the transformer module includes a plurality of groups of overlapping patch encoding units, transformer blocks and random path discarding operation units connected in sequence;

[0096] The overlapping patch encoding unit uses a 3×3 convolution kernel, a stride of 1, and a padding of 1 to segment the image into overlapping patch embeddings;

[0097] The transformer block includes layer normalization, an 8-head attention mechanism, and a 2-layer multilayer perceptron to focus on and process key features;

[0098] Random path dropout units are used for the entire embedding to help regularize and avoid overfitting.

[0099] Specifically, the input received by the transformer network includes the RGB image and the corresponding heat map of each image. The size of these images is set to 480 pixels in width (W) and 640 pixels in height (H). The RGB image consists of three color channels: red (R), green (G), and blue (B), while the heat map is a single temperature channel. All these image data will be extracted through the Segformer-b4 backbone network.

[0100] The first transformer module and the fifth transformer module are connected in sequence by three groups of units. The number of convolution kernels of the teacher transformer network is 64, and the number of convolution kernels of the student transformer network is 32. The output feature map sets are recorded as The width of the feature map is Height is

[0101] The second transformer module and the sixth transformer module structure are connected in sequence by 4 groups of units. The number of convolution kernels of the teacher transformer network is 128, and the number of convolution kernels of the student transformer network is 64. The output feature map sets are recorded as The width of the feature map is Height is

[0102] The third transformer module and the seventh transformer module structure are connected in sequence by 6 groups of units. The number of convolution kernels of the teacher transformer network is 320, and the number of convolution kernels of the student transformer network is 160. The output feature map sets are recorded as The width of the feature map is Height is

[0103] The fourth transformer module and the eighth transformer module structure are connected in sequence by three groups of units. The number of convolution kernels of the teacher transformer network is 512, and the number of convolution kernels of the student transformer network is 256. The output feature map sets are recorded as The width of the feature map is Height is

[0104] Further, see Figure 3 As shown, in the tri-modal fusion module, the RGB image features are passed through the ninth transformer block to obtain output seven;

[0105] The heat map features are sequentially passed through the second denoising block and the eleventh transformer block to obtain output nine;

[0106] The depth map features and the heat map features are multiplied after the first denoising block and the second denoising block respectively to obtain a multiplication result, which is added element-by-element with output seven to obtain output ten. Output ten is respectively subjected to matrix multiplication and fully connected layer with output seven and output nine. The output data of the fully connected layer is multiplied element-by-element and then input into the activation function. The result of the activation function is multiplied with the RGB map features and then added to obtain the output of the three-modal fusion module.

[0107] Further, see Figure 4 As shown, the efficient context alignment module includes four pooling layers, four upsampling layers, a transformer module and an activation function;

[0108] The pooling layer is divided into two adaptive maximum pooling layers and two adaptive average pooling layers. Each pooling layer is connected to the upsampling layer to obtain contextual features in different directions of the input image.

[0109] The upsampling layer is used to restore the input image size, and the obtained image features are concatenated according to the channel dimension and input into the transformer module;

[0110] The transformer module output is added to the input image element-by-element and input to the activation function to obtain the output of the efficient context alignment module.

[0111] Further, see Figure 5 As shown, the decoder module has two adaptive upsampling layers, two convolutional layers, two batch normalization layers, and three activation functions;

[0112] The output of the trimodal fusion module passes through the first adaptive sampling layer, the first convolution layer, the first batch normalization layer and the first ReLu layer in sequence; the output of the efficient context alignment module passes through the second convolution layer, the second batch normalization layer and the second ReLu layer in sequence; the output of the first ReLu layer is added to the output of the second ReLu layer and passes through the second adaptive upsampling layer and the activation function in sequence to obtain the decoder output.

[0113] Furthermore, S1-4 includes: the image is sent as input to the transformer network for training. During the training, the model can output semantic segmentation prediction images and binary segmentation prediction images. These prediction outputs constitute a set of predicted images of urban scenes. The obtained sets of predicted urban scene images are respectively recorded as and In this way, not only the diversity of training data is increased, but also the analysis and prediction effect of the transformer network on complex urban environments is improved.

[0114] Further, S1-5 includes: calculating the loss function value includes: using the cross entropy loss function as the loss function of the multi-category task, which evaluates the error between the true semantic segmentation map (which has been converted into a series of one-hot encoded images) and the predicted segmentation map At the same time, the hybrid contrast learning similarity loss function is used to measure the true binary and semantics One-hot encoded image set and predicted binary values and semantics The intra-class and inter-class similarity gaps between graphs and The results of these two loss calculations are added together to form the final total loss value, which will guide the neural network to optimize learning to improve the accuracy of the model in image semantic segmentation tasks.

[0115] Further, see Figure 6 As shown, constructing knowledge distillation in S1-8 includes inputting the heat map dataset and the color map dataset into the teacher transformer network and the student transformer network respectively, and the heat map dataset and the color map dataset are sequentially distilled through the first teacher fusion feature, the second teacher fusion feature, the third teacher fusion feature, the fourth teacher fusion feature, the teacher response output and the soft label response relationship in the teacher transformer network;

[0116] The heat map dataset and the color map dataset are sequentially distilled through the first student fusion feature, the second student fusion feature, the third student fusion feature, the fourth student fusion feature, the student response output, and the soft label response relationship in the student transformer network;

[0117] The first teacher fusion feature, the second teacher fusion feature, the third teacher fusion feature, the fourth teacher fusion feature and the teacher response output correspond vertically to the first student fusion feature, the second student fusion feature, the third student fusion feature, the fourth student fusion feature and the student response output;

[0118] The vertically corresponding fused features are input into the multi-scale graph relation distillation, and the teacher response output and the student response output are input into the hybrid contrastive learning block with the hard labels respectively.

[0119] Furthermore, in the distillation of the soft label response relationship, the output of the teacher model Correlation modeling is used to adjust the predictions of the student model The correlation loss function is This involves not simply copying the output of the teacher model, but adjusting the output of the student model by taking into account the relational and structural information in the teacher's output to make it closer to the high-level representation of the teacher model. Soft label response relation distillation enhances the student model's ability to understand complex relations in the data. By correcting the soft labels output by the teacher model, the student model can better simulate the teacher's decision-making process, thereby improving its performance and generalization ability while maintaining lower complexity;

[0120] In multi-scale graph relation distillation, the nodes in the graph represent the features in the teacher and student models, denoted as and The edges represent the relationship between features. By applying graph convolution operations on this graph, semantic relationships at different levels (e.g., pixel level and region level) can be captured. The correlation loss established between the multi-scale enhanced teacher features and the multi-scale enhanced student features is This multi-scale relationship modeling allows the student model to not only learn the outputs of the teacher model, but also understand the structure and semantic dependencies behind these outputs, thereby achieving more accurate predictions.

[0121] Furthermore, in the knowledge distillation step, the student transformer network learns the teacher transformer network by repeatedly performing data processing, basic loss calculation, and knowledge distillation loss calculation until the transformer network converges stably and the value of the loss function drops to the lowest point. During the entire training process, the loss values ​​are recorded Q times. From these records, the lowest loss value is selected, and its corresponding weight and bias are set as the optimal parameters of the convolutional neural network, named as the optimal weight vector and the optimal bias, respectively, and denoted as W. kd _ best and b kd _ best .

[0122] In a specific embodiment, a multimodal urban scene parsing network based on trimodal contrastive learning distillation is constructed using the deep learning framework Pytorch 3.7 in a Python environment. The network uses the test set of the MFNet multimodal urban scene database to verify the scene parsing effect of the method of the present invention. In order to objectively evaluate the performance of semantic segmentation, two common evaluation indicators are used: Mean Class Accuracy (mAcc) and Mean Intersection over Union (MIoU). These two indicators measure the accuracy of the predicted scene parsing image and the degree of overlap with the real label image, respectively.

[0123] By applying this method to the test set of the MFNet multimodal urban scene image database, each image is predicted and the corresponding predicted scene analysis image is generated. The analysis of these results is shown in Table 1 and Figure 7a-Figure 10b As shown, it includes the average category accuracy (mAcc) and the intersection over union (MIoU) of the scene parsed image and the labeled image.

[0124] Table 1 Evaluation results on the test set

[0125]

[0126] It is not difficult to conclude that the scene analysis results of urban scene images obtained using the method of the present invention perform excellently, which proves that the method of this paper is not only feasible but also effective in predicting scene analysis of urban scene images.

[0127] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0128] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for urban scene parsing prediction based on a multimodal urban scene parsing network with trimodal contrastive learning distillation, characterized in that: It includes two processes: training phase and testing phase; The training phase consists of the following steps: S1-1, obtaining the original urban scene image and the thermal infrared image corresponding to the original urban image, and the real urban scene parsed image, and generating an initial data set; S1-2, convert the real city scene parsed images in the initial data set into one-hot encoded images as a heat map data set, construct a multispectral image from the original city scene images in the initial data set and the corresponding thermal infrared images, enhance the multispectral image data to generate a color map data set, and divide the heat map data set and the color map data set into a training set and a test set according to a certain ratio; S1-3, build a teacher transformer network; S1-4. After the training set data is enhanced, it is input into the teacher transformer network and the semantic segmentation prediction image is output. And binary segmentation prediction image S1-5. Use the cross entropy loss function to evaluate the one-hot encoded image and the semantic segmentation prediction image in S2 The error loss value is , and the hybrid contrast learning similarity loss function is used to evaluate the unique hot encoding image and the binary segmentation prediction image in S2 Predicting images with semantic segmentation And binary segmentation prediction image The intra-class and inter-class similarity gap loss value between them is calculated, and the error loss value and the intra-class and inter-class similarity gap loss value are accumulated to form the total loss value; S1-6, repeat steps S1-4-S1-5 for V times until the transformer network converges and the value of the loss function drops to the lowest point to obtain the trained teacher transformer network, and set the weights and biases corresponding to the lowest loss value as the optimal weight vector W t _ best and the optimal bias b t _ best ; S1-7, build the student transformer network, repeat steps S1-4-S1-5 for V times until the transformer network converges and the value of the loss function drops to the lowest point to obtain the trained student transformer network, and set the weights and biases corresponding to the lowest loss value as the optimal weight vector W s _ best and the optimal bias b s _ best ; S1-8. Construct knowledge distillation including soft label response relation distillation and multi-scale graph relation distillation. Soft label response relation distillation uses the true label to correct the teacher transformer network output, and obtains the correlation loss function established between the corrected teacher output and the student output. Multi-scale graph relation distillation is used to construct a multi-scale graph, and multi-scale graph convolution is used to obtain multi-scale enhanced teacher features and multi-scale enhanced student features and establish a correlation loss function. S1-9, repeat steps S1-4, S1-5 and S1-8 until the transformer network converges stably and the value of the loss function drops to the lowest point to obtain a well-learned student transformer network, and set the weights and biases corresponding to the lowest loss value as the optimal weight vector W kd _ best and the optimal bias b kd _ best ; The testing phase includes the following steps: S2-1, select the Pth group of urban scene images to be tested in the test set and mark them as I p , where 1≤p≤P; S2-2, input various channel components of the Pth group of urban scene images to be detected into the learned student transformer network, and obtain the urban scene analysis prediction map corresponding to the original urban scene image, denoted as 2. According to claim 1, the urban scene parsing prediction method of a multimodal urban scene parsing network based on trimodal contrastive learning distillation is characterized in that: In S1-3, the teacher transformer network includes feature extraction and feature fusion; Feature extraction includes RGB image input layer, thermal image input layer, eight transformer modules, and an efficient context alignment module; Feature fusion includes four tri-modal fusion modules.

3. The urban scene parsing prediction method of a multimodal urban scene parsing network based on trimodal contrastive learning distillation according to claim 2 is characterized in that: The RGB image passes through the first transformer module, the second transformer module, the third transformer module and the fourth transformer module in sequence; The thermal image passes through the fifth transformer module, the sixth transformer module, the seventh transformer module and the eighth transformer module in sequence; The RGB image generates a depth map through a depth map generation algorithm; The first transformer module, the second transformer module, the third transformer module and the fourth transformer module correspond vertically to the fifth transformer module, the sixth transformer module, the seventh transformer module and the eighth transformer module, and the vertically corresponding transformer module outputs and the depth map are respectively input into the first trimodal fusion module, the second trimodal fusion module, the third trimodal fusion module and the fourth trimodal fusion module; The fourth trimodal fusion module is connected to the efficient context module, the first decoder module, the second decoder module, the third decoder module and the fourth decoder module in sequence; The fourth trimodal fusion module, the third trimodal fusion module, the second trimodal fusion module, and the first trimodal fusion module are jump-connected to the first decoder module, the second decoder module, the third decoder module, and the fourth decoder module, respectively.

4. The urban scene parsing prediction method of a multimodal urban scene parsing network based on trimodal contrastive learning distillation according to claim 3 is characterized in that: The transformer module includes a plurality of groups of overlapping patch encoding units, transformer blocks and random path discarding operation units connected in sequence; The overlapping patch encoding unit uses a 3×3 convolution kernel, a stride of 1, and a padding of 1 to segment the image into overlapping patch embeddings; The transformer block includes layer normalization, an 8-head attention mechanism, and a 2-layer multilayer perceptron to focus on and process key features; Random path dropout units are used to integrate embeddings, helping regularize and avoid overfitting.

5. The urban scene parsing prediction method of a multimodal urban scene parsing network based on trimodal contrastive learning distillation according to claim 2 is characterized in that: In the trimodal fusion module, the RGB image features are passed through the ninth transformer block to obtain output seven; The heat map features are sequentially passed through the second denoising block and the eleventh transformer block to obtain output nine; The depth map features and the heat map features are multiplied after the first denoising block and the second denoising block respectively to obtain a multiplication result, which is added element-by-element with output seven to obtain output ten. Output ten is respectively subjected to matrix multiplication and fully connected layer with output seven and output nine. The output data of the fully connected layer is multiplied element-by-element and then input into the activation function. The result of the activation function is multiplied with the RGB map features and then added to obtain the output of the three-modal fusion module.

6. The urban scene parsing prediction method of a multimodal urban scene parsing network based on trimodal contrastive learning distillation according to claim 2 is characterized in that: The efficient context alignment module consists of four pooling layers, four upsampling layers, a transformer module, and an activation function; The pooling layer is divided into two adaptive maximum pooling layers and two adaptive average pooling layers. Each pooling layer is connected to the upsampling layer to obtain contextual features in different directions of the input image. The upsampling layer is used to restore the input image size, and the obtained image features are concatenated according to the channel dimension and input into the transformer module; The transformer module output is added to the input image element-by-element and input to the activation function to obtain the output of the efficient context alignment module.

7. The urban scene parsing prediction method of a multimodal urban scene parsing network based on trimodal contrastive learning distillation according to claim 2 is characterized in that: The decoder module has two adaptive upsampling layers, two convolutional layers, two batch normalization layers, and three activation functions; The output of the trimodal fusion module passes through the first adaptive sampling layer, the first convolution layer, the first batch of normalization layers, and the first ReLu layer in sequence; The output of the efficient context alignment module passes through the second convolution layer, the second batch normalization layer and the second ReLu layer in sequence; the output of the first ReLu layer is added to the output of the second ReLu layer and then passes through the second adaptive upsampling layer and the activation function in sequence to obtain the decoder output.

8. The urban scene parsing prediction method of a multimodal urban scene parsing network based on trimodal contrastive learning distillation according to claim 1 is characterized in that: The knowledge distillation in S1-8 includes inputting the heat map dataset and the color map dataset into the teacher transformer network and the student transformer network respectively, and the heat map dataset and the color map dataset are sequentially distilled through the first teacher fusion feature, the second teacher fusion feature, the third teacher fusion feature, the fourth teacher fusion feature, the teacher response output and the soft label response relationship in the teacher transformer network; The heat map dataset and the color map dataset are sequentially distilled through the first student fusion feature, the second student fusion feature, the third student fusion feature, the fourth student fusion feature, the student response output, and the soft label response relationship in the student transformer network; The first teacher fusion feature, the second teacher fusion feature, the third teacher fusion feature, the fourth teacher fusion feature and the teacher response output correspond vertically to the first student fusion feature, the second student fusion feature, the third student fusion feature, the fourth student fusion feature and the student response output; The vertically corresponding fused features are input into the multi-scale graph relation distillation, and the teacher response output and the student response output are input into the hybrid contrastive learning block with the hard labels respectively.

Citation Information

Cited By

  • Emergency command rescue data distributed storage method and system based on artificial intelligence

    CN120492546A

  • Intelligent urban road waterlogging depth classification method and system based on knowledge distillation and diffusion model

    CN120852842A

  • An urban road waterlogging depth intelligent classification method and system based on knowledge distillation and diffusion model

    CN120852842B