A RGB-D semantic segmentation method based on multimodal contrastive learning

Abstract

The present invention discloses a semantic segmentation method based on multimodal contrastive learning. A training set is constructed, and a contrastive learning semantic segmentation network consisting of a semantic segmentation basic network and a contrastive learning module is established; the semantic segmentation basic network includes an RGB branch, a depth branch, an integration module and a contrastive learning module, and the segmentation results of the RGB branch and the depth branch are combined by the integration module to obtain the final segmentation result and supervise the training in the training stage. The intermediate features of the RGB branch and the depth branch are input into the contrastive learning module in the training stage and optimized with cross-modal contrast loss and same-modal contrast loss. The RGB image to be segmented and the corresponding depth image are input into the semantic segmentation basic network in the trained contrastive learning semantic segmentation network for semantic segmentation to obtain the semantic segmentation result. The present invention can better explore the commonalities of RGB images and depth images while retaining the characteristics of the two types of images, thereby improving the accuracy of semantic segmentation.

Description

Technical Field

[0001] The present invention relates to an RGB-D semantic segmentation method in the technical field of computer vision, and in particular to an RGB-D semantic segmentation method based on multimodal contrast learning. Background Art

[0002] With the development of sensors, RGB-D semantic segmentation has attracted more and more research interests in recent years. RGB-D semantic segmentation uses RGB images and their corresponding depth images to label semantic categories for each pixel in the image, thereby achieving the effect of segmenting the scene. Since depth images provide geometric information that RGB images do not have, RGB-D semantic segmentation methods have more robust performance when the lighting conditions vary greatly and the color and texture differences are small, so they have potential application value in fields such as autonomous driving and robot vision.

[0003] For example, "Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation" by Chen et al. in "Computer Vision--ECCV 2020:16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XI 16" and "Pattern-structure diffusion for multi-task learning" by Zhou et al. in "Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition" are both effective RGB-D semantic segmentation methods that utilize depth information. Summary of the invention

[0004] In order to solve the problems and needs in the background technology, for RGB-D scenes, the present invention provides an RGB-D semantic segmentation method based on multimodal contrastive learning, and applies contrastive learning to semantic segmentation tasks. The present invention uses cross-modal and same-modal contrastive learning losses for optimization in different feature embedding spaces. The former is used to mine the common features of the two modalities, and the latter is used to learn modality-specific features. In order to make full use of the information of the two modalities of RGB and depth, the situation in which RGB images are difficult to segment due to similar color and texture is improved, and the accuracy of semantic segmentation is improved.

[0005] The technical solution of the present invention is as follows:

[0006] The present invention comprises the following steps:

[0007] 1) Collect multiple RGB images and corresponding depth images of scenes that need semantic segmentation, and the required training set is composed of multiple RGB images and corresponding depth images as well as the category of each pixel in each RGB image;

[0008] 2) After the required training set is processed in batches, the RGB images and their corresponding depth images are input into the RGB-D contrast learning semantic segmentation network in batches. The RGB-D contrast learning semantic segmentation network consists of an RGB-D semantic segmentation basic network and a contrast learning module; the RGB-D semantic segmentation basic network includes an RGB branch, a depth branch and an integration module, wherein the RGB branch includes an RGB feature encoder and an RGB semantic segmentation decoder, and the depth branch includes a depth feature encoder and a depth semantic segmentation decoder. The RGB image is input into the RGB feature encoder, and the output of the RGB feature encoder is respectively input into the RGB semantic segmentation decoder and the contrast learning module, the depth image is input into the depth feature encoder, and the output of the depth feature encoder is respectively input into the deep semantic segmentation decoder and the contrast learning module, and the output of the RGB semantic segmentation decoder and the deep semantic segmentation decoder are both input into the integration module, and the output of the integration module is used as the RGB-D semantic segmentation basic network; the semantic segmentation loss of the RGB-D semantic segmentation basic network and the multimodal contrast loss of the contrast learning module are calculated to obtain the multimodal total loss, and the RGB-D contrast learning semantic segmentation network is trained based on the multimodal total loss to obtain a trained RGB-D contrast learning semantic segmentation network;

[0009] 3) Input the RGB image to be segmented and the corresponding depth image into the RGB-D semantic segmentation basic network in the trained RGB-D contrast learning semantic segmentation network for semantic segmentation to obtain the semantic segmentation result.

[0010] The integration module of step 2) concatenates the output of the RGB semantic segmentation decoder and the output of the deep semantic segmentation decoder and inputs them into the first convolutional layer. The first convolutional layer outputs a weight matrix, and then normalizes the weight matrix to obtain a normalized weight matrix as the confidence E of the RGB branch prediction probability. r , 1 and the confidence E of the RGB branch prediction probability r The difference between 1 and 1 is used as the confidence of the prediction probability of the deep branch 1-E r , and then the output of the RGB semantic segmentation decoder is compared with the confidence E of the RGB branch prediction probability rAfter multiplication, we get the RGB category score map, and we also multiply the output of the deep semantic segmentation decoder with the confidence 1-E of the deep branch prediction probability. r After multiplication, a depth category score map is obtained. Finally, the RGB category score map and the depth category score map are added to obtain an integrated category score map, which is used as the output of the integration module.

[0011] The contrast learning module includes an RGB same-modality contrast learning submodule, a deep same-modality contrast learning submodule and a dual-stream cross-modality contrast learning submodule; the output of the RGB feature encoder in the step 2) is respectively input into the RGB same-modality contrast learning submodule and the dual-stream cross-modality contrast learning submodule, and the output of the deep feature encoder is respectively input into the deep same-modality contrast learning submodule and the dual-stream cross-modality contrast learning submodule.

[0012] The RGB homomodal contrast learning submodule and the deep homomodal contrast learning submodule have the same structure, but process different objects. The input of the RGB homomodal contrast learning submodule is the RGB modal features output by the RGB feature encoder, the input of the deep homomodal contrast learning submodule is the deep modal features output by the deep feature encoder, the output of the RGB homomodal contrast learning submodule is the RGB homomodal anchor sample set and the RGB homomodal positive and negative difficult sample set, and the output of the deep homomodal contrast learning submodule is the deep homomodal anchor sample set and the deep homomodal positive and negative difficult sample set;

[0013] In the RGB homomodal contrast learning submodule, the RGB modal features of the current batch of RGB images are projected by the first projection module to obtain multiple RGB homomodal pixel features, and then the multiple RGB homomodal pixel features are sampled by the following steps to obtain the RGB homomodal anchor sample set and the RGB homomodal positive and negative difficult sample set of the current batch of RGB images, and the RGB homomodal contrast loss of the current batch of RGB images is calculated:

[0014] S1: In each RGB image of the current batch, the pixel features belonging to the same category in multiple RGB homomodal pixel features of each RGB image are pooled to obtain the RGB homomodal central pixel features under each category of the current RGB image, and the remaining RGB images of the current batch are traversed to perform category pixel feature pooling to finally obtain the RGB homomodal central pixel features under each category of the current batch;

[0015] S2: Use the semantic-aware anchor sample sampling method to select anchor samples for multiple RGB homomodal pixel features of the current batch to obtain the RGB homomodal anchor sample set of the current batch ;

[0016] The S2 is specifically:

[0017] S2.1: Randomly sample pixel positions of multiple RGB homomodal pixel features of the current batch, and use the sampled pixel positions as half of the anchor points; sample pixel positions that are misclassified in the RGB semantic segmentation decoder, and use the sampled pixel positions as the other half of the anchor points;

[0018] S2.2: extract pixel features corresponding to all anchor points in S2.1 from multiple RGB homomodal pixel features and use them as individual RGB homomodal anchor point samples, and the individual RGB homomodal anchor point samples constitute the RGB homomodal anchor point sample set of the current batch;

[0019] S3: According to the RGB homomodal anchor sample set of the current batch, for each RGB homomodal anchor sample, use the positive and negative sample sampling method of difficult sample mining to select positive and negative samples for the RGB homomodal central pixel features under each category of the current batch, obtain the RGB homomodal positive and negative difficult sample set corresponding to the current RGB homomodal anchor sample, and calculate the RGB homomodal contrast loss of the current RGB homomodal anchor sample;

[0020] The S3 is specifically:

[0021] S3.1: For each RGB homomodal anchor sample, among the RGB homomodal center pixel features under each category of the current batch, the pixel features that belong to the same category as the current RGB homomodal anchor sample are used as the positive sample set of the current RGB homomodal anchor sample, and the pixel features that do not belong to the same category as the current RGB homomodal anchor sample are used as the negative sample set of the current RGB homomodal anchor sample;

[0022] S3.2: Calculate the similarity between the current RGB homomodal anchor sample and each positive sample in the positive sample set of the current RGB homomodal anchor sample and sort the similarities in ascending order, take the positive samples with the top 10% similarity as the difficult sample set of the current positive sample, and then randomly sample K1 samples from the difficult sample set of the current positive sample set as the RGB homomodal positive difficult sample set of the current RGB homomodal anchor sample;

[0023] S3.3: Calculate the similarity between the current RGB homomodal anchor sample and each negative sample in the negative sample set of the current RGB homomodal anchor sample and sort the similarities in descending order, take the negative samples with the top 10% similarity as the difficult sample set of the current negative sample, and then randomly sample K2 samples from the difficult sample set of the current negative sample set as the RGB homomodal negative difficult sample set of the current RGB homomodal anchor sample;

[0024] S3.4: Based on the RGB homomodal positive difficult sample set and the RGB homomodal negative difficult sample set obtained in S3.2 and S3.3, the RGB homomodal contrast loss of the current RGB homomodal anchor sample is calculated by the following formula:

[0025]

[0026] in, represents the RGB homomodal contrast loss of the current RGB homomodal anchor sample, r i Represents the current RGB homomodal anchor point sample, and Respectively, the current RGB same-modal anchor point samples r i RGB homomodal positive difficult samples and RGB homomodal negative difficult samples, and They represent the RGB homomodal positive difficult sample set and the RGB homomodal negative difficult sample set where pixel i is located, τ represents the temperature coefficient, Indicates the size of the RGB homomodal positive difficult sample set where the pixel i is calculated;

[0027] S4: Repeat S3, traverse the remaining RGB homomodal anchor samples in the current RGB homomodal anchor sample set, obtain the RGB homomodal positive and negative difficult sample sets of each RGB homomodal anchor sample, and calculate the RGB homomodal contrast loss of each RGB homomodal anchor sample, and take the sum of the RGB homomodal contrast losses of each RGB homomodal anchor sample as the RGB homomodal contrast loss of the current batch of RGB images.

[0028] The calculation formula of the deep homomodal contrast loss of each deep homomodal anchor point sample in the deep homomodal contrast learning submodule is as follows:

[0029]

[0030] in, represents the depth homomodal contrast loss of the current depth homomodal anchor sample, d i Indicates the current depth homomodal anchor point sample and Respectively represent the current depth and modality anchor point samples d i The deep positive and negative samples of and They represent the deep homomodal positive difficult sample set and the deep homomodal negative difficult sample set where pixel i is located, τ represents the temperature coefficient, Indicates the size of the set of positive difficult samples of the same modality at the depth where the pixel i is calculated.

[0031] The dual-stream cross-modal contrast learning submodule includes an RGB cross-modal contrast learning submodule and a deep cross-modal contrast learning submodule. The RGB cross-modal contrast learning submodule and the deep cross-modal contrast learning submodule have the same structure, but different processing objects. The input of the RGB cross-modal contrast learning submodule is the RGB modal features output by the RGB feature encoder, and the input of the deep cross-modal contrast learning submodule is the deep modal features output by the deep feature encoder. The output of the RGB cross-modal contrast learning submodule is an RGB cross-modal anchor sample set and an RGB cross-modal positive and negative difficult sample set, and the output of the deep cross-modal contrast learning submodule is a deep cross-modal anchor sample set and a deep cross-modal positive and negative difficult sample set.

[0032] In the RGB cross-modal contrast learning submodule, the RGB modal features of the current batch of RGB images are projected by the second projection module to obtain multiple RGB cross-modal pixel features, and then the multiple RGB cross-modal pixel features are sampled through the following steps to obtain the RGB cross-modal anchor sample set and the RGB cross-modal positive and negative difficult sample set of the current batch of RGB images, and the RGB cross-modal contrast loss of the current batch of RGB images is calculated:

[0033] A1: In each RGB image of the current batch, the pixel features belonging to the same category in multiple RGB cross-modal pixel features of each RGB image are pooled to obtain the RGB cross-modal center pixel features of each category of the current RGB image, and the remaining RGB images of the current batch are traversed to perform category pixel feature pooling to finally obtain the RGB cross-modal center pixel features of each category of the current batch;

[0034] A2: Randomly sample multiple RGB cross-modal pixel features of the current batch to obtain a set of RGB cross-modal anchor point samples of the current batch;

[0035] A3: According to the RGB cross-modal anchor sample set of the current batch, for each RGB cross-modal anchor sample, use the positive and negative sample sampling method of difficult sample mining to select positive and negative samples for the RGB cross-modal central pixel features under each category of the current batch, obtain the RGB cross-modal positive and negative difficult sample set corresponding to the current RGB cross-modal anchor sample, and calculate the RGB cross-modal contrast loss of the current RGB cross-modal anchor sample;

[0036] The A3 is specifically:

[0037] A3.1: For each RGB cross-modal anchor sample, among the RGB cross-modal center pixel features under each category of the current batch, the pixel features that belong to the same category as the current RGB cross-modal anchor sample are used as the positive sample set of the current RGB cross-modal anchor sample, and the pixel features that do not belong to the same category as the current RGB cross-modal anchor sample are used as the negative sample set of the current RGB cross-modal anchor sample;

[0038] A3.2: Calculate the similarity between the current RGB cross-modal anchor sample and each positive sample in the positive sample set of the current RGB cross-modal anchor sample and sort the similarities in ascending order. Take the positive samples with the top 10% similarity as the difficult sample set of the current positive sample. Then randomly sample K1 samples from the difficult sample set of the current positive sample set as the RGB cross-modal positive difficult sample set of the current RGB cross-modal anchor sample.

[0039] A3.3: Calculate the similarity between the current RGB cross-modal anchor sample and each negative sample in the negative sample set of the current RGB cross-modal anchor sample and sort the similarities in descending order. Take the negative samples with the top 10% similarity as the difficult sample set of the current negative sample. Then randomly sample K2 samples from the difficult sample set of the current negative sample set as the RGB cross-modal negative difficult sample set of the current RGB cross-modal anchor sample.

[0040] A3.4: Based on the RGB cross-modal positive difficult sample set and RGB cross-modal negative difficult sample set obtained in A3.2 and A3.3, the r←d flow cross-modal contrast loss of the current RGB cross-modal anchor sample is calculated by the following formula:

[0041]

[0042] in, represents the r←d flow cross-modal contrast loss of the current RGB cross-modal anchor sample, Represents the current RGB cross-modal anchor sample, and Represent the current RGB cross-modal anchor point samples respectively RGB cross-modal positive difficult samples and RGB cross-modal negative difficult samples, and They represent the RGB cross-mode positive difficult sample set and the RGB cross-mode negative difficult sample set where the pixel i is located, τ represents the temperature coefficient, Indicates the size of the RGB cross-modal positive sample set where the pixel i is calculated;

[0043] A4: Repeat A3, traverse the remaining RGB cross-modal anchor samples in the current RGB cross-modal anchor sample set, obtain the RGB cross-modal positive and negative difficult sample sets of each RGB cross-modal anchor sample, and calculate the r←d flow cross-modal contrast loss of each RGB cross-modal anchor sample. The sum of the r←d flow cross-modal contrast losses of each RGB cross-modal anchor sample is used as the r←d flow cross-modal contrast loss of the current batch of RGB images.

[0044] The calculation formula of the d←r flow cross-modal contrast loss of each deep cross-modal anchor point sample in the deep cross-modal contrast learning submodule is as follows:

[0045]

[0046] in, represents the d←r flow cross-modal contrast loss of the current deep cross-modal anchor sample, d i represents the current depth cross-modal anchor point sample, and Represents deep cross-modal anchor samples respectively d i Deep cross-modal positive hard samples and deep cross-modal negative hard samples, and They represent the deep cross-modal positive difficult sample set and the deep cross-modal negative difficult sample set where pixel i is located, τ represents the temperature coefficient, Indicates the size of the deep cross-modal positive difficult sample set where the pixel i is calculated.

[0047] The multimodal total loss is calculated by the following formula:

[0048]

[0049] in, represents the total multimodal loss, represents the semantic segmentation loss, represents the r←d flow cross-modal contrast loss of the current RGB cross-modal anchor sample, represents the d←r flow cross-modal contrast loss of the current deep cross-modal anchor sample, represents the RGB homomodal contrast loss of the current RGB homomodal anchor sample, represents the depth homomodal contrast loss of the current depth homomodal anchor sample, λ1 represents the cross-modal contrast loss balance weight, λ2 represents the homomodal contrast loss balance weight, represents the RGB cross-modal anchor sample set, represents a set of deep cross-modal anchor samples, represents the RGB homomodal anchor sample set, Represents a set of deep homomodal anchor samples.

[0050] The structures of the first projection module and the second projection module of the dual-stream cross-modal contrastive learning sub-module are the same. The first projection module is mainly composed of a second convolutional layer, a batch normalization layer, an activation function layer and a third convolutional layer connected in sequence. The input of the first projection module is input to the second convolutional layer, and the output of the third convolutional layer is used as the output of the first projection module.

[0051] The semantic segmentation loss adopts pixel-by-pixel cross entropy loss.

[0052] The multiple depth images collected in step 1) are decomposed into a three-channel form of HHA: parallax, height, and normal vector.

[0053] The present invention has the following beneficial effects:

[0054] 1. Compared with the semantic segmentation method that only uses RGB images, the RGB-D semantic segmentation method has better performance in cases where segmentation is difficult due to similar color and texture because it introduces the geometric information of the depth image, so it can better segment scenes under different lighting conditions.

[0055] 2. A new framework is proposed for the RGB-D semantic segmentation task. Compared with most RGB-D semantic segmentation methods, it does not require fusion modules at each level of the encoder and can be applied to existing unimodal semantic segmentation networks without changing their basic networks.

[0056] 3. The present invention proposes dual-stream cross-modal contrastive learning and RGB / depth same-modal contrastive learning for the RGB-D setting, so as to better mine the common features between modalities and retain the features specific to the modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the network model of the present invention.

[0058] Figure 2 It is a schematic diagram of the structure of the RGB same-modality contrast learning submodule of the present invention.

[0059] Figure 3 It is a schematic diagram of the structure of the dual-flow cross-modal contrast learning submodule of the present invention.

[0060] Figure 4 It is a schematic diagram of the structure of the deep isomodal contrastive learning submodule of the present invention.

[0061] Figure 5 It is a schematic diagram of the integrated module structure of the present invention. DETAILED DESCRIPTION

[0062] The present invention is further described below in conjunction with the accompanying drawings. The embodiments of the present invention and their implementation process are as follows:

[0063] The present invention comprises the following steps:

[0064] 1) Collect multiple RGB images and corresponding depth images of scenes that need semantic segmentation, and the required training set is composed of multiple RGB images and corresponding depth images and the category of each pixel in each RGB image; in the specific implementation, the multiple depth images collected in step 1) are calculated and decomposed into the HHA three-channel form of disparity-height-normal vector.

[0065] 2) After the required training set is processed in batches, the RGB images and their corresponding depth images are input into the RGB-D contrast learning semantic segmentation network in batches, such as Figure 1 As shown in the figure, the RGB-D contrastive learning semantic segmentation network consists of an RGB-D semantic segmentation basic network and a contrastive learning module; the RGB-D semantic segmentation basic network includes an RGB branch, a depth branch and an integration module, wherein the RGB branch includes an RGB feature encoder and an RGB semantic segmentation decoder, the depth branch includes a depth feature encoder and a depth semantic segmentation decoder, the RGB image is input into the RGB feature encoder, the output of the RGB feature encoder is respectively input into the RGB semantic segmentation decoder and the RGB same-modal contrastive learning module and the dual-stream cross-modal contrastive learning module of the contrastive learning module, the depth image is input into the depth feature encoder, the output of the depth feature encoder is respectively input into the deep semantic segmentation decoder and the deep same-modal contrastive learning module and the dual-stream cross-modal contrastive learning module of the contrastive learning module, the output of the RGB semantic segmentation decoder and the deep semantic segmentation decoder are both input into the integration module, and the output of the integration module is used as the RGB-D semantic segmentation basic network, that is, the output of the RGB-D contrastive learning semantic segmentation network; the semantic segmentation loss of the RGB-D semantic segmentation basic network and the multimodal contrast loss of the contrastive learning module are calculated to obtain the multimodal total loss, and the semantic segmentation loss adopts the pixel-by-pixel cross entropy loss. Based on the multimodal total loss, an RGB-D contrastive learning semantic segmentation network is trained to obtain a trained RGB-D contrastive learning semantic segmentation network;

[0066] In the specific implementation, the RGB branch and the depth branch of the RGB-D semantic segmentation basic network can theoretically adopt any deep learning semantic segmentation method based on the encoder-decoder structure. Here, DeepLabV3+ is taken as an example. In addition, ResNet-101 can be selected and the ResNet-101 trained with the ImageNet dataset can be used as the feature encoder of the semantic segmentation basic network. The network structure of the semantic segmentation decoder is still the network structure of the decoder in DeepLabV3+.

[0067] like Figure 5 As shown in FIG. 2 , the integration module of step 2) concatenates the output of the RGB semantic segmentation decoder and the output of the deep semantic segmentation decoder and inputs them into the first convolutional layer. The first convolutional layer outputs a weight matrix, which is then normalized to obtain a normalized weight matrix and used as the confidence E of the RGB branch prediction probability. r , 1 and the confidence E of the RGB branch prediction probability r The difference between 1 and 1 is used as the confidence of the prediction probability of the deep branch 1-E r , and then the output of the RGB semantic segmentation decoder is compared with the confidence E of the RGB branch prediction probability r After multiplication, we get the RGB category score map, and we also multiply the output of the deep semantic segmentation decoder with the confidence 1-E of the deep branch prediction probability. r After multiplication, a deep category score map is obtained. Finally, the RGB category score map and the depth category score map are added to obtain an integrated category score map. The integrated category score map is used as the output of the integration module for the calculation of semantic segmentation loss.

[0068] The contrastive learning module includes an RGB homomodal contrastive learning submodule, a deep homomodal contrastive learning submodule and a dual-stream cross-modal contrastive learning submodule; in step 2), the outputs of the RGB feature encoder are respectively input into the RGB homomodal contrastive learning submodule and the dual-stream cross-modal contrastive learning submodule, and the outputs of the deep feature encoder are respectively input into the deep homomodal contrastive learning submodule and the dual-stream cross-modal contrastive learning submodule.

[0069] like Figure 2 and Figure 4 As shown in the figure, the structures of the RGB homomodal contrast learning submodule and the deep homomodal contrast learning submodule are the same, but the processing objects are different. The input of the RGB homomodal contrast learning submodule is the RGB modal features output by the RGB feature encoder, and the input of the deep homomodal contrast learning submodule is the deep modal features output by the deep feature encoder. The output of the RGB homomodal contrast learning submodule is the RGB homomodal anchor sample set and the RGB homomodal positive and negative difficult sample set, and the output of the deep homomodal contrast learning submodule is the deep homomodal anchor sample set and the deep homomodal positive and negative difficult sample set.

[0070] In the RGB homomodal contrast learning submodule, the RGB modal features of the current batch of RGB images are projected through the first projection module to obtain multiple RGB homomodal pixel features. Then, after sampling multiple RGB homomodal pixel features through the following steps, the RGB homomodal anchor sample set and RGB homomodal positive and negative difficult sample set of the current batch of RGB images are obtained, and the RGB homomodal contrast loss of the current batch of RGB images is calculated:

[0071] S1: In each RGB image of the current batch, the pixel features belonging to the same category in multiple RGB homomodal pixel features of each RGB image are pooled to obtain the RGB homomodal central pixel features under each category of the current RGB image, and the remaining RGB images of the current batch are traversed to perform category pixel feature pooling to finally obtain the RGB homomodal central pixel features under each category of the current batch;

[0072] S2: Use the semantic-aware anchor sample sampling method to select anchor samples for multiple RGB homomodal pixel features of the current batch to obtain the RGB homomodal anchor sample set of the current batch

[0073] S2 is specifically:

[0074] S2.1: Randomly sample pixel positions of multiple RGB homomodal pixel features of the current batch, and use the sampled pixel positions as half of the anchor points; sample pixel positions that are misclassified in the RGB semantic segmentation decoder, and use the sampled pixel positions as the other half of the anchor points;

[0075] S2.2: Extract the pixel features corresponding to all anchor points in S2.1 from multiple RGB homomodal pixel features and use them as individual RGB homomodal anchor point samples. Each RGB homomodal anchor point sample constitutes the RGB homomodal anchor point sample set of the current batch.

[0076] S3: According to the RGB homomodal anchor sample set of the current batch, for each RGB homomodal anchor sample, use the positive and negative sample sampling method of difficult sample mining to select positive and negative samples for the RGB homomodal central pixel features under each category of the current batch, obtain the RGB homomodal positive and negative difficult sample set corresponding to the current RGB homomodal anchor sample, and calculate the RGB homomodal contrast loss of the current RGB homomodal anchor sample;

[0077] S3 is specifically:

[0078] S3.1: For each RGB homomodal anchor sample, among the RGB homomodal center pixel features under each category of the current batch, the pixel features that belong to the same category as the current RGB homomodal anchor sample are used as the positive sample set of the current RGB homomodal anchor sample, and the pixel features that do not belong to the same category as the current RGB homomodal anchor sample are used as the negative sample set of the current RGB homomodal anchor sample;

[0079] S3.2: Calculate the similarity between the current RGB homomodal anchor sample and each positive sample in the positive sample set of the current RGB homomodal anchor sample and sort the similarities in ascending order, take the positive samples with the top 10% similarity as the difficult sample set of the current positive sample, and then randomly sample K1 samples from the difficult sample set of the current positive sample set as the RGB homomodal positive difficult sample set of the current RGB homomodal anchor sample;

[0080] S3.3: Calculate the similarity between the current RGB homomodal anchor sample and each negative sample in the negative sample set of the current RGB homomodal anchor sample and sort the similarities in descending order, take the negative samples with the top 10% similarity as the difficult sample set of the current negative sample, and then randomly sample K2 samples from the difficult sample set of the current negative sample set as the RGB homomodal negative difficult sample set of the current RGB homomodal anchor sample;

[0081] S3.4: Based on the RGB homomodal positive difficult sample set and the RGB homomodal negative difficult sample set obtained in S3.2 and S3.3, the RGB homomodal contrast loss of the current RGB homomodal anchor sample is calculated by the following formula:

[0082]

[0083] in, represents the RGB homomodal contrast loss of the current RGB homomodal anchor sample, r i represents the current RGB homomodal anchor sample, that is, the RGB homomodal pixel feature corresponding to pixel position i, and Respectively, the current RGB same-modal anchor point samples r i RGB homomodal positive difficult samples and RGB homomodal negative difficult samples, and They represent the RGB homomodal positive difficult sample set and the RGB homomodal negative difficult sample set where pixel i is located, τ represents the temperature coefficient, It indicates the size of the set of RGB homomodal positive difficult samples where the pixel i is located. In the specific implementation, τ=0.1.

[0084] S4: Repeat S3, traverse the remaining RGB homomodal anchor samples in the current RGB homomodal anchor sample set, obtain the RGB homomodal positive and negative difficult sample sets of each RGB homomodal anchor sample, and calculate the RGB homomodal contrast loss of each RGB homomodal anchor sample, and take the sum of the RGB homomodal contrast losses of each RGB homomodal anchor sample as the RGB homomodal contrast loss of the current batch of RGB images.

[0085] The calculation formula of the deep homomodal contrast loss of each deep homomodal anchor sample in the deep homomodal contrast learning submodule is as follows:

[0086]

[0087] in, represents the depth homomodal contrast loss of the current depth homomodal anchor sample, d i represents the current depth homomodal anchor sample, that is, the depth homomodal pixel feature corresponding to pixel position i, and Respectively represent the current depth and modality anchor point samples d i The deep positive and negative samples of and They represent the deep homomodal positive difficult sample set and the deep homomodal negative difficult sample set where pixel i is located, τ represents the temperature coefficient, Indicates the size of the set of positive difficult samples of the same modality at the depth where the pixel i is calculated.

[0088] like Figure 3 As shown, the dual-stream cross-modal contrastive learning submodule includes an RGB cross-modal contrastive learning submodule and a deep cross-modal contrastive learning submodule. The RGB cross-modal contrastive learning submodule and the deep cross-modal contrastive learning submodule have the same structure, but process different objects. The input of the RGB cross-modal contrastive learning submodule is the RGB modal features output by the RGB feature encoder, and the input of the deep cross-modal contrastive learning submodule is the deep modal features output by the deep feature encoder. The output of the RGB cross-modal contrastive learning submodule is an RGB cross-modal anchor sample set and an RGB cross-modal positive and negative difficult sample set, and the output of the deep cross-modal contrastive learning submodule is a deep cross-modal anchor sample set and a deep cross-modal positive and negative difficult sample set.

[0089] In the RGB cross-modal contrast learning submodule, the RGB modal features of the current batch of RGB images are projected through the second projection module to obtain multiple RGB cross-modal pixel features. Then, after sampling the multiple RGB cross-modal pixel features through the following steps, the RGB cross-modal anchor sample set and the RGB cross-modal positive and negative difficult sample set of the current batch of RGB images are obtained, and the RGB cross-modal contrast loss of the current batch of RGB images is calculated:

[0090] A1: In each RGB image of the current batch, the pixel features belonging to the same category in multiple RGB cross-modal pixel features of each RGB image are pooled to obtain the RGB cross-modal center pixel features of each category of the current RGB image, and the remaining RGB images of the current batch are traversed to perform category pixel feature pooling to finally obtain the RGB cross-modal center pixel features of each category of the current batch;

[0091] A2: Randomly sample multiple RGB cross-modal pixel features of the current batch to obtain a set of RGB cross-modal anchor point samples of the current batch;

[0092] A3: According to the RGB cross-modal anchor sample set of the current batch, for each RGB cross-modal anchor sample, use the positive and negative sample sampling method of difficult sample mining to select positive and negative samples for the RGB cross-modal central pixel features under each category of the current batch, obtain the RGB cross-modal positive and negative difficult sample set corresponding to the current RGB cross-modal anchor sample, and calculate the RGB cross-modal contrast loss of the current RGB cross-modal anchor sample;

[0093] A3 specifically:

[0094] A3.1: For each RGB cross-modal anchor sample, among the RGB cross-modal center pixel features under each category of the current batch, the pixel features that belong to the same category as the current RGB cross-modal anchor sample are used as the positive sample set of the current RGB cross-modal anchor sample, and the pixel features that do not belong to the same category as the current RGB cross-modal anchor sample are used as the negative sample set of the current RGB cross-modal anchor sample;

[0095] A3.2: Calculate the similarity between the current RGB cross-modal anchor sample and each positive sample in the positive sample set of the current RGB cross-modal anchor sample and sort the similarities in ascending order. Take the positive samples with the top 10% similarity as the difficult sample set of the current positive sample. Then randomly sample K1 samples from the difficult sample set of the current positive sample set as the RGB cross-modal positive difficult sample set of the current RGB cross-modal anchor sample.

[0096] A3.3: Calculate the similarity between the current RGB cross-modal anchor sample and each negative sample in the negative sample set of the current RGB cross-modal anchor sample and sort the similarities in descending order. Take the negative samples with the top 10% similarity as the difficult sample set of the current negative sample. Then randomly sample K2 samples from the difficult sample set of the current negative sample set as the RGB cross-modal negative difficult sample set of the current RGB cross-modal anchor sample.

[0097] A3.4: Based on the RGB cross-modal positive difficult sample set and RGB cross-modal negative difficult sample set obtained in A3.2 and A3.3, the r←d flow cross-modal contrast loss of the current RGB cross-modal anchor sample is calculated by the following formula:

[0098]

[0099] in, represents the r←d flow cross-modal contrast loss of the current RGB cross-modal anchor sample, represents the current RGB cross-modal anchor point sample, that is, the RGB cross-modal pixel feature corresponding to pixel position i, and Represent the current RGB cross-modal anchor point samples respectively RGB cross-modal positive difficult samples and RGB cross-modal negative difficult samples, and They represent the RGB cross-mode positive difficult sample set and the RGB cross-mode negative difficult sample set where the pixel i is located, τ represents the temperature coefficient, Indicates the size of the RGB cross-modal positive sample set where the pixel i is calculated;

[0100] A4: Repeat A3, traverse the remaining RGB cross-modal anchor samples in the current RGB cross-modal anchor sample set, obtain the RGB cross-modal positive and negative difficult sample sets of each RGB cross-modal anchor sample, and calculate the r←d flow cross-modal contrast loss of each RGB cross-modal anchor sample, and take the sum of the r←d flow cross-modal contrast losses of each RGB cross-modal anchor sample as the r←d flow cross-modal contrast loss of the current batch of RGB images.

[0101] The calculation formula of the d←r flow cross-modal contrast loss of each deep cross-modal anchor point sample in the deep cross-modal contrastive learning submodule is as follows:

[0102]

[0103] in, represents the d←r flow cross-modal contrast loss of the current deep cross-modal anchor sample, d i represents the current depth cross-modal anchor sample, that is, the RGB homomodal pixel feature corresponding to pixel position i, and Represents deep cross-modal anchor samples respectively d i Deep cross-modal positive hard samples and deep cross-modal negative hard samples, and They represent the deep cross-modal positive difficult sample set and the deep cross-modal negative difficult sample set where pixel i is located, τ represents the temperature coefficient, Indicates the size of the deep cross-modal positive difficult sample set where the pixel i is calculated.

[0104] The structures of the fourth projection module of the deep homomodal contrastive learning submodule, the first projection module, the second projection module of the RGB cross-modal contrastive learning submodule in the dual-stream cross-modal contrastive learning submodule, and the third projection module of the deep cross-modal contrastive learning submodule are all the same. The first projection module is mainly composed of a second convolutional layer, a batch normalization layer, an activation function layer and a third convolutional layer connected in sequence. The input of the first projection module is input into the second convolutional layer, and the output of the third convolutional layer is used as the output of the first projection module.

[0105] The total multimodal loss is calculated using the following formula:

[0106]

[0107] in, represents the total multimodal loss, represents the semantic segmentation loss, is the multimodal contrast loss, represents the r←d flow cross-modal contrast loss of the current RGB cross-modal anchor sample, represents the d←r flow cross-modal contrast loss of the current deep cross-modal anchor sample, represents the RGB homomodal contrast loss of the current RGB homomodal anchor sample, represents the depth homomodal contrast loss of the current depth homomodal anchor sample, λ1 represents the cross-modal contrast loss balance weight, λ2 represents the homomodal contrast loss balance weight, represents the RGB cross-modal anchor sample set, represents a set of deep cross-modal anchor samples, represents the RGB homomodal anchor sample set, Indicates the set of deep homomodal anchor samples. In the specific implementation, λ1=λ2=1.

[0108] 3) Input the RGB image to be segmented and the corresponding depth image into the RGB-D semantic segmentation basic network in the trained RGB-D contrast learning semantic segmentation network for semantic segmentation to obtain the semantic segmentation result.

[0109] In the specific implementation, the SGD algorithm with momentum is used for optimization training. The specific settings of the optimizer are: the initial learning rate is set to 0.0025, the momentum is set to 0.9, and the weight decay coefficient is set to 0.0005. In addition, the poly learning rate strategy is adopted to automatically adjust the learning rate size as the number of iterations increases.

[0110] In order to illustrate the effectiveness of the network framework and contrastive learning module proposed in this invention, verification is carried out on the public dataset NYU Depth V2. The NYU Depth V2 dataset consists of video sequences of various indoor scenes collected by Microsoft Kinect, covering 464 scenes in 3 cities, and provides 1449 paired RGB-D images. It is the most commonly used dataset in the task of RGB-D semantic segmentation of indoor scenes. In order to evaluate the performance of semantic segmentation, the commonly used pixel accuracy (Pixel Accuracy, pAcc), mean pixel accuracy (Mean Pixel Accuracy, mAcc) and mean intersection over Union (Mean Intersection over Union, mIoU) are used as evaluation indicators. The semantic segmentation evaluation results of different model variants are shown in Table 1:

[0111] The five model variants in Table 1 are: M1 means only cross entropy loss is used The RGB-D semantic segmentation basic network for supervised training; M2 represents the RGB-D contrast learning semantic segmentation network for supervised training using cross entropy loss and cross-modal contrast learning loss of r←d flow; M3 represents the cross entropy loss and two-way cross-modal contrastive learning loss RGB-D contrast learning semantic segmentation network for supervised training; M4 means using cross entropy loss and two-way cross-modal contrastive learning loss and homomodal contrastive learning loss RGB-D contrastive learning semantic segmentation network with supervised training; M5 represents the complete method of the present invention.

[0112] Table 1

[0113]

[0114] By comparing different model variants, it is proved that adding the complete contrastive learning module (M5) proposed in the present invention can effectively improve the performance and enhance all indicators.

[0115] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention, but are not intended to limit the present invention. Any changes and modifications made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A RGB-D semantic segmentation method based on multimodal contrastive learning, characterized in that: The steps include: 1) Collect multiple RGB images and corresponding depth images of scenes that need semantic segmentation, and the required training set is composed of multiple RGB images and corresponding depth images as well as the category of each pixel in each RGB image; 2) After the required training set is processed in batches, the RGB images and their corresponding depth images are input into the RGB-D contrast learning semantic segmentation network in batches. The RGB-D contrast learning semantic segmentation network consists of an RGB-D semantic segmentation basic network and a contrast learning module; the RGB-D semantic segmentation basic network includes an RGB branch, a depth branch and an integration module, wherein the RGB branch includes an RGB feature encoder and an RGB semantic segmentation decoder, and the depth branch includes a depth feature encoder and a depth semantic segmentation decoder. The RGB image is input into the RGB feature encoder, and the output of the RGB feature encoder is respectively input into the RGB semantic segmentation decoder and the contrast learning module, the depth image is input into the depth feature encoder, and the output of the depth feature encoder is respectively input into the deep semantic segmentation decoder and the contrast learning module, and the output of the RGB semantic segmentation decoder and the deep semantic segmentation decoder are both input into the integration module, and the output of the integration module is used as the RGB-D semantic segmentation basic network; the semantic segmentation loss of the RGB-D semantic segmentation basic network and the multimodal contrast loss of the contrast learning module are calculated to obtain the multimodal total loss, and the RGB-D contrast learning semantic segmentation network is trained based on the multimodal total loss to obtain a trained RGB-D contrast learning semantic segmentation network; 3) Input the RGB image to be segmented and the corresponding depth image into the RGB-D semantic segmentation basic network in the trained RGB-D contrast learning semantic segmentation network for semantic segmentation to obtain the semantic segmentation result.

2. The RGB-D semantic segmentation method based on multimodal contrastive learning according to claim 1, characterized in that: The integration module of step 2) concatenates the output of the RGB semantic segmentation decoder and the output of the deep semantic segmentation decoder and inputs them into the first convolutional layer. The first convolutional layer outputs a weight matrix, and then normalizes the weight matrix to obtain a normalized weight matrix as the confidence E of the RGB branch prediction probability. r , 1 and the confidence E of the RGB branch prediction probability r The difference between 1 and 1 is used as the confidence of the prediction probability of the deep branch 1-E r , and then the output of the RGB semantic segmentation decoder is compared with the confidence E of the RGB branch prediction probability r After multiplication, we get the RGB category score map, and we also multiply the output of the deep semantic segmentation decoder with the confidence 1-E of the deep branch prediction probability. r After multiplication, a depth category score map is obtained. Finally, the RGB category score map and the depth category score map are added to obtain an integrated category score map, which is used as the output of the integration module.

3. The RGB-D semantic segmentation method based on multimodal contrastive learning according to claim 1, characterized in that: The contrast learning module includes an RGB same-modality contrast learning submodule, a deep same-modality contrast learning submodule and a dual-stream cross-modality contrast learning submodule; the output of the RGB feature encoder in the step 2) is respectively input into the RGB same-modality contrast learning submodule and the dual-stream cross-modality contrast learning submodule, and the output of the deep feature encoder is respectively input into the deep same-modality contrast learning submodule and the dual-stream cross-modality contrast learning submodule.

4. The RGB-D semantic segmentation method based on multimodal contrastive learning according to claim 3, characterized in that: The RGB homomodal contrast learning submodule and the deep homomodal contrast learning submodule have the same structure, but process different objects. The input of the RGB homomodal contrast learning submodule is the RGB modal features output by the RGB feature encoder, the input of the deep homomodal contrast learning submodule is the deep modal features output by the deep feature encoder, the output of the RGB homomodal contrast learning submodule is the RGB homomodal anchor sample set and the RGB homomodal positive and negative difficult sample set, and the output of the deep homomodal contrast learning submodule is the deep homomodal anchor sample set and the deep homomodal positive and negative difficult sample set; In the RGB homomodal contrast learning submodule, the RGB modal features of the current batch of RGB images are projected by the first projection module to obtain multiple RGB homomodal pixel features, and then the multiple RGB homomodal pixel features are sampled by the following steps to obtain the RGB homomodal anchor sample set and the RGB homomodal positive and negative difficult sample set of the current batch of RGB images, and the RGB homomodal contrast loss of the current batch of RGB images is calculated: S1: In each RGB image of the current batch, the pixel features belonging to the same category in multiple RGB homomodal pixel features of each RGB image are pooled to obtain the RGB homomodal central pixel features under each category of the current RGB image, and the remaining RGB images of the current batch are traversed to perform category pixel feature pooling to finally obtain the RGB homomodal central pixel features under each category of the current batch; S2: Use the semantic-aware anchor sample sampling method to select anchor samples for multiple RGB homomodal pixel features of the current batch to obtain the RGB homomodal anchor sample set of the current batch The S2 is specifically: S2.1: Randomly sample the pixel positions of multiple RGB homomodal pixel features of the current batch, and use the sampled pixel positions as half of the anchor points; Sample the pixel positions that are misclassified in the RGB semantic segmentation decoder, and use the sampled pixel positions as the other half of the anchor points; S2.2: extract pixel features corresponding to all anchor points in S2.1 from multiple RGB homomodal pixel features and use them as individual RGB homomodal anchor point samples, and the individual RGB homomodal anchor point samples constitute the RGB homomodal anchor point sample set of the current batch; S3: According to the RGB homomodal anchor sample set of the current batch, for each RGB homomodal anchor sample, use the positive and negative sample sampling method of difficult sample mining to select positive and negative samples for the RGB homomodal central pixel features under each category of the current batch, obtain the RGB homomodal positive and negative difficult sample set corresponding to the current RGB homomodal anchor sample, and calculate the RGB homomodal contrast loss of the current RGB homomodal anchor sample; The S3 is specifically: S3.1: For each RGB homomodal anchor sample, among the RGB homomodal center pixel features under each category of the current batch, the pixel features that belong to the same category as the current RGB homomodal anchor sample are used as the positive sample set of the current RGB homomodal anchor sample, and the pixel features that do not belong to the same category as the current RGB homomodal anchor sample are used as the negative sample set of the current RGB homomodal anchor sample; S3.2: Calculate the similarity between the current RGB homomodal anchor sample and each positive sample in the positive sample set of the current RGB homomodal anchor sample and sort the similarities in ascending order, take the positive samples with the top 10% similarity as the difficult sample set of the current positive sample, and then randomly sample K1 samples from the difficult sample set of the current positive sample set as the RGB homomodal positive difficult sample set of the current RGB homomodal anchor sample; S3.3: Calculate the similarity between the current RGB homomodal anchor sample and each negative sample in the negative sample set of the current RGB homomodal anchor sample and sort the similarities in descending order, take the negative samples with the top 10% similarity as the difficult sample set of the current negative sample, and then randomly sample K2 samples from the difficult sample set of the current negative sample set as the RGB homomodal negative difficult sample set of the current RGB homomodal anchor sample; S3.4: Based on the RGB homomodal positive difficult sample set and the RGB homomodal negative difficult sample set obtained in S3.2 and S3.3, the RGB homomodal contrast loss of the current RGB homomodal anchor sample is calculated by the following formula: in, represents the RGB homomodal contrast loss of the current RGB homomodal anchor sample, r i Represents the current RGB homomodal anchor point sample, and Respectively, the current RGB same-modal anchor point samples r i RGB homomodal positive difficult samples and RGB homomodal negative difficult samples, and They represent the RGB homomodal positive difficult sample set and the RGB homomodal negative difficult sample set where pixel i is located, τ represents the temperature coefficient, Indicates the size of the RGB homomodal positive difficult sample set where the pixel i is calculated; S4: Repeat S3, traverse the remaining RGB homomodal anchor samples in the current RGB homomodal anchor sample set, obtain the RGB homomodal positive and negative difficult sample sets of each RGB homomodal anchor sample, and calculate the RGB homomodal contrast loss of each RGB homomodal anchor sample, and take the sum of the RGB homomodal contrast losses of each RGB homomodal anchor sample as the RGB homomodal contrast loss of the current batch of RGB images.

5. The RGB-D semantic segmentation method based on multimodal contrastive learning according to claim 4, characterized in that: The calculation formula of the deep homomodal contrast loss of each deep homomodal anchor point sample in the deep homomodal contrast learning submodule is as follows: in, represents the depth homomodal contrast loss of the current depth homomodal anchor sample, d i Indicates the current depth homomodal anchor point sample and Respectively represent the current depth and modality anchor point samples d i Deep positive and deep negative samples, and They represent the deep homomodal positive difficult sample set and the deep homomodal negative difficult sample set where pixel i is located, τ represents the temperature coefficient, Indicates the size of the set of positive difficult samples of the same modality at the depth where the pixel i is calculated.

6. The RGB-D semantic segmentation method based on multimodal contrastive learning according to claim 3, characterized in that: The dual-stream cross-modal contrast learning submodule includes an RGB cross-modal contrast learning submodule and a deep cross-modal contrast learning submodule. The RGB cross-modal contrast learning submodule and the deep cross-modal contrast learning submodule have the same structure, but different processing objects. The input of the RGB cross-modal contrast learning submodule is the RGB modal features output by the RGB feature encoder, and the input of the deep cross-modal contrast learning submodule is the deep modal features output by the deep feature encoder. The output of the RGB cross-modal contrast learning submodule is an RGB cross-modal anchor sample set and an RGB cross-modal positive and negative difficult sample set, and the output of the deep cross-modal contrast learning submodule is a deep cross-modal anchor sample set and a deep cross-modal positive and negative difficult sample set. In the RGB cross-modal contrast learning submodule, the RGB modal features of the current batch of RGB images are projected by the second projection module to obtain multiple RGB cross-modal pixel features, and then the multiple RGB cross-modal pixel features are sampled through the following steps to obtain the RGB cross-modal anchor sample set and the RGB cross-modal positive and negative difficult sample set of the current batch of RGB images, and the RGB cross-modal contrast loss of the current batch of RGB images is calculated: A1: In each RGB image of the current batch, the pixel features belonging to the same category in multiple RGB cross-modal pixel features of each RGB image are pooled to obtain the RGB cross-modal center pixel features of each category of the current RGB image, and the remaining RGB images of the current batch are traversed to perform category pixel feature pooling to finally obtain the RGB cross-modal center pixel features of each category of the current batch; A2: Randomly sample multiple RGB cross-modal pixel features of the current batch to obtain a set of RGB cross-modal anchor point samples of the current batch; A3: According to the RGB cross-modal anchor sample set of the current batch, for each RGB cross-modal anchor sample, use the positive and negative sample sampling method of difficult sample mining to select positive and negative samples for the RGB cross-modal central pixel features under each category of the current batch, obtain the RGB cross-modal positive and negative difficult sample set corresponding to the current RGB cross-modal anchor sample, and calculate the RGB cross-modal contrast loss of the current RGB cross-modal anchor sample; The A3 is specifically: A3.1: For each RGB cross-modal anchor sample, among the RGB cross-modal center pixel features under each category of the current batch, the pixel features that belong to the same category as the current RGB cross-modal anchor sample are used as the positive sample set of the current RGB cross-modal anchor sample, and the pixel features that do not belong to the same category as the current RGB cross-modal anchor sample are used as the negative sample set of the current RGB cross-modal anchor sample; A3.2: Calculate the similarity between the current RGB cross-modal anchor sample and each positive sample in the positive sample set of the current RGB cross-modal anchor sample and sort the similarities in ascending order. Take the positive samples with the top 10% similarity as the difficult sample set of the current positive sample. Then randomly sample K1 samples from the difficult sample set of the current positive sample set as the RGB cross-modal positive difficult sample set of the current RGB cross-modal anchor sample. A3.3: Calculate the similarity between the current RGB cross-modal anchor sample and each negative sample in the negative sample set of the current RGB cross-modal anchor sample and sort the similarities in descending order. Take the negative samples with the top 10% similarity as the difficult sample set of the current negative sample. Then randomly sample K2 samples from the difficult sample set of the current negative sample set as the RGB cross-modal negative difficult sample set of the current RGB cross-modal anchor sample. A3.4: Based on the RGB cross-modal positive difficult sample set and RGB cross-modal negative difficult sample set obtained in A3.2 and A3.3, the r←d flow cross-modal contrast loss of the current RGB cross-modal anchor sample is calculated by the following formula: in, represents the r←d flow cross-modal contrast loss of the current RGB cross-modal anchor sample, Represents the current RGB cross-modal anchor sample, and Represent the current RGB cross-modal anchor point samples respectively RGB cross-modal positive difficult samples and RGB cross-modal negative difficult samples, and They represent the RGB cross-mode positive difficult sample set and the RGB cross-mode negative difficult sample set where the pixel i is located, τ represents the temperature coefficient, Indicates the size of the RGB cross-modal positive sample set where the pixel i is calculated; A4: Repeat A3, traverse the remaining RGB cross-modal anchor samples in the current RGB cross-modal anchor sample set, obtain the RGB cross-modal positive and negative difficult sample sets of each RGB cross-modal anchor sample, and calculate the r←d flow cross-modal contrast loss of each RGB cross-modal anchor sample. The sum of the r←d flow cross-modal contrast losses of each RGB cross-modal anchor sample is taken as the r←d flow cross-modal contrast loss of the current batch of RGB images. The calculation formula of the d←r flow cross-modal contrast loss of each deep cross-modal anchor point sample in the deep cross-modal contrast learning submodule is as follows: in, represents the d←r flow cross-modal contrast loss of the current deep cross-modal anchor sample, d i represents the current depth cross-modal anchor point sample, and Represents deep cross-modal anchor samples respectively d i Deep cross-modal positive hard samples and deep cross-modal negative hard samples, and They represent the deep cross-modal positive difficult sample set and the deep cross-modal negative difficult sample set where pixel i is located, τ represents the temperature coefficient, Indicates the size of the deep cross-modal positive difficult sample set where the pixel i is calculated.

7. The RGB-D semantic segmentation method based on multimodal contrastive learning according to claim 1, characterized in that: The multimodal total loss is calculated by the following formula: in, represents the total multimodal loss, represents the semantic segmentation loss, represents the r←d flow cross-modal contrast loss of the current RGB cross-modal anchor sample, represents the d←r flow cross-modal contrast loss of the current deep cross-modal anchor sample, represents the RGB homomodal contrast loss of the current RGB homomodal anchor sample, represents the depth homomodal contrast loss of the current depth homomodal anchor sample, λ1 represents the cross-modal contrast loss balance weight, λ2 represents the homomodal contrast loss balance weight, represents the RGB cross-modal anchor sample set, represents a set of deep cross-modal anchor samples, represents the RGB homomodal anchor sample set, Represents a set of deep homomodal anchor samples.

8. The RGB-D semantic segmentation method based on multimodal contrastive learning according to claim 4 or 6, characterized in that: The structures of the first projection module and the second projection module of the dual-stream cross-modal contrastive learning sub-module are the same. The first projection module is composed of a second convolutional layer, a batch normalization layer, an activation function layer and a third convolutional layer connected in sequence. The input of the first projection module is input to the second convolutional layer, and the output of the third convolutional layer is used as the output of the first projection module.

9. The RGB-D semantic segmentation method based on multimodal contrastive learning according to claim 1, characterized in that: The semantic segmentation loss adopts pixel-by-pixel cross entropy loss.

10. The RGB-D semantic segmentation method based on multimodal contrastive learning according to claim 1, characterized in that: The multiple depth images collected in step 1) are decomposed into a three-channel form of HHA: parallax, height, and normal vector.

Images