A Deep Privileged Semantic Segmentation Method Based on Multimodal Contrastive Learning

Abstract

The present invention discloses a deep privileged semantic segmentation method based on multimodal contrastive learning. RGB images and depth images are collected and together with categories form a training set, and an RGB branch and a depth branch are established, both of which include a semantic segmentation basic network and a projection module; the two semantic segmentation basic networks are supervised and trained with pixel-by-pixel cross entropy loss; the embedded features obtained by the projection module are optimized with cross-modal contrast loss, and the feature encoder in the semantic segmentation basic network is enabled to mine the commonality of RGB features and depth features through the back propagation algorithm, which improves the situation in which RGB images are difficult to segment due to similar color and texture to a certain extent, thereby improving the accuracy of semantic segmentation. The present invention only needs the RGB image of the scene without depth information in the reasoning stage. Since the depth information is introduced in the training stage, it has better segmentation ability and generalization performance.

Description

Technical Field

[0001] The present invention relates to a deep privileged semantic segmentation method in the technical field of computer vision, and in particular to a deep privileged semantic segmentation method based on multimodal contrastive learning. Background Art

[0002] With the development of sensors, RGB-D semantic segmentation has attracted more and more research interests in recent years. RGB-D semantic segmentation uses RGB images and their corresponding depth images to label semantic categories for each pixel in the image, thereby achieving the effect of segmenting the scene. Since depth images provide geometric information that RGB images do not have, RGB-D semantic segmentation methods have more robust performance when the lighting conditions vary greatly and the color and texture differences are small, so they have potential application value in fields such as autonomous driving and robot vision.

[0003] For example, "Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation" by Chen et al. in "Computer Vision--ECCV 2020:16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XI 16" and "Pattern-structure diffusion for multi-task learning" by Zhou et al. in "Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition" are both effective RGB-D semantic segmentation methods that utilize depth information.

[0004] Some existing RGB-D semantic segmentation methods require RGB and depth data in both training and inference stages. Some do not require depth data during inference, but are often implemented through multiple interactive decoders, which affects the inference speed. Summary of the invention

[0005] In view of the situation that RGB images and depth images can be provided during training, but depth images cannot be provided during testing, the present invention provides a deep privileged semantic segmentation method based on multimodal contrastive learning, using DeepLabV3+ based on encoder-decoder deconstruction as the semantic segmentation basic network model, and using cross-modal contrastive learning to allow the feature encoder to automatically learn to mine the commonality of RGB features and depth features, so as to improve the situation that RGB images are difficult to segment due to similar color and texture, thereby improving the accuracy of semantic segmentation. The present invention only uses contrastive learning to introduce depth information during training, and can obtain better segmentation results when inferring, under the premise of only requiring RGB images and not affecting the inference speed.

[0006] The technical solution of the present invention is as follows:

[0007] The present invention comprises the following steps:

[0008] 1) Collect multiple RGB images and corresponding depth images of scenes that need semantic segmentation, and the required training set is composed of multiple RGB images and corresponding depth images as well as the category of each pixel in each RGB image;

[0009] 2) After the required training set is processed in batches, the RGB images and the corresponding depth images are input into the multimodal contrastive learning semantic segmentation network in batches. The multimodal contrastive learning semantic segmentation network consists of an RGB image branch, a depth image branch and a contrastive learning module. The RGB image branch is connected to the depth image branch through the contrastive learning module. Then, the semantic segmentation losses of the RGB image branch and the depth image branch are calculated respectively, and the cross-modal contrast loss of the contrastive learning module is also calculated. Then, the multimodal total loss is calculated according to the semantic segmentation loss and the cross-modal contrast loss of the RGB image branch and the depth image branch. The multimodal contrastive learning semantic segmentation network is trained based on the multimodal total loss to obtain a trained multimodal contrastive learning semantic segmentation network; wherein, the RGB image branch includes an RGB feature encoder, an RGB semantic decoder and an RGB projection module, and the output of the RGB feature encoder is respectively input into the RGB semantic decoder and the RGB projection module, the RGB projection module is connected to the contrastive learning module, the RGB feature encoder and the RGB semantic decoder constitute an RGB semantic segmentation basic network, and the output of the RGB semantic decoder is used as the output of the RGB semantic segmentation basic network;

[0010] 3) The RGB image to be segmented is input into the RGB semantic segmentation basic network of the trained multimodal contrastive learning semantic segmentation network for semantic segmentation to obtain the RGB image after semantic segmentation.

[0011] The depth image branch includes a depth feature encoder, a depth semantic decoder and a depth projection module. The output of the depth feature encoder is respectively input into the depth semantic decoder and the depth projection module, and the depth projection module is connected to the contrast learning module.

[0012] The RGB projection module and the depth projection module have the same structure, specifically, they are composed of a first convolutional layer, a batch normalization layer, an activation function layer, and a second convolutional layer connected in sequence.

[0013] In step 2), the RGB projection module outputs multiple RGB pixel embedding features, and the depth projection module outputs multiple depth pixel embedding features. The multiple RGB pixel embedding features and the depth pixel embedding features of each batch are input into the contrast learning module. The contrast learning module samples the multiple RGB pixel embedding features and the depth pixel embedding features of the current batch, obtains the positive and negative difficult sample sets of the current batch and outputs them, specifically:

[0014] S1: Randomly sample P pixel positions of all RGB images in each batch, extract pixel features corresponding to P pixel positions in multiple deep pixel embedding features and use them as deep random pixel features of the current batch;

[0015] S2: In each depth image of the current batch, the pixel features belonging to the same category in the multiple depth pixel embedding features of each depth image are pooled to obtain the depth center pixel features of each category of the current depth image, and the remaining depth images of the current batch are traversed to perform category pixel feature pooling to finally obtain the depth center pixel features of the current batch;

[0016] S3: Use the RGB-D-aware anchor sample sampling method to select anchor samples for multiple RGB pixel embedding features of the current batch to obtain the anchor sample set of the current batch;

[0017] S4: According to the anchor sample set of the current batch, the positive and negative sample sampling method of mining some difficult samples is used to select positive and negative samples for the depth random pixel features and depth center pixel features of the current batch to obtain the positive and negative sample set of the current batch. By measuring the similarity between the anchor sample set and the positive and negative sample set, comparative learning is performed to obtain the positive and negative difficult sample set of the current batch.

[0018] The S3 is specifically:

[0019] S3.1: Randomly sample pixel positions of multiple RGB pixel embedding features of the current batch, and use the sampled pixel positions as half of the anchor points; sample pixel positions that are misclassified in the RGB semantic decoder but correctly classified in the deep semantic decoder, and use the sampled pixel positions as the other half of the anchor points;

[0020] S3.2: Extract pixel features corresponding to all anchor points in S3.1 from multiple RGB pixel embedding features as anchor point samples, and each anchor point sample constitutes an anchor point sample set for calculating the cross-modal contrast loss.

[0021] The S4 is specifically:

[0022] S4.1: The pixel features in the depth random pixel features and the depth center pixel features that belong to the same category as each anchor sample in the current anchor sample set are taken as the positive sample set of the current anchor sample; the pixel features in the depth random pixel features and the depth center pixel features that do not belong to the same category as the current anchor sample are taken as the negative sample set of the current anchor sample;

[0023] S4.2: Calculate the similarity between the current anchor sample and each positive sample in the positive sample set of the current anchor sample and sort the similarities in ascending order, take the positive samples with the top 10% similarity as the difficult sample set of the current positive sample, and then randomly sample K1 samples from the difficult sample set of the current positive sample set as the positive and difficult sample set of the current anchor sample; the positive and negative difficult sample sets of the current anchor sample are composed of the positive and negative difficult sample sets of the current anchor sample;

[0024] S4.3: Calculate the similarity between the current anchor sample and each negative sample in the negative sample set of the current anchor sample and sort the similarities in descending order, take the negative samples with the top 10% similarity as the difficult sample set of the current negative sample, and then randomly sample K2 samples from the difficult sample set of the current negative sample set as the negative difficult sample set of the current anchor sample;

[0025] S4.4: Repeat S4.1-S4.3, traverse the remaining anchor point samples in the current anchor point sample set, obtain the positive and negative difficult sample sets of each anchor point sample, and the positive and negative difficult sample sets of each anchor point sample constitute the positive and negative difficult sample sets of the current batch.

[0026] Based on the output of the contrastive learning module in step 2), the cross-modal contrast loss of the contrastive learning module is calculated by the following formula:

[0027]

[0028] in, represents the cross-modal contrast loss of pixel i, r i Represents the RGB pixel embedding feature of pixel i, using and Represent the positive and negative difficult samples of pixel i respectively, and They represent the positive and negative difficult sample sets where pixel i is located, τ represents the temperature coefficient, Indicates the size of the positive difficult sample set where the pixel i is calculated.

[0029] The multimodal total loss is calculated by the following formula:

[0030]

[0031]

[0032] in, represents the total multimodal loss, represents the joint loss of semantic segmentation, λ1 and λ2 represent the first and second loss weights respectively, and M represents the number of anchor samples in the anchor sample set; denote the semantic segmentation losses in the RGB image branch and the depth image branch, respectively.

[0033] The semantic segmentation losses of the RGB image branch and the depth image branch both use pixel-by-pixel cross entropy loss.

[0034] The multiple depth images collected in step 1) are decomposed into a three-channel form of HHA: parallax, height, and normal vector.

[0035] The present invention has the following beneficial effects:

[0036] 1. Compared with the semantic segmentation method that only uses RGB images, this method introduces the geometric information of the depth image, which has better performance in cases where segmentation is difficult due to similar color and texture, so it can better segment scenes under different lighting conditions.

[0037] 2. Because this method is based on the depth privilege setting, it does not require the participation of depth images when segmenting unknown scene images. It only needs to retain the RGB semantic segmentation basic network for segmentation. Therefore, under the premise of using the same semantic segmentation basic network, the model size and inference speed are better than most RGB-D semantic segmentation methods.

[0038] 3. This method is applicable to most existing semantic segmentation basic networks based on encoder-decoder structures, so it has good versatility and universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the network model of the present invention.

[0040] Figure 2 This is an example of the semantic segmentation result of the present invention.

[0041] Figure 3 It is a flow chart of the contrastive learning module of the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with the accompanying drawings. The embodiments of the present invention and their implementation process are as follows:

[0043] The present invention comprises the following steps:

[0044] 1) Collect multiple RGB images and corresponding depth images of scenes that need semantic segmentation, and the required training set is composed of multiple RGB images and corresponding depth images as well as the category of each pixel in each RGB image;

[0045] In a specific implementation, the multiple depth images collected in step 1) are decomposed into a three-channel form of HHA of disparity-height-normal vector.

[0046] 2) After the required training set is processed in batches, the RGB images and the corresponding depth images are input into the multimodal contrastive learning semantic segmentation network in batches. The structural diagram of the multimodal contrastive learning semantic segmentation network is shown in Figure 1 As shown, the multimodal contrastive learning semantic segmentation network consists of an RGB image branch, a deep image branch and a contrastive learning module. The RGB image branch is connected to the deep image branch through the contrastive learning module. Then, the semantic segmentation losses of the RGB image branch and the deep image branch are calculated respectively, and the cross-modal contrast loss of the contrastive learning module is also calculated. Then, the multimodal total loss is calculated according to the semantic segmentation loss and the cross-modal contrast loss of the RGB image branch and the deep image branch. The multimodal contrastive learning semantic segmentation network is trained based on the multimodal total loss to obtain a trained multimodal contrastive learning semantic segmentation network; wherein, the RGB image branch includes an RGB feature encoder, an RGB semantic decoder and an RGB projection module. The output of the RGB feature encoder is respectively input into the RGB semantic decoder and the RGB projection module. The RGB projection module is connected to the contrastive learning module. The RGB feature encoder and the RGB semantic decoder constitute an RGB semantic segmentation basic network. The output of the RGB semantic decoder is used as the output of the RGB semantic segmentation basic network to obtain a trained RGB semantic segmentation basic network.

[0047] The deep image branch includes a deep feature encoder, a deep semantic decoder and a deep projection module. The output of the deep feature encoder is input into the deep semantic decoder and the deep projection module respectively. The deep projection module is connected to the contrastive learning module. The deep feature encoder and the deep semantic decoder constitute the deep semantic segmentation basic network. The output of the deep semantic decoder is used as the output of the deep semantic segmentation basic network.

[0048] In the specific implementation, the semantic segmentation basic network can theoretically adopt any deep learning semantic segmentation method based on the encoder-decoder structure. Here, DeepLabV3+ is taken as an example. In addition, ResNet-101 can be selected and the ResNet-101 trained with the ImageNet dataset can be used as the feature encoder of the semantic segmentation basic network. The network structure of the semantic decoder is still the network structure of the decoder in DeepLabV3+.

[0049] The semantic segmentation losses of both the RGB image branch and the depth image branch use pixel-by-pixel cross entropy loss.

[0050] The structure of the RGB projection module is the same as that of the depth projection module, which is composed of a first convolution layer with a convolution kernel size of 1×1, a batch normalization layer, a RELU activation function layer, and a second convolution layer with a convolution kernel size of 1×1. The projection module obtains the RGB features after the input RGB image passes through the RGB feature encoder. Dimensionality reduction to RGB embedding feature R′∈R H×W×C′ , the input depth image is passed through the depth feature encoder to obtain the depth feature Dimensionality reduction to deep embedding feature D′∈R H×W×C′ ,In the specific implementation, the feature after dimensionality reduction is a 256-dimensional feature vector.

[0051] In step 2), the RGB projection module outputs multiple RGB pixel embedding features, and the depth projection module outputs multiple depth pixel embedding features. The multiple RGB pixel embedding features and the depth pixel embedding features of each batch are input into the contrast learning module. The contrast learning module samples the multiple RGB pixel embedding features and the depth pixel embedding features of the current batch to obtain and output the positive and negative difficult sample sets of the current batch, specifically, as follows: Figure 3 As shown:

[0052] S1: Randomly sample P pixel positions of all RGB images in each batch, extract pixel features corresponding to P pixel positions in multiple deep pixel embedding features and use them as deep random pixel features of the current batch; in the specific implementation, P = 500.

[0053] S2: In each depth image of the current batch, the pixel features belonging to the same category in the multiple depth pixel embedding features of each depth image are pooled to obtain the depth center pixel features of each category of the current depth image, and the remaining depth images of the current batch are traversed to perform category pixel feature pooling to finally obtain the depth center pixel features of the current batch;

[0054] S3: Use the RGB-D-aware anchor sample sampling method to select anchor samples for multiple RGB pixel embedding features of the current batch to obtain the anchor sample set of the current batch;

[0055] S3 is specifically:

[0056] S3.1: Randomly sample pixel positions of multiple RGB pixel embedding features of the current batch, and use the sampled pixel positions as half of the anchor points; sample pixel positions that are misclassified in the RGB semantic decoder but correctly classified in the deep semantic decoder, and use the sampled pixel positions as the other half of the anchor points;

[0057] S3.2: Extract pixel features corresponding to all anchor points in S3.1 from multiple RGB pixel embedding features as anchor point samples, and each anchor point sample constitutes an anchor point sample set for calculating the cross-modal contrast loss.

[0058] S4: According to the anchor sample set of the current batch, the positive and negative sample sampling method of mining some difficult samples is used to select positive and negative samples for the depth random pixel features and depth center pixel features of the current batch to obtain the positive and negative sample set of the current batch. By measuring the similarity between the anchor sample set and the positive and negative sample set, comparative learning is performed to obtain the positive and negative difficult sample set of the current batch.

[0059] S4 is specifically:

[0060] S4.1: The pixel features in the depth random pixel features and the depth center pixel features that belong to the same category as each anchor sample in the current anchor sample set are taken as the positive sample set of the current anchor sample; the pixel features in the depth random pixel features and the depth center pixel features that do not belong to the same category as the current anchor sample are taken as the negative sample set of the current anchor sample;

[0061] S4.2: Calculate the similarity between the current anchor sample and each positive sample in the positive sample set of the current anchor sample and sort the similarities in ascending order, take the positive samples with the top 10% similarity as the difficult sample set of the current positive sample, and then randomly sample K1 samples from the difficult sample set of the current positive sample set as the positive and difficult sample set of the current anchor sample; the positive and negative difficult sample sets of the current anchor sample are composed of the positive and negative difficult sample sets of the current anchor sample;

[0062] S4.3: Calculate the similarity between the current anchor sample and each negative sample in the negative sample set of the current anchor sample and sort the similarities in descending order, take the negative samples with the top 10% similarity as the difficult sample set of the current negative sample, and then randomly sample K2 samples from the difficult sample set of the current negative sample set as the negative difficult sample set of the current anchor sample; in the specific implementation, the similarity is specifically the dot product of the two samples. In the specific implementation, K1=K2=1024.

[0063] S4.4: Repeat S4.1-S4.3, traverse the remaining anchor point samples in the current anchor point sample set, obtain the positive and negative difficult sample sets of each anchor point sample, and the positive and negative difficult sample sets of each anchor point sample constitute the positive and negative difficult sample sets of the current batch.

[0064] Based on the output of the contrastive learning module in step 2), the cross-modal contrast loss of the contrastive learning module is calculated by the following formula:

[0065]

[0066] in, represents the cross-modal contrast loss of pixel i, r i Represents the RGB pixel embedding feature of pixel i, using and Represent the positive and negative difficult samples of pixel i respectively, and They represent the positive and negative difficult sample sets where pixel i is located, respectively. τ represents the temperature coefficient. The temperature coefficient τ is an artificially set hyperparameter, where τ = 0.1. Indicates the size of the positive difficult sample set where the pixel i is calculated.

[0067] The total multimodal loss is calculated using the following formula:

[0068]

[0069]

[0070] in, represents the total multimodal loss, represents the semantic segmentation joint loss, λ1 and λ2 represent the first and second loss weights respectively, λ1 is a manually adjusted parameter used to balance the values ​​of the semantic segmentation loss function of the two branches, and λ2 is a manually set hyperparameter, where λ2=1; M represents the number of anchor samples in the anchor sample set, and in the specific implementation, M=500; denote the semantic segmentation losses in the RGB image branch and the depth image branch, respectively.

[0071] 3) The RGB image to be segmented is input into the RGB semantic segmentation basic network of the trained multimodal contrastive learning semantic segmentation network for semantic segmentation to obtain the RGB image after semantic segmentation.

[0072] In the specific implementation, the SGD algorithm with momentum is used for optimization training. The specific settings of the optimizer are: the initial learning rate is set to 0.0025, the momentum is set to 0.9, and the weight decay coefficient is set to 0.0005. In addition, the poly learning rate strategy is adopted to automatically adjust the learning rate size as the number of iterations increases.

[0073] Figure 2 Some segmentation results of the embodiment on the public dataset NYU Depth V2 are shown, and the segmentation effect is improved to a certain extent.

[0074] In order to illustrate the effectiveness of the network framework and cross-modal comparison performance proposed in this paper, the public dataset NYU Depth V2 is used for verification. The NYU Depth V2 dataset consists of video sequences of various indoor scenes captured by Microsoft Kinect, covering 464 scenes in three cities, and provides 1449 paired RGB-D images. It is the most commonly used dataset in the task of RGBD semantic segmentation of indoor scenes. In order to evaluate the performance of semantic segmentation, the commonly used pixel accuracy (Pixel Accuracy, pAcc) and average pixel are used.

[0075] The accuracy (Mean Pixel Accuracy, mAce) and mean intersection over Union (Mean Intersection over Union, mIoU) are used as evaluation indicators. The semantic segmentation evaluation results of different model variants are shown in Table 1:

[0076] The five model variants in Table 1 are: M1 means only cross entropy loss is used Supervised training of the semantic segmentation base network of the RGB branch; M2 represents the use of joint cross entropy loss The semantic segmentation basic network model of the optimized RGB branch and depth branch; M3 represents the use of joint cross entropy semantic segmentation loss And the cross-modal contrast loss proposed in this invention The complete network model for optimization.

[0077] Table 1

[0078]

[0079] By comparing M2 and M1, it can be observed that the simple combination of the RGB branch and the depth branch is of little help, but by adding the cross-modal contrastive learning (M3) proposed in this invention, the performance can be effectively improved, and all indicators can be improved. Figure 2 , which performs better for situations where segmentation is difficult due to similar color and texture.

[0080] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention, but are not intended to limit the present invention. Any changes and modifications made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A deep privileged semantic segmentation method based on multimodal contrastive learning, characterized in that: The steps include: 1) Collect multiple RGB images and corresponding depth images of scenes that need semantic segmentation, and the required training set is composed of multiple RGB images and corresponding depth images as well as the category of each pixel in each RGB image; 2) After the required training set is processed in batches, the RGB images and the corresponding depth images are input into the multimodal contrastive learning semantic segmentation network in batches. The multimodal contrastive learning semantic segmentation network consists of an RGB image branch, a depth image branch and a contrastive learning module. The RGB image branch is connected to the depth image branch through the contrastive learning module. Then, the semantic segmentation losses of the RGB image branch and the depth image branch are calculated respectively, and the cross-modal contrast loss of the contrastive learning module is also calculated. Then, the multimodal total loss is calculated according to the semantic segmentation loss and the cross-modal contrast loss of the RGB image branch and the depth image branch. The multimodal contrastive learning semantic segmentation network is trained based on the multimodal total loss to obtain a trained multimodal contrastive learning semantic segmentation network; wherein, the RGB image branch includes an RGB feature encoder, an RGB semantic decoder and an RGB projection module, and the output of the RGB feature encoder is respectively input into the RGB semantic decoder and the RGB projection module, the RGB projection module is connected to the contrastive learning module, the RGB feature encoder and the RGB semantic decoder constitute an RGB semantic segmentation basic network, and the output of the RGB semantic decoder is used as the output of the RGB semantic segmentation basic network; 3) The RGB image to be segmented is input into the RGB semantic segmentation basic network of the trained multimodal contrastive learning semantic segmentation network for semantic segmentation to obtain the RGB image after semantic segmentation.

2. According to claim 1, a deep privileged semantic segmentation method based on multimodal contrastive learning is characterized in that: The depth image branch includes a depth feature encoder, a depth semantic decoder and a depth projection module. The output of the depth feature encoder is respectively input into the depth semantic decoder and the depth projection module, and the depth projection module is connected to the contrast learning module.

3. The deep privileged semantic segmentation method based on multimodal contrastive learning according to claim 1, characterized in that: The RGB projection module and the depth projection module have the same structure, specifically, they are composed of a first convolutional layer, a batch normalization layer, an activation function layer, and a second convolutional layer connected in sequence.

4. The deep privileged semantic segmentation method based on multimodal contrastive learning according to claim 1, characterized in that: In step 2), the RGB projection module outputs multiple RGB pixel embedding features, and the depth projection module outputs multiple depth pixel embedding features. The multiple RGB pixel embedding features and the depth pixel embedding features of each batch are input into the contrast learning module. The contrast learning module samples the multiple RGB pixel embedding features and the depth pixel embedding features of the current batch, obtains the positive and negative difficult sample sets of the current batch and outputs them, specifically: S1: Randomly sample P pixel positions of all RGB images in each batch, extract pixel features corresponding to P pixel positions in multiple deep pixel embedding features and use them as deep random pixel features of the current batch; S2: In each depth image of the current batch, the pixel features belonging to the same category in the multiple depth pixel embedding features of each depth image are pooled to obtain the depth center pixel features of each category of the current depth image, and the remaining depth images of the current batch are traversed to perform category pixel feature pooling to finally obtain the depth center pixel features of the current batch; S3: Use the RGB-D-aware anchor sample sampling method to select anchor samples for multiple RGB pixel embedding features of the current batch to obtain the anchor sample set of the current batch; S4: According to the anchor sample set of the current batch, the positive and negative sample sampling method of mining some difficult samples is used to select positive and negative samples for the depth random pixel features and depth center pixel features of the current batch to obtain the positive and negative sample set of the current batch. By measuring the similarity between the anchor sample set and the positive and negative sample set, comparative learning is performed to obtain the positive and negative difficult sample set of the current batch.

5. The deep privileged semantic segmentation method based on multimodal contrastive learning according to claim 4 is characterized in that: The S3 is specifically: S3.1: Randomly sample pixel positions of multiple RGB pixel embedding features of the current batch, and use the sampled pixel positions as half of the anchor points; Sample the pixel positions that are misclassified in the RGB semantic decoder but correctly classified in the depth semantic decoder, and use the sampled pixel positions as the other half of the anchor points; S3.2: Extract pixel features corresponding to all anchor points in S3.1 from multiple RGB pixel embedding features as anchor point samples, and each anchor point sample constitutes an anchor point sample set for calculating the cross-modal contrast loss.

6. The deep privileged semantic segmentation method based on multimodal contrastive learning according to claim 4, characterized in that: The S4 is specifically: S4.1: The pixel features in the depth random pixel features and the depth center pixel features that belong to the same category as each anchor sample in the current anchor sample set are taken as the positive sample set of the current anchor sample; the pixel features in the depth random pixel features and the depth center pixel features that do not belong to the same category as the current anchor sample are taken as the negative sample set of the current anchor sample; S4.2: Calculate the similarity between the current anchor sample and each positive sample in the positive sample set of the current anchor sample and sort the similarities in ascending order, take the positive samples with the top 10% similarity as the difficult sample set of the current positive sample, and then randomly sample K1 samples from the difficult sample set of the current positive sample set as the positive and difficult sample set of the current anchor sample; the positive and negative difficult sample sets of the current anchor sample are composed of the positive and negative difficult sample sets of the current anchor sample; S4.3: Calculate the similarity between the current anchor sample and each negative sample in the negative sample set of the current anchor sample and sort the similarities in descending order, take the negative samples with the top 10% similarity as the difficult sample set of the current negative sample, and then randomly sample K2 samples from the difficult sample set of the current negative sample set as the negative difficult sample set of the current anchor sample; S4.4: Repeat S4.1-S4.3, traverse the remaining anchor point samples in the current anchor point sample set, obtain the positive and negative difficult sample sets of each anchor point sample, and the positive and negative difficult sample sets of each anchor point sample constitute the positive and negative difficult sample sets of the current batch.

7. The deep privileged semantic segmentation method based on multimodal contrastive learning according to claim 1, characterized in that: Based on the output of the contrastive learning module in step 2), the cross-modal contrast loss of the contrastive learning module is calculated by the following formula: in, represents the cross-modal contrast loss of pixel i, r i Represents the RGB pixel embedding feature of pixel i, using and Represent the positive and negative difficult samples of pixel i respectively, and They represent the positive and negative difficult sample sets where pixel i is located, τ represents the temperature coefficient, Indicates the size of the positive difficult sample set where the pixel i is calculated.

8. The method for deep privileged semantic segmentation based on multimodal contrastive learning according to claim 1, characterized in that: The multimodal total loss is calculated by the following formula: in, represents the total multimodal loss, represents the joint loss of semantic segmentation, λ1 and λ2 represent the first and second loss weights respectively, and M represents the number of anchor samples in the anchor sample set; denote the semantic segmentation losses in the RGB image branch and the depth image branch, respectively.

9. The method for deep privileged semantic segmentation based on multimodal contrastive learning according to claim 1, characterized in that: The semantic segmentation losses of the RGB image branch and the depth image branch both use pixel-by-pixel cross entropy loss.

10. The method for deep privileged semantic segmentation based on multimodal contrastive learning according to claim 1, characterized in that: The multiple depth images collected in step 1) are decomposed into a three-channel form of HHA: parallax, height, and normal vector.

Images