Unsupervised domain-adaptive semantic segmentation methods, apparatuses, devices, and readable storage media

Abstract

This invention provides an unsupervised domain-adaptive semantic segmentation method, apparatus, device, and readable storage medium. The method includes: calculating the cross-entropy loss between the label of the source domain image and the first predicted label output by the student network; calculating the consistency loss between the pseudo-label of the fused image and the second predicted label output by the student network; calculating the depth loss of the student network; calculating the gradients of the cross-entropy loss, consistency loss, and depth loss, backpropagating the gradients to update the parameters of the student network, and updating the parameters of the teacher network using an exponential moving average method; iteratively updating the student network and the teacher network, and when the number of iterations reaches a preset number, using the latest teacher network to perform semantic segmentation prediction on the unlabeled target domain image to obtain the pseudo-label of the unlabeled target domain image. Through this invention, while ensuring reasonable GPU memory usage, a segmentation result including detailed segmentation is obtained, improving the segmentation performance in the target domain.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an unsupervised domain-adaptive semantic segmentation method, apparatus, device, and readable storage medium. Background Technology

[0002] Semantic segmentation is a very broad research area, which mainly involves assigning a label to each pixel in an image that corresponds to its semantic content.

[0003] Training a high-performing deep semantic segmentation model typically requires a large amount of pixel-level labeled data. Currently, virtual synthetic images are mainly used as the source domain for training segmentation models. However, due to the appearance difference between synthetic and real images, known as domain difference, models trained only on synthetic images usually have poor generalization performance on real images. To mitigate the domain gap between the source and target domains, researchers have proposed various unsupervised domain adaptation methods. However, unsupervised domain adaptation methods for semantic segmentation usually require a large amount of GPU memory, and most previous methods only operate on scaled-down images, resulting in the inability to accurately locate contours of fine categories, thus preventing accurate detail segmentation of the unlabeled target domain. Therefore, it is necessary to design a method that can integrate contextual information and object detail information while ensuring reasonable GPU memory usage during unsupervised domain adaptation segmentation. Summary of the Invention

[0004] The main objective of this invention is to provide an unsupervised domain-adaptive semantic segmentation method, apparatus, device, and readable storage medium, which aims to obtain segmentation results including detail segmentation while ensuring reasonable GPU memory usage.

[0005] In a first aspect, the present invention provides an unsupervised domain-adaptive semantic segmentation method, the unsupervised domain-adaptive semantic segmentation method comprising:

[0006] The preprocessed source domain image is input into the student network to obtain the first predicted label output by the student network. The first predicted label is obtained by fusing the label of the semantic prediction map containing long-dependency context information with the label of the semantic prediction map containing detail segmentation.

[0007] Calculate the cross-entropy loss between the label of the source domain image and the first predicted label;

[0008] The preprocessed source domain image and the preprocessed target domain image are fused, and the fused image is input into the student network to obtain the second predicted label output by the student network.

[0009] Calculate the consistency loss between the pseudo-label of the fused image and the second predicted label;

[0010] Calculate the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label, or calculate the second depth loss between the second depth information pseudo-label corresponding to the target domain image and the second predicted label;

[0011] Calculate the gradients of cross-entropy loss, consistency loss, and either the first depth loss or the second depth loss. Backpropagate the gradients to update the parameters of the student network and use the exponential moving average method to update the parameters of the teacher network.

[0012] Using the new preprocessed source domain image as the preprocessed source domain image and the new preprocessed target domain image as the preprocessed target domain image, return to the step of inputting the preprocessed source domain image into the student network and calculating the cross-entropy loss between the first predicted label output by the student network and the label of the source domain image.

[0013] When the number of iterations reaches the preset number, the latest teacher network is used to perform semantic segmentation prediction on the unlabeled target domain image to obtain pseudo-labels for the unlabeled target domain image.

[0014] Optionally, before the step of inputting the preprocessed source domain image into the student network to obtain the first predicted label output by the student network, the following steps are included:

[0015] Obtain the source domain image and the target domain image;

[0016] The source image and the target image are cropped, flipped, color-dithered, and Gaussian blurred to obtain preprocessed source images and preprocessed target images.

[0017] The depth estimation model is trained based on the source domain image and the target domain image to obtain the trained depth estimation model;

[0018] The trained depth estimation model yields the first depth information pseudo-label corresponding to the source domain image and the second depth information pseudo-label corresponding to the target domain image.

[0019] Optionally, the step of inputting the preprocessed source domain image into the student network to obtain the first predicted label output by the student network includes:

[0020] The preprocessed source domain image is input into the student network, where the encoder performs deep feature extraction on the preprocessed source domain image to obtain backbone features at different scales. , where i is a positive integer;

[0021] Skeletal features at different scales The input is scaled into the first decoder for scale fusion and prediction, resulting in a label of a semantic prediction graph containing contextual information about long dependencies. ;

[0022] Skeletal features at different scales Input the two bottleneck layers separately to obtain depth information. and semantic information The bottleneck layer includes a 1×1 convolution for dimensionality reduction, a 3×3 convolution, and a convolution for restoring the 1×1 dimension.

[0023] The deep information features are extracted using a channel attention mechanism and then fused with semantic information to obtain semantic information that incorporates the deep features. The fusion formula is as follows:

[0024]

[0025] This represents the semantic information that incorporates deep features at the i-th scale. Represents semantic information at the i-th scale. Represents depth information at the i-th scale. This represents element-wise matrix multiplication. Indicates a linear layer. It is the normalized sigmoid function. These are learnable parameters. and These represent the parameters of the two linear layers respectively;

[0026] The semantic information, which incorporates deep features, is input into the second decoder to obtain labels from a semantic prediction map that includes detail segmentation. ;

[0027] The labeling of the semantic prediction graph, which contains contextual information including long dependencies, is obtained through a channel attention mechanism. Labels with semantic prediction graphs containing detail segmentation The data is then fused to obtain the first predicted label.

[0028] Optionally, the step of calculating the cross-entropy loss between the label of the source domain image and the first predicted label includes:

[0029] The cross-entropy loss between the first predicted label output by the student network and the label of the source domain image is calculated using the first loss function formula, where the first loss function formula is as follows:

[0030]

[0031] in, Labels representing the source domain image. Represents the first prediction label, These are height, width, and number of channels, where C represents the number of channels. This represents the cross-entropy loss.

[0032] Optionally, the pseudo-labels of the fused image are obtained in the following manner:

[0033] The target domain image is input into the teacher network, and the obtained segmentation prediction results are used as pseudo-labels for the target domain image.

[0034] The labels of the source domain image and the pseudo labels of the target domain image are blended with the same mask to obtain the pseudo labels of the fused image.

[0035] Optionally, the step of calculating the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label includes:

[0036] The difference is obtained by subtracting the first predicted label from the first depth information pseudo-label corresponding to the source domain image. ;

[0037] The difference Substituting into the second loss function formula, we obtain the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label. The second loss function formula is as follows:

[0038]

[0039] It is a constant.

[0040] Optionally, the step of updating the parameters of the teacher network using the exponential moving average method includes:

[0041] Substitute the updated parameters for students' networks and the parameters for teachers' networks before the update into the preset formula. The updated parameters of the teacher network are obtained, among which, The parameters represent the updated parameters of the teacher network. The parameters represent those before the teacher network update. The parameters represent the updated student network information. These are weighting coefficients, 0 < <1.

[0042] Secondly, the present invention also provides an unsupervised domain-adaptive semantic segmentation apparatus, the unsupervised domain-adaptive semantic segmentation apparatus comprising:

[0043] The first acquisition module is used to input the preprocessed source domain image into the student network to obtain the first predicted label output by the student network. The first predicted label is obtained by fusing the label of the semantic prediction map containing long-dependency context information with the label of the semantic prediction map containing detail segmentation.

[0044] The first calculation module is used to calculate the cross-entropy loss between the label of the source domain image and the first predicted label;

[0045] The second acquisition module is used to fuse the preprocessed source domain image and the preprocessed target domain image, input the fused image into the student network, and obtain the second predicted label output by the student network.

[0046] The second calculation module is used to calculate the consistency loss between the pseudo-label of the fused image and the second predicted label;

[0047] The third calculation module is used to calculate the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first prediction label, or to calculate the second depth loss between the second depth information pseudo-label corresponding to the target domain image and the second prediction label.

[0048] The parameter update module is used to calculate the gradients of cross-entropy loss, consistency loss, and first depth loss or second depth loss, backpropagate the gradients to update the parameters of the student network, and use the exponential moving average method to update the parameters of the teacher network.

[0049] The loop module is used to take the new preprocessed source domain image as the preprocessed source domain image and the new preprocessed target domain image as the preprocessed target domain image, and return to execute the step of inputting the preprocessed source domain image into the student network and calculating the cross-entropy loss between the first predicted label output by the student network and the label of the source domain image.

[0050] The segmentation prediction module is used to perform semantic segmentation prediction on the unlabeled target domain image using the latest teacher network when the number of iterations reaches a preset number, so as to obtain pseudo-labels for the unlabeled target domain image.

[0051] Thirdly, the present invention also provides an unsupervised domain-adaptive semantic segmentation device, the unsupervised domain-adaptive semantic segmentation device including a processor, a memory, and an unsupervised domain-adaptive semantic segmentation program stored in the memory and executable by the processor, wherein when the unsupervised domain-adaptive semantic segmentation program is executed by the processor, it implements the steps of the unsupervised domain-adaptive semantic segmentation method as described above.

[0052] Fourthly, the present invention also provides a readable storage medium storing an unsupervised domain-adaptive semantic segmentation program, wherein when the unsupervised domain-adaptive semantic segmentation program is executed by a processor, it implements the steps of the unsupervised domain-adaptive semantic segmentation method as described above.

[0053] In this invention, a preprocessed source domain image is input into a student network to obtain a first predicted label output by the student network. This first predicted label is obtained by fusing the label of a semantic prediction graph containing long-dependency contextual information with the label of a semantic prediction graph containing detail segmentation. The cross-entropy loss between the label of the source domain image and the first predicted label is calculated. The preprocessed source domain image and the preprocessed target domain image are then fused, and the fused image is input into the student network to obtain a second predicted label output by the student network. The consistency loss between the pseudo-label of the fused image and the second predicted label is calculated. A first depth loss is calculated between the first depth information pseudo-label corresponding to the source domain image and the first predicted label, or a second depth loss is calculated between the pseudo-label corresponding to the target domain image and the first predicted label. The second depth loss is calculated between the pseudo-label of the depth information and the second predicted label; the gradient of the cross-entropy loss, consistency loss, and the first depth loss or the second depth loss is calculated, the gradient is backpropagated to update the parameters of the student network, and the parameters of the teacher network are updated using the exponential moving average method; the new preprocessed source domain image is used as the preprocessed source domain image, and the new preprocessed target domain image is used as the preprocessed target domain image, and the process is returned to the step of inputting the preprocessed source domain image into the student network and calculating the cross-entropy loss between the first predicted label output by the student network and the label of the source domain image; when the number of iterations reaches the preset number, the latest teacher network is used to perform semantic segmentation prediction on the unlabeled target domain image to obtain the pseudo-label of the unlabeled target domain image. This invention addresses the challenge that, because semantic segmentation requires significant GPU resources, most previous methods only operated on scaled-down images. However, low-resolution predictions often fail to preserve details. While this method also operates on scaled-down images, the student and teacher networks, each containing an encoder, a decoder for predicting long-dependency contextual information, and a decoder for predicting detail information, ensure that the semantic segmentation predictions from both networks do not increase GPU usage while preserving prediction details. Therefore, by employing the latest teacher network—the fully trained teacher network—for semantic segmentation prediction of unlabeled target domain images, segmentation results containing detail segmentation can be obtained while ensuring reasonable GPU memory usage, thus improving segmentation performance in the target domain. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating an embodiment of the unsupervised domain-adaptive semantic segmentation method of the present invention;

[0055] Figure 2 for Figure 1 A detailed flowchart of step S10;

[0056] Figure 3 This is a schematic diagram of the student network architecture process according to an embodiment of the unsupervised domain adaptive semantic segmentation method of the present invention;

[0057] Figure 4 This is a schematic diagram of the functional modules of an embodiment of the unsupervised domain-adaptive semantic segmentation device of the present invention;

[0058] Figure 5 This is a schematic diagram of the hardware structure of the unsupervised domain-adaptive semantic segmentation device involved in the embodiments of the present invention.

[0059] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0060] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0061] In a first aspect, embodiments of the present invention provide an unsupervised domain-adaptive semantic segmentation method.

[0062] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the unsupervised domain-adaptive semantic segmentation method of the present invention. Figure 1 As shown, unsupervised domain-adaptive semantic segmentation methods include:

[0063] Step S10: Input the preprocessed source domain image into the student network to obtain the first predicted label output by the student network. The first predicted label is obtained by fusing the label of the semantic prediction graph containing long-dependency context information with the label of the semantic prediction graph containing detail segmentation.

[0064] In this embodiment, the source domain image is an image with corresponding pixel-level semantic annotations. The preprocessed source domain image is input into the student network to obtain the first predicted label output by the student network. The first predicted label output by the student network is obtained by fusing the label of the semantic prediction map containing contextual information about long dependencies with the label of the semantic prediction map containing detail segmentation.

[0065] Step S20: Calculate the cross-entropy loss between the label of the source domain image and the first predicted label;

[0066] In this embodiment, after obtaining the first predicted label output by the student network, the cross-entropy loss between the label of the source domain image and the first predicted label is calculated, so as to provide supervised training of the student network based on the cross-entropy loss.

[0067] Further, in one embodiment, step S20 includes:

[0068] The cross-entropy loss between the first predicted label output by the student network and the label of the source domain image is calculated using the first loss function formula, where the first loss function formula is as follows:

[0069]

[0070] in, Labels representing the source domain image. Represents the first prediction label, These are height, width, and number of channels, where C represents the number of channels. This represents the cross-entropy loss.

[0071] In this embodiment, since cross-entropy compares the model's prediction results with the true labels of the data, the value of cross-entropy decreases as the prediction becomes more accurate, and reaches zero if the prediction is completely correct. Therefore, cross-entropy can be used as a loss function when training the classification model. Substituting the first predicted label output by the student network and the label of the source domain image into the first loss function formula, the cross-entropy loss between the first predicted label output by the student network and the label of the source domain image is calculated. The first loss function formula is as follows:

[0072]

[0073] in, Labels representing the source domain image. Represents the first prediction label, These are height, width, and number of channels, where C represents the number of channels. This represents the cross-entropy loss.

[0074] Step S30: The preprocessed source domain image and the preprocessed target domain image are fused, and the fused image is input into the student network to obtain the second predicted label output by the student network.

[0075] In this embodiment, in order to speed up convergence, the ClassMix algorithm is used to fuse the preprocessed source domain image and the preprocessed target domain image. Specifically, some categories in the source domain image are randomly selected and pasted into the target domain image for fusion to obtain a fused image.

[0076] The fused image is input into the student network to obtain the second predicted label output by the student network. The second predicted label output by the student network is obtained by fusing the label of the semantic prediction graph containing contextual information about long dependencies with the label of the semantic prediction graph containing detail segmentation.

[0077] Step S40: Calculate the consistency loss between the pseudo-label of the fused image and the second predicted label;

[0078] In this embodiment, to ensure that the prediction results of the teacher network and the student network for the common parts of the image are as consistent as possible, a consistency loss is used as a constraint. The pseudo-label of the fused image and the second predicted label are substituted into the third loss function formula to calculate the consistency loss between the pseudo-label and the second predicted label of the fused image. The third loss function formula is as follows:

[0079]

[0080] in, Pseudo-labels representing the merged image. Represents the second prediction label, These are height, width, and number of channels, where C represents the number of channels. This represents a loss of consistency.

[0081] Furthermore, in one embodiment, the pseudo-labels of the fused image are obtained in the following manner:

[0082] The target domain image is input into the teacher network, and the obtained segmentation prediction results are used as pseudo-labels for the target domain image.

[0083] The labels of the source domain image and the pseudo labels of the target domain image are blended with the same mask to obtain the pseudo labels of the fused image.

[0084] In this embodiment, the target domain image is input into the teacher network. Since the target domain image is unlabeled, the segmentation prediction result output by the teacher network, i.e., the pseudo-label, is obtained. .

[0085] Label the source domain image pseudo-labels of the target domain image Blend using the same mask to obtain a merged image. pseudo-tags .

[0086] Step S50: Calculate the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label, or calculate the second depth loss between the second depth information pseudo-label corresponding to the target domain image and the second predicted label.

[0087] In this embodiment, the pseudo-label of the first depth information corresponding to the source domain image is used. Subtract the first prediction label The difference is obtained. The difference Substituting into the second loss function formula, we obtain the first depth information pseudo-label corresponding to the source domain image. With the first prediction label First depth loss between .

[0088] Alternatively, pseudo-labels based on the second depth information of the target domain image can be used. Subtract the second predicted label The difference is obtained. The difference Substituting into the second loss function formula, we obtain the first depth information pseudo-label corresponding to the target domain image. With the second prediction label First depth loss between .

[0089] Further, in one embodiment, the step of calculating the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label includes:

[0090] The difference is obtained by subtracting the first predicted label from the first depth information pseudo-label corresponding to the source domain image. ;

[0091] The difference Substituting into the second loss function formula, we obtain the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label. The second loss function formula is as follows:

[0092]

[0093] It is a constant.

[0094] In this embodiment, the Huber loss function is used to calculate the loss for depth estimation. The formula corresponding to the Huber loss function is the second loss function formula. The second loss function formula is as follows:

[0095]

[0096] This is a constant. That is, the difference is obtained by subtracting the first predicted label from the first depth information pseudo-label corresponding to the source domain image. The difference Substituting into the second loss function formula, we obtain the first depth information pseudo-label corresponding to the source domain image. With the first prediction label First depth loss between .

[0097] Furthermore, the pseudo-label of the second depth information corresponding to the target domain image is... Subtract the second predicted label The difference Substituting into the second loss function formula, we obtain the first depth information pseudo-label corresponding to the target domain image. With the second prediction label First depth loss between Specifically:

[0098] .

[0099] Step S60: Calculate the gradients of cross-entropy loss, consistency loss, and first depth loss or second depth loss, backpropagate the gradients to update the parameters of the student network, and use the exponential moving average method to update the parameters of the teacher network.

[0100] In this embodiment, the gradients of cross-entropy loss, consistency loss, and first depth loss or second depth loss are calculated. The parameters of the student network are updated using the gradient backpropagation algorithm, and the parameters of the teacher network are updated using the exponential moving average method.

[0101] Furthermore, in one embodiment, the step of updating the parameters of the teacher network using the exponential moving average method includes:

[0102] Substitute the updated parameters for students' networks and the parameters for teachers' networks before the update into the preset formula. The updated parameters of the teacher network are obtained, among which, The parameters represent the updated parameters of the teacher network. The parameters represent those before the teacher network update. The parameters represent the updated student network information. These are weighting coefficients, 0 < <1.

[0103] In this embodiment, the exponential moving average method is simply referred to as the exponential smoothing method. It is a forecasting method that uses the actual value and the predicted value (estimated value) of the previous period, assigns them different weights, and obtains an exponentially smoothed value as the predicted value for the next period. Specifically, it uses the updated parameters of the student network to update the parameters of the teacher network, thus obtaining the updated parameters of the teacher network. Specifically, the updated parameters of the student network and the unupdated parameters of the teacher network are substituted into a preset formula. The updated parameters of the teacher network are obtained, among which, The parameters represent the updated parameters of the teacher network. The parameters represent those before the teacher network update. The parameters represent the updated student network information. These are weighting coefficients, 0 < <1.

[0104] Step S70: Using the new preprocessed source domain image as the preprocessed source domain image and the new preprocessed target domain image as the preprocessed target domain image, return to the step of inputting the preprocessed source domain image into the student network and calculating the cross-entropy loss between the first predicted label output by the student network and the label of the source domain image.

[0105] In this embodiment, the student network and teacher network need to be trained multiple times. Therefore, after updating the parameters of the student network and the teacher network, the new preprocessed source domain image is used as the preprocessed source domain image, and the new preprocessed target domain image is used as the preprocessed target domain image. Then, the step of inputting the preprocessed source domain image into the student network and calculating the cross-entropy loss between the first predicted label output by the student network and the label of the source domain image is returned to be executed. That is, the student network and teacher network are trained again.

[0106] Step S80: When the number of iterations reaches the preset number, the latest teacher network is used to perform semantic segmentation prediction on the unlabeled target domain image to obtain the pseudo-label of the unlabeled target domain image.

[0107] In this embodiment, when the number of iterations reaches a preset number, the latest teacher network is the teacher network that has been trained after the last parameter update. The trained teacher network is then used to perform semantic segmentation prediction on the unlabeled target domain image, thus obtaining the pseudo-labels for the unlabeled target domain image predicted by the teacher network. The pseudo-labels output by the teacher network fuse the labels from the long-dependency contextual information semantic prediction map and the labels from the detail segmentation semantic prediction map.

[0108] In this embodiment, the preprocessed source domain image is input into the student network to obtain the first predicted label output by the student network. The first predicted label is obtained by fusing the label of a semantic prediction graph containing long-dependency contextual information with the label of a semantic prediction graph containing detail segmentation. The cross-entropy loss between the label of the source domain image and the first predicted label is calculated. The preprocessed source domain image and the preprocessed target domain image are fused, and the fused image is input into the student network to obtain the second predicted label output by the student network. The consistency loss between the pseudo-label of the fused image and the second predicted label is calculated. The first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label is calculated, or the second depth loss between the pseudo-label corresponding to the target domain image and the first predicted label is calculated. The second depth loss is calculated between the pseudo-label of the depth information and the second predicted label; the gradient of the cross-entropy loss, consistency loss, and the first depth loss or the second depth loss is calculated, the gradient is backpropagated to update the parameters of the student network, and the parameters of the teacher network are updated using the exponential moving average method; the new preprocessed source domain image is used as the preprocessed source domain image, and the new preprocessed target domain image is used as the preprocessed target domain image, and the process is returned to the step of inputting the preprocessed source domain image into the student network and calculating the cross-entropy loss between the first predicted label output by the student network and the label of the source domain image; when the number of iterations reaches the preset number, the latest teacher network is used to perform semantic segmentation prediction on the unlabeled target domain image to obtain the pseudo-label of the unlabeled target domain image. In this embodiment, because semantic segmentation requires a large amount of GPU, most previous methods only operate on scaled-down images. However, low-resolution predictions often fail to preserve details. In this method, although the operation is also performed on scaled-down images, the student and teacher networks contain an encoder, a decoder for predicting contextual information for long dependencies, and a decoder for predicting details. This allows the semantic segmentation prediction results of the student and teacher networks to not only avoid increasing GPU usage but also preserve prediction details. Therefore, by using the latest teacher network, i.e., the trained teacher network, to perform semantic segmentation prediction on unlabeled target domain images, we can obtain segmentation results containing details while ensuring reasonable GPU memory usage, thus improving segmentation performance in the target domain.

[0109] Further, in one embodiment, before step S10, the following steps are included:

[0110] Obtain the source domain image and the target domain image;

[0111] The source image and the target image are cropped, flipped, color-dithered, and Gaussian blurred to obtain preprocessed source images and preprocessed target images.

[0112] The depth estimation model is trained based on the source domain image and the target domain image to obtain the trained depth estimation model;

[0113] The trained depth estimation model yields the first depth information pseudo-label corresponding to the source domain image and the second depth information pseudo-label corresponding to the target domain image.

[0114] In this embodiment, a large number of source domain images and target domain images are acquired, and it is assumed that... The source domain image set, where each source domain image For color RGB images, the corresponding For the relevant pixel-level semantic annotations. This represents the i-th image in the source domain image set. The image represents the pixel-level semantic annotation, or label, corresponding to the i-th image in the source domain image set. The target domain image set T contains only unlabeled images, and H and W represent the height and width of the image, respectively.

[0115] Each source and target domain image is cropped to a size of 512*512 pixels. Then, data augmentation is performed by flipping, color dithering, and Gaussian blurring to obtain the preprocessed source and target domain images.

[0116] A depth estimation model is trained in a self-supervised manner based on each source domain image and each target domain image to obtain the trained depth estimation model;

[0117] The depth of the source domain image is predicted by the trained depth estimation model, and the first depth information pseudo-label corresponding to the source domain image is obtained. The depth of the target domain image is predicted by the trained depth estimation model, and the pseudo-label of the second depth information corresponding to the target domain image is obtained. .

[0118] Furthermore, in one embodiment, reference is made to Figure 2 , Figure 2 for Figure 1 A detailed flowchart of step S10. (See attached diagram.) Figure 2 As shown, step S10 includes:

[0119] Step S101: The preprocessed source domain image is input into the student network. The encoder in the student network performs deep feature extraction on the preprocessed source domain image to obtain backbone features at different scales. , where i is a positive integer;

[0120] Step S102: The skeletal features at different scales are... The input is scaled into the first decoder for scale fusion and prediction, resulting in a label of a semantic prediction graph containing contextual information about long dependencies. ;

[0121] Step S103: The skeletal features at different scales are... Input the two bottleneck layers separately to obtain depth information. and semantic information The bottleneck layer includes a 1×1 convolution for dimensionality reduction, a 3×3 convolution, and a convolution for restoring the 1×1 dimension.

[0122] Step S104: Features of the depth information are extracted using a channel attention mechanism and fused with the depth information to obtain semantic information that incorporates the depth features. The fusion formula is as follows:

[0123]

[0124] This represents the semantic information that incorporates deep features at the i-th scale. Represents semantic information at the i-th scale. Represents depth information at the i-th scale. This represents element-wise matrix multiplication. Indicates a linear layer. It is the normalized sigmoid function. These are learnable parameters. and These represent the parameters of the two linear layers respectively;

[0125] Step S105: Input the semantic information fused with deep features into the second decoder to obtain the label of the semantic prediction map containing detail segmentation. ;

[0126] Step S106: Using a channel attention mechanism, the labels of the semantic prediction graph containing long-term dependencies are generated. Labels with semantic prediction graphs containing detail segmentation The data is then fused to obtain the first predicted label.

[0127] In this embodiment, refer to Figure 3 , Figure 3 This is a schematic diagram of the student network architecture process according to an embodiment of the unsupervised domain adaptive semantic segmentation method of the present invention. Figure 3 As shown, the student network mainly consists of three parts: a Transformer-based hierarchical encoder F, a first decoder for global scene information prediction, and a third decoder. and a second decoder for detail information prediction Among them, the Transformer is a model that utilizes an attention mechanism to improve model training speed. First, the preprocessed source domain image is input into the student network. The encoder F in the student network performs deep feature extraction on the preprocessed source domain image, obtaining backbone features at different scales. , where i is a positive integer.

[0128] These features at different scales are then fed into the first decoder. Multi-scale fusion is performed, and global scene prediction is conducted to obtain a semantic prediction graph containing contextual information and long dependencies. .

[0129] Then, based on the idea of ​​multi-task learning, a multi-scale task interaction module is proposed. This module interacts deep features and semantic features at each scale to generate a prediction map containing details. Specifically, it combines the extracted backbone features from different scales... The data is fed into two bottleneck layers respectively to obtain high-dimensional depth information. and semantic information The bottleneck layer consists of a 1×1 convolution for dimensionality reduction, a 3×3 convolution, and a 1×1 convolution for dimensionality recovery. Then, a channel attention mechanism is used to extract depth features, which are then fused with semantic information in an additive manner to obtain semantic information incorporating depth features. This enhancement of semantic information with depth features facilitates fine-grained edge segmentation of small objects. The same operation is applied to each scale, as different scales focus on different features, thus accommodating objects at different distances in the image. The semantic information incorporating depth features is then input into the second decoder. It also performs detail information prediction to obtain labels from a semantic prediction graph containing detail segmentation. The fusion formula is as follows:

[0130]

[0131] This represents the semantic information that incorporates deep features at the i-th scale. Represents semantic information at the i-th scale. Represents depth information at the i-th scale. This represents element-wise matrix multiplication. Indicates a linear layer. It is the normalized sigmoid function. These are learnable parameters. and These represent the parameters of two linear layers. Among them, sigmoid is the most widely used activation function.

[0132] Finally, the label of the semantic prediction graph containing long-dependency contextual information is obtained through a channel attention mechanism. Labels with semantic prediction graphs containing detail segmentation The data is then fused to obtain the prediction result, which is the first prediction label. .

[0133] Furthermore, semantic information features are extracted using a channel attention mechanism and then fused with deep information in an additive manner to obtain deep information that incorporates semantic features. The fusion formula is as follows:

[0134] .

[0135] Secondly, embodiments of the present invention also provide an unsupervised domain-adaptive semantic segmentation apparatus.

[0136] In one embodiment, reference is made to Figure 4 , Figure 4 This is a functional module diagram of an embodiment of the unsupervised domain-adaptive semantic segmentation device of the present invention. Figure 4 As shown, the unsupervised domain-adaptive semantic segmentation device includes:

[0137] The first acquisition module 10 is used to input the preprocessed source domain image into the student network to obtain the first predicted label output by the student network. The first predicted label is obtained by fusing the label of the semantic prediction map containing long-dependency context information with the label of the semantic prediction map containing detail segmentation.

[0138] The first calculation module 20 is used to calculate the cross-entropy loss between the label of the source domain image and the first predicted label;

[0139] The second acquisition module 30 is used to fuse the preprocessed source domain image and the preprocessed target domain image, input the fused image into the student network, and obtain the second predicted label output by the student network.

[0140] The second calculation module 40 is used to calculate the consistency loss between the pseudo-label of the fused image and the second predicted label;

[0141] The third calculation module 50 is used to calculate the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first prediction label, or to calculate the second depth loss between the second depth information pseudo-label corresponding to the target domain image and the second prediction label.

[0142] The parameter update module 60 is used to calculate the gradients of cross-entropy loss, consistency loss, and first depth loss or second depth loss, backpropagate the gradients to update the parameters of the student network, and use the exponential moving average method to update the parameters of the teacher network.

[0143] The loop module 70 is used to take the new preprocessed source domain image as the preprocessed source domain image and the new preprocessed target domain image as the preprocessed target domain image, and return to the step of inputting the preprocessed source domain image into the student network and calculating the cross-entropy loss between the first predicted label output by the student network and the label of the source domain image.

[0144] The segmentation prediction module 80 is used to perform semantic segmentation prediction on the unlabeled target domain image using the latest teacher network when the number of iterations reaches a preset number, so as to obtain the pseudo-label of the unlabeled target domain image.

[0145] Furthermore, in one embodiment, the unsupervised domain-adaptive semantic segmentation apparatus further includes a third acquisition module, used for:

[0146] Obtain the source domain image and the target domain image;

[0147] The source image and the target image are cropped, flipped, color-dithered, and Gaussian blurred to obtain preprocessed source images and preprocessed target images.

[0148] The depth estimation model is trained based on the source domain image and the target domain image to obtain the trained depth estimation model;

[0149] The trained depth estimation model yields the first depth information pseudo-label corresponding to the source domain image and the second depth information pseudo-label corresponding to the target domain image.

[0150] Furthermore, in one embodiment, the first acquisition module 10 is specifically used for:

[0151] The preprocessed source domain image is input into the student network, where the encoder performs deep feature extraction on the preprocessed source domain image to obtain backbone features at different scales. , where i is a positive integer;

[0152] Skeletal features at different scales The input is scaled into the first decoder for scale fusion and prediction, resulting in a label of a semantic prediction graph containing contextual information about long dependencies. ;

[0153] Skeletal features at different scales Input the two bottleneck layers separately to obtain depth information. and semantic information The bottleneck layer includes a 1×1 convolution for dimensionality reduction, a 3×3 convolution, and a convolution for restoring the 1×1 dimension.

[0154] The deep information features are extracted using a channel attention mechanism and then fused with semantic information to obtain semantic information that incorporates the deep features. The fusion formula is as follows:

[0155]

[0156] This represents the semantic information that incorporates deep features at the i-th scale. Represents semantic information at the i-th scale. Represents depth information at the i-th scale. This represents element-wise matrix multiplication. Indicates a linear layer. It is the normalized sigmoid function. These are learnable parameters. and These represent the parameters of the two linear layers respectively;

[0157] The semantic information, which incorporates deep features, is input into the second decoder to obtain labels from a semantic prediction map that includes detail segmentation. ;

[0158] The labeling of the semantic prediction graph, which contains contextual information including long dependencies, is obtained through a channel attention mechanism. Labels with semantic prediction graphs containing detail segmentation The data is then fused to obtain the first predicted label.

[0159] Furthermore, in one embodiment, the first computing module 20 is used for:

[0160] The cross-entropy loss between the first predicted label output by the student network and the label of the source domain image is calculated using the first loss function formula, where the first loss function formula is as follows:

[0161]

[0162] in, Labels representing the source domain image. Represents the first prediction label, These are height, width, and number of channels, where C represents the number of channels. This represents the cross-entropy loss.

[0163] Furthermore, in one embodiment, the unsupervised domain-adaptive semantic segmentation apparatus further includes a fourth acquisition module, used for:

[0164] The target domain image is input into the teacher network, and the obtained segmentation prediction results are used as pseudo-labels for the target domain image.

[0165] The labels of the source domain image and the pseudo labels of the target domain image are blended with the same mask to obtain the pseudo labels of the fused image.

[0166] Furthermore, in one embodiment, the third calculation module 50 is used for:

[0167] The difference is obtained by subtracting the first predicted label from the first depth information pseudo-label corresponding to the source domain image. ;

[0168] The difference Substituting into the second loss function formula, we obtain the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label. The second loss function formula is as follows:

[0169]

[0170] It is a constant.

[0171] Furthermore, in one embodiment, the parameter update module 60 is used for:

[0172] Substitute the updated parameters for students' networks and the parameters for teachers' networks before the update into the preset formula. The updated parameters of the teacher network are obtained, among which, The parameters represent the updated parameters of the teacher network. The parameters represent those before the teacher network update. The parameters represent the updated student network information. These are weighting coefficients, 0 < <1.

[0173] The functions of each module in the above-mentioned unsupervised domain adaptive semantic segmentation device correspond to the steps in the above-mentioned unsupervised domain adaptive semantic segmentation method embodiment, and their functions and implementation processes will not be described in detail here.

[0174] Thirdly, embodiments of the present invention provide an unsupervised domain-adaptive semantic segmentation device, which can be a device with data processing capabilities such as a personal computer (PC), a laptop computer, or a server.

[0175] Reference Figure 5 , Figure 5This is a schematic diagram of the hardware structure of the unsupervised domain-adaptive semantic segmentation device involved in the embodiments of the present invention. In this embodiment, the unsupervised domain-adaptive semantic segmentation device may include a processor 1001 (e.g., a Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement communication between these components; the user interface 1003 may include a display screen or an input unit such as a keyboard; the network interface 1004 may optionally include a standard wired interface or a wireless interface (e.g., Wireless Fidelity, Wi-Fi); the memory 1005 may be high-speed random access memory (RAM) or stable memory (non-volatile memory), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001. Those skilled in the art will understand that… Figure 5 The hardware structure shown does not constitute a limitation of the invention and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0176] Continue to refer to Figure 5 , Figure 5 The memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an unsupervised domain-adaptive semantic segmentation program. The processor 1001 can call the unsupervised domain-adaptive semantic segmentation program stored in the memory 1005 and execute the unsupervised domain-adaptive semantic segmentation method provided in this embodiment of the invention.

[0177] Fourthly, embodiments of the present invention also provide a readable storage medium.

[0178] The present invention stores an unsupervised domain-adaptive semantic segmentation program on a readable storage medium, wherein when the unsupervised domain-adaptive semantic segmentation program is executed by a processor, it implements the steps of the unsupervised domain-adaptive semantic segmentation method as described above.

[0179] The method implemented when the unsupervised domain adaptive semantic segmentation program is executed can be referred to in various embodiments of the unsupervised domain adaptive semantic segmentation method of the present invention, and will not be repeated here.

[0180] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0181] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of the present invention.

[0183] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An unsupervised domain-adaptive semantic segmentation method, characterized in that, The unsupervised domain-adaptive semantic segmentation method includes: The preprocessed source domain image is input into the student network to obtain the first predicted label output by the student network. The first predicted label is obtained by fusing the label of the semantic prediction map containing long-dependency context information with the label of the semantic prediction map containing detail segmentation. Calculate the cross-entropy loss between the label of the source domain image and the first predicted label; The preprocessed source domain image and the preprocessed target domain image are fused, and the fused image is input into the student network to obtain the second predicted label output by the student network. Calculate the consistency loss between the pseudo-label of the fused image and the second predicted label; Calculate the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label, or calculate the second depth loss between the second depth information pseudo-label corresponding to the target domain image and the second predicted label; Calculate the gradients of cross-entropy loss, consistency loss, and either the first depth loss or the second depth loss. Backpropagate the gradients to update the parameters of the student network and use the exponential moving average method to update the parameters of the teacher network. Using the new preprocessed source domain image as the preprocessed source domain image and the new preprocessed target domain image as the preprocessed target domain image, return to the step of inputting the preprocessed source domain image into the student network and calculating the cross-entropy loss between the first predicted label output by the student network and the label of the source domain image. When the number of iterations reaches the preset number, the latest teacher network is used to perform semantic segmentation prediction on the unlabeled target domain image to obtain the pseudo-label of the unlabeled target domain image. The step of calculating the cross-entropy loss between the label of the source domain image and the first predicted label includes: The cross-entropy loss between the first predicted label output by the student network and the label of the source domain image is calculated using the first loss function formula, where the first loss function formula is as follows: in, Labels representing the source domain image. Represents the first prediction label, These are height, width, and number of channels, where C represents the number of channels. Represents cross-entropy loss; The step of calculating the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label includes: The difference is obtained by subtracting the first predicted label from the first depth information pseudo-label corresponding to the source domain image. ; The difference Substituting into the second loss function formula, we obtain the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label. The second loss function formula is as follows: It is a constant.

2. The unsupervised domain-adaptive semantic segmentation method as described in claim 1, characterized in that, Before the step of inputting the preprocessed source domain image into the student network to obtain the first predicted label output by the student network, the following steps are included: Obtain the source domain image and the target domain image; The source image and the target image are cropped, flipped, color-dithered, and Gaussian blurred to obtain preprocessed source images and preprocessed target images. The depth estimation model is trained based on the source domain image and the target domain image to obtain the trained depth estimation model; The trained depth estimation model yields the first depth information pseudo-label corresponding to the source domain image and the second depth information pseudo-label corresponding to the target domain image.

3. The unsupervised domain-adaptive semantic segmentation method as described in claim 1, characterized in that, The step of inputting the preprocessed source domain image into the student network to obtain the first predicted label output by the student network includes: The preprocessed source domain image is input into the student network, where the encoder performs deep feature extraction on the preprocessed source domain image to obtain backbone features at different scales. , where i is a positive integer; Skeletal features at different scales The input is scaled into the first decoder for scale fusion and prediction, resulting in a label of a semantic prediction graph containing contextual information about long dependencies. ; Skeletal features at different scales Input the two bottleneck layers separately to obtain depth information. and semantic information The bottleneck layer includes a 1×1 convolution for dimensionality reduction, a 3×3 convolution, and a convolution for restoring the 1×1 dimension. The deep information features are extracted using a channel attention mechanism and then fused with semantic information to obtain semantic information that incorporates the deep features. The fusion formula is as follows: This represents the semantic information that incorporates deep features at the i-th scale. Represents semantic information at the i-th scale. Represents depth information at the i-th scale. This represents element-wise matrix multiplication. Indicates a linear layer. It is the normalized sigmoid function. These are learnable parameters. and These represent the parameters of the two linear layers respectively; The semantic information, which incorporates deep features, is input into the second decoder to obtain labels from a semantic prediction map that includes detail segmentation. ; The labeling of the semantic prediction graph, which contains contextual information including long dependencies, is obtained through a channel attention mechanism. Labels with semantic prediction graphs containing detail segmentation The data is then fused to obtain the first predicted label.

4. The unsupervised domain-adaptive semantic segmentation method as described in claim 1, characterized in that, The pseudo-labels of the fused image are obtained in the following way: The target domain image is input into the teacher network, and the obtained segmentation prediction results are used as pseudo-labels for the target domain image. The labels of the source domain image and the pseudo labels of the target domain image are blended with the same mask to obtain the pseudo labels of the fused image.

5. The unsupervised domain-adaptive semantic segmentation method as described in claim 1, characterized in that, The step of updating the parameters of the teacher network using the exponential moving average method includes: Substitute the updated parameters for students' networks and the parameters for teachers' networks before the update into the preset formula. The updated parameters of the teacher network are obtained, among which, The parameters represent the updated parameters of the teacher network. The parameters represent those before the teacher network update. The parameters represent the updated student network information. These are weighting coefficients, 0 < <1.

6. An unsupervised domain-adaptive semantic segmentation device, characterized in that, The unsupervised domain-adaptive semantic segmentation device includes: The first acquisition module is used to input the preprocessed source domain image into the student network to obtain the first predicted label output by the student network. The first predicted label is obtained by fusing the label of the semantic prediction map containing long-dependency context information with the label of the semantic prediction map containing detail segmentation. The first calculation module is used to calculate the cross-entropy loss between the label of the source domain image and the first predicted label; The second acquisition module is used to fuse the preprocessed source domain image and the preprocessed target domain image, input the fused image into the student network, and obtain the second predicted label output by the student network. The second calculation module is used to calculate the consistency loss between the pseudo-label of the fused image and the second predicted label; The third calculation module is used to calculate the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first prediction label, or to calculate the second depth loss between the second depth information pseudo-label corresponding to the target domain image and the second prediction label. The parameter update module is used to calculate the gradients of cross-entropy loss, consistency loss, and first depth loss or second depth loss, backpropagate the gradients to update the parameters of the student network, and use the exponential moving average method to update the parameters of the teacher network. The loop module is used to take the new preprocessed source domain image as the preprocessed source domain image and the new preprocessed target domain image as the preprocessed target domain image, and return to execute the step of inputting the preprocessed source domain image into the student network and calculating the cross-entropy loss between the first predicted label output by the student network and the label of the source domain image. The segmentation prediction module is used to perform semantic segmentation prediction on the unlabeled target domain image using the latest teacher network when the number of iterations reaches a preset number, so as to obtain the pseudo-label of the unlabeled target domain image. The first calculation module is used for: The cross-entropy loss between the first predicted label output by the student network and the label of the source domain image is calculated using the first loss function formula, where the first loss function formula is as follows: in, Labels representing the source domain image. Represents the first prediction label, These are height, width, and number of channels, where C represents the number of channels. Represents cross-entropy loss; The third calculation module is used for: The difference is obtained by subtracting the first predicted label from the first depth information pseudo-label corresponding to the source domain image. ; The difference Substituting into the second loss function formula, we obtain the first depth loss between the first depth information pseudo-label corresponding to the source domain image and the first predicted label. The second loss function formula is as follows: It is a constant.

7. An unsupervised domain-adaptive semantic segmentation device, characterized in that, The unsupervised domain-adaptive semantic segmentation device includes a processor, a memory, and an unsupervised domain-adaptive semantic segmentation program stored in the memory and executable by the processor, wherein when the unsupervised domain-adaptive semantic segmentation program is executed by the processor, it implements the steps of the unsupervised domain-adaptive semantic segmentation method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, The readable storage medium stores an unsupervised domain-adaptive semantic segmentation program, wherein when the unsupervised domain-adaptive semantic segmentation program is executed by a processor, it implements the steps of the unsupervised domain-adaptive semantic segmentation method as described in any one of claims 1 to 5.

Images