Unsupervised multi-source domain adaptation image semantic segmentation method based on weighted mutual learning

Abstract

The present invention discloses an unsupervised multi-source domain adaptation image semantic segmentation method based on weighted mutual learning, which belongs to the field of computer vision and includes the following steps: determining a target domain and multiple source domain data sets, constructing a corresponding image semantic segmentation model, training a model with labeled data in the source domain, calculating the confidence weight of the model with unlabeled data in the target domain, performing weighted mutual learning, judging training convergence, joint reasoning of multiple models, and generating prediction results and models of the target domain data set. The present invention can be applied to domain adaptation training from multiple source domains to an unsupervised target domain. Aiming at the domain offset problem between source domains in the multi-source domain problem, the present invention proposes to use weighted mutual learning to realize knowledge fusion and alignment between source domains. The applied multi-source domain scenario is closer to the real setting, and can achieve better domain adaptation effect, which is beneficial to expand the application scope of image semantic segmentation technology.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an unsupervised multi-source domain adaptation image semantic segmentation method based on weighted mutual learning. Background Art

[0002] Image semantic segmentation is one of the important tasks in the field of computer vision, which aims to predict the category of an image pixel by pixel. The semantic segmentation map generated by the prediction result can be used to accurately identify the shape, position and category of objects in the image, and finally be used to assist the computer to understand the natural scene. It is of great significance in both emerging fields with rapid development such as autonomous driving, human-computer interaction and virtual reality, and in traditional tasks such as remote sensing and medical image recognition. With the development of deep learning technology, deep neural networks have achieved remarkable results in semantic segmentation tasks. However, deep neural networks usually rely on large-scale supervised training of manually annotated data to obtain reliable performance. Deep neural networks have strong data fitting ability, so for scenes with the same task but not covered by training data, the performance of the model will be affected and decreased. It is very expensive and time-consuming to collect and annotate large-scale data sets for each new scene.

[0003] Unsupervised domain adaptation technology provides effective theoretical guidance for solving the problem of missing annotations in new scenarios. The idea is to use a labeled data set similar to the target scenario (called the source domain) for supervised training so that the model can learn knowledge related to the target task. The unlabeled data in the target scenario is called the target domain. Usually, there are differences in the sample distribution between the source domain and the target domain. If the model trained in the source domain is directly applied to the target domain, its accuracy will drop significantly. The focus of unsupervised domain adaptation technology is to overcome domain differences and more effectively transfer the knowledge learned from the source domain data to the target domain. Its "unsupervised" means that no annotations of the target domain are introduced, and only the unlabeled data that is easily obtained in the target domain is used to complete the migration. According to the number of source domains, unsupervised domain adaptation can be subdivided into unsupervised domain adaptation of a single source domain and unsupervised domain adaptation of multiple source domains.

[0004] Existing domain adaptation image semantic segmentation methods are mainly aimed at a single source domain. For example, the literature Tsai YH, Hung W C, Schulter S, et al. Learning to adapt structured output space for semantic segmentation [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7472-7481. proposed to train a domain discriminator for the output of the model (after the softmax layer) to distinguish the source domain from the target domain. On this basis, the domain discriminator and the segmentation model are used to form an adversarial learning mechanism to align the output of the segmentation model between the source domain and the target domain. For example, the literature Wang, Z., et al. Differential treatment for stuff and things: A simple unsupervised domain adaptation method for semantic segmentation. in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2020. proposed to align the features of background and foreground objects separately to encourage the model to learn the structured information in the scene graph and achieve more refined feature alignment. By constraining the distance between feature vectors of different domains through loss functions, alignment of the same category between domains can be achieved. For example, Yang Y, Soatto S. Fda: Fourier domain adaptation for semantic segmentation [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 4085-4095. proposed a method for domain alignment at the surface level, which pointed out that the low-frequency components of the source domain image and the target domain image can be exchanged through Fourier transform to more conveniently achieve style transfer between the two domains. By transferring the style of the source domain data to the target domain, the segmentation model is trained on this basis to achieve better adaptability in the unlabeled target domain.Currently, there are few domain adaptation methods for multi-source domains. For example, Zhang, P., Zhang, B., Zhang, T., Chen, D., Wang, Y., & Wen, F. (2021). Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 12414-12424). A multi-source domain adaptation method based on collaborative learning is proposed. During model training, the corresponding semantic segmentation networks are trained separately for multiple source domains. The input images of each model are simultaneously input into other partner networks, and the KL distance between the output features of the partner network and the output features of this network is used to achieve collaborative learning between models.

[0005] There are several problems with existing methods: (1) Most domain adaptation methods are targeted at a single source domain and cannot be applied to multi-source domain situations that are closer to actual application scenarios. When faced with target domain data, multiple similar labeled data sets are usually available as source domains. Multi-source domain adaptation can make better use of existing knowledge and achieve better domain adaptation effects; (2) Existing multi-source domain adaptation methods use collaborative learning between source domains to achieve knowledge fusion between source domains. The amount of computation grows in a quadratic relationship with the number of source domains, which has the disadvantage of high computational complexity. (3) The offset problem between source domains in the multi-source domain adaptation problem has not been analyzed in more depth. Summary of the invention

[0006] In view of the shortcomings of the existing technology, the present invention proposes an unsupervised multi-source domain adaptive image semantic segmentation method based on weighted mutual learning, which can more efficiently achieve the effect of transferring knowledge from multiple source domains to the target domain and improve the application scenarios of semantic segmentation technology; based on the entropy value of the model's predicted probability map in the target domain data set, the present invention proposes to calculate the confidence weights of different models. Based on this weight, a more confident mutual learning strategy is achieved.

[0007] The technical solution of the present invention is as follows:

[0008] An unsupervised multi-source domain adaptation image semantic segmentation method based on weighted mutual learning comprises the following steps:

[0009] Collect and construct unlabeled target domain datasets and multiple labeled source domain datasets, where the labeling refers to the labeling of pixel categories in the dataset images;

[0010] For each source domain dataset, build a corresponding semantic segmentation model with the same structure;

[0011] Using the multiple labeled source domain data sets to train a corresponding semantic segmentation model through cross entropy loss, so that the model acquires knowledge in the corresponding source domain data sets;

[0012] Using the unlabeled target domain dataset to input each semantic segmentation model, and using the category probability map output by the semantic segmentation model to calculate the confidence weight of the semantic segmentation model;

[0013] Based on the confidence weight of the semantic segmentation model, weighted mutual learning of the semantic segmentation model at the feature and probability levels of the target domain dataset is achieved;

[0014] Determine whether the semantic segmentation model training has converged. If not, continue to train the semantic segmentation model. If converged, use multiple semantic segmentation models trained to perform joint reasoning on the target domain dataset to obtain the final prediction result.

[0015] Furthermore, the image semantic segmentation model is based on the DeepLab model, and the backbone network adopts ResNet.

[0016] Furthermore, the method for training the semantic segmentation model is as follows:

[0017] Let the source domain dataset be represented as Where i represents the number of the source domain dataset, the number of source domains is I, and x S represents the source domain image, y S represents the source domain annotation, H, W represent the height and width of the image respectively, R represents a real number set, and 3 is the number of channels of the image; the annotated data in the source domain is used to train the semantic segmentation model to obtain the knowledge in the source domain, and the formula of the cross entropy loss function used is as follows:

[0018]

[0019] In the formula, F represents the trained semantic segmentation model, K is the total number of categories, and F k (x S ) represents the probability that the predicted input belongs to class k, is the indicator function.

[0020] Furthermore, the step of calculating the confidence weight of the model using the category probability map output by the semantic segmentation model includes:

[0021] Let the target domain dataset be represented as D T ={x T |x T ∈R H×W×3}, the semantic segmentation model predicts the probability distribution of the input pixels The information entropy of is used to estimate the confidence of the semantic segmentation model for the pixel; the semantic segmentation model F is used to estimate the confidence of the target domain image x T The information entropy of the predicted probability distribution is calculated by the following formula:

[0022]

[0023] In the formula, the base of log is e;

[0024] For multiple semantic segmentation models, the confidence weight of the semantic segmentation model is further calculated through the information entropy of each semantic model. The formula is as follows:

[0025]

[0026] In the formula, F i represents the semantic segmentation model corresponding to the source domain dataset numbered i, I represents the number of source domains, τ is the temperature coefficient, and H upper The upper bound of the information entropy representing the predicted probability distribution is related to the number of categories and is calculated as follows:

[0027]

[0028] Here, the base of log is e.

[0029] Furthermore, the method of using the confidence weight of the semantic segmentation model to achieve weighted mutual learning at the probability level is:

[0030] Through unlabeled target domain images, knowledge fusion and domain alignment between different source domain models are achieved through mutual learning between semantic segmentation models. In the multi-source domain adaptation task, considering the domain offset factor between source domains, the weighted average probability distribution is calculated to generate the predicted probability distribution for the unlabeled data, and then the MSE loss function is used to constrain all semantic segmentation models to align to the average probability distribution, so as to achieve alignment and mutual learning of all semantic segmentation models at the probability level.

[0031] Among them, the formula of MSE loss function is as follows:

[0032]

[0033] Where P mean (x T ) represents the weighted average probability distribution, which is calculated as follows:

[0034]

[0035] Furthermore, the method of using the confidence weight of the semantic segmentation model to achieve weighted mutual learning at the feature level is:

[0036] Let G(x T ) represents the representative feature map extracted by the semantic segmentation model for the input image, and the feature map output by the previous layer of the fully connected layer is used as the representative feature map of the input; based on the feature map, the knowledge alignment between models is achieved at the feature level; by constraining the consistency of features, different semantic segmentation models learn the same representation; through weighted mutual learning, all semantic segmentation models are made to approach the model with high confidence weight, and learn more reliable feature representation;

[0037] Weighted mutual learning at the feature level is performed based on the following cosine distance loss function:

[0038]

[0039] In the formula, G mean (x T ) represents the weighted average feature, which is calculated as follows:

[0040]

[0041] Furthermore, during the training of the semantic segmentation model, in each iteration cycle, the overall loss function of the semantic segmentation model consists of three parts: the cross entropy loss function of the labeled data in the source domain, the MSE loss and cosine distance loss of the unlabeled data in the target domain. The expression of the overall loss function is as follows:

[0042] L All (F i ,x S ,y S ,x T )=L seg (F i ,x S ,y s )+λ·(L MSE (F i ,x T )+L Cosine (F i ,x T ));

[0043] Where λ is the weight coefficient of the weighted mutual learning loss function.

[0044] Furthermore, one method to determine whether the semantic segmentation model training has converged is to set the maximum number of iterations of model training before the semantic segmentation model training begins, test the performance of the model using the validation set data at certain step intervals during the training process, and determine whether the model has converged by the accuracy of the validation set; wherein, the indicator used to evaluate the model performance is mIoU.

[0045] Furthermore, another method for determining whether the semantic segmentation model training has converged is to determine whether the semantic segmentation model has converged based on the training loss function curve.

[0046] Furthermore, the average inference of multiple trained semantic segmentation models is used as the final model output, which is calculated as follows:

[0047]

[0048] The advantages and positive effects of the present invention compared with the prior art are as follows:

[0049] (1) The present invention proposes a new unsupervised multi-source domain adaptation image semantic segmentation method based on weighted mutual learning. Compared with a large number of existing single-source domain adaptation methods, the multi-source domain setting is closer to the actual application scenario. The multi-source domain adaptation algorithm established through mutual learning has better domain adaptation performance, can make more full use of existing data and knowledge, and expands the application scope of the domain adaptation method.

[0050] (2) The weighted mutual learning strategy proposed in the present invention has better performance than the existing mutual learning methods. The method of the present invention uses the information entropy based on the probability distribution of model output to calculate the confidence weight, which can make the model with higher confidence occupy a larger weight in mutual learning, so that the model is closer to the model with higher confidence during alignment. Therefore, weighted mutual learning can achieve more reliable alignment between models.

[0051] (3) The weighted mutual learning strategy proposed in the present invention includes the probability level and the feature level, and verifies the complementarity between the two levels. Compared with the existing mutual learning in the domain that only considers the mutual learning at a single level, the method of the present invention has better performance and extends the mutual learning method to the situation where more than two models perform mutual learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of the unsupervised multi-source domain adaptation image semantic segmentation method proposed in an embodiment of the present invention;

[0053] Figure 2 This is a framework diagram of the weighted mutual learning method proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the above features and advantages of the present invention more obvious and easy to understand, embodiments are given below and described in detail with reference to the accompanying drawings.

[0055] like Figure 1As shown, this embodiment proposes an unsupervised multi-source domain adaptation image semantic segmentation method based on weighted mutual learning, including the following steps: First, construct an unlabeled target domain dataset according to the application scenario, and determine a variety of related labeled datasets as source domains, and the above two datasets are both image datasets. Secondly, a corresponding semantic segmentation model is constructed for each source domain dataset, and the semantic segmentation model is an image semantic segmentation model. A weighted mutual learning framework is formed using multiple semantic segmentation models. The source domain data and the target domain data are used in the weighted mutual learning training process. The source domain labeled data trains the corresponding semantic segmentation model through cross entropy loss. The output category probability map of the semantic segmentation model of the target domain unlabeled data is used to calculate the confidence weight of the semantic segmentation model. Weighted mutual learning between semantic segmentation models is realized using unlabeled datasets and confidence weights. Finally, after the training is completed, a domain-adapted available semantic segmentation model can be obtained, and the prediction result of the target domain data can be output. This method can realize domain adaptation training from multiple source domains to one target domain. The multi-source domain scenario is closer to the real setting, can achieve better domain adaptation effect, and is beneficial to expand the application scope of image semantic segmentation technology.

[0056] The details are explained below.

[0057] Step 1: Collect and construct an unlabeled target domain dataset and a variety of labeled source domain datasets.

[0058] In practical applications, data annotations in the target domain are missing, and manual annotation has a high cost, but unlabeled data is easy to collect and does not require too much cost. Therefore, this embodiment can easily construct an unlabeled target domain dataset. For the target domain dataset, this embodiment can usually find multiple similar labeled datasets as the source domain. The purpose is to achieve the migration of knowledge in multiple source domain datasets to the target domain through domain adaptation technology. The above two datasets are both image datasets, and the unlabeled data refers to data that does not label the pixel category in the image, and the labeled data refers to data that labels the category of each pixel in the image.

[0059] Step 2: Build a corresponding semantic segmentation model for each source domain dataset. The constructed multiple semantic segmentation models have the same model structure and are all based on the DeepLab model.

[0060] like Figure 2 As shown, in mutual learning, this embodiment maintains multiple learning bodies for joint training. Therefore, this embodiment builds an independent semantic segmentation model for each source domain dataset. The model only needs to learn the knowledge in the corresponding source domain dataset. The structure of all models is the same, but the parameters and update directions are different. In the implementation, the model uses the DeepLabV2 model, and the backbone network uses ResNet101.

[0061] Step 3: Use the multiple labeled source domain datasets in step 1 to train the corresponding semantic segmentation model through cross entropy loss so that the model can acquire the knowledge in the corresponding dataset.

[0062] Figure 2 The dashed arrow on the right side of represents the calculation of the cross entropy loss and the reverse propagation object. Let the source domain dataset be represented as Where i represents the number of the source domain dataset, and the number of source domains is I. The labeled data in the source domain is used to train the model to obtain knowledge in the source domain, and the cross entropy loss function used is given by the following formula:

[0063]

[0064] Where: F represents the trained model, K is the total number of categories, and F k (x S ) represents the probability that the predicted input belongs to class k, is the indicator function.

[0065] Step 4: Use the unlabeled target domain dataset in step 1 to input each semantic segmentation model, and use the category probability map output by the model to calculate the confidence weight of the model.

[0066] Figure 2 The mutual learning weights in are calculated using the output class probability maps of all models on the target domain data. Let the target domain dataset be represented as D T ={x T |x T ∈R H×W×3 This embodiment proposes to use the model to predict the probability distribution of the input pixels. The information entropy of the output probability distribution is used to estimate the confidence of the model for the pixel. More specifically, if the information entropy of the output probability distribution is smaller, it means that the model is more confident in the prediction of the pixel. Conversely, if the information entropy of the output probability distribution is larger, it means that the model is less confident in the prediction of the pixel. T The information entropy of the predicted probability distribution is calculated by the following formula:

[0067]

[0068] For multiple training models, the confidence weight of the model can be further calculated through the information entropy of each model, and the formula is given by the following formula:

[0069]

[0070] Where F irepresents the model corresponding to the source domain data set numbered i, τ is the temperature coefficient, H upper The upper bound of the information entropy representing the predicted probability distribution depends on the number of categories and is calculated as follows:

[0071]

[0072] The model confidence weight obtained by formula (3) represents the weight map of different models for the same image. This embodiment proposes that this weight can be used as an estimate of the model confidence weight to achieve weighted mutual learning.

[0073] Step 5: Use the model confidence weights obtained in step 4 to implement weighted mutual learning of the multi-semantic segmentation model at both the feature and probability levels of the target domain dataset.

[0074] The model confidence weights are used to achieve weighted mutual learning at the probability level. Through unlabeled target domain images, knowledge fusion and domain alignment between different source domain models can be achieved through mutual learning between models. In multi-source domain adaptation tasks, domain offset between source domains is also a factor that needs to be considered. This embodiment proposes a weighted mutual learning method at the probability level, and its MSE loss function is given by the following formula:

[0075]

[0076] Where P mean (x T ) represents the weighted average probability distribution, which is calculated as follows:

[0077]

[0078] By calculating the weighted average probability distribution, a more reliable prediction probability distribution can be generated for unlabeled data. The MSE loss function is used to constrain all models to align to the average probability distribution, so that all models can be aligned and learn from each other at the probability level.

[0079] The model confidence weights are used to achieve weighted mutual learning at the feature level. In addition to mutual learning at the probability level, this embodiment also proposes a weighted mutual learning method at the feature level. Let G(x T) represents the representative feature map extracted by the model for the input image. This embodiment uses the feature map output by the previous layer of the fully connected layer as the representative feature map of the input. Based on the feature map, this embodiment proposes to achieve knowledge alignment between models at the feature level. Due to the domain offset between source domains, there will be large differences in the model at the feature level. By constraining the consistency of features, different models can be prompted to learn the same representation. In addition, weighted mutual learning can make all models closer to models with high confidence weights, which is conducive to learning more reliable feature representations. This embodiment proposes a weighted mutual learning method at the feature level, and the cosine distance loss function is given by the following formula:

[0080]

[0081] In the formula, G mean (x T ) represents the weighted average feature, which is calculated as follows:

[0082]

[0083] This embodiment proposes to combine mutual learning at the probability and feature levels to obtain better mutual learning effects. In the experiment, the experimental results show that the two are complementary, and combined use can achieve better domain adaptation effects. In training, in each iteration cycle, the overall loss function of the model consists of three parts: the cross entropy loss function of labeled data in the source domain, the MSE loss of unlabeled data in the target domain, and the cosine distance loss. The overall loss function is expressed as follows:

[0084] L All (F i ,x S ,y S ,x T )=L seg (F i ,x S ,y s )+λ·(L MSE (F i ,x T )+L Cosine (F i ,x T )) (9)

[0085] Where λ is the weight coefficient of the weighted mutual learning loss function.

[0086] Step 6: Determine whether the model training has converged. If it has converged, proceed to step 7; otherwise, jump to step 3.

[0087] Before model training begins, set the maximum number of iterations for model training. During the training process, use the validation set data to test the performance of the model at certain intervals. The indicators used to evaluate model performance are the commonly used mean-Intersection-over-Union (mIoU). The accuracy of the validation set is used to determine whether the model has converged. If validation set data is missing, the training loss function curve can also be used to determine whether the model has converged. After the training is completed, the optimal model is selected as the output model.

[0088] Step 7: For the target domain dataset, use multiple trained models to perform joint reasoning to obtain more accurate prediction results.

[0089] The weighted mutual learning proposed in this embodiment will establish a corresponding model for each source domain data set, so after the training is completed, multiple available models will be output. This embodiment uses the average inference of multiple models as the final model output, and its calculation method is shown in the following formula:

[0090]

[0091] Step 8: The above steps obtain the prediction results of the target domain dataset and the semantic segmentation model available in this domain.

[0092] Experimental test:

[0093] The test scene used in this test is the migration from virtual synthetic images to real scene images. The virtual synthetic images include two datasets, GTA5 and Synscapes. GTA5 is collected from the game rendering engine and contains 24,966 images with a resolution of 1914×1052. Each image has a corresponding fine category annotation. Synscapes is generated using image realistic rendering technology. It contains 25,000 images with a resolution of 1440×720. The target domain dataset uses the Cityscapes dataset, which contains 5,000 real street scene images with a resolution of 2048×1024. The training set of this dataset contains 2,975 images and the validation set contains 500 images. The remaining 1,525 images are used as the test set. Due to the lack of test set annotations and in line with existing work, this test uses the training set for unsupervised domain adaptation training and the validation set for model evaluation. This test is trained and tested on 19 categories shared between datasets. The evaluation index is mean-Intersection-over-Union (mIoU). The experimental comparison results of this test are shown in Table 1 below:

[0094] Table 1 GTA5&Synscapes-to-Cityscapes experimental results

[0095]

[0096] The first two rows in Table 1 represent the accuracy of using GTA5 and Synscapes to independently supervise the training model and then apply it to the target domain dataset. The third row represents the accuracy obtained by randomly mixing the two datasets for supervised training, which is set as the benchmark for comparison in this test. Ours (w / o feat) means that only the mutual learning method at the probability level proposed in the present invention is used, and the result reaches 52.7mIoU. By further adding weighted mutual learning at the feature level, the final performance can reach 53.4mIoU, which is 5.2mIoU ahead of the baseline method. The above experimental results demonstrate the superiority of the method proposed in the present invention.

[0097] The parts not described in detail in the present invention belong to the known technology in the art.

[0098] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should all be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.

Claims

1. An unsupervised multi-source domain adaptation image semantic segmentation method based on weighted mutual learning, characterized in that: The following steps are involved: Collect and construct unlabeled target domain datasets and multiple labeled source domain datasets, where the labeling refers to the labeling of pixel categories in the dataset images; For each source domain dataset, build a corresponding semantic segmentation model with the same structure; The corresponding semantic segmentation model is trained by using the multiple labeled source domain data sets through cross entropy loss, so that the model obtains the knowledge in the corresponding source domain data sets; the method for training the semantic segmentation model is as follows: Let the source domain dataset be represented as Where i represents the number of the source domain dataset, the number of source domains is I, and x S represents the source domain image, y S represents the source domain annotation, H, W represent the height and width of the image respectively, R represents a real number set, and 3 is the number of channels of the image; the annotated data in the source domain is used to train the semantic segmentation model to obtain the knowledge in the source domain, and the formula of the cross entropy loss function used is as follows: In the formula, F represents the trained semantic segmentation model, K is the total number of categories, and F k (x S ) represents the probability that the predicted input belongs to class k, is the indicator function; Using the unlabeled target domain dataset to input each semantic segmentation model, and using the category probability map output by the semantic segmentation model to calculate the confidence weight of the semantic segmentation model; The steps of calculating the confidence weight of the model using the category probability map output by the semantic segmentation model include: Let the target domain dataset be represented as D T ={x T |x T ∈R H×W×3 }, the semantic segmentation model predicts the probability distribution of the input pixels The information entropy of is used to estimate the confidence of the semantic segmentation model for the pixel; the semantic segmentation model F is used to estimate the confidence of the target domain image x T The information entropy of the predicted probability distribution is calculated by the following formula: In the formula, the base of log is e; For multiple semantic segmentation models, the confidence weight of the semantic segmentation model is further calculated through the information entropy of each semantic model. The formula is as follows: In the formula, F i represents the semantic segmentation model corresponding to the source domain dataset numbered i, I represents the number of source domains, τ is the temperature coefficient, and H upper The upper bound of the information entropy representing the predicted probability distribution is related to the number of categories and is calculated as follows: In the formula, the base of log is e; Based on the confidence weight of the semantic segmentation model, weighted mutual learning of the semantic segmentation model at the feature and probability levels of the target domain dataset is achieved; Determine whether the semantic segmentation model training has converged. If not, continue to train the semantic segmentation model. If converged, use multiple semantic segmentation models trained to perform joint reasoning on the target domain dataset to obtain the final prediction result.

2. The method according to claim 1, characterized in that The image semantic segmentation model is based on the DeepLab model, and the backbone network uses ResNet.

3. The method according to claim 1, characterized in that The method of using the confidence weights of the semantic segmentation model to achieve weighted mutual learning at the probability level is: Through unlabeled target domain images, knowledge fusion and domain alignment between different source domain models are achieved through mutual learning between semantic segmentation models. In the multi-source domain adaptation task, considering the domain offset factor between source domains, the weighted average probability distribution is calculated to generate the predicted probability distribution for the unlabeled data, and then the MSE loss function is used to constrain all semantic segmentation models to align to the average probability distribution, so as to achieve alignment and mutual learning of all semantic segmentation models at the probability level. Among them, the formula of MSE loss function is as follows: Where P mean (x T ) represents the weighted average probability distribution, which is calculated as follows:

4. The method according to claim 3, characterized in that The method of using the confidence weights of the semantic segmentation model to achieve weighted mutual learning at the feature level is: Let G(x T ) represents the representative feature map extracted by the semantic segmentation model for the input image, and the feature map output by the previous layer of the fully connected layer is used as the representative feature map of the input; based on the feature map, the knowledge alignment between models is achieved at the feature level; by constraining the consistency of features, different semantic segmentation models learn the same representation; through weighted mutual learning, all semantic segmentation models are made to approach the model with high confidence weight, and learn more reliable feature representation; Weighted mutual learning at the feature level is performed based on the following cosine distance loss function: In the formula, G mean (x T ) represents the weighted average feature, which is calculated as follows:

5. The method according to claim 4, characterized in that During the training of the semantic segmentation model, in each iteration cycle, the overall loss function of the semantic segmentation model consists of three parts: the cross entropy loss function of the labeled data in the source domain, the MSE loss of the unlabeled data in the target domain, and the cosine distance loss. The expression of the overall loss function is as follows: L All (F i ,x S ,y S ,x T )=L seg (F i ,x S ,y S )+λ·(L MSE (F i ,x T )+L Cosine (F i ,x T )); Where λ is the weight coefficient of the weighted mutual learning loss function.

6. The method according to claim 1, characterized in that One method to determine whether the semantic segmentation model training has converged is to set the maximum number of iterations of the model training before the semantic segmentation model training begins, test the performance of the model using the validation set data at certain step intervals during the training process, and determine whether the model has converged by the accuracy of the validation set; the indicator used to evaluate the model performance is mIoU.

7. The method according to claim 1, characterized in that Another method to determine whether the semantic segmentation model training has converged is to determine whether the semantic segmentation model has converged based on the training loss function curve.

8. The method according to claim 1, characterized in that The average inference of multiple trained semantic segmentation models is used as the final model output, and the calculation formula is as follows: In the formula, F i represents the semantic segmentation model corresponding to the source domain dataset numbered i, where I represents the number of source domains, and x T Represents the target domain image.

Images