Unsupervised domain adaptive semantic segmentation method, system and device based on inter-domain consistency and medium

Abstract

The invention discloses an unsupervised domain adaptive semantic segmentation method, system and device based on inter-domain consistency and a medium. The method comprises the following steps: translating source domain image data to a target domain style while keeping the content by using a GAN-based multi-scale generative adversarial network; inputting the source domain image data into an inter-domain consistency training network based on a Transform model for adaptive training, wherein the source domain image data subjected to data enhancement is sent to a student network for source domain semantic segmentation training, and an EMA strategy is used for updating a teacher network weight; inputting the target domain label-free image data into a teacher network to generate a pseudo label so as to further guide students to train the network; randomly combining the source domain image data and the target domain image data by using a hybrid strategy to guide students to learn domain invariant knowledge in a network; and finally, adjusting and optimizing the student network according to the consistency of the attention feature map, outputting an unsupervised field adaptive semantic segmentation model, and completing segmentation of target domain image data under the condition of not using a target domain label. According to the method, a model with excellent performance can be trained by using fewer real image data sets in the unmanned driving field, so that the training time and cost are greatly reduced, and the working efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned driving perception system and transfer learning technology, and specifically relates to an unsupervised domain adaptive semantic segmentation method, system, device and medium based on inter-domain consistency. Background Art

[0002] In the field of autonomous driving visual perception technology, existing Transformer-based models have outperformed models based on traditional CNN architectures. Training the most advanced semantic segmentation models for autonomous driving perception systems based on Transformer architectures requires large amounts of manually annotated image training data. However, in the real world, collecting and labeling training data is very expensive and requires a lot of time and money. Therefore, using virtual synthetic image datasets to train models has become a more promising solution, as it is easier to obtain annotation information than real image data.

[0003] However, when a model trained on a virtual synthetic image dataset is directly used to predict a real-world image dataset, it usually performs poorly because there are domain differences between the two image datasets, and the model is very sensitive to this. In order to narrow the performance gap, the unsupervised domain adaptation (UDA) method was proposed. The model trained on synthetic image data is adjusted using unlabeled real-world image data to make the model more adaptable to the target domain.

[0004] Unsupervised domain adaptation technology is a new solution to the problems of expensive and time-consuming labeled image data in the field of autonomous driving perception. However, while using virtual synthetic image data to train models brings convenience, this technology still brings the following important challenges:

[0005] 1) There is a huge domain gap between virtual synthetic image datasets and real image datasets, and existing methods still cannot completely overcome it.

[0006] 2) Limited by the capacity of the model, existing methods perform poorly in the segmentation of small objects.

[0007] With the development of artificial intelligence and the introduction of more and more unsupervised domain adaptation technologies, many emerging deep learning-based algorithms are widely used in the field of unsupervised domain adaptation semantic segmentation. As an emerging technology in the field of autonomous driving, unsupervised domain adaptation technology can greatly reduce the need for manually labeled image data in autonomous driving algorithm training, thereby greatly saving manpower and material resources, and also improving the accuracy of perception models.

[0008] Most of the existing unsupervised domain adaptation semantic segmentation methods only focus on single space adaptation of neural networks (such as input space, feature space and output space), and there is almost no method to propose a unified training framework to narrow the inter-domain differences from multiple spaces to improve model performance.

[0009] The 2022 European Conference on Computer Vision (ECCV) conference paper HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation uses a multi-resolution training method to improve segmentation accuracy by combining high-resolution image cropping to retain fine segmentation details and low-resolution image cropping to capture the advantages of learning long-range contextual dependencies. However, this method does not take into account the improvement of the model's learning ability through the construction of intermediate domains and the construction of attention mechanism adaptation, and has disadvantages such as poor segmentation accuracy of small targets and weak ability to distinguish easily confused classes. Summary of the invention

[0010] In order to overcome the defects of the above-mentioned prior art, the purpose of the present invention is to provide an unsupervised domain adaptive semantic segmentation method, system, device and medium based on inter-domain consistency, which can realize high-accuracy prediction on the real image dataset through auxiliary training of the semantic segmentation model trained on the labeled virtual synthetic image dataset (source domain image data) with the aid of unlabeled real dataset (target domain image data); greatly reduce the time and money cost of training, and have the advantages of fast, accurate and efficient.

[0011] In order to achieve the above object, the technical solution adopted by the present invention is:

[0012] An unsupervised domain adaptive semantic segmentation method based on inter-domain consistency specifically includes the following steps:

[0013] Step 1: Input the labeled source domain image data into the GAN-based multi-scale image translation network for style transfer to construct intermediate domain image data to reduce the distribution difference between the source domain and target domain image data;

[0014] Step 2: Input the intermediate domain image data constructed in step 1 into the inter-domain consistency training network based on the Transformer model, and output the semantic segmentation prediction result for the target domain real image dataset through the joint adjustment of the target domain image data and the intermediate domain image data.

[0015] In the style transfer, style refers to the visual characteristics of an image including color scheme, texture, line form, lighting conditions, and detail complexity. Style transfer refers to the transfer of the above characteristics from one image to another without changing the image content.

[0016] The specific method of step one is:

[0017] The labeled source domain image data is input into the GAN-based multi-scale image translation network and translated into an intermediate domain with the same content as the source domain image data, but the image data style is consistent with the target domain image data style:

[0018] 1.1: Use simulated street scene images artificially generated by computer engines as the source domain image data, i.e., virtual synthetic dataset; use real images collected from real scene roads as the target domain, i.e., real image dataset;

[0019] 1.2: Assume that the GAN-based multi-scale image translation network has n scales, where the value of n is artificially selected, and each scale has a CNN-based generator and a CNN-based discriminator; each generator is responsible for translating image data at that scale; each scale generator has two inputs: the first is the generated image data from the generator of the previous scale enlarged to the current scale size, and the second is a scaled version of the source domain image data that matches the current generator scale; the generator of the lowest scale only receives the second input; the output of each scale generator is the translated image data of the current scale with source domain content and target domain style;

[0020] 1.3: The discriminator of each scale has two inputs: the first is the output image data of the generator of the current scale, and the second is a scaled version of the target domain image data that matches the current scale;

[0021] Starting from the lowest scale generator and discriminator, the discriminator and generator are trained in sequence using the loss function. When the scale n is finally reached, the final output of the generator is the intermediate domain image data with the content of the source domain image data and the style of the target domain.

[0022] The specific method of step 2 is:

[0023] A self-learning training method is used to simultaneously train two semantic segmentation networks based on the Transformer structure with exactly the same structure. One of them is a student network that receives normal gradient descent training, and the other is a teacher network that is periodically updated according to the student network parameters.

[0024] 2.1: The student network first accepts the translated image data X generated in step 1 IAs input, it is randomly cropped into small blocks of m×m pixels (patch), and the patch is enhanced by color jitter, Gaussian blur and ClassMix data enhancement methods to obtain the data-enhanced intermediate domain image data; the data-enhanced intermediate domain image data is sent to the student network f θ The training is performed for source domain semantic segmentation. The loss function of the source domain semantic segmentation is:

[0025]

[0026] in, is a weight parameter, equal to N is the total number of pixels, N c is the number of predicted pixels corresponding to the ground truth, meaning that the pixel belongs to the predicted specific category c; is a one-hot vector. Since the image translation module does not change the content and structural information of the image data, the ground truth of the translated image data remains unchanged from the source image data. but When the prediction result is wrong, is the probability that the pixel belongs to category c, which is the prediction result of the student network and is further expressed as:

[0027] The teacher network shares the same architecture as the student network, and instead of performing gradient descent weight updates on the teacher network during the training of the student network, an EMA strategy is used to update the parameters of the teacher network:

[0028]

[0029] in, and θ are the parameters of the teacher network and the student network respectively; β is a hyperparameter used to adjust the update speed;

[0030] 2.2: Mix the intermediate domain image data generated in step 1 with the target domain image data to promote more stable training; select the complementary parts of the source domain image data and the target domain image data for mixing:

[0031] x Mix =M I x I +M T x T

[0032] Among them, M I is the mask, M T is the complementary part of the mask; based on the pseudo-label generated by the teacher network, a mixed pseudo-label with the corresponding part is generated as a supervisory signal:

[0033]

[0034] Among them, y I Equal to the source domain label y S ;

[0035] The student network is trained using mixed image data and mixed pseudo labels, and the mixed cross entropy loss is as follows:

[0036]

[0037] is the prediction result of the teacher network, further expressed as

[0038] 2.3: First, extract the attention feature map corresponding to the intermediate domain image data and the target domain image data input into the student network in step 2 S1 from the decoder head of the current student network, and at the same time extract the attention feature map of the mixed image data from the decoder head of the teacher network The attention features of the intermediate domain and target domain image data are mixed through the mask M:

[0039]

[0040] Where M is the image mask M I Resize to get;

[0041] Minimize the Kullback-Leibler divergence to minimize and The difference between the two attention feature maps:

[0042]

[0043] The global loss function consists of two pixel-level cross entropy losses and attention adaptation loss. The inter-domain consistency training network is trained under the constraints of these three losses:

[0044]

[0045] Finally, the inter-domain consistency training network outputs the student network as an adaptive semantic segmentation network to complete the semantic segmentation task on the target domain image dataset.

[0046] An unsupervised domain-adaptive semantic segmentation system based on inter-domain consistency, comprising:

[0047] The multi-scale image translation network module is used in step 1 to translate the labeled source domain image data, i.e., the virtual synthetic image dataset, into an intermediate domain with the same content as the source domain image but the same style as the target domain, i.e., the real image dataset;

[0048] The inter-domain consistency training network module, in step 2, is used to simultaneously train two semantic segmentation networks based on the Transformer structure with exactly the same structure, one of which is a student network that undergoes normal gradient descent training, and the other is a teacher network that is periodically updated according to the student network parameters. The student network is finally output as an adaptive semantic segmentation network to complete the semantic segmentation task on the target domain image dataset.

[0049] An unsupervised domain adaptive semantic segmentation device based on inter-domain consistency, comprising:

[0050] Memory for storing computer programs;

[0051] A processor is used to implement the unsupervised domain adaptive semantic segmentation method based on inter-domain consistency described in steps 1 to 2 when executing the computer program.

[0052] A computer-readable storage medium stores a computer program, characterized in that when the computer program is executed by a processor, it can implement unsupervised domain adaptive semantic segmentation based on the unsupervised domain adaptive semantic segmentation method based on inter-domain consistency described in steps one to two.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. This invention is a new breakthrough in unsupervised domain adaptive semantic segmentation based on deep learning algorithms of Transformer architecture. Existing methods usually only focus on minimizing inter-domain differences at a single level (such as pixels, features, and predictions), while ignoring the mutual promotion of knowledge learning at different levels. The method proposed in this invention combines knowledge learning at the pixel, feature, and prediction levels, and proposes a complete self-training framework to output a domain adaptive semantic segmentation model with high accuracy.

[0055] 2. Use the image translation module to map the source domain image data to an intermediate domain with a similar distribution to the target domain image data, so as to reduce the huge image data distribution difference between the source domain and the target domain. At the same time, by combining two pixel-level cross entropy losses and attention adaptation losses, the semantic segmentation model learns the domain-invariant knowledge shared by the source domain and the target domain, and can accurately and efficiently apply the model trained on the virtual synthetic image dataset to the real-world dataset.

[0056] 3. Through quantitative verification under the GTA→Cityscapes experimental setting, the present invention improves +1.3mIoU in the Cityscapes real street view image dataset compared with the existing optimal method without using the real image dataset annotation data, achieving the most advanced performance, and has excellent segmentation effects for small targets and easily confused categories. As a new method in the field of unmanned driving perception, the present invention can save a lot of manual annotation costs, reduce the difficulty of training unmanned driving semantic segmentation models, and improve accuracy in practical applications.

[0057] In summary, the present invention can use the semantic segmentation model trained on a labeled virtual synthetic image dataset (source domain image data) to achieve high-accuracy prediction on a real image dataset through auxiliary training with an unlabeled real dataset (target domain image data); it greatly reduces the time and money costs of training, and has the advantages of being fast, accurate, and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a block diagram of the adaptive training of the semantic segmentation network proposed in the present invention.

[0059] Figure 2 This is a visual schematic diagram of step 1 of the present invention.

[0060] Figure 3 This is a visual schematic diagram of step 1 of the present invention.

[0061] Figure 4 This is a visualization diagram of step 2.1 of step 2 of the present invention.

[0062] Figure 5 This is a visualization diagram of step 2.2 of step 2 of the present invention.

[0063] Figure 6 This is a visualization diagram of step 2.3 of step 2 of the present invention. DETAILED DESCRIPTION

[0064] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0065] See also Figure 1 , an unsupervised domain adaptive semantic segmentation method based on inter-domain consistency, specifically comprising the following steps:

[0066] Step 1: Input the labeled source domain image data into the GAN-based multi-scale image translation network for style transfer to construct intermediate domain image data to reduce the distribution difference between the source domain and target domain image data;

[0067] In the style transfer, style refers to the visual characteristics of an image, including color scheme, texture, line form, lighting conditions, and detail complexity. Style transfer refers to the transfer of the above characteristics from one image to another without changing the image content.

[0068] See also Figure 2 , the specific method of step one is:

[0069] 1.1: Use the simulated street scene images artificially generated by computer engines as the source domain image data, i.e., the virtual synthetic dataset; use the real image data collected by real scene roads as the target domain, i.e., the real image dataset;

[0070] 1.2: See Figure 3 , suppose that the GAN-based multi-scale image translation network has n scaling scales, the value of n is artificially selected, and each scale has a CNN-based generator and a CNN-based discriminator; each generator is responsible for translating image data at this scale; the generator of each scale has two inputs: the first is the generated image data from the generator of the previous scale enlarged to the current scaling ratio, and the second is a scaled version of the source domain image data matching the current generator scale; the generator of the lowest scale only receives the second input; the output of each scale generator is the translated image data of the current scale with source domain content and target domain style;

[0071] 1.3: The discriminator of each scale has two inputs: the first is the output image data of the generator of the current scale, and the second is a scaled version of the target domain image data that matches the current scale;

[0072] Starting from the lowest scale generator and discriminator, use the loss function With L G =-E(logD(G(z))) trains the discriminator and generator in sequence, where D(x) refers to the output of the scale discriminator for the real image data x, and G(z) is the output of the scale generator for the two inputs; when the scale n is finally reached, the final output of the generator is the intermediate domain image data with the content of the source domain image data and the style of the target domain.

[0073] Step 2: Input the intermediate domain image data constructed in step 1 into the inter-domain consistency training network based on the Transformer model, and output the semantic segmentation prediction result of the target domain real image dataset through the joint adjustment of the target domain image data and the intermediate domain image data;

[0074] The specific method is:

[0075] A self-learning training method is used to simultaneously train two semantic segmentation networks based on the Transformer structure with exactly the same structure. One of them is a student network that receives normal gradient descent training, and the other is a teacher network that is periodically updated according to the student network parameters.

[0076] See also Figure 4 , 2.1: The student network first accepts the translated image data X generated in step 1 I As input, it is randomly cropped into small patches of m×m pixels, and the patches are enhanced by color jitter, Gaussian blur and ClassMix data enhancement methods to obtain the data-enhanced intermediate domain image data; the data-enhanced intermediate domain image data is sent to the student network f θ The training is performed for source domain semantic segmentation. The loss function of the source domain semantic segmentation is:

[0077]

[0078] in, is a weight parameter, equal to N is the total number of pixels, N c is the number of predicted pixels corresponding to the ground truth, meaning that the pixel belongs to the predicted specific category c; is a one-hot vector. Since the image translation module does not change the content and structural information of the image data, the ground truth of the translated image data remains unchanged from the source image data. but When the prediction result is wrong, This is because in the semantic segmentation cross entropy loss, we only care about the prediction probability of the correct category. By training, the prediction probability of the model for the correct category is increased. Since the total prediction probability of all categories is 1, the prediction probability of the remaining wrong categories will be indirectly reduced, and the accuracy of the model will be improved. is the probability that the pixel belongs to category c, which is the prediction result of the student network and is further expressed as:

[0079] The teacher network shares the same architecture as the student network, and instead of performing gradient descent weight updates on the teacher network during the training of the student network, an EMA strategy is used to update the parameters of the teacher network:

[0080]

[0081] in, and θ are the parameters of the teacher network and the student network respectively; β is a hyperparameter used to adjust the update speed; through the EMA update strategy, the prediction of the teacher network changes steadily, which ensures the smooth progress of the training process;

[0082] See also Figure 5 , 2.2: Mix the intermediate domain image data generated in step 1 with the target domain image data to promote more stable training; Select the complementary parts of the source domain image data and the target domain image data for mixing:

[0083] x Mix =M I x I +M T x T

[0084] Among them, M I is the mask, M T is the complementary part of the mask; based on the pseudo-label generated by the teacher network, a mixed pseudo-label with the corresponding part is generated as a supervisory signal:

[0085]

[0086] Among them, y I Equal to the source domain label y S ;

[0087] The student network is trained using mixed image data and mixed pseudo labels, and the mixed cross entropy loss is as follows:

[0088]

[0089] The difference from S1 is that is the prediction result of the teacher network, which can be further expressed as

[0090] See also Figure 6 ,2.3: First, extract the attention feature map corresponding to the intermediate domain image data and the target domain image data input into the student network in step 2 S1 from the decoder head of the current student network, and at the same time extract the attention feature map of the mixed image data from the decoder head of the teacher network The attention features of the intermediate domain and target domain image data are mixed through the mask M:

[0091]

[0092] Where M is the image mask M I Resize to get;

[0093] Minimize the Kullback-Leibler divergence to minimize and The difference between the two attention feature maps:

[0094]

[0095] The global loss function consists of two pixel-level cross entropy losses and attention adaptation loss. The inter-domain consistency training network is trained under the constraints of these three losses:

[0096]

[0097] Finally, the inter-domain consistency training network outputs the student network as an adaptive semantic segmentation network to complete the semantic segmentation task on the target domain image dataset.

[0098] An unsupervised domain-adaptive semantic segmentation system based on inter-domain consistency, comprising:

[0099] The multi-scale image translation network module is used in step 1 to translate the labeled source domain image data (virtual synthetic image dataset) into an intermediate domain with the same content as the source domain image data but the same style as the target domain (real image dataset). ;

[0100] The inter-domain consistency training network module, in step 2, is used to simultaneously train two semantic segmentation networks based on the Transformer structure with exactly the same structure, one of which is a student network that undergoes normal gradient descent training, and the other is a teacher network that is periodically updated according to the student network parameters. The student network is finally output as an adaptive semantic segmentation network to complete the semantic segmentation task on the target domain image dataset.

[0101] An unsupervised domain adaptive semantic segmentation device based on inter-domain consistency, comprising:

[0102] Memory for storing computer programs;

[0103] A processor is used to implement the unsupervised domain adaptive semantic segmentation method based on inter-domain consistency described in steps 1 to 2 when executing the computer program.

[0104] A computer-readable storage medium stores a computer program, characterized in that when the computer program is executed by a processor, it can implement unsupervised domain adaptive semantic segmentation based on the unsupervised domain adaptive semantic segmentation method based on inter-domain consistency described in steps one to two.

[0105] Experimental verification

[0106] We tested the proposed method on the GTA→Cityscapes experimental setting. GTAV is a synthetic image dataset with pixel-level annotations generated by a game engine, containing 24,966 images with a resolution of 1914 × 1052. Cityscapes is a real-world street scene image dataset, containing 2,975 training set images and 500 validation set images for evaluation.

[0107] In GTA→Cityscapes, the proposed method has a significant improvement over the existing methods. When the method is applied to the HRDA network structure, its mIoU reaches 75.1, exceeding HRDA by 1.3mIoU. When the method is applied to the MIC network structure, it improves by +1.3mIoU, reaching the current state-of-the-art performance of 77.2mIoU, proving the effectiveness of the method. Moreover, the accuracy of the traffic light and bicycle categories is improved by +3.2IoU and +4.0IoU respectively, proving that the method helps to improve the discriminability of easily confused categories, especially for small targets.

Claims

1. An unsupervised domain adaptive semantic segmentation method based on inter-domain consistency, characterized in that: The specific steps include: Step 1: Input the labeled source domain image data into the GAN-based multi-scale image translation network for style transfer to construct intermediate domain image data to reduce the distribution difference between the source domain and target domain image data; Step 2: Input the intermediate domain image data constructed in step 1 into the inter-domain consistency training network based on the Transformer model, and output the semantic segmentation prediction result for the target domain real image dataset through the joint adjustment of the target domain image data and the intermediate domain image data.

2. The unsupervised domain adaptive semantic segmentation method based on inter-domain consistency according to claim 1, characterized in that: In the style transfer, style refers to the visual characteristics of an image including color scheme, texture, line form, lighting conditions, and detail complexity. Style transfer refers to the transfer of the above characteristics from one image to another without changing the image content.

3. The unsupervised domain adaptive semantic segmentation method based on inter-domain consistency according to claim 1, characterized in that: The specific method of step one is: The labeled source domain image data is input into the GAN-based multi-scale image translation network and translated into an intermediate domain with the same content as the source domain image data but the same style as the target domain image: 1.1: The simulated street scene images artificially generated by computer engines are used as the source domain image data, i.e., the virtual synthetic image dataset; the real image data collected by real scene roads are used as the target domain, i.e., the real image dataset; 1.2: Assume that the GAN-based multi-scale image translation network has n scales, where the value of n is artificially selected, and each scale has a CNN-based generator and a CNN-based discriminator; each generator is responsible for translating image data at that scale; each scale generator has two inputs: the first is the generated image data from the generator of the previous scale enlarged to the current scale size, and the second is a scaled version of the source domain image data that matches the current generator scale; the generator of the lowest scale only receives the second input; the output of each scale generator is the translated image data of the current scale with source domain content and target domain style; 1.3: The discriminator of each scale has two inputs: the first is the output image data of the generator of the current scale, and the second is a scaled version of the target domain image data that matches the current scale; Starting from the lowest scale generator and discriminator, use the loss function: - Train the discriminator and generator in sequence. When the scale n is finally reached, the final output of the generator is the intermediate domain image data with the content of the source domain image data and the style of the target domain.

4. The unsupervised domain adaptive semantic segmentation method based on inter-domain consistency according to claim 1, characterized in that: The specific method of step 2 is: A self-learning training method is used to simultaneously train two semantic segmentation networks based on the Transformer structure with exactly the same structure. One of them is a student network that receives normal gradient descent training, and the other is a teacher network that is periodically updated according to the student network parameters. 2.1: The student network first accepts the translated image data X generated in step 1 I As input, it is randomly cropped into small patches of m×m pixels, and the patches are enhanced by color jitter, Gaussian blur and ClassMix data enhancement methods to obtain the data-enhanced intermediate domain image data; the data-enhanced intermediate domain image data is sent to the student network f θ The training is performed for source domain semantic segmentation. The loss function of the source domain semantic segmentation is: in, is a weight parameter, equal to N is the total number of pixels, N c is the number of predicted pixels corresponding to the ground truth, meaning that the pixel belongs to the predicted specific category c; is a one-hot vector. Since the image translation module does not change the content and structural information of the image data, the ground truth of the translated image data remains unchanged from the source image data. but When the prediction result is wrong, is the probability that the pixel belongs to category c, which is the prediction result of the student network and is further expressed as: The teacher network shares the same architecture as the student network, and instead of performing gradient descent weight updates on the teacher network during the training of the student network, an EMA strategy is used to update the parameters of the teacher network: in, and θ are the parameters of the teacher network and the student network respectively; β is a hyperparameter used to adjust the update speed; 2.2: Mix the intermediate domain image data generated in step 1 with the target domain image data to promote more stable training; select the complementary parts of the source domain image data and the target domain image data for mixing: x Mix =M I x I +M T x T Among them, M I is the mask, M T is the complementary part of the mask; based on the pseudo-label generated by the teacher network, a mixed pseudo-label with the corresponding part is generated as a supervisory signal: Among them, y I Equal to the source domain label y S ; The student network is trained using mixed image data and mixed pseudo labels, and the mixed cross entropy loss is as follows: is the prediction result of the teacher network, further expressed as 2.3: First, extract the attention feature map corresponding to the intermediate domain image data and the target domain image data input into the student network in step 2 S1 from the decoder head of the current student network, and at the same time extract the attention feature map of the mixed image data from the decoder head of the teacher network The attention features of the intermediate domain and target domain image data are mixed through the mask M: Where M is the image mask M I Resize to get; Minimize the Kullback-Leibler divergence to minimize and The difference between the two attention feature maps: The global loss function consists of two pixel-level cross entropy losses and attention adaptation loss. The inter-domain consistency training network is trained under the constraints of these three losses: Finally, the inter-domain consistency training network outputs the student network as an adaptive semantic segmentation network to complete the semantic segmentation task on the target domain image dataset.

5. An unsupervised domain-adaptive semantic segmentation system based on inter-domain consistency, characterized in that: include: The multi-scale image translation network module is used in step 1 to translate the labeled source domain image data, i.e., the virtual synthetic image dataset, into an intermediate domain having the same content as the source domain image data but the same style as the target domain, i.e., the real image dataset; The inter-domain consistency training network module, in step 2, is used to simultaneously train two semantic segmentation networks based on the Transformer structure with exactly the same structure, one of which is a student network that undergoes normal gradient descent training, and the other is a teacher network that is periodically updated according to the student network parameters. The student network is finally output as an adaptive semantic segmentation network to complete the semantic segmentation task on the target domain image dataset.

6. An unsupervised domain adaptive semantic segmentation device based on inter-domain consistency, characterized in that: include: Memory for storing computer programs; A processor is used to implement the unsupervised domain adaptive semantic segmentation method based on inter-domain consistency described in steps 1 to 2 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement unsupervised domain adaptive semantic segmentation based on the unsupervised domain adaptive semantic segmentation method based on inter-domain consistency described in steps one to two.

Images