Semantic Segmentation Method Based on Feature-Level Unsupervised Domain Adaptation Network

Abstract

The present invention relates to a semantic segmentation method based on a feature-level unsupervised domain adaptation network, comprising the following steps: acquiring image data; inputting the image data into the feature-level unsupervised domain adaptation network to visualize the segmentation effect of the image data, where the feature-level unsupervised domain adaptation network is a self-training network including three shared-weight branches, and the self-training network includes a source branch part, a target branch part, and a target-source branch part; the source branch part is used to learn the distribution within the source domain, the target branch part is used to learn the distribution within the target domain, and the target-source branch part is used to align the inter-domain distribution between the source domain and the target domain at the feature level. The present invention can effectively enhance the domain adaptation ability and achieve a better segmentation effect on the target domain dataset.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly to a semantic segmentation method based on a feature-level unsupervised domain adaptation network. Background Art

[0002] In the past few years, methods based on deep neural networks have made significant progress in semantic segmentation and have been widely applied to tasks such as autonomous driving, medical image analysis, remote sensing image observation, and augmented reality. However, the training of neural networks usually requires a large amount of labeled data that matches the considered scenario. However, collecting and annotating large datasets for each task and domain is very expensive, time-consuming, and error-prone. Especially for semantic segmentation tasks, the cost of annotation is particularly high because each pixel must be labeled. One solution is to use computer graphics technology to render and synthesize a large number of synthetic datasets based on real-world images (e.g., computer game images). In this way, the data collection cost is relatively low and pixel-level annotation can be automatically generated. Furthermore, rich labeled synthetic data (source domain) and insufficient unlabeled real data (target domain) can be utilized to improve the network's processing performance for real data. However, there are significant domain differences between synthetic data and real data, such as differences in texture, lighting conditions, etc. Therefore, simply applying the neural network learned on synthetic data to real images may lead to large generalization errors due to domain differences, and a certain domain adaptation method must be used to narrow the performance gap.

[0003] According to the location where the domain alignment strategy is implemented, in the field of semantic segmentation, existing unsupervised domain adaptation methods can be roughly divided into three categories: input-level, output-level, and stage-output-level. Among them, the input-level methods can be further divided into two categories. The first category is to directly mix input images from different domains. The second category is to construct an additional style transfer network to achieve visual appearance consistency of images from different domains. For output-level alignment methods, some methods attempt to create different decoders through dropout or generative adversarial ideas, and then guide the encoder to learn domain-invariant features by forcing the output consistency of different decoders. Other methods choose to train a domain discriminator based on the output of the decoder to provide additional domain-level supervision for domain distribution alignment. To avoid the complexity of the high-dimensional feature space, some methods also directly apply the discriminator to the output of the encoder, and this improved discriminant method is classified as stage-output-level. However, because the semantic representations and local details learned at different stages of the network are different, for domain adaptation tasks, simply aligning the domain distributions of the input, output, and stage output is insufficient. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a semantic segmentation method based on a feature-level unsupervised domain adaptation network, which can effectively enhance the domain adaptation ability and achieve better segmentation results on the target domain dataset.

[0005] The technical solution adopted by the present invention to solve its technical problems is: to provide a semantic segmentation method based on a feature-level unsupervised domain adaptation network, including the following steps:

[0006] Obtain image data;

[0007] Input the image data into the feature-level unsupervised domain adaptation network to visualize the segmentation effect of the image data. The feature-level unsupervised domain adaptation network is a self-training network including three shared-weight branches, namely a source branch part, a target branch part, and a target-source branch part. The source branch part is used to learn the distribution within the source domain, the target branch part is used to learn the distribution within the target domain, and the target-source branch part is used to align the inter-domain distribution between the source domain and the target domain at the feature level.

[0008] On each layer of the encoder of the target branch part, the distribution within the target domain is learned based on the multi-head self-attention module, and the attention output is obtained through a residual connection and layer normalization for the distribution within the target domain.

[0009] The learning of the distribution within the target domain on each layer of the encoder of the target branch part based on the multi-head self-attention module is expressed as: F msa (Q t ,K t ,V t ) = Concat(Attn1(Q t ,K t ,V t ),...,Attn N (Q t ,K t ,V t ))W MSA , where F msa (Q t ,K t ,V t ) represents the distribution within the target domain, N represents the number of self-attention heads, W MSA ∈R C×C represents learnable parameters, Q t ,K t ,V t represent the query set, key set, and value set of the target domain respectively.

[0010] On each layer of the encoder in the source branch part, the distribution within the source domain is learned based on the multi-head self-attention module, and the attention output is obtained from the distribution within the source domain through a residual connection and layer normalization.

[0011] On each layer of the encoder in the target-source branch part, there are two modules based on the cross-attention mechanism, namely the inter-domain feature alignment module and the domain-invariant feature learning module. The inter-domain feature alignment module is connected to the domain-invariant feature learning module through a linear transformation module. The inter-domain feature alignment module is used to learn the correlation between the features of the source domain and the target domain, and then find the source domain features with a similar feature style to the target domain features based on the correlation to update the target domain features. The query set of the inter-domain feature alignment module comes from the target domain, and the key set and value set come from the source domain. The query set of the domain-invariant feature learning module comes from the linear transformation module, and the key set and value set come from the target domain.

[0012] The inter-domain feature alignment module is expressed as: The domain-invariant feature learning module is expressed as F dflm (Q ts ,K t ,V t ) = Concat(Attn1(Q ts ,K t ,V t ),..., Attn N (Q ts ,K t ,V t ))W MSA , where F dflm (Q ts ,K t ,V t ) represents the output of the domain-invariant feature learning module, N represents the number of self-attention heads, W MSA ∈ R C×C is the shared parameter of the source branch part and the target branch part, Q ts = Attn(Q t ,K s ,V s )W ifam represents the output of the inter-domain feature alignment module after passing through the linear transformation module, represents the learnable parameter, , where Q t represents the query set of the target domain, K s ,V s respectively represent the key set and value set of the source domain, K t ,V trespectively represent the key set and the value set of the target domain.

[0013] When training the feature-level unsupervised domain adaptation network, first construct a teacher network with the same structure as the target branch part by using the method based on moving exponential average; input the target domain data into the teacher network to obtain pseudo-labels, input the target domain data into the target branch part and the target-source branch part, and calculate the cross-entropy loss of the target branch part and the cross-entropy loss of the target-source branch part according to the corresponding network prediction results and the pseudo-labels respectively; input the source domain data into the source branch part, and directly calculate the cross-entropy loss of the source branch part according to the corresponding network prediction result and the label.

[0014] Beneficial effects

[0015] Due to the adoption of the above technical solutions, compared with the prior art, the present invention has the following advantages and positive effects: The present invention proposes to align the domain distributions between the source domain and the target domain at the feature level, and by migrating the feature style on each layer of the network encoder and learning domain-invariant features, the feature-level unsupervised domain adaptation network proposed by the present invention effectively enhances the domain adaptation ability and achieves better segmentation results on the target domain dataset. Brief description of the drawings

[0016] Figure 1 is the framework diagram of the feature-level unsupervised domain adaptation network in the embodiment of the present invention;

[0017] Figure 2 is the schematic diagram of the feature-level domain alignment operation in the embodiment of the present invention;

[0018] Figure 3 is the visualization result diagram of the embodiment of the present invention. Specific embodiments

[0019] The following further elaborates the present invention in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0020] The embodiment of the present invention relates to a semantic segmentation method based on a feature-level unsupervised domain adaptation network, including the following steps:

[0021] Obtain image data;

[0022] Input the image data into the feature-level unsupervised domain adaptation network to realize the visualization of the segmentation effect of the image data.

[0023] This embodiment uses labeled source domain data and unlabeled target domain data as the input for training a feature-level unsupervised domain adaptation network, aiming to reduce the domain difference through a feature-level domain alignment strategy, so as to obtain better semantic segmentation results on the target domain data.

[0024] As Figure 1 shown, the feature-level unsupervised domain adaptation network in this embodiment is a self-training network that contains three shared-weight branches (source branch, target branch, and target-source branch). Among them, the source branch and the target branch respectively learn the intra-domain distributions of the source domain and the target domain, and the middle target-source branch is used to align the inter-domain distribution between the two domains at the feature level.

[0025] Specifically, as Figure 2 shown, on each layer of the encoder of the target branch, this embodiment learns the intra-domain distribution representation based on the multi-head self-attention module (MSA) as:

[0026]

[0027] F msa (Q t ,K t ,V t ) = Concat(Attn1(Q t ,K t ,V t ),..., Attn N (Q t ,K t ,V t ))W MSA

[0028] where the learnable parameter W MSA ∈R C×C , N is the number of attention heads, Q t ,K t ,V t respectively represent the query set, key set, and value set of the target domain. Then, a residual connection and layer normalization (LN) are applied to F msa (Q t ,K t ,V t ) to obtain the attention output:

[0029] Y t = LN(F msa (Q t ,K t ,V t ) + X t )

[0030] On each layer of the encoder in the source branch, this embodiment uses the same multi-head self-attention module to learn the intra-domain distribution Y of the source domain s .

[0031] The values in Attn(Q, K, V) reflect the correlation between the feature points to be updated and all other feature points in the current feature map. This embodiment believes that this correlation can be used as a criterion for measuring the consistency of feature styles. Therefore, the role of multi-head self-attention can be considered to find all other feature points in the current feature map that have the same feature style as the feature points to be updated, and give them higher weights to update the current feature value. Theoretically, feature points of the same semantic category should have a consistent feature style. Considering that in the unsupervised domain adaptation task, the semantic categories are the same between different domains, on each layer of the encoder in the target-source branch, this embodiment designs an inter-domain feature alignment module (IFAM) based on the cross-attention mechanism to learn the correlation between the source domain features and the target domain features, and then find the source domain features with a similar feature style to the target domain features based on the correlation to update the target domain features. Different from the query set Q, key set K, and value set V in the multi-head self-attention module all coming from the same domain, the query set Q of the inter-domain feature alignment module t comes from the target domain, and the key set K s and the value set V s come from the source domain:

[0032]

[0033] The inter-domain feature alignment module can transfer the style of features with the same semantic category in the source domain to the current target domain features. If no feature points with the same semantic category are found in the source domain, this module can also transfer the overall feature style of the source domain. The present invention believes that the difference in feature style is an important manifestation of domain difference at the feature level. Through the inter-domain feature alignment module, the present invention realizes feature-level domain alignment while ensuring that the semantic content in the original target domain features remains unchanged.

[0034] In addition, the output Attn(Q t , K s , V s ) of the inter-domain feature alignment module needs to be subjected to an additional linear transformation:

[0035] F ifam (Q t , K s , V s ) = Attn(Q t , K s , V s )W ifam

[0036] Different from the multi-head self-attention module, multiple attention heads are not merged before this linear transformation, so the learnable parameters

[0037] In the target branch, Attn(Q t ,K t ,V t ) can only select highly relevant feature points, but it cannot guarantee that feature points with strong relevance also have small domain differences. Therefore, this embodiment further proposes a domain-invariant feature learning module (DFLM), that is, using F ifam (Q t ,K s ,V s ) to replace the query set Q t of the multi-head attention module in the target branch. Since F ifam (Q t ,K s ,V s ) migrates the source domain feature style, so compared with Attn(Q t ,K t ,V t ) in the target branch, Attn(Q ifam (Q t ,K s ,V s ) calculated using F ts ,K t ,V t ) can select feature points with strong relevance and small domain differences, further enhancing the domain adaptation effect of the network. The domain-invariant feature learning module can be expressed as:

[0038] Q ts =F ifam (Q t ,K s ,V s )

[0039]

[0040] F dflm (Q ts ,K t ,V t )=Concat(Attn1(Q ts ,K t ,V t ),...,Attn N (Q ts ,K t ,V t ))W MSA

[0041] Among them, WMSA ∈R C×C shares parameters with the source branch and the target branch. Then, for F dflm (Q ts , K t , V t ), a residual connection and layer normalization are applied to obtain the attention output:

[0042] Y ts = LN(F dflm (Q ts , K t , V t ) + X ts )

[0043] As can be seen from Figure 1 , due to the lack of labels in the target domain data, in this embodiment, a teacher network with the same structure as the segmentation network but more robust to noise is additionally constructed based on the moving average method. At the beginning of each round of training, the target domain data is first input into the teacher network to obtain pseudo-labels. Then the same target domain data is input into the target branch and the target-source branch in the segmentation network, and the cross-entropy loss is calculated between the corresponding network prediction results and the pseudo-labels. The source domain data is input into the source branch, and the cross-entropy loss is directly calculated between the corresponding network prediction results and the labels.

[0044] It should be noted that in the inference stage, only the target domain data is input into the network, that is, there is no source branch and target branch in the inference stage. Therefore, compared with the existing benchmark method DAFormer, the inference efficiency of this embodiment does not decrease at all.

[0045] This embodiment realizes a more sufficient feature-level unsupervised domain alignment operation, effectively reduces the domain difference, and obtains better semantic segmentation results on the unlabeled target domain data. Figure 3 shows the visualization of the segmentation results of some complex scenes in the GTA-to-Cityscapes task by the feature-level domain adaptation semantic segmentation network of this embodiment, and compares the method proposed in this embodiment with the existing benchmark method DAFormer and the current best method SePiCo. It can be clearly seen that the present invention can more accurately avoid regional segmentation errors, especially for some semantic categories with similar appearances in the image, and the present invention can give more accurate discrimination.

Claims

1. A semantic segmentation method based on a feature-level unsupervised domain adaptation network, characterized in that Including the following steps: Obtain image data; Input the image data into a feature-level unsupervised domain adaptation network to visualize the segmentation effect of the image data. The feature-level unsupervised domain adaptation network is a self-training network with three shared-weight branches, including a source branch part, a target branch part, and a target-source branch part. The source branch part is used to learn the distribution within the source domain, the target branch part is used to learn the distribution within the target domain, and the target-source branch part is used to align the inter-domain distribution between the source domain and the target domain at the feature level; Among them, on each layer of the encoder of the target branch part, the distribution within the target domain is learned based on a multi-head self-attention module, and the attention output is obtained through a residual connection and layer normalization for the distribution within the target domain; On each layer of the encoder of the source branch part, the distribution within the source domain is learned based on a multi-head self-attention module, and the attention output is obtained through a residual connection and layer normalization for the distribution within the source domain; On each layer of the encoder of the target-source branch part, there are two modules based on the cross-attention mechanism, namely an inter-domain feature alignment module and a domain-invariant feature learning module. The inter-domain feature alignment module is connected to the domain-invariant feature learning module through a linear transformation module. The inter-domain feature alignment module is used to learn the correlation between the features of the source domain and the features of the target domain, and then find the source domain features with a similar feature style to the features of the target domain based on the correlation to update the target domain features. The query set of the inter-domain feature alignment module comes from the target domain, and the key set and value set come from the source domain. The query set of the domain-invariant feature learning module comes from the linear transformation module, and the key set and value set come from the target domain.

2. The semantic segmentation method based on a feature-level unsupervised domain adaptation network according to claim 1, wherein On each layer of the encoder of the target branch part, the distribution representation within the target domain is learned based on the multi-head self-attention module as: F msa (Q t ,K t ,V t ) = Concat(Attn1(Q t ,K t ,V t ),..., Attn N (Q t ,K t ,V t ))W MSA , where F msa (Q t ,K t ,V t ) represents the distribution within the target domain, N represents the number of self-attention heads, W MSA ∈ R C×C represents learnable parameters, Q t ,K t ,V t represent the query set, key set, and value set of the target domain respectively.

3. The semantic segmentation method based on a feature-level unsupervised domain adaptation network according to claim 1, characterized in that, The inter-domain feature alignment module is denoted as: The domain-invariant feature learning module is denoted as: F dflm (Q ts , K t , V t ) = Concat(Attn1(Q ts , K t , V t ),..., Attn N (Q ts , K t , V t ))W MSA , where F dflm (Q ts , K t , V t ) represents the output of the domain-invariant feature learning module, N represents the number of self-attention heads, and W MSA ∈ R C×C is the shared parameter of the source branch part and the target branch part. Q ts = Attn(Q t , K s , V s )W ifam represents the output of the inter-domain feature alignment module after passing through the linear transformation module. represents learnable parameters. where Q t represents the query set of the target domain, K s , V s represent the key set and value set of the source domain respectively, and K t , V t represent the key set and value set of the target domain respectively.

4. The semantic segmentation method based on a feature-level unsupervised domain adaptation network according to claim 1, wherein When training the feature-level unsupervised domain adaptation network, first construct a teacher network with the same structure as the target branch part using the method based on moving exponential average; input the target domain data into the teacher network to obtain pseudo-labels, input the target domain data into the target branch part and the target-source branch part, and calculate the cross-entropy loss of the target branch part and the cross-entropy loss of the target-source branch part respectively according to the corresponding network prediction results and the pseudo-labels; Input the source domain data into the source branch part, and directly calculate the cross-entropy loss of the source branch part according to the corresponding network prediction result and the label.

Images