An unsupervised domain adaptation semantic segmentation method based on region feature alignment

Abstract

The present invention relates to an unsupervised domain adaptation semantic segmentation method based on regional feature alignment, the method comprising: training a student-teacher model: inputting a source domain image into a student model to obtain a segmentation prediction result, performing cross entropy loss calculation on the segmentation prediction result of the source domain image and the source domain label, performing category-based fusion on the pixels of the source domain image and the target domain image, performing consistency loss calculation on the first fused image obtained through the segmentation prediction result of the student model and the pseudo label of the first fused image, performing region-based fusion on the pixels of the source domain image and the target domain image, performing regional feature alignment on the second fused image and the target domain image to obtain contrast loss, calculating the gradient of the three types of losses, back-propagating the gradient to update the parameters of the student model, and updating the parameters of the teacher model at the same time; and using the trained teacher model to perform semantic segmentation on the target domain image. Compared with the prior art, the robustness and stability of the segmentation effect of the present invention are greatly improved.

Description

Technical Field

[0001] The present invention relates to a semantic segmentation method, and in particular to an unsupervised domain adaptation semantic segmentation method based on regional feature alignment. Background Art

[0002] Semantic segmentation is a fundamental research direction in the field of computer vision. Its task is to assign a category label to each pixel in an image. It has many real-world application scenarios, such as autonomous driving, medical image analysis, and remote sensing systems. With the continuous development of deep learning, especially since 2017, deep learning models and technologies have made great progress in the field of semantic segmentation. At the same time, these deep learning methods require a large amount of labeled data, which requires high overhead and time costs to obtain.

[0003] In order to solve the above problems, unsupervised domain adaptation semantic segmentation has become a hot research direction. Its goal is to learn a model that can achieve segmentation performance close to that of the source domain in the target domain based on labeled source domain data and unlabeled target domain data. Its research focus is on how to align the source domain data and target domain data with different distributions. The current mainstream domain alignment methods are mainly concentrated in three levels: input level (image level), feature level, and output level (label level). Among them, feature alignment is the most widely studied and most effective domain alignment method.

[0004] From the perspective of the research object, the current feature alignment methods mainly focus on the global feature alignment of the image, that is, penalizing the inconsistency of features between the source domain image and the target domain image at the image level. This method ignores the importance of regional feature alignment. Due to the existence of domain differences, objects in the source domain and the target domain often exist in different distribution environments. After a large amount of supervised learning of the same object in the source domain, its segmentation effect will be very dependent on the contextual information of the source domain, resulting in a sharp decline in the segmentation effect of the object in different distribution environments in the target domain. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an unsupervised domain adaptation semantic segmentation method based on regional feature alignment.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] An unsupervised domain adaptation semantic segmentation method based on regional feature alignment, the method comprising:

[0008] Training the Student-Teacher Model:

[0009] Input the source domain image into the student model to obtain the segmentation prediction result, and calculate the cross entropy loss between the segmentation prediction result of the source domain image and the source domain label.

[0010] The pixels of the source domain image and the target domain image are fused based on categories, and the first fused image is obtained by calculating the consistency loss of the segmentation prediction result of the student model and the pseudo label of the first fused image.

[0011] The pixels of the source domain image and the target domain image are fused based on the region, and the regional features of the second fused image and the target domain image are aligned to obtain the contrast loss.

[0012] Calculate the gradients of cross entropy loss, consistency loss, and contrast loss, backpropagate the gradients to update the parameters of the student model, and use the exponential moving average method to update the parameters of the teacher model;

[0013] The trained teacher model is used to perform semantic segmentation on the target domain images.

[0014] Preferably, the category-based fusion method includes: randomly selecting a set number of pixel points of a category on the source domain image, and overlaying the selected pixel points on the same position on the target domain image to obtain a first fused image.

[0015] Preferably, the pseudo label of the first fused image is obtained by:

[0016] Input the target domain image into the teacher model, and the obtained segmentation prediction result is used as the pseudo label of the target domain image;

[0017] The source domain label and the pseudo label of the target domain image are fused based on the category according to the pixel points selected when obtaining the first fused image, so as to obtain the pseudo label of the first fused image.

[0018] Preferably, the region-based fusion method includes: randomly cutting out pixels of a rectangular region from the target domain image, and overlaying the pixels of the region on the same position of the source domain image to obtain a second fused image.

[0019] Preferably, a specific method of performing regional feature alignment on the second fused image and the target domain image to obtain the contrast loss includes:

[0020] Input the target domain image into the teacher model, and the obtained segmentation prediction result is used as the pseudo label of the target domain image;

[0021] Input the second fused image into the student model, and obtain a pseudo label of the second fused image based on the source domain label and the pseudo label of the target domain image;

[0022] The teacher model projection structure is used to project the features extracted from the target domain image at the teacher model backbone network to obtain the projection vector of the target domain image;

[0023] The student model projection structure is used to project the features extracted from the second fused image at the student model backbone network to obtain a projection vector of the second fused image;

[0024] Feature alignment is performed on the overlapping area between the projection vector of the target domain image and the projection vector of the second fused image to obtain the contrast loss of the overlapping area on the projection vector of the second fused image.

[0025] Preferably, the pseudo label of the second fused image is obtained by: performing region-based fusion on the source domain label and the pseudo label of the target domain image according to the pixel points selected when obtaining the second fused image, to obtain the pseudo label of the second fused image.

[0026] Preferably, the teacher model projection structure and the student model projection structure respectively include two 1*1 convolutions after the backbone network and in parallel with the segmentation head.

[0027] Preferably, the specific method of obtaining the contrast loss of the overlapping area on the projection vector of the second fused image is:

[0028] For each position of the overlapping area of ​​the projection vector of the second fused image, respectively select positive and negative samples;

[0029] Based on the screened positive and negative samples, the contrast loss of the projection vectors at each position in the overlapping area of ​​the second fused image is calculated and summed to obtain the contrast loss of the overlapping area.

[0030] Preferably, the method of screening positive and negative samples includes:

[0031] For each position in the overlapping area, its positive sample includes the projection vector of the same position of the target domain image. The position is screened whether to participate in the calculation of the overlapping area contrast loss according to the predicted probability of the segmentation prediction result of the target domain image at this position by the teacher model. If the predicted probability of the segmentation prediction result of the target domain image at this position by the teacher model is greater than the set threshold, the corresponding position participates in the calculation of the overlapping area contrast loss, and then the negative sample corresponding to the position is screened. The negative sample includes the projection vector corresponding to other positions in the target domain image and the second fused image and does not belong to the same label category as the current screening position. The negative sample also includes the negative sample generated during the historical iteration process.

[0032] Preferably, the contrast loss is obtained by the following formula:

[0033]

[0034]

[0035] in, Represents the projection vector E at position (i, j) in the overlapping area of ​​the second fused image m2(i,j) The contrast loss is is the projection vector E m2(i,j) The corresponding positive sample projection vector, E (m,n) is the projection vector E m2(i,j) The corresponding negative sample projection vector, N represents the projection vector E m2(i,j) The corresponding negative sample set, P is the set of positions that participate in the contrast loss calculation, NUM is the total number of positions in the set P that participate in the contrast loss calculation, exp is the exponential operation, sim(u,v)=(u T v) / (||u||||v||), sim(u,v) represents the cosine similarity of the projection vectors u and v, and τ is the parameter factor.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] (1) The present invention solves the problem that common unsupervised domain adaptation semantic segmentation methods only focus on global feature alignment of images. The method based on regional feature alignment can prompt the model to learn features that are more focused on the object itself, improve the stability of the segmentation effect of the object in different environments, make the segmentation effect of the object in the target domain environment closer to the source domain, and enhance the robustness and stability of the segmentation effect of the object in different environments of the source domain and the target domain;

[0038] (2) Since the confidence of the teacher model is higher than that of the student model, the extracted features are more accurate. Different from the common contrast loss function of two-way approach, the present invention uses a learning mode of one-way approach from the projection vector of the student model to the projection vector of the teacher model, which can avoid the inaccurate output of the student model during two-way approach from interfering with the learning effect of the contrast loss, thereby improving the accuracy of the model;

[0039] (3) When calculating the contrast loss, the present invention designs a positive and negative sample screening strategy to screen the positive and negative samples, thereby enabling more effective regional feature alignment and improving the segmentation prediction effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a general flow chart of an unsupervised domain adaptation semantic segmentation method based on regional feature alignment of the present invention;

[0041] Figure 2 A flowchart of the cross entropy loss calculation of the present invention;

[0042] Figure 3A flow chart of the consistency loss calculation of the present invention;

[0043] Figure 4 This is a flow chart of the contrast loss calculation of the present invention. DETAILED DESCRIPTION

[0044] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the following embodiments are merely illustrative in nature, and the present invention is not intended to limit its applicable objects or uses, and the present invention is not limited to the following embodiments.

[0045] Example

[0046] like Figure 1 As shown, the present invention provides an unsupervised domain adaptation semantic segmentation method based on regional feature alignment, which uses the existing student-teacher model as a basic framework and specifically includes the following steps:

[0047] Training the Student-Teacher Model:

[0048] Input the source domain image into the student model to obtain the segmentation prediction result, and calculate the cross entropy loss between the segmentation prediction result of the source domain image and the source domain label.

[0049] The pixels of the source domain image and the target domain image are fused based on categories, and the first fused image is obtained by calculating the consistency loss of the segmentation prediction result of the student model and the pseudo label of the first fused image.

[0050] The pixels of the source domain image and the target domain image are fused based on the region, and the regional features of the second fused image and the target domain image are aligned to obtain the contrast loss.

[0051] Calculate the gradients of cross entropy loss, consistency loss, and contrast loss, backpropagate the gradients to update the parameters of the student model, and use the exponential moving average method to update the parameters of the teacher model;

[0052] The trained teacher model is used to perform semantic segmentation on the target domain images.

[0053] The specific implementation steps are as follows:

[0054] See also Figure 2 In step 1 of the present invention, the source domain image is input into the student model to obtain the segmentation prediction result of the source domain image, and the cross entropy loss is calculated for the result and the source domain label.

[0055] S11: Input the source domain image into the student model, and extract the feature map at the backbone network of the student model;

[0056] S12: Input the feature map extracted by S11 into the student model segmentation head;

[0057] S13: Process the output of the segmentation head in S12 to obtain the segmentation result of the source domain image;

[0058] S14: Calculate the cross entropy loss between the source domain image segmentation result obtained in S13 and the source domain label.

[0059] See also Figure 3 , step 2 of the present invention uses the source domain image and the target domain image to generate a first fused image.

[0060] S21: randomly selecting half of the pixel points of the categories on the source domain image according to the label of the source domain image, and covering the pixel points of this part with the same positions on the target domain image to obtain a first fused image;

[0061] See also Figure 3 In step 2 of the present invention, the target domain image is placed into the teacher model, the projection vector of the target domain image is obtained at the output end of the projection structure, the category prediction of each pixel point and the prediction probability corresponding to the category are obtained at the segmentation head, and the category prediction is used as a pseudo label of the target domain image.

[0062] S22: Input the target domain image into the teacher model, and extract the feature map at the teacher model backbone network;

[0063] S23: Input the feature map extracted by S22 into the teacher model projection structure;

[0064] S24: Process the output of the projection structure in S23 to obtain a projection vector of the target domain image;

[0065] S25: Input the feature map extracted by S22 into the teacher model segmentation head;

[0066] S26: Process the output of the segmentation head in S25 to obtain the predicted probability of each position of the target domain image;

[0067] S27: Process the output of the segmentation head in S25 to obtain a pseudo label at each position of the target domain image;

[0068] See also Figure 3 In step 2 of the present invention, the label of the source domain image and the pseudo label of the target domain image are fused to obtain a pseudo label of the first fused image.

[0069] S28: In the same manner as S21, the source domain image label and the target domain pseudo label obtained in S27 are fused to obtain a pseudo label of the first fused image.

[0070] See also Figure 3In step 2 of the present invention, the fused image 1 is placed into the student model, the category prediction of each pixel is obtained in the segmentation head, and the consistency loss function is used to penalize the category prediction result and the fused label 1 obtained in S28.

[0071] S29: inputting the first fused image into the student model, and extracting a feature map at the backbone network of the student model;

[0072] S210: Input the feature map extracted by S29 into the student model segmentation head;

[0073] S211: Processing the output of the segmentation head in S210 to obtain a segmentation result of the first fused image;

[0074] S212: Calculate the consistency loss of the segmentation result of the fused image 1 obtained in S211 and the fused label 1 obtained in S28.

[0075] See also Figure 4 In step 3 of the present invention, image fusion is used to create environmental changes by pasting a random rectangular area on the target domain to the same position in the source domain to create two samples in different environments for the rectangular area.

[0076] S31: randomly cropping pixels of a rectangular area from the target domain image, and overlaying the pixels of the area on the same position of the source domain image to obtain a second fused image;

[0077] See also Figure 4 In step 3 of the present invention, the label of the source domain image and the pseudo label of the target domain image are fused to obtain a pseudo label of the second fused image.

[0078] S32: In the same manner as S31, the source domain image label and the target domain pseudo label obtained in S27 are fused to obtain a pseudo label of a second fused image.

[0079] See also Figure 4 In step 3 of the present invention, the fused image 2 is placed into the student model, and the projection vector of the second fused image is obtained at the output end of the projection structure. The contrast loss function is used to penalize the overlapping rectangular area between the projection vector of the second fused image and the projection vector of the target domain image obtained in S24.

[0080] S33: inputting the second fused image into the student model, and extracting a feature map at the backbone network of the student model;

[0081] S34: input the feature map extracted by S33 into the student model projection structure;

[0082] S35: Processing the output of the projection structure in S34 to obtain a projection vector of the second fused image;

[0083] See also Figure 4 In step 3 of the present invention, different sampling strategies are used for positive and negative samples to further exert the effect of the contrast loss function. For negative samples, all positions on the image are randomly screened first, and then negative samples of the same category at different positions are filtered out according to fusion label 2; for positive samples, a threshold is set to filter out low-confidence samples, and only those positions whose predicted probability at the segmented head is greater than the threshold are involved in the calculation of the contrast loss function.

[0084] S36: filtering out a portion of samples using a positive and negative sample filtering strategy for the projection vector of the second fused image obtained in S35 and the projection vector of the target domain image obtained in S24;

[0085] See also Figure 4 , step 3 of the present invention uses a storage mechanism to perform more effective regional feature alignment. The method sets a projection vector memory to store negative samples generated in the past few iterations, and these negative samples participate in the contrast loss calculation of the current round together with the negative samples of the round.

[0086] S37: Calculate the contrast loss using the projection vector of the second fused image obtained in S35, the projection vector of the target domain image obtained in S24, and the negative sample vector in the projection vector memory;

[0087] S38: putting the projection vector of the second fused image obtained in S35 and the projection vector of the target domain image obtained in S24 into a projection vector memory for storage;

[0088] The projection structures used in step 2 and step 3 of the present invention are respectively connected to the backbone networks of the student model and the teacher model, and are implemented by two 1*1 convolutions.

[0089] If the screening of positive and negative samples is not considered, the projection vector obtained by the student model for the second fused image is recorded as E m2 , the projection vector at position (i, j) is denoted as E m2(i，j) ,The projection vector of the target domain image obtained by the teacher model is recorded as The projection vector at position (i, j) is denoted by Since the confidence of the teacher model is higher than that of the student model, this method uses a learning mode that moves the projection vector of the student model toward the projection vector of the teacher model in one direction, that is, only the E output by the student model is calculated. m2 The contrast loss of the projection vectors belonging to the overlapping rectangular regions. m2 The projection vector E of each position on the overlapping rectangular area m2(i，j) , the definition of positive sample is the output of teacher model The projection vector at the same position on It belongs to E m2(i，j) The only positive sample of E m2 and The projection vectors of other positions on the surface are not considered in this process. m2 and The projection vectors of other positions on the image are screened for the same category as the current position, and other negative samples are introduced. The screening of the position for calculating the total contrast loss is also not considered. In order to improve the effect of regional feature alignment, this embodiment designs a corresponding sample screening strategy to screen positive and negative samples and calculate the contrast loss.

[0090] In summary, the specific method of obtaining the contrast loss of the overlapping area on the projection vector of the second fused image is:

[0091] For each position of the overlapping area of ​​the projection vector of the second fused image, respectively select positive and negative samples;

[0092] Based on the screened positive and negative samples, the contrast loss of the projection vectors at each position in the overlapping area of ​​the second fused image is calculated and summed to obtain the contrast loss of the overlapping area.

[0093] Methods for screening positive and negative samples include:

[0094] For each position in the overlapping area, its positive sample includes the projection vector of the same position of the target domain image. The position is screened whether to participate in the calculation of the overlapping area contrast loss according to the predicted probability of the segmentation prediction result of the target domain image at this position by the teacher model. If the predicted probability of the segmentation prediction result of the target domain image at this position by the teacher model is greater than the set threshold, the corresponding position participates in the calculation of the overlapping area contrast loss, and then the negative sample corresponding to the position is screened. The negative sample includes the projection vector corresponding to other positions in the target domain image and the second fused image and does not belong to the same label category as the current screening position. The negative sample also includes the negative sample generated during the historical iteration process.

[0095] The contrast loss is obtained as follows:

[0096]

[0097]

[0098] in, Represents the projection vector E at position (i, j) in the overlapping area of ​​the second fused image m2(i,j) The contrast loss is is the projection vector E m2(i,j) The corresponding positive sample projection vector, E (m,n) is the projection vector E m2(i,j)The corresponding negative sample projection vector, N represents the projection vector E m2(i,j) The corresponding negative sample set, P is the set of positions that participate in the contrast loss calculation, NUM is the total number of positions in the set P that participate in the contrast loss calculation, exp is the exponential operation, sim(u,v)=(u T v) / (||u||||v||), sim(u,v) represents the cosine similarity of the projection vectors u and v, and τ is the parameter factor.

[0099] See also Figure 1 , step 4 of the present invention calculates the gradients of the three loss functions of the student model, back-propagates the gradients and updates the student model parameters.

[0100] S41: Update the student model backbone network parameters;

[0101] S42: Update the student model segmentation head parameters;

[0102] S43: updating the projection structure parameters of the student model;

[0103] See also Figure 1 , Step 4 of the present invention adopts the exponential moving average method to update the teacher model parameters with the student model parameters.

[0104] S44: Use the exponential moving average method to update the parameters of the teacher model backbone network with the parameters of the student model backbone network;

[0105] S45: Use the exponential moving average method to update the parameters of the teacher model segmentation head with the parameters of the student model segmentation head;

[0106] S46: Using the exponential moving average method, the parameters of the student model projection structure are used to update the teacher model projection structure parameters.

[0107] This embodiment also provides an unsupervised domain adaptation semantic segmentation device based on region feature alignment, including a memory and a processor, the memory is used to store a computer program, and the processor is used to implement the above-mentioned unsupervised domain adaptation semantic segmentation method based on region feature alignment when executing the computer program.

[0108] In addition, this embodiment also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned unsupervised domain adaptation semantic segmentation method based on regional feature alignment is implemented.

[0109] The above embodiments are merely examples and do not limit the scope of the present invention. These embodiments can also be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the technical concept of the present invention.

Claims

1. An unsupervised domain adaptation semantic segmentation method based on regional feature alignment, characterized in that: The method includes: Training the Student-Teacher Model: Input the source domain image into the student model to obtain the segmentation prediction result, and calculate the cross entropy loss between the segmentation prediction result of the source domain image and the source domain label. The pixels of the source domain image and the target domain image are fused based on categories, and the first fused image is obtained by calculating the consistency loss of the segmentation prediction result of the student model and the pseudo label of the first fused image. The pixels of the source domain image and the target domain image are fused based on the region, and the regional features of the second fused image and the target domain image are aligned to obtain the contrast loss. Calculate the gradients of cross entropy loss, consistency loss, and contrast loss, backpropagate the gradients to update the parameters of the student model, and use the exponential moving average method to update the parameters of the teacher model; Use the trained teacher model to perform semantic segmentation on the target domain images; The specific method of performing regional feature alignment on the second fusion image and the target domain image to obtain the contrast loss includes: Input the target domain image into the teacher model, and the obtained segmentation prediction result is used as the pseudo label of the target domain image; Input the second fused image into the student model, and obtain a pseudo label of the second fused image based on the source domain label and the pseudo label of the target domain image; The teacher model projection structure is used to project the features extracted from the target domain image at the teacher model backbone network to obtain the projection vector of the target domain image; The student model projection structure is used to project the features extracted from the second fused image at the student model backbone network to obtain a projection vector of the second fused image; Perform feature alignment on the overlapping area between the projection vector of the target domain image and the projection vector of the second fused image, and obtain the contrast loss of the overlapping area on the projection vector of the second fused image; The specific method of obtaining the contrast loss of the overlapping area on the projection vector of the second fused image is: For each position of the overlapping area of ​​the projection vectors of the second fused image, respectively, positive and negative samples are selected; Based on the screened positive and negative samples, the contrast loss of the projection vectors at each position in the overlapping area of ​​the second fused image is calculated and summed to obtain the contrast loss of the overlapping area; Methods for screening positive and negative samples include: For each position in the overlapping area, its positive sample includes the projection vector of the same position of the target domain image, and the position is screened whether to participate in the calculation of the overlapping area contrast loss according to the predicted probability of the segmentation prediction result of the target domain image at the position by the teacher model. If the predicted probability of the segmentation prediction result of the target domain image at the position by the teacher model is greater than the set threshold, the corresponding position participates in the calculation of the overlapping area contrast loss, and then the negative sample corresponding to the position is screened, and the negative sample includes the projection vector corresponding to other positions in the target domain image and the second fused image and the current screening position does not belong to the same label category position, and the negative sample also includes the negative sample generated in the historical iteration process; The contrast loss is obtained as follows: in, Represents the projection vector E at position (i, j) in the overlapping area of ​​the second fused image m2(i,j) The contrast loss, is the projection vector E m2(i,j ) corresponds to the positive sample projection vector, E (m,n) is the projection vector E m2(i,j) The corresponding negative sample projection vector, N represents the projection vector E m2(i,j) The corresponding negative sample set, P is the set of positions that participate in the contrast loss calculation, NUM is the total number of positions in the set P that participate in the contrast loss calculation, exp is the exponential operation, sim(u,v)=(u T v) / (||u||||v||), sim(u,v) represents the cosine similarity of the projection vectors u and v, and τ is the parameter factor.

2. The unsupervised domain adaptation semantic segmentation method based on regional feature alignment according to claim 1, characterized in that: The category-based fusion method includes: randomly selecting a set number of pixel points of a category on the source domain image, and covering the selected pixel points at the same position on the target domain image to obtain a first fused image.

3. The unsupervised domain adaptation semantic segmentation method based on regional feature alignment according to claim 2, characterized in that: The pseudo label of the first fused image is obtained in the following way: Input the target domain image into the teacher model, and the obtained segmentation prediction result is used as the pseudo label of the target domain image; The source domain label and the pseudo label of the target domain image are fused based on the category according to the pixel points selected when obtaining the first fused image, so as to obtain the pseudo label of the first fused image.

4. The unsupervised domain adaptation semantic segmentation method based on regional feature alignment according to claim 1, characterized in that: The region-based fusion method includes: randomly cutting out pixels of a rectangular region from the target domain image, and overlaying the pixels of the region on the same position of the source domain image to obtain a second fused image.

5. The unsupervised domain adaptation semantic segmentation method based on regional feature alignment according to claim 1, characterized in that: The pseudo label of the second fused image is obtained in the following manner: the source domain label and the pseudo label of the target domain image are fused based on the region according to the pixel points selected when obtaining the second fused image, so as to obtain the pseudo label of the second fused image.

6. The unsupervised domain adaptation semantic segmentation method based on regional feature alignment according to claim 1, characterized in that: The teacher model projection structure and the student model projection structure respectively include two 1*1 convolutions after the backbone network and in parallel with the segmentation head.

Images