A domain adaptation segmentation method based on feature perception enhancement network

Abstract

The application provides a domain adaptation segmentation method based on a feature perception enhancement network, comprising the following steps: step S1: acquiring a data set, and improving the quality of pseudo labels through a dynamic pseudo label generation mechanism; step S2: through a rare class sampling method, enabling the model to fully learn the less sample classes; step S3: through a feature distance measurement method, improving the representation ability of features; and step S4: model training, and improving the convergence stability of the model through preheating of the model training process.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition and computer vision technology, and specifically relates to a domain adaptation segmentation method based on feature perception enhancement network. Background Technology

[0002] The development of artificial intelligence has dramatically changed the pace and lifestyle of modern life, with neural networks receiving increasing attention. Currently, most neural networks rely on training on large datasets to obtain a model for various tasks. However, in some tasks, the object to be identified remains the same, but its form changes, such as a real car and a car in a painting. Although both are cars, their forms are different. In this case, if a dataset for one type already exists and the goal is to identify the other type, a separate dataset needs to be prepared, a cumbersome process. Therefore, domain adaptation methods have emerged. Domain adaptation can predict the target domain dataset even with only the source domain dataset and a small amount of target domain data, thereby reducing the intermediate dataset preparation process and increasing the flexibility of applications. Summary of the Invention

[0003] Therefore, in response to real-world needs and gaps in existing technologies, this invention provides a domain adaptation segmentation method based on feature-aware augmentation networks, which can effectively solve the domain adaptation segmentation problem.

[0004] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0005] A domain adaptation segmentation method based on feature-aware enhancement networks includes the following steps:

[0006] Step S1: Obtain the dataset and improve the quality of pseudo-labels through a dynamic pseudo-label generation mechanism;

[0007] Step S2: By using the rare class sampling method, the model can fully learn the class with fewer samples;

[0008] Step S3: Improve the representational power of features through feature distance measurement methods;

[0009] Step S4: Perform model training and warm up the model training process to improve the convergence stability of the model.

[0010] Furthermore, step S1 specifically includes the following steps:

[0011] Step S11: Obtain the publicly available domain-adaptive image dataset and obtain the relevant annotations for the training data;

[0012] Step S12: Calculate the frequency f of each L class in the source dataset. L It is calculated based on the number of L-type pixels, and the calculation method is as follows:

[0013]

[0014] Where i, j, and c represent the row, column, and channel positions of the pixel, respectively; H and W represent the height and width of the image, respectively; and N represents the width and height of the image. S y represents the total number of data in the source domain. S The label represents the source domain data; the sampling probability P(L) of class L is calculated as follows:

[0015]

[0016] Where e represents the exponent, Tem is a manually set hyperparameter, and L... , Which specific category?

[0017] Furthermore, step S2 specifically includes the following steps:

[0018] Step S21: Input the source domain dataset and target domain dataset from the dataset into the student network and teacher network composed of the Acmix50 backbone network, respectively; use the teacher network Generate pseudo-labels for target domain data

[0019]

[0020] Where [·] represents Iverson brackets, T represents the target domain data, and arg max represents taking the maximum value. Represent the teacher network; Step S22: Calculate the pseudo-label generation quality estimate

[0021]

[0022] Where r is the threshold; pseudo-labels and their quality estimates are used to additionally train the network g in the target domain. θ Calculate the loss function Optimize:

[0023]

[0024] Where C represents the number of channels in the image, and θ represents the student network;

[0025] Step S23: During the generation of pseudo-tags, According to g during training θ Update; weight Set g after each training step t θ The exponential moving average of the weights is used to increase the stability of the prediction; teacher network. The method for generating pseudo-tags is as follows:

[0026]

[0027] Where t represents the number of iterations and α is the mixing ratio.

[0028] Furthermore, step S3 specifically includes the following steps:

[0029] Step S31: Based on the semantic segmentation model g θ Bottleneck characteristics F θ Bottleneck features F of the ImageNet model IN The model is regularized using the feature distance, and the feature distance d is calculated. (i,j) :

[0030]

[0031] Where x S This represents a sample of source domain data.

[0032] Step S32: Calculate the class C containing the entity described by the binary mask. things Feature distance loss of image region The calculation method is shown in the following formula:

[0033]

[0034] mask For the C class of things things The mask, from the downsampled source domain data label y Ss get:

[0035]

[0036] Where C represents the number of channels, and c' refers to a specific channel;

[0037] Step S33: Downsample the label to the bottleneck feature size, and set the patch size to H. F ×W F Average pooling is applied to each class channel, and when the ratio r is exceeded, one class is retained, representing the source domain sample label. The calculation method is shown in the following formula:

[0038]

[0039] Where AvgPool represents the average pooling operation.

[0040] Furthermore, in step S4:

[0041] L is obtained through steps S22 and S32. T and L FD Then through L Tand L FD Calculate and minimize the loss; the formula for calculating the total loss is:

[0042] Loss = L T +L FD

[0043] Training is performed by minimizing the total loss (Loss).

[0044] Furthermore, during the training process, a learning rate warm-up method is used in the first 20 iterations, by setting the learning rate to 0.00001 and the momentum to 0.95 to warm up the training process; after 20 iterations until the training is completed, the training process is trained with the learning rate to 0.0001 and the momentum to 0.99.

[0045] And an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a domain adaptation segmentation method based on a feature-aware augmentation network as described above.

[0046] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a domain adaptation segmentation method based on a feature-aware augmentation network as described above.

[0047] Compared with the prior art, the present invention and its preferred embodiments have at least the following outstanding effects:

[0048] 1. It can improve the quality of pseudo-labels in target domain data by dynamically adjusting the generation method of pseudo-labels during training and improving the credibility of labels by generating confidence estimates for pseudo-labels.

[0049] 2. It enables the model to learn the sample categories that appear less frequently more stably. By frequently sampling images of rare categories, the network can learn more stably.

[0050] 3. It can improve the representational ability of features, extract knowledge from diverse and expressive ImageNet features, and enable the model to extract richer feature representations.

[0051] 4. It can conduct a stable training process, and improve the stability of model training by warming up the training in the early stage. Attached Figure Description

[0052] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0053] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0054] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below for detailed explanation:

[0055] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0056] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0057] like Figure 1 As shown, this embodiment provides a domain adaptation segmentation method based on a feature-aware enhancement network, including the following steps:

[0058] Step S1: Obtain the dataset and improve the quality of pseudo-labels through a dynamic pseudo-label generation mechanism.

[0059] In this embodiment, step S1 specifically includes the following steps:

[0060] Step S11: Obtain the publicly available domain-adaptive image dataset and obtain the relevant annotations for the training data;

[0061] Step S12: Calculate the frequency f of each L class in the source dataset. L It is calculated based on the number of L-type pixels, and the calculation method is as follows:

[0062]

[0063] Where i, j, and c represent the row, column, and channel positions of the pixel, respectively; H and W represent the height and width of the image, respectively; and N represents the width and height of the image. S y represents the total number of data in the source domain. S The label represents the source domain data; the sampling probability P(L) of class L is calculated as follows:

[0064]

[0065] Where e represents the exponent, Tem is a manually set hyperparameter, and L... , Which specific category?

[0066] Step S2: By using the rare class sampling method, the model can learn fully about the classes with fewer samples.

[0067] In this embodiment, step S2 specifically includes the following steps:

[0068] Step S21: Input the source domain dataset and target domain dataset from the dataset into the student network and teacher network composed of the Acmix50 backbone network, respectively; use the teacher network Generate pseudo-labels for target domain data

[0069]

[0070] Where [·] represents Iverson brackets, T represents the target domain data, and arg max represents taking the maximum value. Represent the teacher network; Step S22: Calculate the pseudo-label generation quality estimate

[0071]

[0072] Where r is the threshold; pseudo-labels and their quality estimates are used to additionally train the network g in the target domain. θ Calculate the loss function Optimize:

[0073]

[0074] Where C represents the number of channels in the image, and θ represents the student network;

[0075] Step S23: During the generation of pseudo-tags, According to g during training θ Update; weight Set g after each training step t θ The exponential moving average of the weights is used to increase the stability of the prediction; teacher network. The method for generating pseudo-tags is as follows:

[0076]

[0077] Where t represents the number of iterations and α is the mixing ratio.

[0078] Step S3: Improve the representational ability of features through feature distance measurement methods.

[0079] In this embodiment, step S3 specifically includes the following steps:

[0080] Step S31: Based on the semantic segmentation model g θ Bottleneck characteristics F θBottleneck features F of the ImageNet model IN The model is regularized using the feature distance, and the feature distance d is calculated. (i,j) :

[0081]

[0082] Where x S This represents a sample of source domain data.

[0083] Step S32: Calculate the class C containing the entity described by the binary mask. things Feature distance loss of image region The calculation method is shown in the following formula:

[0084]

[0085] mask For the C class of things things The mask is derived from the downsampled source domain data label y. Ss Obtained:

[0086]

[0087] Where C represents the number of channels, and c' represents which specific channel;

[0088] Step S33: Downsample the label to the bottleneck feature size, and set the patch size to H. F ×W F Average pooling is applied to each class channel, and a class is retained when the ratio r is exceeded, label The calculation method is shown in the following formula:

[0089]

[0090] Where AvgPool represents the average pooling operation.

[0091] Step S4: Perform model training and improve the convergence stability of the model by preheating the model training process.

[0092] In this embodiment, step S4 specifically includes the following steps:

[0093] L is obtained through steps S22 and S32. T and L FD Then through L T and L FD Calculate and minimize the loss; the formula for calculating the total loss is:

[0094] Loss = L T +L FD

[0095] Training is performed by minimizing the total loss (Loss).

[0096] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0102] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of domain adaptation segmentation methods based on feature-aware enhancement networks under the guidance of this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A domain-adaptive segmentation method based on feature-aware enhancement networks, characterized in that, Includes the following steps: Step S1: Obtain the dataset and improve the quality of pseudo-labels through a dynamic pseudo-label generation mechanism; Step S2: By using the rare class sampling method, the model can fully learn the class with fewer samples; Step S3: Improve the representational power of features through feature distance measurement methods; Step S3 specifically includes the following steps: Step S31: Based on the semantic segmentation model g θ Bottleneck characteristics F θ Bottleneck features F of the ImageNet model IN The model is regularized using the feature distance, and the feature distance d is calculated. (i,j) : Where x S This represents a sample of source domain data. Step S32: Calculate the class C containing the entity described by the binary mask. things Feature distance loss of image region The calculation method is shown in the following formula: mask For the C class of things things The mask, from the downsampled source domain data label y Ss get: Where C represents the number of channels, and c' refers to a specific channel; Step S33: Downsample the label to the bottleneck feature size, and set the patch size to H. F ×W F Average pooling is applied to each class channel, and when the ratio r is exceeded, one class is retained, representing the source domain sample label. The calculation method is shown in the following formula: Where AvgPool represents the average pooling operation; Step S4: Perform model training and warm up the model training process to improve the convergence stability of the model.

2. The domain adaptation segmentation method based on feature-aware enhancement networks according to claim 1, characterized in that: Step S1 specifically includes the following steps: Step S11: Obtain the publicly available domain-adaptive image dataset and obtain the relevant annotations for the training data; Step S12: Calculate the frequency f of each L class in the source dataset. L It is calculated based on the number of L-type pixels, and the calculation method is as follows: Where i, j, and c represent the row, column, and channel positions of the pixel, respectively; H and W represent the height and width of the image, respectively; and N represents the width and height of the image. S y represents the total number of data in the source domain. S The label represents the source domain data; the sampling probability P(L) of class L is calculated as follows: Where e represents the exponent, Tem is a set hyperparameter, and the subscript L refers to a specific category.

3. The domain adaptation segmentation method based on feature-aware enhancement networks according to claim 2, characterized in that: Step S2 specifically includes the following steps: Step S21: Input the source domain dataset and target domain dataset from the dataset into the student network and teacher network composed of the Acmix50 backbone network, respectively; use the teacher network Generate pseudo-labels for target domain data Where [·] represents Iverson brackets, T represents the target domain data, and arg max represents taking the maximum value. Indicates a teacher network; Step S22: Calculate the pseudo-label generation quality estimate Where r is the threshold; pseudo-labels and their quality estimates are used to additionally train the network g in the target domain. θ Calculate the loss function Optimize: Where C represents the number of channels in the image, and θ represents the student network; Step S23: During the generation of pseudo-tags, According to g during training θ Update; weight Set g after each training step t θ The exponential moving average of the weights is used to increase the stability of the prediction; teacher network. The method for generating pseudo-tags is as follows: Where t represents the number of iterations and α is the mixing ratio.

4. The domain adaptation segmentation method based on feature-aware enhancement networks according to claim 1, characterized in that: In step S4: L is obtained through steps S22 and S32. T and L FD Then through L T and L FD Calculate and minimize the loss; the formula for calculating the total loss is: Loss=L T +L FD Training is performed by minimizing the total loss (Loss).

5. The domain adaptation segmentation method based on feature-aware enhancement network according to claim 4, characterized in that: During the training process, a learning rate warm-up method was used in the first 20 iterations, with the training parameters set to a learning rate of 0.00001 and a momentum of 0.95 to warm up the training process. After 20 iterations until the training was completed, the training process was trained with a learning rate of 0.0001 and a momentum of 0.

99.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of a domain adaptation segmentation method based on a feature-aware enhancement network as described in any one of claims 1-5.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of a domain adaptation segmentation method based on a feature-aware augmentation network as described in any one of claims 1-5.

Images