A domain generalization image classification method based on semi-supervised learning

Abstract

The present invention discloses a domain generalization image classification method based on semi-supervised learning, which obtains an image data set and pre-processes the image; constructs a network framework including a dual-branch domain adversarial network and an integrated network; the dual-branch domain adversarial network includes two domain adversarial networks with the same structure, each domain adversarial network is composed of a feature extractor, a classifier and a domain discriminator; the integrated network is composed of a feature extractor and a classifier that are the same as the domain adversarial network; the two domain adversarial networks in the step are trained using the pre-processed image data set, and the pseudo labels of the unlabeled data are updated and saved; the target domain image is tested using the trained model to obtain the classification result. The structural framework of the dual-branch domain adversarial network constructed by the present invention simultaneously achieves the goals of semi-supervised learning and domain generalization, improves the test accuracy of the model in the target domain, and has a high use value.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning and artificial intelligence, and in particular relates to a domain generalization image classification method based on semi-supervised learning. Background Art

[0002] Machine learning learns parameters by providing empirical data to the machine learning model, and the trained model can be applied to provide prediction results for new test data. Therefore, in machine learning, if there is a distribution inconsistency problem (domain shift) between the test set and the training set, the accuracy of the model will decrease during the testing phase. In order to alleviate this phenomenon, the domain generalization technology improves the versatility and generalization of the model by jointly training multiple source domains during the training phase, so that the model can be generalized to unknown target domains. Domain generalization requires multiple labeled source domains, but in actual applications, obtaining labels is often the most difficult. Therefore, in actual domain generalization scenarios, if the source domain consists of one or more labeled domains and multiple completely unlabeled domains, it is necessary to invent a domain generalization algorithm based on semi-supervised learning.

[0003] Domain generalization has received extensive attention in recent years. Existing domain generalization technologies mainly address the domain shift problem from the following perspectives: (1) enhancing the diversity of training data based on data augmentation or image generation technology; (2) extracting domain-independent feature representations based on methods such as domain adversarial training; and (3) improving model generalization based on learning strategies such as ensemble learning and meta-learning.

[0004] In traditional semi-supervised learning, only a small amount of data is labeled, and its main purpose is to use a large amount of unlabeled data to improve the performance of the model. Pseudo-labeling, entropy minimization, and consistency regularization are three effective methods. However, in the problem of domain generalization, how to perform semi-supervised learning between source domains with inconsistent distributions is a difficult problem. In addition, because there is a distribution difference between the target domain and the source domain, how to improve the generalization of the model based on semi-supervised learning is the overall goal of semi-supervised domain generalization.

[0005] In view of the shortcomings of traditional semi-supervised methods in solving domain generalization problems, the present invention proposes a semi-supervised learning method under domain differences and a domain generalization image classification framework based on semi-supervised learning, so that the domain generalization technology can be applied to more practical scenarios with insufficient labels. Summary of the invention

[0006] Purpose of the invention: In order to solve the problem of domain generalization image classification in which the source domain is not fully labeled, the present invention proposes a domain generalization image classification method based on semi-supervised learning, which effectively utilizes unlabeled domain samples, while improving the generalization of the model and the test accuracy of the model in the target domain.

[0007] Technical solution: The domain generalization image classification method based on semi-supervised learning described in the present invention specifically includes the following steps:

[0008] (1) Obtain image data sets and preprocess images;

[0009] (2) constructing a network framework including a dual-branch domain adversarial network and an integrated network; the dual-branch domain adversarial network includes two domain adversarial networks with the same structure, each domain adversarial network is composed of a feature extractor, a classifier and a domain discriminator; the integrated network is composed of a feature extractor and a classifier that are the same as the domain adversarial network;

[0010] (3) Use the preprocessed image dataset in step (1) to train the two domain adversarial networks in step (2), and update and save the pseudo labels of the unlabeled data;

[0011] (4) Use the trained model to test the target domain image and obtain the classification results.

[0012] Furthermore, the implementation process of step (1) is as follows:

[0013] Obtain an image dataset containing multiple source domains; in the semi-supervised domain generalization task, given a set of labeled source domains S l ={D1, ..., D m} and the set S of unlabeled source domains u ={D1, ..., D n},S l The jth field in can be expressed as Each sample consists of input x, category label y and domain label z, where n j is the number of samples in the field; S u The jth field in can be expressed as The sample does not contain the category label; the target domain where x i is the i-th input image, n t is the number of samples in the target domain;

[0014] The pre-acquired dataset images are cropped into images of uniform size, and the R, G, and B channels of the images are standardized separately, where the mean and variance use the mean and variance of ImageNet.

[0015] Furthermore, the implementation process of step (2) is as follows:

[0016] The preprocessed image is input into the feature extractor, and the feature extraction network outputs a high-dimensional feature vector, which is input into the classifier and the domain discriminator respectively, and finally outputs the prediction of the sample category and the prediction of the domain; the two domain adversarial networks are represented as P θ1 (y,z|x) and P θ2 (y,z|x), where θ1 and θ2 are network parameters, y is the category value, and z is the domain value; in addition, the network used to predict pseudo labels is composed of P θ1 (y,z|x network integration, that is, the network does not participate in training, through P θ1 The parameters of the (y,z|x) network update their own parameters, and the integrated network is represented by P θ1’ (y,z|x), where θ1' is the network parameter.

[0017] Furthermore, the step (3) comprises the following steps:

[0018] (31) Obtaining an image dataset containing multiple source domains;

[0019] (32) Difference is performed between the labeled source domain and the unlabeled source domain to expand the mixed samples between the domains as follows:

[0020]

[0021]

[0022]

[0023] in, is the pseudo label of the unlabeled sample, λ follows the Beta distribution; new training samples are obtained after interpolation between domains

[0024] (33) Using mixed samples to train P θ2 (y,z|x), using the original labeled samples and the unlabeled samples with pseudo labels to train P θ1 (y,z|x), assuming that the feature extractor, classifier and domain discriminator in the domain adversarial network are represented as F g 、F c and F d , P θ2 (y,z|x) and P θ1 The classification losses of (y,z|x) are:

[0025]

[0026]

[0027] Among them, N p is the number of unlabeled samples with pseudo labels, N l is the number of labeled samples, represents the cross entropy loss function; the losses of domain adversarial are:

[0028]

[0029]

[0030] Among them, N represents the total number of samples; entropy loss is used to train P for unlabeled samples. θ1 (y,z|x), the loss function is as follows:

[0031]

[0032] Among them, H represents the entropy function, N u is the number of unlabeled samples that have not yet been labeled with pseudo-labels;

[0033] In addition, the mixed samples are constrained to have consistent features to train P θ2 (y,z|x), the loss function is as follows:

[0034]

[0035] in, represents the mean square error function, N l is the number of labeled samples, N p is the number of unlabeled samples with pseudo labels, and λ represents the value of the hyperparameter in the sample mixing interpolation operation;

[0036] (34) After each forward propagation and backward propagation of all samples, a pseudo-label update and prediction is performed. The network used to predict the pseudo-label is composed of P θ1 The network obtained by integrating (y, z|x) through exponential moving average, the algorithm formula of the exponential moving average process is as follows:

[0037] θ′ t+ 1=βθ1′ t +(1-β)θ1 t+1

[0038] Among them, β is a hyperparameter and θ1′ is P θ1 '(y, z|x) network parameters, θ1 is P θ1 Network parameters of (y, z|x);

[0039] (35) When predicting pseudo labels, the similarity between the sample features and the category features of the field to which the sample belongs is superimposed on the network's prediction output for the unlabeled sample. This is the final prediction for the unlabeled sample. Samples exceeding the threshold are selected and assigned pseudo labels. The final prediction formula is as follows:

[0040] s(x u )=γF c (F g (x u ))+(1-γ)sim(x u , M).

[0041] Among them, sim() is the function for calculating cosine similarity, γ is a hyperparameter, and M is x u The category feature representation matrix of the field belongs to the category. The category feature representation matrix is ​​composed of the feature representation of the samples with the highest prediction confidence rate for each category in the field. The M matrix is ​​also iteratively updated during the update of the pseudo-label set.

[0042] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention improves the pseudo-labeling method in semi-supervised learning, proposes a domain-based pseudo-labeling method, and improves the pseudo-label prediction accuracy between domains with inconsistent distributions; 2. The present invention proposes an expansion method for interpolation mixing between domains, and constrains the category prediction, domain prediction and feature prediction result values ​​of mixed samples to improve the generalization of the model; 3. The present invention uses a dual-branch domain adversarial network structure, so that label prediction and domain generalization do not interfere with each other, while improving the pseudo-label accuracy and the test performance of the model in the target domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the present invention;

[0044] Figure 2 A schematic diagram of the network framework constructed for the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0046] The present invention proposes a domain generalization image classification method based on semi-supervised learning, such as Figure 1 As shown, the specific steps include:

[0047] Step 1: Obtain an image dataset and preprocess the dataset images.

[0048] Get an image dataset, including a set S of labeled source domains l ={D1, ..., D m} and the set S of unlabeled source domains u={D1, ..., D n},S l The jth field in can be expressed as Each sample consists of input x, semantic label y and domain label z, where n j is the number of samples in the domain. The semantic label y is a value between 0 and the number of dataset categories, and z is a value between 0 and the number of source domains. During network training, y and z need to be converted into vectors with one-bit efficient encoding. For example, for a sample in the 0th domain, assuming there are 3 source domains in total, the domain value z of the sample needs to be represented as a vector such as [1,0,0]. S u The jth field in can be expressed as The sample does not contain semantic labels, where n j is the number of samples in the domain. where x i is the i-th image, n t is the number of samples in the target domain.

[0049] In image classification tasks, standardizing images during preprocessing helps accelerate the convergence of deep networks and improve accuracy. For a given image sample x, the three channels of R, G, and B are standardized, and the mean and variance use the mean and variance of ImageNet. The means of the three channels are 0.485, 0.456, and 0.406, and the standard deviations are 0.229, 0.224, and 0.225, respectively. The standardization calculation formula is x = (x-mean) / std.

[0050] Step 2: Construct a network framework including a dual-branch domain adversarial network and an integrated network; the dual-branch domain adversarial network contains two domain adversarial networks with the same structure, each domain adversarial network consists of a feature extractor, a classifier and a domain discriminator; the integrated network consists of a feature extractor and a classifier that are the same as the domain adversarial network, such as Figure 2 shown.

[0051] Specifically, each domain adversarial network contains a feature extractor mainly composed of convolutional layers, a classifier composed of fully connected layers, and a domain discriminator composed of fully connected layers. During training, the image is input to the feature extractor, and the feature extractor outputs a high-dimensional feature vector, which is input to the classifier and the domain discriminator respectively, and finally outputs the prediction of the sample category and the prediction of the domain.

[0052] Select a convolutional neural network suitable for image classification problems as a feature extractor, such as a deep residual network. Add a domain discriminator composed of a fully connected layer at the output of the feature. The domain discriminator and the feature extractor are trained adversarially. The adversarial effect is to enable the feature layer to extract features that are not related to the domain. The preprocessed image is input to the feature extractor. The feature extraction network outputs a high-dimensional feature vector. The feature vector is input to the classifier and the domain discriminator respectively, and finally outputs the prediction of the sample category and the prediction of the domain; the two domain adversarial networks are represented as P θ1 (y,z|x) and P θ2 (y,z|x), where θ1 and θ2 are network parameters, y is the category value, and z is the domain value; in addition, the network used to predict pseudo labels is composed of P θ1 (y,z|x) network integration, that is, the network does not participate in training, through P θ1 The parameters of the (y,z|x) network update their own parameters, and the integrated network is represented by P θ1’ (y,z|x), where θ1' is the network parameter.

[0053] Step 3: Use the preprocessed image dataset in step (1) to train the two domain adversarial networks in step (2), and update and save the pseudo labels of the unlabeled data.

[0054] Taking the original sample as an example, the loss of adversarial training is as follows:

[0055]

[0056] in, represents a labeled sample, Represents the domain value to which it belongs. represents an unlabeled sample, Represents the domain value to which it belongs, N l Represents the total number of labeled samples, N u Represents the total number of unlabeled samples, the total number of all samples in N generations.

[0057] For the domain discriminator F d Use L adv Training, for feature extractor F g Using -w(t)L adv The weight w(t) increases from 0 to 1 over time, and the calculation formula is as follows:

[0058] w(t)=2 / (1+e -10*(t / n) )-1

[0059] Among them, n is the total number of training rounds, and t represents the round at the current moment.

[0060] For the set of labeled source domains and S l ={D1, ..., D m} and the pseudo-labeled sample set S p To interpolate between domains, S p It is a set of unlabeled samples and assigned pseudo labels constructed during the training process. After each training round, samples with higher confidence rates are selected from the unlabeled samples to form a set S u ′, the selection method will be described in detail in the following steps. p and S u Update according to the following formula:

[0061] S p =S p ∪S u′

[0062] S u =S u -S u′

[0063] Multiple random selections of S l and S p The two samples in the interpolation domain are interpolated to obtain the extended samples between domains. The formula is as follows:

[0064]

[0065]

[0066]

[0067] in, is the pseudo label of the unlabeled sample. Where λ follows the Beta(α, α) distribution, and α is a hyperparameter. New training samples are obtained after interpolation between domains

[0068] Use mixed samples and original samples to train the network P respectively θ2 (y, z|x) and P θ1 (y, z|x), and use the exponential moving average to get P θ1’ (y, z|x). θ2 The three loss functions of (y, z|x) are as follows:

[0069]

[0070]

[0071]

[0072] in, represents the cross entropy loss function, represents the mean square error function, Nl is the number of labeled samples, N p is the number of unlabeled samples with pseudo labels, and λ represents the value of the hyperparameter in the sample mixture. θ2 The overall objective function of the (y, z|x) network is as follows:

[0073]

[0074]

[0075] where w(t) is a coefficient that changes with the number of training rounds.

[0076] P θ1 The three loss functions of the (y, z|x) network are as follows:

[0077]

[0078]

[0079]

[0080] in, represents the cross entropy loss function, H represents the entropy loss function, Nl is the number of labeled samples, N p is the number of unlabeled samples with pseudo labels, N u Represents the total number of unlabeled samples. The total number of all samples in N generations. θ1 The overall objective function of the (y, z|x) network is as follows:

[0081]

[0082]

[0083] Among them, w(t) is the coefficient that changes with the number of training rounds.

[0084] Using P θ1’ (y, z|x) predicts unlabeled samples, where P θ1’ (y, z|x) is given by P θ1 The network obtained by integrating (y, z|x) through exponential moving average, the algorithm formula of the exponential moving average process is as follows:

[0085] θ1′ t+1 =βθ1′ t +(1-β)θ1 t+1

[0086] Among them, β is a hyperparameter, θ1′ is the parameter of the integrated network;

[0087] Select unlabeled samples whose prediction confidence is greater than the threshold to form S u′ , assign pseudo labels to these samples and update S p and S u First, obtain P θ1’ The prediction result of (y, z|x) network is:

[0088] q(x u )=F c (F g (x u ))

[0089] Then, based on the domain to which the unlabeled sample belongs, the similarity between the sample features and the category representation matrix of the domain to which it belongs is calculated. The purpose of calculating the similarity is to make domain-based corrections to the network's predictions to improve the accuracy of the pseudo-labels.

[0090] ψ(x u ) = sim(x u , M)

[0091] Where M is the unlabeled sample x u The category feature representation of the field, F g YesP θ1’ (y, z|x) feature extractor. sim() calculates the cosine similarity between a feature vector and each vector in a matrix and returns a vector. The final prediction is q(x u ) and ψ(x u ) is the weighted sum of:

[0092] S(x u )=γq(x u )+(1-γ)ψ(x u )

[0093] Among them, γ is a hyperparameter. Finally, according to S(x u ) decides whether to assign pseudo labels to unlabeled samples:

[0094]

[0095] Among them, δ is the threshold for selecting pseudo labels, S i′ (x u ) is the value of the i'th position of the prediction result vector, representing the probability that the sample belongs to the i'th category. After predicting all samples, for each category in each field, the features of the sample with the highest confidence rate are selected to update the category feature representation of the field.

[0096] Step 4: Use the trained model to test the target domain image and obtain the classification results.

[0097] The target domain image is used as the input image, and the P obtained by step 2 training θ2 (y, z|x) model, testing the classification results of the target domain images.

Claims

1. A domain generalization image classification method based on semi-supervised learning, characterized in that: The following steps are involved: (1) Obtain image data sets and preprocess images; (2) constructing a network framework including a dual-branch domain adversarial network and an integrated network; the dual-branch domain adversarial network includes two domain adversarial networks with the same structure, each domain adversarial network is composed of a feature extractor, a classifier and a domain discriminator; the integrated network is composed of a feature extractor and a classifier that are the same as the domain adversarial network; (3) Use the preprocessed image dataset in step (1) to train the two domain adversarial networks in step (2), and update and save the pseudo labels of the unlabeled data; (4) Use the trained model to test the target domain image and obtain the classification result; The implementation process of step (1) is as follows: Obtain an image dataset containing multiple source domains; in the semi-supervised domain generalization task, a set of labeled source domains is given and the set of unlabeled source domains The jth field in can be expressed as Each sample consists of input x, category label y and domain label z, where n j is the number of samples in the domain; The jth field in can be expressed as The sample does not contain the category label; the target domain where x i is the i-th input image, n t is the number of samples in the target domain; The pre-acquired dataset images are cropped into images of uniform size, and the R, G, and B channels of the images are standardized respectively, where the mean and variance use the mean and variance of ImageNet; The implementation process of step (2) is as follows: The preprocessed image is input into the feature extractor, and the feature extraction network outputs a high-dimensional feature vector, which is input into the classifier and the domain discriminator respectively, and finally outputs the prediction of the sample category and the prediction of the domain; the two domain adversarial networks are represented as P θ1 (y,z|x) and P θ2 (y,z|x), where θ1 and θ2 are network parameters, y is the category value, and z is the domain value; in addition, the network used to predict pseudo labels is composed of P θ1 (y,z|x) network integration, that is, the network does not participate in training, through P θ1 The parameters of the (y,z|x) network update their own parameters, and the integrated network is represented by P θ1’ (y,z|x), where θ1' is the network parameter; The step (3) comprises the following steps: (31) Obtaining an image dataset containing multiple source domains; (32) Difference is performed between the labeled source domain and the unlabeled source domain to expand the mixed samples between the domains as follows: in, is the pseudo label of the unlabeled sample, λ follows the Beta distribution; new training samples are obtained after interpolation between domains (33) Using mixed samples to train P θ2 (y,z|x), using the original labeled samples and the unlabeled samples with pseudo labels to train P θ1 (y,z|x), assuming that the feature extractor, classifier and domain discriminator in the domain adversarial network are represented as F g 、F c and F d , P θ2 (y,z|x) and P θ1 The classification losses of (y,z|x) are: Among them, N p is the number of unlabeled samples with pseudo labels, N l is the number of labeled samples, l ce represents the cross entropy loss function; the losses of domain adversarial are: Among them, N represents the total number of samples; entropy loss is used to train P for unlabeled samples. θ1 (y,z|x), the loss function is as follows: Among them, H represents the entropy function, N u is the number of unlabeled samples that have not yet been labeled with pseudo-labels; In addition, the mixed samples are constrained to have consistent features to train P θ2 (y,z|x), the loss function is as follows: Among them, l mse represents the mean square error function, N l is the number of labeled samples, N p is the number of unlabeled samples with pseudo labels, and λ represents the value of the hyperparameter in the sample mixing interpolation operation; (34) After each forward propagation and backward propagation of all samples, a pseudo-label update and prediction is performed. The network used to predict the pseudo-label is composed of P θ1 The network obtained by integrating (y,z|x) through exponential moving average, the algorithm formula of the exponential moving average process is as follows: θ′ t+1 =βθ1 t +(1-β)θ1 t+1 Among them, β is a hyperparameter and θ1′ is P θ1′ (y, z|x), θ1 is the network parameter of P θ1 Network parameters of (y, z|x); (35) When predicting pseudo labels, the similarity between the sample features and the category features of the field to which the sample belongs is superimposed on the network's prediction output for the unlabeled sample. This is the final prediction for the unlabeled sample. Samples exceeding the threshold are selected and assigned pseudo labels. The final prediction formula is as follows: s(x u )=γF c (F g (x u ))+(1-γ)sim(x u ,M). Among them, sim() is the function for calculating cosine similarity, γ is a hyperparameter, and M is x u The category feature representation matrix of the field belongs to the category. The category feature representation matrix is ​​composed of the feature representation of the samples with the highest prediction confidence rate for each category in the field. The M matrix is ​​also iteratively updated during the update of the pseudo-label set.

Images