Unsupervised adversarial domain adaptation image classification method based on pseudo-label loss of Gaussian uniform mixture model

Abstract

The present invention discloses an unsupervised adversarial domain adaptation image classification method based on pseudo-label loss of Gaussian uniform mixture model, which utilizes a large amount of available annotation data of a related source domain to transfer knowledge across domains to a related target domain to obtain labeled target data through transfer learning or domain adaptation methods; the proposed domain adaptation method integrates Gaussian uniform mixture model to detect outliers and deep neural network to perform image classification, and uses Gaussian uniform mixture model to model the cosine distance from the target sample feature of each class to the class mean, and obtains the posterior probability of the target sample as the importance of estimating the pseudo-label of the target sample; the auxiliary pseudo-label loss proposed based on the pseudo-label of the target sample generated in the training process is added to the training of the neural network; at the same time, the conditional entropy loss is minimized so that the learned features are far away from the decision boundary; a large number of experiments are comprehensively conducted to prove that the proposed method can improve the image classification accuracy of the deep network model.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning image classification, and mainly to an unsupervised adversarial domain adaptation image classification method based on a Gaussian uniform mixture model with pseudo-label loss. Background Art

[0002] The current state-of-the-art has achieved impressive results on large-scale datasets for image classification or action recognition. However, obtaining such large annotated datasets is very expensive and requires transferring knowledge from existing annotated datasets to specific unlabeled data. If the labeled and unlabeled data have different characteristics, then they are sampled from two different domains. In particular, datasets collected from the Internet, such as those collected from platforms for sharing videos or images, are very different from the data that applications need to process. Due to the common domain shift in computer vision, the data distribution between the source domain and the target domain may be very different. Therefore, a deep neural network trained with enough labeled images may not adapt well to the target domain. In order to solve the domain shift problem between the labeled dataset (source domain) and the unlabeled data in the target domain, various unsupervised domain adaptation methods have been proposed. If the data from the target source is partially labeled, this problem is called semi-supervised domain adaptation. In this work, image classification in the context of unsupervised domain adaptation is addressed.

[0003] Adversarial adaptation is an important component of domain adaptation methods that solve the visual domain adaptation problem. It aims to reduce the difference in cross-domain distribution through two adversarial objectives of a domain discriminator. The idea of ​​these methods mainly comes from generative adversarial networks, which consist of two neural networks: a generator and a discriminator. The purpose of the former is to generate new images that confuse the latter, trying to distinguish them from real images. Many existing visual domain adaptation methods have exploited this idea to ensure that the network cannot distinguish between images from the source and target domains.

[0004] Domain adaptation mainly attempts to reduce the domain shift between the source domain and the target domain. Previous shallow domain adaptation methods either learn the invariant feature representation of the source domain data, or learn the prediction model of the target domain through supervised learning of the source domain data, ignoring the pseudo-labels of the target domain data, which have been shown to improve domain adaptation learning. Since pseudo-labels inevitably contain noise, it is crucial to correctly select labeled data when using pseudo-labels to guide domain adaptation tasks.

[0005] Although current domain adaptation methods have shown good performance in practical applications, they cannot effectively use pseudo-labels to improve image classification accuracy and still face huge challenges. Summary of the invention

[0006] Purpose of the invention: In view of the problems existing in the above-mentioned background technology, the present invention proposes an unsupervised adversarial domain adaptation image classification method based on a pseudo-label loss of a Gaussian uniform mixture model, which utilizes the pseudo-labels generated during training by training the learning network model in steps. The cosine distance from the target sample feature of each class to the class mean is modeled using a Gaussian uniform mixture model, and the posterior probability of the target sample is obtained as an estimate of the importance of the pseudo-label of the target sample. The posterior probability of the target sample is used to screen out the sample pseudo-labels with high confidence and add them to the learning of the deep neural network. The auxiliary pseudo-label loss proposed based on the target sample pseudo-labels generated during the training process is added to the training of the neural network. Minimizing the conditional entropy loss keeps the learned features away from the decision boundary.

[0007] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0008] A pseudo-label loss unsupervised adversarial domain adaptation image classification method based on Gaussian uniform mixture model, comprising the following steps:

[0009] Step S1, respectively obtaining images of a source domain and a target domain, wherein the source domain is a known domain and the images are labeled, and the target domain is a domain to be classified and the images are unlabeled; preprocessing and enhancing the obtained images based on multi-operation splicing;

[0010] Step S2, building a pre-trained deep neural network; learning the pre-trained neural network based on the source domain supervised classification loss and adversarial loss in step S1 to obtain a pre-trained feature extractor and classifier;

[0011] Step S3, establishing a Gaussian uniform mixture model; based on the pre-trained feature extractor and classifier obtained in step S2, first input the sample image of the target domain into the pre-trained feature extractor for feature extraction; input the extracted features into the classifier to obtain the target sample pseudo-label; the features extracted by the feature extractor are the sample features of the target domain; calculate the cosine distance of each type of target sample to the class feature mean of the target domain; model the cosine distance calculated for each type of target sample, and update the Gaussian uniform mixture model parameters by the expectation maximization method to obtain the posterior probability of the target sample as an indicator to measure the confidence of the pseudo-label;

[0012] Step S4, build an adversarial domain adaptation network and train it; build a feature extractor, classifier, and discriminator network, where the feature extractor is a feature extractor network pre-trained on a conventional ImageNet dataset; input source and target domain images, and obtain the outputs of the source and target domain image classifiers and discriminators respectively through the forward propagation of the adversarial domain adaptation network; for the labeled source domain, use the standard supervised cross-entropy loss to train the feature extractor and classifier; propose domain adversarial loss based on the domain labels of the source and target domains, and learn the feature extractor and discriminator; based on the posterior probability of the target sample obtained in step S3, select the pseudo-label with a high posterior probability of the target sample in step S3, combine the target sample prediction of the adversarial domain adaptation network classifier to calculate the mean absolute error as the pseudo-label loss, and learn the adversarial domain adaptation network through the pseudo-label loss; use the minimized conditional entropy loss to learn features away from the decision boundary; use the finally learned feature extractor and classifier as the pre-trained feature extractor and classifier of step S3;

[0013] Step S5, repeating steps S3-S4, respectively training the adversarial domain adaptation network and the Gaussian uniform mixture model through iteration;

[0014] When the Gaussian uniform mixture model is first trained, the initialized neural network is used as the feature extractor and classifier pre-trained in step S2; the parameters of the pre-trained feature extractor and classifier are fixed, and the Gaussian uniform mixture model is trained to obtain the posterior probability of the target sample; when the adversarial domain adaptation network is first trained, the feature extractor is used as the feature extractor network pre-trained on the conventional ImageNet dataset, and the domain adaptation task is learned using the pseudo-label loss of the posterior probability of the target sample; after training a round of adversarial domain adaptation network, the feature extractor and classifier learned by adversarial domain adaptation are saved and applied to the next step of Gaussian uniform mixture model learning;

[0015] Step S6: Based on the trained adversarial domain adaptation network, input the target domain image sample, extract the features through the feature extractor, input the target sample features into the classifier, and use the softmax method to calculate the dimension with the largest probability that the target sample output by the classifier belongs to, which is the predicted label category of the target sample image.

[0016] Furthermore, the step S1 also includes the steps of preprocessing and enhancing the source domain and target domain images; specifically,

[0017] Adjust the size of the source and target domain images. Under the condition of fixed aspect ratio, adjust the image to 256×256 pixel value; randomly crop the adjusted image to 224×224 pixel value; randomly flip the cropped image horizontally with a flip probability of 0.5;

[0018] Convert the image from PIL.image with a value range of [0,255] to Tensor, and normalize the value range to the interval [0,1]; standardize the normalized image Tensor in turn, subtract the mean, divide by the standard deviation, and splice the operations together with the splicing operation.

[0019] Furthermore, the specific steps of obtaining the pre-trained feature extractor and classifier in step S2 include:

[0020] Step S2.1, build a pre-trained deep neural network, including a feature extractor G, a classifier C and a discriminator D; the pre-trained feature extractor G is a ResNet residual network model and a fully connected neural network with 256 neurons; the 256-dimensional features output by the feature extractor G are standardized as follows:

[0021]

[0022] Where f is the sample feature obtained by the feature extractor of the input image, r is the coefficient, and ||f||2 is the l2 norm;

[0023] The classifier D is a one-layer fully connected neural network, and the number of neurons is equal to the number of image categories; the discriminator D is a two-layer fully connected neural network, and the last layer contains one neuron, which is the output layer; a gradient reversal layer is also included between the feature extractor F and the discriminator D, which realizes the identity transformation in the forward propagation of the network; in the reverse propagation of the network, when the gradient is reversed through the discriminator and then propagated to the feature extractor, the gradient reversal layer continues to back-propagate to the feature extractor after taking the inverse multiplication coefficient of the gradient;

[0024] Step S2.3, learning pre-trained feature extractor G, classifier C, and discriminator D; for the deep neural network feature extractor G and classifier C composed of convolution and fully connected layers, the source domain image samples and the target domain image samples that have undergone the data preprocessing and enhancement steps described in step S1 are respectively input into the feature extractor G to obtain 256-dimensional source domain sample features and target domain sample features, and the source domain sample features and the target domain sample features are standardized in step S2.1; the standardized source domain sample features are input into the classifier; the classification loss minimization of the source domain sample features is based on the following standard supervised classification loss:

[0025]

[0026] Where G represents the feature extractor, C represents the classifier, and n S is the number of source samples in the source domain, is the i-th sample in the source domain, is the i-th sample label in the source domain, is the feature of the i-th sample in the source domain, is the predicted output of the classifier for the i-th sample in the source domain, and J(·,·) is the cross entropy loss. The feature extractor G and classifier C are learned through the cross entropy loss to minimize the source classification loss.

[0027] The standardized source domain sample features and target domain sample features are respectively input into the domain discriminator D. The output of the domain discriminator D passes through the Sigmod activation function with a value range of (0, 1). The function is defined as follows:

[0028]

[0029] The domain adversarial loss is minimized based on the standard supervised binary classification loss as follows:

[0030]

[0031] Where G represents the feature extractor, D represents the domain discriminator, and n S is the number of source samples in the source domain, n t is the number of target samples in the target domain, x m is the mth sample of all samples, d m is the mth sample label among all samples, G(x m ) is the feature of the mth sample among all samples, D(G(x m )) is the predicted output of the domain discriminator of the mth sample among all samples, and BCE(·,·) is the binary cross entropy loss. The feature extractor G and the domain discriminator D are learned by the binary cross entropy loss to minimize the source classification loss.

[0032] The binary cross entropy loss is calculated during forward propagation. During back propagation, when the gradient passes through the gradient reversal layer after the domain discriminator, the gradient is negated and multiplied by the coefficient, and then back propagated to the feature extractor.

[0033] Step S2.4: After 500 rounds of learning for the pre-trained deep neural network model, save the learned feature extractor G and classifier C as a pre-training model for learning the Gaussian uniform mixture model.

[0034] Furthermore, the establishment of the Gaussian uniform mixture model in step S3 specifically includes:

[0035] Step S3.1, using the target domain image preprocessed and enhanced as described in step S1;

[0036] Step S3.2, establish a Gaussian uniform mixture model, wherein the feature extractor G is the pre-trained ResNet residual network model and the fully connected neural network of 256 neurons obtained in step S2, and the classifier C is the fully connected neural network of the pre-trained layer obtained in step S2; when training the Gaussian uniform mixture model, the pre-trained feature extractor G and classifier C only have forward propagation but no back propagation, that is, some parameters of the neural network are fixed; the target domain image sample obtained in step S3.1 is input into the feature extractor G to obtain a 256-dimensional target sample feature, and the 256-dimensional feature of the target sample is standardized as follows:

[0037]

[0038] in The target sample features are obtained by the feature extractor after the j-th target sample image in the input target domain; the standardized target domain image sample features are input into the classifier C to obtain the pseudo label of the target domain image sample as follows:

[0039]

[0040] in represents the classification prediction of the jth target sample in the target domain after the feature extractor and classifier, where c represents the number of image samples in the target domain, k represents the one-dimensional classification prediction with the highest probability in the output of the classifier, and the pseudo label The category is represented as the dimension with the largest probability in the classification prediction;

[0041] Based on the pseudo-labels of the target domain image samples obtained, the target domain is classified into C categories, and the cosine distance from each target sample in category C to the class center is calculated. The cosine distance is defined as follows:

[0042]

[0043] in Represented by the target sample The cosine distance from the feature of the jth target sample in the target domain obtained by the input feature extractor G to the mean of the target category, represents the mean of all target sample features of the kth category in the target domain, k is the number of categories of target domain samples, ||·|| represents the modulus, Represents the target sample Input the feature extractor G to obtain the features of the jth target sample in the target domain;

[0044] Assume that the internal points obey Gaussian distribution and the outliers obey uniform distribution. For calculating the k-th target sample in the target domain, Perform Gaussian uniform mixture modeling:

[0045]

[0046] where π k represents the prior probability of the interior point, σ k represents the standard deviation of the Gaussian distribution, δ k represents the uniform distribution parameter; Represents all parameters of the Gaussian uniform mixture model, and introduces a random variable z for each pseudo-labeled target sample j ∈{0,1}, when the target sample If the pseudo label of the target sample is correctly marked, it is recorded as 1, otherwise it is recorded as 0; when the pseudo label of the target sample is correctly marked, the probability is expressed as The definition is as follows:

[0047]

[0048] The Gaussian component models the correctly labeled target data, and the uniform component models the incorrectly labeled data. The maximum expectation algorithm is used to update the parameters of the hybrid model. First, the parameters of the feature extractor G and the classifier C are fixed, the initial value of the model parameter θ is initialized, the number of iterations is set, and the E step is started first to calculate the conditional probability expectation of the joint distribution. The posterior probability of the jth target data being correctly labeled is:

[0049]

[0050] Where l represents the number of iterations of the EM algorithm. The E step uses the existing estimated values ​​of the latent variables to calculate the probability that each sample comes from the Gaussian component. The M step maximizes the maximum likelihood value obtained in the E step to calculate the parameter value as follows:

[0051]

[0052]

[0053]

[0054]

[0055] in

[0056]

[0057] Iterate the E and M steps, exit the loop when the maximum likelihood value reaches the maximum, fix the Gaussian uniform mixture model parameters, and obtain the posterior probability of each target data being correctly labeled as the basis for target sample screening.

[0058] Furthermore, the specific steps of building the adversarial domain adaptation network and training it in step S4 include:

[0059] Step S4.1, using the image preprocessing and enhancement of the target domain image and the source domain image described in step S1;

[0060] Step S4.2, training an adversarial domain adaptation network; unsupervised adversarial domain adaptation learning includes a labeled source domain and an unlabeled target domain, wherein images of the labeled source domain and the unlabeled target domain are used as training sets, and the unlabeled target domain is used as a test set; the adversarial domain adaptation network includes a feature extractor G, a classifier C, and a discriminator D, wherein the feature extractor G is a residual network component pre-trained on a conventional ImageNet dataset; the pre-processed source domain image samples and target domain image samples are respectively input into the feature extractor G, and step S2.3 is repeated;

[0061] Step S4.3: Use the pseudo-label loss of the posterior probability of the target sample to learn the domain adaptation task; set the auxiliary pseudo-label loss as follows:

[0062]

[0063] MAE stands for mean absolute error. in The definition is as follows:

[0064]

[0065] in The target data whose target sample labeling probability is less than 0.5 is discarded, and the conditional entropy loss is defined as:

[0066]

[0067] H(·) is the entropy loss; the adversarial domain adaptation network is trained based on the above loss and the source classification loss and domain adversarial loss in step S2.3. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a network structure diagram of the pseudo-label loss adversarial domain adaptation method provided by the present invention. DETAILED DESCRIPTION

[0069] The present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] The present invention provides an unsupervised adversarial domain adaptation image classification method based on a Gaussian uniform mixture model with pseudo-label loss. The fundamental motivation is that there are no images available for training in the target domain to be classified. The specific steps are as follows:

[0071] Step S1, respectively obtain images of the source domain and the target domain, wherein the source domain is a known domain and the images are labeled, and the target domain is a domain to be classified and the images are unlabeled; preprocess and enhance the obtained images based on multi-operation splicing. Specifically,

[0072] Adjust the size of the source and target domain images. Under the condition of fixed aspect ratio, adjust the image to 256×256 pixel value; randomly crop the adjusted image to 224×224 pixel value; randomly flip the cropped image horizontally with a flip probability of 0.5;

[0073] Convert the image from PIL.image with a value range of [0,255] to Tensor, and normalize the value range to the interval [0,1]; standardize the normalized image Tensor in turn, subtract the mean, divide by the standard deviation, and splice the operations together with the splicing operation.

[0074] Step S2: build a pre-trained deep neural network; learn the pre-trained neural network based on the source domain supervised classification loss and adversarial loss in step S1 to obtain a pre-trained feature extractor and classifier. Specifically,

[0075] Step S2.1, build a pre-trained deep neural network, including a feature extractor G, a classifier C and a discriminator D; the pre-trained feature extractor G is a ResNet residual network model and a fully connected neural network with 256 neurons; the 256-dimensional features output by the feature extractor G are standardized as follows:

[0076]

[0077] Where f is the sample feature obtained by the feature extractor of the input image, r is the coefficient, and ||f||2 is the l2 norm;

[0078] The classifier D is a one-layer fully connected neural network, and the number of neurons is equal to the number of image categories; the discriminator D is a two-layer fully connected neural network, and the last layer contains one neuron, which is the output layer; a gradient reversal layer is also included between the feature extractor F and the discriminator D, which realizes the identity transformation in the forward propagation of the network; in the reverse propagation of the network, when the gradient is reversed through the discriminator and then propagated to the feature extractor, the gradient reversal layer continues to back-propagate to the feature extractor after taking the inverse multiplication coefficient of the gradient;

[0079] Step S2.2, learning pre-trained feature extractor G, classifier C, discriminator D; for the deep neural network feature extractor G and classifier C composed of convolution and fully connected layers, the source domain image samples and the target domain image samples that have undergone the data preprocessing and enhancement steps described in step S1 are respectively input into the feature extractor G to obtain 256-dimensional source domain sample features and target domain sample features, and the source domain sample features and the target domain sample features are standardized in step S2.1; the standardized source domain sample features are input into the classifier; the classification loss minimization of the source domain sample features is based on the following standard supervised classification loss:

[0080]

[0081] Where G represents the feature extractor, C represents the classifier, and n S is the number of source samples in the source domain, is the i-th sample in the source domain, is the i-th sample label in the source domain, is the feature of the i-th sample in the source domain, is the predicted output of the classifier for the i-th sample in the source domain, and J(·,·) is the cross entropy loss. The feature extractor G and classifier C are learned through the cross entropy loss to minimize the source classification loss.

[0082] The standardized source domain sample features and target domain sample features are respectively input into the domain discriminator D. The output of the domain discriminator D passes through the Sigmod activation function with a value range of (0, 1). The function is defined as follows:

[0083]

[0084] The domain adversarial loss is minimized based on the standard supervised binary classification loss as follows:

[0085]

[0086] Where G represents the feature extractor, D represents the domain discriminator, and n S is the number of source samples in the source domain, n t is the number of target samples in the target domain, x m is the mth sample of all samples, d m is the mth sample label among all samples, G(x m ) is the feature of the mth sample among all samples, D(G(x m )) is the predicted output of the domain discriminator of the mth sample among all samples, and BCE(·,·) is the binary cross entropy loss. The feature extractor G and the domain discriminator D are learned by the binary cross entropy loss to minimize the source classification loss.

[0087] The binary cross entropy loss is calculated during forward propagation. During back propagation, when the gradient passes through the gradient reversal layer after the domain discriminator, the gradient is negated and multiplied by the coefficient, and then back propagated to the feature extractor.

[0088] Step S2.3: After 500 rounds of learning for the pre-trained deep neural network model, save the learned feature extractor G and classifier C as a pre-trained model for learning the Gaussian uniform mixture model.

[0089] Step S3: Establish a Gaussian uniform mixture model; specifically,

[0090] Step S3.1, using the image preprocessing and enhancement of the target domain image described in step S1;

[0091] Step S3.2, establish a Gaussian uniform mixture model, wherein the feature extractor G is the pre-trained ResNet residual network model and the fully connected neural network of 256 neurons obtained in step S2, and the classifier C is the fully connected neural network of the pre-trained layer obtained in step S2; when training the Gaussian uniform mixture model, the pre-trained feature extractor G and classifier C only have forward propagation but no back propagation, that is, some parameters of the neural network are fixed; the target domain image sample obtained in step S3.1 is input into the feature extractor G to obtain a 256-dimensional target sample feature, and the 256-dimensional feature of the target sample is standardized as follows:

[0092]

[0093] in The target sample features are obtained by the feature extractor after the j-th target sample image in the input target domain; the standardized target domain image sample features are input into the classifier C to obtain the pseudo label of the target domain image sample as follows:

[0094]

[0095] in represents the classification prediction of the jth target sample in the target domain after the feature extractor and classifier, where c represents the number of image samples in the target domain, k represents the one-dimensional classification prediction with the highest probability in the output of the classifier, and the pseudo label The category is represented as the dimension with the largest probability in the classification prediction;

[0096] Based on the obtained pseudo-labels of the target domain image samples, the target domain is classified into C categories, and the cosine distance from each target sample in category C to the class center is calculated. The cosine distance is defined as follows:

[0097]

[0098] in Represented by the target sample The cosine distance from the feature of the jth target sample in the target domain obtained by the input feature extractor G to the mean of the target category, represents the mean of all target sample features of the kth category in the target domain, k is the number of categories of target domain samples, ||·|| represents the modulus, Represents the target sample Input the feature extractor G to obtain the features of the jth target sample in the target domain;

[0099] The goal is to train a model to detect outliers and reduce their role in the prediction of network output. Since there is no prior information about the percentage and distribution of outliers, it is assumed that the inliers follow a Gaussian distribution and the outliers follow a uniform distribution. Perform Gaussian uniform mixture modeling:

[0100]

[0101] where π k represents the prior probability of the interior point, σ k represents the standard deviation of the Gaussian distribution, δ k represents the uniform distribution parameter; Represents all parameters of the Gaussian uniform mixture model, and introduces a random variable z for each pseudo-labeled target sample j ∈{0,1}, when the target sample If the pseudo label of the target sample is correctly marked, it is recorded as 1, otherwise it is recorded as 0; when the pseudo label of the target sample is correctly marked, the probability is expressed as The definition is as follows:

[0102]

[0103] The Gaussian component models the correctly labeled target data, and the uniform component models the incorrectly labeled data. The maximum expectation algorithm is used to update the parameters of the hybrid model. First, the parameters of the feature extractor G and the classifier C are fixed, the initial value of the model parameter θ is initialized, the number of iterations is set, and the E step is started first to calculate the conditional probability expectation of the joint distribution. The posterior probability of the jth target data being correctly labeled is:

[0104]

[0105] Where l represents the number of iterations of the EM algorithm. The E step uses the existing estimated values ​​of the latent variables to calculate the probability that each sample comes from the Gaussian component. The M step maximizes the maximum likelihood value obtained in the E step to calculate the parameter value as follows:

[0106]

[0107]

[0108]

[0109]

[0110] in

[0111]

[0112] Iterate the E and M steps, exit the loop when the maximum likelihood value reaches the maximum, fix the Gaussian uniform mixture model parameters, and obtain the posterior probability of each target data being correctly labeled as the basis for target sample screening.

[0113] Step S4: building an adversarial domain adaptation network and training it;

[0114] Step S4.1, using the target domain and source domain images preprocessed and enhanced as described in step S1;

[0115] Step S4.2, training an adversarial domain adaptation network; unsupervised adversarial domain adaptation learning includes a labeled source domain and an unlabeled target domain, wherein images of the labeled source domain and the unlabeled target domain are used as training sets, and the unlabeled target domain is used as a test set; the adversarial domain adaptation network includes a feature extractor G, a classifier C, and a discriminator D, wherein the feature extractor G is a residual network component pre-trained on a conventional ImageNet dataset; the pre-processed source domain image samples and target domain image samples are respectively input into the feature extractor G, and step S2.3 is repeated;

[0116] Step S4.3: Use the pseudo-label loss of the posterior probability of the target sample to learn the domain adaptation task; set the auxiliary pseudo-label loss as follows:

[0117]

[0118] MAE stands for mean absolute error. in The definition is as follows:

[0119]

[0120] in The target data whose target sample labeling probability is less than 0.5 is discarded, and the conditional entropy loss is defined as:

[0121]

[0122] H(·) is the entropy loss; the adversarial domain adaptation network is trained based on the above loss and the source classification loss and domain adversarial loss in step 2.3.

[0123] Step S5, repeating steps S3-S4, respectively training the adversarial domain adaptation network and the Gaussian uniform mixture model through iteration;

[0124] When the Gaussian uniform mixture model is first trained, the initialized neural network is used as the feature extractor and classifier pre-trained in step S2; the parameters of the pre-trained feature extractor and classifier are fixed, and the Gaussian uniform mixture model is trained to obtain the posterior probability of the target sample; when the adversarial domain adaptation network is first trained, the feature extractor is used as the feature extractor network pre-trained on the conventional ImageNet dataset, and the domain adaptation task is learned using the pseudo-label loss of the posterior probability of the target sample; after training a round of adversarial domain adaptation network, the feature extractor and classifier learned by adversarial domain adaptation are saved and applied to the next step of Gaussian uniform mixture model learning;

[0125] Step S6: Based on the trained adversarial domain adaptation network, save the feature extractor and classifier learned by adversarial domain adaptation. The preprocessing and enhancement steps in step S1 are applied to the target domain data, and the target domain image sample is input. After the feature extractor extracts the features, the target sample features are input to the classifier, and the softmax method is used to calculate the dimension with the largest probability that the target sample output by the classifier belongs to, which is the predicted label category of the target sample image.

[0126] The pseudo-label loss adversarial domain adaptation method provided by the present invention is further illustrated below based on experimental results.

[0127] In order to verify the effectiveness of the present invention, experiments were conducted on Office31, ImageCLEF-DA, and Office-Caltech10.

[0128] The Office31 dataset contains three domains: Amazon, Webcam, and Dslr. One of the domains is selected as the labeled source domain and the other domain is selected as the target domain to be classified. As shown in Table 1 below, this method has a significant improvement in multiple domains and average precision.

[0129] Table 1 Recognition accuracy on the office31 dataset

[0130]

[0131] The ImageCLEF-DA dataset contains three domains: I, P, and C, as shown in Table 2 below. It can be clearly seen that the method proposed in this patent has a significant improvement compared to other methods.

[0132] Table 2 Recognition accuracy on the ImageCLEF-DA dataset

[0133]

[0134] The Office-Caltech10 dataset contains four different domains, which are replaced by A, C, D, and W respectively. As shown in Table 3, the method proposed in this paper achieves the best average accuracy.

[0135] Table 3 Recognition accuracy on the Office-Caltech10 dataset

[0136]

[0137] In summary, the pseudo-label loss adversarial domain adaptation method based on the Gaussian uniform mixture model of the present invention, on the one hand, proposes an iterative method for unsupervised outlier detection and adversarial domain adaptation unsupervised image classification using a Gaussian uniform mixture model, which uses the pseudo-labels generated during training by step-by-step training of the learning network model, and effectively utilizes the target domain pseudo-labels during training. On the other hand, the auxiliary pseudo-label loss proposed based on the target sample pseudo-labels generated during the training process is added to the training of the neural network. Minimizing the conditional entropy loss keeps the learned features away from the decision boundary and improves the classification accuracy. The method improves the utilization rate of the target domain pseudo-labels, and better assists the domain adaptation task and learns the classification network.

[0138] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A pseudo-label loss unsupervised adversarial domain adaptation image classification method based on Gaussian uniform mixture model, characterized in that: The following steps are involved: Step S1, respectively obtaining images of a source domain and a target domain, wherein the source domain is a known domain and the images are labeled, and the target domain is a domain to be classified and the images are unlabeled; preprocessing and enhancing the obtained images based on multi-operation splicing; Step S2, building a pre-trained deep neural network; learning the pre-trained neural network based on the source domain supervised classification loss and adversarial loss in step S1 to obtain a pre-trained feature extractor and classifier; Step S3, establishing a Gaussian uniform mixture model; based on the pre-trained feature extractor and classifier obtained in step S2, first inputting the sample image of the target domain into the pre-trained feature extractor for feature extraction; inputting the extracted features into the pre-trained classifier to obtain the target sample pseudo label; the features extracted by the feature extractor are the sample features of the target domain; Calculate the cosine distance from each type of target sample to the class feature mean of the target domain; model the cosine distance calculated for each type of target sample, fix the pre-trained feature extractor and classifier parameters, and update the Gaussian uniform mixture model parameters through the expectation maximization method to detect outliers, and obtain the posterior probability of the target sample as an indicator to measure the confidence of the pseudo label; Step S4: building an adversarial domain adaptation network and training it; Build a feature extractor, classifier, and discriminator network, where the feature extractor is a feature extractor network pre-trained on the regular ImageNet dataset; input source and target domain images, and obtain the outputs of the source and target domain image classifiers and discriminators respectively through the forward propagation of the adversarial domain adaptation network; use the standard supervised cross-entropy loss to train the feature extractor and classifier for the labeled source domain; propose domain adversarial loss based on the domain labels of the source and target domains, and learn the feature extractor and discriminator; based on the posterior probability of the target sample obtained in step S3, select the pseudo-label with high posterior probability of the target sample in step S3, combine the target sample prediction of the adversarial domain adaptation network classifier to calculate the mean absolute error as the pseudo-label loss, and learn the adversarial domain adaptation network through the pseudo-label loss; use the minimized conditional entropy loss to learn features away from the decision boundary; use the finally learned feature extractor and classifier as the pre-trained feature extractor and classifier of step S3; Step S5, repeating steps S3-S4, respectively training the adversarial domain adaptation network and the Gaussian uniform mixture model through iteration; When the Gaussian uniform mixture model is first trained, the initialized neural network is used as the feature extractor and classifier pre-trained in step S2; the parameters of the pre-trained feature extractor and classifier are fixed, and the Gaussian uniform mixture model is trained to obtain the posterior probability of the target sample; when the adversarial domain adaptation network is first trained, the feature extractor is used as the feature extractor network pre-trained on the conventional ImageNet dataset, and the domain adaptation task is learned using the pseudo-label loss of the posterior probability of the target sample; after training a round of adversarial domain adaptation network, the feature extractor and classifier learned by adversarial domain adaptation are saved and applied to the next step of Gaussian uniform mixture model learning; Step S6: Based on the trained adversarial domain adaptation network, input the target domain image sample, extract the features through the feature extractor, input the target sample features into the classifier, and use the softmax method to calculate the dimension with the largest probability that the target sample output by the classifier belongs to, which is the predicted label category of the target sample image.

2. According to claim 1, the unsupervised adversarial domain adaptation image classification method based on pseudo-label loss of Gaussian uniform mixture model is characterized in that: The step S1 also includes the steps of preprocessing and enhancing the source domain and target domain images; specifically, Adjust the size of the source and target domain images to 256×256 pixels under a fixed aspect ratio. The resized image is randomly cropped to 224×224 pixel values; The cropped image is randomly flipped horizontally with a flip probability of 0.5; Convert the image from PIL.image with a value range of [0,255] to Tensor, and normalize the value range to the interval [0,1]; standardize the normalized image Tensor in turn, subtract the mean, divide by the standard deviation, and splice the operations together with the splicing operation.

3. The unsupervised adversarial domain adaptation image classification method based on pseudo-label loss of Gaussian uniform mixture model according to claim 2, characterized in that: The specific steps of obtaining the pre-trained feature extractor and classifier in step S2 include: Step S2.1, build a pre-trained deep neural network, including a feature extractor G, a classifier C and a discriminator D; the pre-trained feature extractor G is a ResNet residual network model and a fully connected neural network with 256 neurons; the 256-dimensional features output by the feature extractor G are standardized as follows: Where f is the sample feature obtained by the feature extractor of the input image, r is the coefficient, and ||f||2 is the l2 norm; The classifier D is a one-layer fully connected neural network, and the number of neurons is equal to the number of image categories; the discriminator D is a two-layer fully connected neural network, and the last layer contains one neuron, which is the output layer; a gradient reversal layer is also included between the feature extractor F and the discriminator D, which realizes the identity transformation in the forward propagation of the network; in the reverse propagation of the network, when the gradient is reversed through the discriminator and then propagated to the feature extractor, the gradient reversal layer continues to back-propagate to the feature extractor after taking the inverse multiplication coefficient of the gradient; Step S2.2, learning pre-trained feature extractor G, classifier C, discriminator D; for the deep neural network feature extractor G and classifier C composed of convolution and fully connected layers, the source domain image samples and the target domain image samples that have undergone the data preprocessing and enhancement steps described in step S1 are respectively input into the feature extractor G to obtain 256-dimensional source domain sample features and target domain sample features, and the source domain sample features and the target domain sample features are standardized in step S2.1; the standardized source domain sample features are input into the classifier; the classification loss minimization of the source domain sample features is based on the following standard supervised classification loss: Where G represents the feature extractor, C represents the classifier, and n S is the number of source samples in the source domain, is the i-th sample in the source domain, is the i-th sample label in the source domain, is the feature of the i-th sample in the source domain, is the predicted output of the classifier for the i-th sample in the source domain, and J(·,·) is the cross entropy loss. The feature extractor G and classifier C are learned through the cross entropy loss to minimize the source classification loss. The standardized source domain sample features and target domain sample features are respectively input into the domain discriminator D. The output of the domain discriminator D passes through the Sigmod activation function with a value range of (0, 1). The function is defined as follows: The domain adversarial loss is minimized based on the standard supervised binary classification loss as follows: Where G represents the feature extractor, D represents the domain discriminator, and n S is the number of source samples in the source domain, n t is the number of target samples in the target domain, x m is the mth sample of all samples, d m is the mth sample label among all samples, G(x m ) is the feature of the mth sample among all samples, D(G(x m )) is the predicted output of the domain discriminator of the mth sample among all samples, and BCE(·,·) is the binary cross entropy loss. The feature extractor G and the domain discriminator D are learned by the binary cross entropy loss to minimize the source classification loss. The binary cross entropy loss is calculated during forward propagation. During back propagation, when the gradient passes through the gradient reversal layer after the domain discriminator, the gradient is negated and multiplied by the coefficient, and then back propagated to the feature extractor. Step S2.3: After 500 rounds of learning for the pre-trained deep neural network model, save the learned feature extractor G and classifier C as a pre-trained model for learning the Gaussian uniform mixture model.

4. The unsupervised adversarial domain adaptation image classification method based on pseudo-label loss of Gaussian uniform mixture model according to claim 3, characterized in that: The Gaussian uniform mixture model establishment in step S3 specifically includes: Step S3.1, using the image preprocessing and enhancement of the target domain image described in step S1; Step S3.2, establish a Gaussian uniform mixture model, wherein the feature extractor G is the pre-trained ResNet residual network model and the fully connected neural network of 256 neurons obtained in step S2, and the classifier C is the fully connected neural network of the pre-trained layer obtained in step S2; when training the Gaussian uniform mixture model, the pre-trained feature extractor G and classifier C only have forward propagation but no back propagation, that is, all parameters of the neural network are fixed; the target domain image sample obtained in step S3.1 is input into the feature extractor G to obtain a 256-dimensional target sample feature, and the 256-dimensional feature of the target sample is standardized as follows: in The target sample features are obtained by the feature extractor after the j-th target sample image in the input target domain; the standardized target domain image sample features are input into the classifier C to obtain the pseudo label of the target domain image sample as follows: in represents the classification prediction of the jth target sample in the target domain after the feature extractor and classifier, where c represents the number of image samples in the target domain, k represents the one-dimensional classification prediction with the highest probability in the output of the classifier, and the pseudo label The category is represented as the dimension with the largest probability value in the classification prediction; Based on the obtained pseudo-labels of the target domain image samples, the target domain is classified into C categories, and the cosine distance from each target sample in category C to the class center is calculated. The cosine distance is defined as follows: in Represented by the target sample The cosine distance from the feature of the jth target sample in the target domain obtained by the input feature extractor G to the mean of the target category, represents the mean of all target sample features of the kth category in the target domain, k is the number of categories of target domain samples, ||·|| represents the modulus, Represents the target sample Input the feature extractor G to obtain the features of the jth target sample in the target domain; Assume that the internal points obey Gaussian distribution and the outliers obey uniform distribution. For calculating the k-th target sample in the target domain, Perform Gaussian uniform mixture modeling: where π k represents the prior probability of the interior point, σ k represents the standard deviation of the Gaussian distribution, δ k represents the uniform distribution parameter; Represents all parameters of the Gaussian uniform mixture model, and introduces a random variable z for each pseudo-labeled target sample j ∈{0,1}, when the target sample If the pseudo label of the target sample is correctly marked, it is recorded as 1, otherwise it is recorded as 0; when the pseudo label of the target sample is correctly marked, the probability is expressed as The definition is as follows: The Gaussian component models the correctly labeled target data, and the uniform component models the incorrectly labeled data. The maximum expectation algorithm is used to update the parameters of the hybrid model. First, the parameters of the feature extractor G and the classifier C are fixed, the initial value of the model parameter θ is initialized, the number of iterations is set, and the E step is started first to calculate the conditional probability expectation of the joint distribution. The posterior probability of the jth target data being correctly labeled is: Where l represents the number of iterations of the EM algorithm. The E step uses the existing estimated values ​​of the latent variables to calculate the probability that each sample comes from the Gaussian component. The M step maximizes the maximum likelihood value obtained in the E step to calculate the parameter value as follows: in Iterate the E and M steps, exit the loop when the maximum likelihood value reaches the maximum, fix the Gaussian uniform mixture model parameters, and obtain the posterior probability of each target data being correctly labeled as the basis for target sample screening.

5. According to claim 4, the method for unsupervised adversarial domain adaptation image classification based on pseudo-label loss of Gaussian uniform mixture model is characterized in that: The specific steps of building the adversarial domain adaptation network and training it in step S4 include: Step S4.1, using the image preprocessing and enhancement of the target domain image and the source domain image described in step S1; Step S4.2, training an adversarial domain adaptation network; unsupervised adversarial domain adaptation learning includes a labeled source domain and an unlabeled target domain, wherein images of the labeled source domain and the unlabeled target domain are used as training sets, and the unlabeled target domain is used as a test set; the adversarial domain adaptation network includes a feature extractor G, a classifier C, and a discriminator D, wherein the feature extractor G is a residual network component pre-trained on a conventional ImageNet dataset; the pre-processed source domain image samples and target domain image samples are respectively input into the feature extractor G, and step S2.3 is repeated; Step S4.3: Use the pseudo-label loss of the posterior probability of the target sample to learn the domain adaptation task; set the auxiliary pseudo-label loss as follows: MAE stands for mean absolute error. in The definition is as follows: in The target data whose target sample labeling probability is less than 0.5 is discarded, and the conditional entropy loss is defined as: H(·) is the entropy loss; the adversarial domain adaptation network is trained based on the above loss and the source classification loss and domain adversarial loss in step S2.3.

Images