Image pseudo label labeling method based on virtual adversarial training

Abstract

The invention belongs to the field of semi-supervised learning in machine learning, and particularly relates to an image false label labeling method based on virtual adversarial training, which comprises the following steps: acquiring labeled image data and unlabeled image data as training data; calculating a cross entropy loss between the predicted label and the real label as a first loss by using a label image data self-training model; predicting a pseudo label of the label-free image data, and taking a cross entropy of a predicted label of the strongly enhanced label-free image data and the pseudo label as a second loss; adding disturbance into the training data, and taking divergence between a prediction label of the disturbance data and a real label or a pseudo label of the disturbance data as third loss; and predicting labels of the label-free image data by using the first, second and third loss-trained models. According to the method, the invariance of the model to input disturbance is improved and the generalization ability of the model is improved by generating the virtual adversarial sample without depending on generation of the pseudo tag.

Description

Technical Field

[0001] The present invention belongs to the field of semi-supervised learning in machine learning, and in particular relates to an image pseudo-labeling method based on virtual adversarial training. Background Art

[0002] In many practical applications, it is very expensive and time-consuming to obtain a large amount of accurately labeled data. For example, in order to train an autonomous driving system, a large amount of road image data is required, and these data need to be accurately labeled with vehicles, pedestrians, traffic signs, etc. The labeling work needs to distinguish various complex traffic situations and different road users. In natural language processing, high-quality text labeling also requires a lot of work from linguists and data scientists. This has prompted researchers to find ways to reduce their dependence on labeled data. With the development of information technology, it has become easier and easier to obtain unlabeled data. For example, texts on the Internet, posts on social media, and videos from surveillance cameras are huge amounts of data, but most of them are unlabeled. Semi-supervised learning provides a way to utilize these unlabeled data to improve the performance of learning models.

[0003] Semi-supervised learning is an important field in machine learning. It aims to solve the problem of high cost of labeling data in supervised learning, while overcoming the limitation of lack of clear learning objectives in unsupervised learning. The core idea of ​​semi-supervised learning is to use a large amount of unlabeled data and a small amount of labeled data to train the model in order to achieve or approach the performance of fully supervised learning. Most of the existing semi-supervised adversarial training relies on screening out the prediction with the maximum confidence as a pseudo-label to the unlabeled data, and then putting the labeled unlabeled data into the labeled data set for model training. If there is noise or error in the unlabeled data, inaccurate pseudo-labels may be generated, and if the prediction ability of the model is not strong, the generated pseudo-labels may have large errors. These erroneous pseudo-labels may mislead the learning process of the model, resulting in poor model performance. Summary of the invention

[0004] In order to eliminate the influence of pseudo labels of unlabeled data on the model to a certain extent, the present invention proposes an image pseudo labeling method based on virtual adversarial training, which uses a trained label prediction model to predict the label of image data. The training process of the label prediction model specifically includes the following steps:

[0005] Obtain a training data set, the training data set includes labeled image data and unlabeled image data, and use the labeled image data to train a label prediction model to obtain a first label prediction model;

[0006] Strongly enhance the labeled image data, and input the strongly enhanced data into the first label prediction model to obtain the cross entropy loss between the prediction result and its true label as the first loss;

[0007] Using the first label prediction model to predict the label of the unlabeled image data after weak enhancement, and taking the label whose confidence of the prediction result is greater than a set threshold as the pseudo label of the unlabeled image data;

[0008] The unlabeled image data with pseudo labels is strongly enhanced, the label probability distribution of the enhanced image data is predicted using the first label prediction model, and the cross entropy between the pseudo labels and the probability distribution is calculated as the second loss;

[0009] Calculate a virtual adversarial perturbation for each weakly enhanced data in the training dataset and apply the perturbation to the original data;

[0010] The KL divergence between the data distribution of the real labels or pseudo labels of all training data and the data distribution of the predicted labels obtained by inputting the perturbed data into the first label prediction model is used as the third loss;

[0011] The label prediction model is trained using the first loss, the second loss, and the third loss to obtain the final label prediction model.

[0012] Compared with the prior art, the present invention improves the invariance of the model to input perturbations and the generalization ability of the model by generating virtual adversarial samples without relying on the generation of pseudo labels, thereby eliminating the influence of pseudo labels of unlabeled data on the model to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The present invention is a flowchart of a method for annotating pseudo-labels of images based on virtual adversarial training. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] The present invention proposes a pseudo-labeling method for images based on virtual adversarial training, which uses a trained label prediction model to predict the label of image data. Figure 1 , the training process of the label prediction model specifically includes the following steps:

[0016] Obtain a training data set, the training data set includes labeled image data and unlabeled image data, and use the labeled image data to train a label prediction model to obtain a first label prediction model;

[0017] Strongly enhance the labeled image data, and input the strongly enhanced data into the first label prediction model to obtain the cross entropy loss between the prediction result and its true label as the first loss;

[0018] Using the first label prediction model to predict the label of the unlabeled image data after weak enhancement, and taking the label whose confidence of the prediction result is greater than a set threshold as the pseudo label of the unlabeled image data;

[0019] The unlabeled image data with pseudo labels is strongly enhanced, the label probability distribution of the enhanced image data is predicted using the first label prediction model, and the cross entropy between the pseudo labels and the probability distribution is calculated as the second loss;

[0020] Calculate a virtual adversarial perturbation for each weakly enhanced data in the training dataset and apply the perturbation to the original data;

[0021] The KL divergence between the data distribution of the real labels or pseudo labels of all training data and the data distribution of the predicted labels obtained by inputting the perturbed data into the first label prediction model is used as the third loss;

[0022] The label prediction model is trained using the first loss, the second loss, and the third loss to obtain the final label prediction model.

[0023] In this embodiment, labeled data and unlabeled data are obtained for preprocessing, pseudo labels of unlabeled data are obtained through self-training, and a strongly enhanced version of the same unlabeled image is predicted; consistency regularization is applied between the pseudo labels and the new predictions and the consistency regularization loss is calculated; perturbations are calculated for each weakly enhanced sample, and virtual adversarial samples are generated for virtual adversarial training; virtual adversarial training is integrated into consistency learning, and the KL divergence of weakly enhanced data and virtual adversarial samples is calculated; the virtual adversarial training loss is combined with the supervision loss and the consistency regularization loss to form the final total loss function, the total loss function is used for model training, and the model parameters are updated through back propagation and gradient descent algorithms to obtain a trained robust neural network model. Next, this embodiment is divided into six parts to illustrate the above scheme.

[0024] S1: Obtain labeled data and unlabeled data for preprocessing.

[0025] 1000 and 4000 labeled data points were randomly selected from the SVHN and CIFAR-10 datasets for training, respectively, and the rest of the data in the datasets were set as unlabeled data. For the weak data enhancement preprocessing method, the image was flipped horizontally or vertically with a probability of 50%; for the strong data enhancement method, the image contrast was adjusted to maximize the image processing.

[0026] S2: Predict a strongly enhanced version of the same unlabeled image by obtaining pseudo labels for unlabeled data through self-training.

[0027] In this step, it is necessary to train the label prediction model with labeled data to obtain a first label prediction model, and then predict the pseudo label of the unlabeled data according to the first label prediction model, which specifically includes the following steps:

[0028] S21. First, weakly enhance the labeled data, flip the image horizontally or vertically with a probability of 50%, and convert the enhanced labeled data x l Input the model, get the predicted probability distribution, and calculate the model's loss l on labeled data s :

[0029]

[0030] Among them, D ce represents the cross entropy loss function, y l is the true label of the labeled data, P(y l |τ(x l );θ) is the model parameter θ, for the weakly enhanced labeled data x l The predicted probability distribution of l is the number of labeled samples, is the true label of the i-th labeled sample, It is a weak enhancement operation performed on the i-th labeled sample.

[0031] S22. Then use the trained model to predict the unlabeled data and generate pseudo labels. The model will output a probability distribution for each unlabeled sample, indicating the probability of belonging to each category. Only when the model has a high confidence prediction probability for a category of the unlabeled data, will a pseudo label be generated and retained for the data point. The initial setting of the confidence threshold starts from 0.5-0.7, and then adjusted according to the performance on the validation set. The cross entropy loss function for unlabeled data is defined as follows:

[0032]

[0033] Among them, N uis the number of unlabeled samples; 1(·) is an indicator function, which is used in conditional expressions. It takes the value 1 when the condition is met, otherwise it takes the value 0; γ represents a predefined confidence threshold. Only when the prediction probability exceeds this threshold, the prediction of the sample will be used as a pseudo-label. Technical personnel in this field generally adjust and select it according to the effect of the validation set. The validation set does not participate in the model training process and is used to evaluate the model performance. By observing whether the model is overfitting or underfitting, hyperparameters such as learning rate and number of iterations are adjusted to achieve the best effect of the model on the validation set.

[0034] S23, perform strong enhancement on the same unlabeled image, adjust the image contrast to maximize the image contrast, and convert the enhanced unlabeled data x u Input the model and obtain the probability distribution. After obtaining the strong and weak enhanced versions of the same image, for the samples that meet the conditions (i.e. max(q (i) )≥γ), calculate the cross entropy loss between its pseudo-label and the prediction after strong enhancement, that is, the second loss, expressed as:

[0035]

[0036] Among them, κ(x) is a strong data enhancement operation, which means a more significant transformation of unlabeled data; P(y|κ(x); θ)) represents the model's transformation of the strongly enhanced unlabeled image. The predicted probability distribution of q (i) Represents an unlabeled image given a weak enhancement by the model The category distribution when It means to find the model parameter θ The cross entropy between P(y|κ(x) and Indicates that the category with the highest probability is selected as the output from the probability distribution predicted by the model, p (i) Represents the probability distribution of the i-th sample belonging to each category; add the cross entropy losses of all qualified samples and then divide them by the total number of qualified samples plus ρ to get the final regularization loss l r , so that the model maintains consistent predictions for the same unlabeled sample under different enhancements (weak enhancement and strong enhancement). This consistency training helps improve the model's robustness to input perturbations.

[0037] In this embodiment, q (i) Represents an unlabeled image given a weak enhancement by the model The category distribution when It means selecting the category with the highest probability from the probability distribution predicted by the model as the output.

[0038] S3: Generate corresponding virtual adversarial samples for each weakly enhanced sample and calculate virtual training loss. Specifically including:

[0039] S31: Calculate virtual adversarial perturbations, generate adversarial samples, and calculate virtual adversarial perturbations r for each weakly enhanced sample vadv , the announcement is as follows:

[0040]

[0041] Among them, x * It can be labeled data or unlabeled data; D kl [p,q] represents KL divergence, which is used to calculate the difference between two probability distributions p and q. The goal of perturbation r is to find a direction so that a small movement in this direction can change the model's prediction to the greatest extent. The model parameters θ represent the model parameters obtained by training the unperturbed data. Represents the model parameters obtained by training the perturbed data.

[0042] The process of calculating virtual adversarial perturbations includes the following steps: first, select a gradient descent algorithm, then initialize a perturbation vector close to zero, and iteratively update the perturbation vector to maximize the output change of the model in the perturbation direction. The size of the perturbation should be limited to ensure that the perturbation is small enough and does not significantly change the appearance of the sample. The calculated perturbation is added to the original sample to generate a virtual adversarial sample.

[0043] S32: In virtual adversarial training, local perturbation sensitivity (LDS) measures the sensitivity of the model output to small perturbations of the input data. A smaller LDS value indicates that the model is more robust to input perturbations. LDS is used as a regularization term. The goal is to train the model to reduce the sensitivity to these small perturbations, thereby improving the generalization ability of the model. Local perturbation sensitivity (LDS) is defined as:

[0044]

[0045] Among them, LDS(x * ; θ) represents the KL divergence between the predicted probability distribution of the computing model on the original sample and the slightly perturbed sample. By minimizing the KL divergence, the model is encouraged to remain unchanged for small perturbations of the input data; ε represents the size of the perturbation; Represents the input sample x * In the model parameters Under this condition, the predicted probability of output label y; P(y|x * +εr vadv ; θ) represents the input sample x * After the disturbance factor ε and the virtual adversarial disturbance r vadv After the effect of , under the condition of parameter model θ, the predicted probability of output label y is obtained.

[0046] The local perturbation sensitivity LDS of all input samples (labeled and unlabeled) is summed to obtain the regularization term on the entire dataset, which is expressed as:

[0047]

[0048] S33: The original samples of virtual adversarial training are weakly enhanced, so the loss function of virtual adversarial training is as follows:

[0049]

[0050] Integrate virtual adversarial training into consistency learning and calculate the KL divergence of weakly enhanced data and virtual adversarial samples. By calculating the KL divergence between all weakly enhanced inputs (labeled data and unlabeled data) and the corresponding virtual adversarial samples, integrate virtual adversarial training into consistency learning, and finally obtain the target virtual adversarial training loss function as follows:

[0051]

[0052] Among them, l vat Denotes the third loss, which is composed of the sum of the divergences between the labels of all samples and their predicted labels, where if the sample is labeled data, its label is its true label, and if the sample is unlabeled data, its label is its pseudo label; D kl Represents KL divergence, which is used to measure the difference between two probability distributions. For each labeled sample, calculate the probability distribution corresponding to its true label and the probability distribution predicted by the model For each unlabeled sample, calculate the probability distribution corresponding to its pseudo label and the probability distribution predicted by the model The KL divergence between .

[0053] S4: Calculate the total loss function and use it to train the model to obtain a trained robust neural network model.

[0054] The total loss function is used for model training, and the model parameters are updated through back propagation and gradient descent algorithms to minimize the total loss. The complete objective function is:

[0055]

[0056] Among them, α and β are weighting coefficients that control the trade-off between supervision loss, regularization loss, and VAT loss. The optimal values ​​of weighting coefficients α and β are determined by grid search, which is a method of finding the best hyperparameter combination by traversing a given set of hyperparameters to improve the performance of the model.

[0057] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for annotating pseudo-labels of images based on virtual adversarial training, characterized in that: The trained label prediction model is used to predict the label of the image data. The training process of the label prediction model specifically includes the following steps: Obtain a training data set, the training data set includes labeled image data and unlabeled image data, and use the labeled image data to train a label prediction model to obtain a first label prediction model; Strongly enhance the labeled image data, and input the strongly enhanced data into the first label prediction model to obtain the cross entropy loss between the prediction result and its true label as the first loss; Using the first label prediction model to predict the label of the unlabeled image data after weak enhancement, and taking the label whose confidence of the prediction result is greater than a set threshold as the pseudo label of the unlabeled image data; The unlabeled image data with pseudo labels is strongly enhanced, the label probability distribution of the enhanced image data is predicted using the first label prediction model, and the cross entropy between the pseudo labels and the probability distribution is calculated as the second loss; Calculate a virtual adversarial perturbation for each weakly enhanced data in the training dataset and apply the perturbation to the original data; The KL divergence between the data distribution of the real labels or pseudo labels of all training data and the data distribution of the predicted labels obtained by inputting the perturbed data into the first label prediction model is used as the third loss; The label prediction model is trained using the first loss, the second loss, and the third loss to obtain the final label prediction model.

2. According to the method of claim 1, the pseudo-labeling method of images based on virtual adversarial training is characterized in that: The weak enhancement processing of the image data includes flipping the image horizontally or vertically with a probability of 50%.

3. According to the method of claim 1, the pseudo-labeling method of an image based on virtual adversarial training is characterized in that: The processing of strong enhancement of image data includes maximizing the image contrast.

4. According to the method of claim 1, the pseudo-labeling method of images based on virtual adversarial training is characterized in that: In the process of obtaining the pseudo label of the unlabeled training data according to the first label model, if and only if the prediction probability of the pseudo label is greater than a set confidence threshold, the predicted label is set as the pseudo label of the unlabeled training data.

5. The method for annotating pseudo-labels of an image based on virtual adversarial training according to claim 1, characterized in that: The first loss is expressed as: Among them, l s Indicates the first loss; D ce (y l ,p(|τ(x l );θ)) represents the probability distribution y of the true label l The probability distribution of the label obtained by the first label prediction model is p(y l |τ(x l ), the cross entropy between θ, x l represents a labeled image data, θ represents the network parameters of the first label prediction model, τ(x l ) represents the labeled image data x l Perform weak enhancement operation; N l Indicates the number of labeled image data, represents the true label of the i-th labeled image data, Indicates a weak enhancement operation on the i-th labeled image sample.

6. The method for annotating pseudo-labels of an image based on virtual adversarial training according to claim 1, characterized in that: The second loss is expressed as: Among them, l r Indicates the second loss; N u is the number of unlabeled samples; 1(·) is an indicator function used in conditional expressions. It takes the value 1 when the condition is met and takes the value 0 otherwise; γ represents the predefined confidence threshold; q (i) Represents an unlabeled image given a weak enhancement by the model The category distribution when It means to find the model parameter θ The cross entropy between P(y|κ(x) and Indicates that the category with the highest probability is selected as the output from the probability distribution predicted by the model, p (i) represents the probability distribution of the i-th sample belonging to each category; P(y|κ(x)) represents the model's response to the strongly enhanced unlabeled image The predicted probability distribution of , κ(x) represents the strong data augmentation operation on the input sample, y represents the probability distribution of output y given the input x, θ represents the model parameters; γ represents the predefined confidence threshold; ρ is a non-zero term.

7. The method for annotating pseudo-labels of an image based on virtual adversarial training according to claim 1, characterized in that: The process of calculating the virtual adversarial perturbation for each weakly enhanced data in the training dataset is expressed as: Among them, r vadv represents the virtual adversarial perturbation corresponding to the sample; It represents the KL divergence between the predicted probability distribution of the computing model on the original sample and the slightly perturbed sample. Represents a given input sample x * and model parameters Under the condition of , the probability distribution of the output label y; P(y|x * +r) means that for a given input sample x * The predicted probability of the output label y under the condition of perturbation r and model parameter θ; * For labeled image data x l Or unlabeled image data x u , represents the model parameters before the perturbation.

8. The method for annotating pseudo labels of an image based on virtual adversarial training according to claim 1, characterized in that: The third loss is expressed as: Among them, l vat represents the third loss, Represents the true label distribution of the i-th sample After weak enhancement with the i-th sample, input into the first prediction model to obtain the predicted label distribution The divergence between Represents the pseudo label distribution predicted by the first prediction model of the input of the i-th sample After weak enhancement with the i-th sample, input into the first prediction model to obtain the predicted label distribution The divergence between .

9. The method for annotating pseudo-labels of an image based on virtual adversarial training according to claim 1, characterized in that: When the label prediction model is trained using the first loss, the second loss, and the third loss, the model parameters are updated through back propagation and gradient descent algorithms with the purpose of minimizing the total loss. The total loss is expressed as: in, represents the total loss; l s Indicates the first loss; l r Indicates the second loss; l vat represents the third loss; α and β are the weighting coefficients of the second loss and the third loss respectively.

Images