A Pseudo-labeling Method for Images Based on Virtual Adversarial Training

Abstract

This invention belongs to the field of semi-supervised learning in machine learning, specifically relating to an image pseudo-labeling method based on virtual adversarial training. The method includes: acquiring labeled and unlabeled image data as training data; calculating the cross-entropy loss between the predicted label and the real label using a model self-trained with the labeled image data as a first loss; predicting pseudo-labels for the unlabeled image data, and using the cross-entropy between the predicted label and its pseudo-label of the strongly enhanced unlabeled image data as a second loss; adding perturbations to the training data, and using the divergence between the predicted label and its real or pseudo-label of the perturbed data as a third loss; and using a model trained with the first, second, and third losses to predict the label of the unlabeled image data. This invention improves the model's invariance to input perturbations and enhances its generalization ability by generating virtual adversarial examples, without relying on the generation of pseudo-labels.

Description

Technical Field

[0001] This invention belongs to the field of semi-supervised learning in machine learning, specifically involving an image pseudo-labeling method based on virtual adversarial training. Background Technology

[0002] In many practical applications, acquiring large amounts of accurately labeled data is extremely expensive and time-consuming. For example, training autonomous driving systems requires vast amounts of road imagery data, which needs to be precisely labeled with vehicles, pedestrians, traffic signs, and more. Labeling work necessitates distinguishing between various complex traffic situations and different road users. In natural language processing, high-quality text annotation also requires significant work from linguists and data scientists. This has prompted researchers to seek methods to reduce their reliance on labeled data. With the development of information technology, acquiring unlabeled data has become increasingly easy. For example, text on the internet, posts on social media, and surveillance camera footage—these data are enormous in volume, but most are unlabeled. Semi-supervised learning offers a way to utilize this unlabeled data to improve the performance of learning models.

[0003] Semi-supervised learning is an important area in machine learning that aims to address the high cost of labeled data in supervised learning while overcoming the limitations of unsupervised learning due to the lack of a clear learning objective. The core idea of ​​semi-supervised learning is to train a model using a large amount of unlabeled data and a small amount of labeled data, aiming to achieve or approach the performance of fully supervised learning. Most existing semi-supervised adversarial training methods rely on selecting the predictions with the highest confidence as pseudo-labels for the unlabeled data, and then feeding the labeled unlabeled data into the labeled dataset for model training. If the unlabeled data contains noise or errors, inaccurate pseudo-labels may be generated. Furthermore, if the model's predictive ability is weak, the generated pseudo-labels may contain significant errors. These erroneous pseudo-labels can mislead the model's learning process, leading to a decline in model performance. Summary of the Invention

[0004] To mitigate the impact of pseudo-labels on the model from unlabeled data, this invention proposes an image pseudo-labeling method based on virtual adversarial training. This method utilizes a trained label prediction model to predict the labels of image data. The training process for the label prediction model specifically includes the following steps:

[0005] Obtain the training dataset, which includes labeled image data and unlabeled image data, and use the labeled image data to train the label prediction model to obtain the first label prediction model;

[0006] The labeled image data is strongly enhanced, and the enhanced data is input into the first label prediction model. The cross-entropy loss between the prediction result and its true label is used as the first loss.

[0007] The first label prediction model is used to predict the label of the unlabeled image data after weak enhancement, and the label with the confidence of the prediction result is greater than a set threshold is used as the pseudo label of the unlabeled image data.

[0008] The unlabeled image data with pseudo-labels is strongly enhanced. The first label prediction model is used to predict the label probability distribution of the enhanced image data. The cross-entropy between the pseudo-labels and the probability distribution is calculated as the second loss.

[0009] For each weakly augmented data in the training dataset, a virtual adversarial perturbation is computed and then applied to the original data.

[0010] The KL divergence between the distribution of the true or pseudo labels of all training data and the distribution of the predicted labels obtained by inputting the perturbated dataset into the first label prediction model is used as the third loss.

[0011] The label prediction model is trained using the first loss, the second loss, and the third loss to obtain the final label prediction model.

[0012] Compared with existing technologies, this invention improves the model's invariance to input perturbations and enhances its generalization ability by generating virtual adversarial examples without relying on the generation of pseudo-labels, thereby eliminating the impact of pseudo-labels on the model to a certain extent. Attached Figure Description

[0013] Figure 1 This is a flowchart of an image pseudo-labeling method based on virtual adversarial training according to the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] This invention proposes an image pseudo-labeling method based on virtual adversarial training, which uses a trained label prediction model to predict the labels of image data, such as... Figure 1 The training process for the label prediction model specifically includes the following steps:

[0016] Obtain the training dataset, which includes labeled image data and unlabeled image data, and use the labeled image data to train the label prediction model to obtain the first label prediction model;

[0017] The labeled image data is strongly enhanced, and the enhanced data is input into the first label prediction model. The cross-entropy loss between the prediction result and its true label is used as the first loss.

[0018] The first label prediction model is used to predict the label of the unlabeled image data after weak enhancement, and the label with the confidence of the prediction result is greater than a set threshold is used as the pseudo label of the unlabeled image data.

[0019] The unlabeled image data with pseudo-labels is strongly enhanced. The first label prediction model is used to predict the label probability distribution of the enhanced image data. The cross-entropy between the pseudo-labels and the probability distribution is calculated as the second loss.

[0020] For each weakly augmented data in the training dataset, a virtual adversarial perturbation is computed and then applied to the original data.

[0021] The KL divergence between the distribution of the true or pseudo labels of all training data and the distribution of the predicted labels obtained by inputting the perturbated dataset into the first label prediction model is used as the third loss.

[0022] The label prediction model is trained using the first loss, the second loss, and the third loss to obtain the final label prediction model.

[0023] In this embodiment, labeled and unlabeled data are preprocessed, and pseudo-labels for the unlabeled data are obtained through self-training. A strongly enhanced version of the same unlabeled image is then predicted. Consistency regularization is applied between the pseudo-labels and the new predictions, and the consistency regularization loss is calculated. A perturbation is calculated for each weakly enhanced sample, generating virtual adversarial samples for virtual adversarial training. This virtual adversarial training is integrated into consistency learning, and the KL divergence between the weakly enhanced data and the virtual adversarial samples is calculated. The virtual adversarial training loss is combined with the supervision loss and the consistency regularization loss to form the final total loss function. This total loss function is used for model training, and the model parameters are updated using backpropagation and gradient descent algorithms to obtain a trained robust neural network model. The following six parts will describe the above scheme.

[0024] S1: Obtain labeled and unlabeled data and perform preprocessing.

[0025] 1000 and 4000 labeled data points were randomly selected from the SVHN and CIFAR-10 datasets, respectively, for training. The remaining data in the datasets were set as unlabeled data. For weak data augmentation preprocessing methods, images were flipped horizontally or vertically with a 50% probability; for strong data augmentation methods, image contrast was maximized.

[0026] S2: Obtain pseudo-labels for unlabeled data through self-training, and predict strongly enhanced versions of the same unlabeled image.

[0027] In this step, a first label prediction model needs to be trained using labeled data to obtain the label prediction model. Then, the pseudo-labels for unlabeled data are predicted based on the first label prediction model. The specific steps include:

[0028] S21. First, perform weak augmentation on the labeled data by horizontally or vertically flipping the image with a 50% probability, and then displaying the augmented labeled data. Input the model, obtain the predicted probability distribution, and calculate the model's loss on labeled data. :

[0029]

[0030] in, Represents the cross-entropy loss function. These are the actual labels in the label data. Model parameters Below, for labeled data that has undergone weak augmentation The predicted probability distribution; It is the number of labeled samples. It is the true label of the i-th labeled sample. It is a weak augmentation operation performed on the i-th labeled sample.

[0031] S22. Then, the trained model is used to predict the unlabeled data and generate pseudo-labels. The model outputs a probability distribution for each unlabeled sample, representing the probability of belonging to each category. A pseudo-label is only generated and retained for a data point if the model has a high-confidence prediction probability for a certain category of the unlabeled data. The initial confidence threshold is set from 0.5 to 0.7, and then adjusted based on performance on the validation set. The cross-entropy loss function for unlabeled data is defined as follows:

[0032]

[0033] in, It is the number of unlabeled samples; It is an indicator function used in conditional expressions. It takes the value 1 when the condition is true and 0 otherwise. This represents a predefined confidence threshold. Only when the predicted probability exceeds this threshold will the sample's prediction be used as a pseudo-label. Those skilled in the art generally adjust and select this threshold based on the performance of the validation set. The validation set does not participate in the model training process but is used to evaluate model performance. Hyperparameters such as learning rate and number of iterations are adjusted by observing whether the model is overfitting or underfitting so that the model achieves its best performance on the validation set.

[0034] S23. Perform strong enhancement on the same unlabeled image, maximize the image contrast, and then apply the enhanced unlabeled data. Input the model to obtain the probability distribution. After obtaining strong and weak enhancement versions of the same image, for samples that meet the conditions (i.e. ), calculate the cross-entropy loss between its pseudo-label and the strongly enhanced prediction, i.e., the second loss, expressed as:

[0035]

[0036] in, It is a strong data augmentation operation, which means performing more significant transformations on unlabeled data; The model represents the effect of strongly enhanced unlabeled images. The predicted probability distribution; This indicates that the model is given a weakly augmented unlabeled image. Category distribution at time; This indicates the search for the model parameter θ. and Cross-entropy between This indicates that the class with the highest probability is selected as the output from the probability distribution predicted by the model. Let represent the probability distribution of the i-th sample belonging to each category; sum the cross-entropy losses of all samples that meet the conditions, then divide by the total number of samples that meet the conditions, and add . To obtain the final regularization loss. This allows the model to make consistent predictions for the same unlabeled sample under different augmentations (weak and strong), and this consistent training helps improve the model's robustness to input perturbations.

[0037] In this embodiment, This indicates that the model is given a weakly augmented unlabeled image. Category distribution at time, This indicates that the class with the highest probability is selected as the output from the probability distribution predicted by the model.

[0038] S3: Generate corresponding virtual adversarial examples for each weakly enhanced sample and calculate the virtual training loss. Specifically, this includes:

[0039] S31: Calculate virtual adversarial perturbations, generate adversarial examples, and calculate virtual adversarial perturbations for each weakly enhanced example. The announcement is as follows:

[0040]

[0041] in, It can be labeled data or unlabeled data; Let represent the KL divergence, used to calculate the difference between two probability distributions p and q; the perturbation r aims to find a direction in which a small movement maximally alters the model's predictions; the model parameters θ represent the model parameters trained on the unperturbed data. This represents the model parameters obtained by training on perturbated data.

[0042] The process of calculating virtual adversarial perturbations involves the following steps: First, a gradient descent algorithm is selected. Then, a perturbation vector close to zero is initialized. The perturbation vector is iteratively updated to maximize the change in the model's output in the perturbation direction. The size of the perturbation must be limited to ensure it is small enough not to significantly alter the appearance of the sample. The calculated perturbation is then added to the original sample to generate the virtual adversarial sample.

[0043] S32: In virtual adversarial training, Local Perturbation Sensitivity (LDS) measures how sensitive the model's output is to small perturbations in the input data. A smaller LDS value indicates that the model is more robust to input perturbations. LDS is used as a regularization term, aiming to train the model to reduce its sensitivity to these small perturbations, thereby improving the model's generalization ability. Local Perturbation Sensitivity (LDS) is defined as follows:

[0044]

[0045] in, This represents the KL divergence between the predicted probability distributions of the computational model on the original samples and the samples after small perturbations. By minimizing the KL divergence, the model is encouraged to remain invariant to small perturbations of the input data. Indicates the magnitude of the disturbance; Indicates input sample In model parameters Given the given conditions, output the predicted probability of label y; Indicates input sample After perturbation factor and virtual counter-disturbance After the effect of θ, under the condition of the parametric model θ, the predicted probability of the label y is output.

[0046] Summing the Local Perturbation Sensitivity (LDS) over all input samples (labeled and unlabeled) yields the regularization term for the entire dataset, denoted as:

[0047]

[0048] S33: The original samples for virtual adversarial training are weakly augmented, so the resulting virtual adversarial training loss function is as follows:

[0049]

[0050] Virtual adversarial training is integrated into consensus learning by calculating the KL divergence between weakly augmented data and virtual adversarial samples. By calculating the KL divergence between all weakly augmented inputs (labeled and unlabeled data) and their corresponding virtual adversarial samples, virtual adversarial training is incorporated into consensus learning, ultimately yielding the target virtual adversarial training loss function as follows:

[0051]

[0052] in, The third loss is represented by the sum of the divergences between the labels of all samples and their predicted labels. If a sample is labeled data, its label is its true label; if a sample is unlabeled data, its label is its pseudo label. This represents the KL divergence, used to measure the difference between two probability distributions. For each labeled sample, calculate the probability distribution corresponding to its true label. and the probability distribution predicted by the model The KL divergence between them. For each unlabeled sample, calculate the probability distribution corresponding to its pseudo-label. and the probability distribution predicted by the model The KL divergence between them.

[0053] S4: Calculate the total loss function, use the total loss function to train the model, and obtain a trained robust neural network model.

[0054] The model is trained using the total loss function, and the model parameters are updated through backpropagation and gradient descent to minimize the total loss. The complete objective function is:

[0055]

[0056] Here, α and β are weighting coefficients that control the trade-off between supervised loss, regularization loss, and VAT loss. The optimal values ​​of the weighting coefficients α and β are determined through grid search, a method that searches for the best combination of hyperparameters from a given set of hyperparameters to improve model performance.

[0057] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An image pseudo-labeling method based on virtual adversarial training, characterized in that, The label prediction model, once trained, is used to predict the labels of image data. The training process for the label prediction model specifically includes the following steps: Obtain the training dataset, which includes labeled image data and unlabeled image data, and use the labeled image data to train the label prediction model to obtain the first label prediction model; The labeled image data is strongly enhanced, and the enhanced data is input into the first label prediction model. The cross-entropy loss between the prediction result and its true label is used as the first loss. The first label prediction model is used to predict the label of the unlabeled image data after weak enhancement, and the label with the confidence of the prediction result is greater than a set threshold is used as the pseudo label of the unlabeled image data. The unlabeled image data with pseudo-labels is strongly enhanced. The first label prediction model is used to predict the label probability distribution of the enhanced image data. The cross-entropy between the pseudo-labels and the probability distribution is calculated as the second loss. For each weakly augmented data in the training dataset, a virtual adversarial perturbation is computed and then applied to the original data. The KL divergence between the distribution of the true or pseudo labels of all training data and the distribution of the predicted labels obtained by inputting the perturbated dataset into the first label prediction model is used as the third loss. The label prediction model is trained using the first loss, the second loss, and the third loss to obtain the final label prediction model.

2. The image pseudo-label annotation method based on virtual adversarial training according to claim 1, characterized in that, Weak enhancement of image data includes flipping the image horizontally or vertically with a 50% probability.

3. The image pseudo-labeling method based on virtual adversarial training according to claim 1, characterized in that, Image data enhancement processes include maximizing image contrast.

4. The image pseudo-labeling method based on virtual adversarial training according to claim 1, characterized in that, In the process of obtaining pseudo-labels for unlabeled training data based on the first label model, the predicted label is set as the pseudo-label for the unlabeled training data if and only if the predicted probability of the pseudo-label is greater than the set confidence threshold.

5. The image pseudo-label annotation method based on virtual adversarial training according to claim 1, characterized in that, The first loss is represented as: in, This indicates the first loss; Probability distribution representing the true labels The probability distribution of labels obtained by the first label prediction model Cross-entropy between x l This represents labeled image data. This represents the network parameters of the first label prediction model. This indicates labeled image data. Perform a weak enhancement operation; This indicates the number of labeled image data. This represents the true label of the i-th labeled image data. This indicates that a weak enhancement operation is performed on the i-th labeled image sample.

6. The image pseudo-label annotation method based on virtual adversarial training according to claim 1, characterized in that, The second loss is represented as: in, Indicates the second loss; It is the number of unlabeled samples; It is an indicator function used in conditional expressions, taking the value 1 when the condition is true and 0 otherwise; γ represents a predefined confidence threshold. This indicates that the model is given a weakly augmented unlabeled image. Category distribution at time; This indicates the search for the model parameter θ. and Cross-entropy between This indicates that the class with the highest probability is selected as the output from the probability distribution predicted by the model; The model represents the strong enhancement of the unlabeled image. The predicted probability distribution This indicates that a strong data augmentation operation is performed on the input sample, and y represents the probability distribution of the output y given the input x. Indicates model parameters; This represents a predefined confidence threshold. It is a non-zero term.

7. The image pseudo-labeling method based on virtual adversarial training according to claim 1, characterized in that, The process of calculating virtual adversarial perturbations for weakly augmented data for each data point in the training dataset is represented as follows: in, This represents the virtual adversarial perturbation corresponding to the sample; This represents the KL divergence between the predicted probability distributions of the computational model on the original samples and on samples with slight perturbations. Indicates a given input sample and model parameters Under the given conditions, the probability distribution of the output label y; Indicates that in a given input sample After perturbation r and model parameters Given the given conditions, output the predicted probability of label y; For labeled image data Or unlabeled image data , This represents the model parameters before the disturbance.

8. The image pseudo-label annotation method based on virtual adversarial training according to claim 1, characterized in that, The third loss is represented as: in, Indicates the third loss. Represents the true label distribution of the i-th sample. After weak enhancement of the i-th sample, the result is input into the first prediction model to obtain the predicted label distribution. The divergence between them; This represents the pseudo-label distribution predicted by the first prediction model based on the input of the i-th sample. After weak enhancement of the i-th sample, the result is input into the first prediction model to obtain the predicted label distribution. The divergence between them.

9. The image pseudo-label annotation method based on virtual adversarial training according to claim 1, characterized in that, When training the label prediction model using the first loss, second loss, and third loss, the model parameters are updated using backpropagation and gradient descent algorithms with the aim of minimizing the total loss. The total loss is expressed as: in, Indicates the total loss; This indicates the first loss; Indicates the second loss; α represents the third loss; α and β are the weighting coefficients for the second and third losses, respectively.

Images