A semi-supervised active learning method for image classification
By employing a semi-supervised active learning method, and utilizing image augmentation and consistency regularization to obtain pseudo-labels, the problems of high cost and low accuracy in image classification are solved, and image classification that efficiently utilizes the features of unlabeled samples is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2022-12-26
- Publication Date
- 2026-04-17
AI Technical Summary
In existing technologies, image classification methods rely on large-scale labeled samples, resulting in high costs. At the same time, semi-supervised learning cannot fully utilize the features of unlabeled samples, leading to low accuracy of classification models.
A semi-supervised active learning method is adopted, which processes labeled and unlabeled samples through image enhancement technology, obtains pseudo-labels by using consistency regularization and entropy minimization, trains the model by combining fully supervised and semi-supervised loss, and manually labels unlabeled samples with large differences.
It improves the accuracy of image classification models, makes full use of unlabeled sample features, reduces the labeling cost of sample datasets, and enhances classification accuracy.
Smart Images
Figure CN115797702B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and in particular relates to a semi-supervised active learning method for image classification. Background Technology
[0002] Currently, image classification methods are mainly based on deep feature learning. These methods heavily rely on large-scale samples, and the high cost of constructing a large number of labeled samples limits their development. Applying active learning to image classification is one way to alleviate this problem. By using information-rich samples selected through active learning, high classification accuracy can be achieved with fewer labeled samples. Compared to supervised learning methods, this significantly reduces the labeling cost of the sample dataset. However, traditional active learning suffers from the technical limitation of using only labeled samples for training, ignoring unlabeled samples.
[0003] Semi-supervised learning only introduces unlabeled samples during model training, which cannot fully utilize the features of labeled samples, resulting in the technical problem of low accuracy in classification models.
[0004] Therefore, a new semi-supervised and active learning method needs to be proposed to solve the technical problems existing in the above-mentioned semi-supervised or active learning. Summary of the Invention
[0005] The purpose of this invention is to provide a semi-supervised active learning method for image classification, in order to solve the technical problems of existing technologies that cannot fully utilize the features of unlabeled samples and have low classification model accuracy.
[0006] The present invention adopts the following technical solution:
[0007] Embodiment 1 of the present invention provides a semi-supervised active learning method for image classification, including...
[0008] Obtain the first set of labeled sample images and the first set of unlabeled sample images;
[0009] A strong image enhancement process is performed on the first labeled sample image set to obtain the first processed sample image set;
[0010] Weak image enhancement processing is performed on the first unlabeled sample image set to obtain the second processed sample image set;
[0011] A strong image enhancement process is applied to the first unlabeled sample image set to obtain the third processed sample image set.
[0012] Predict the first label distribution probability from the first set of processed sample images;
[0013] Predict the second label distribution probability from the second set of processed sample images;
[0014] The probability distribution of the third label is obtained by predicting the third set of processed sample images;
[0015] Align the second label distribution probability with the first label distribution probability to obtain the aligned distribution probability.
[0016] Sharpen the aligned probability distribution to obtain pseudo-labels for unlabeled samples;
[0017] The first loss is calculated based on the probability distribution of the first label and the sample label. The classification model is then trained based on the first loss. The first loss represents the fully supervised loss.
[0018] The second loss is calculated based on the distribution probability of the pseudo-label and the third label. The consistency regularization term is trained based on the second loss. The second loss represents the semi-supervised loss.
[0019] Optionally, calculating the second loss based on the probability distributions of the pseudo-labels and the third label includes:
[0020] Calculate the cross-entropy between the third label distribution and the pseudo-label to obtain the second loss;
[0021] Optionally, the method further includes:
[0022] Calculate the JS divergence of the probability distribution of the third label and the probability distribution of the second label.
[0023] Optionally, the method further includes:
[0024] Based on the JS divergence, a preset number of unlabeled samples are selected from the first set of unlabeled sample images;
[0025] The unlabeled samples were manually labeled to obtain a second set of labeled sample images.
[0026] Optionally, the method further includes:
[0027] The second set of labeled sample images is appended to the first set of labeled sample images to form a new set of labeled sample images.
[0028] The classification model is trained based on the new set of labeled sample images.
[0029] Optionally, the total loss of the classification model is a weighted sum of the first loss and the second loss.
[0030] Embodiment 2 of the present invention provides a semi-supervised active learning classification device, the device comprising:
[0031] Image acquisition module: used to acquire the first labeled sample image set, the second unlabeled sample image set, and the first unlabeled sample image set;
[0032] First strong image enhancement module: used to perform strong image enhancement processing on the first labeled sample image set to obtain the first processed sample image set;
[0033] Weak image enhancement module: used to perform weak image enhancement processing on the first unlabeled sample image set to obtain the second processed sample image set;
[0034] The second strong image enhancement module is used to perform strong image enhancement processing on the first unlabeled sample image set to obtain the third processed sample image set.
[0035] The first prediction module is used to predict the first processed sample image set to obtain the first label distribution probability;
[0036] The second prediction module is used to predict the second processed sample image set to obtain the second label distribution probability;
[0037] The third prediction module is used to predict the probability distribution of the third label on the third processed sample image set.
[0038] Alignment module: used to align the label distributions of the second label distribution based on the first label distribution probability, to obtain the aligned distribution probability;
[0039] Sharpening module: used to sharpen the aligned probability distribution and obtain pseudo-labels for unlabeled samples;
[0040] First loss calculation module: used to calculate the first loss based on the probability distribution of the first label, and to train the classification model based on the first loss. The first loss represents the fully supervised loss.
[0041] The second loss calculation module is used to calculate the second loss based on the distribution probability of the pseudo-label and the third label, and to train the consistency regularization term based on the second loss. The second loss represents the semi-supervised loss.
[0042] Embodiment 3 of the present invention provides a semi-supervised active learning classification device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The device is characterized in that when the processor executes the computer program, it implements a semi-supervised active learning classification method based on image enhancement as described in any of the embodiments in Embodiment 1 above.
[0043] The beneficial effects of this invention are:
[0044] This invention proposes a semi-supervised active learning framework based on image enhancement, combining consistency regularization and entropy minimization in active learning. Entropy minimization is implicitly included while calculating the consistency regularization term. Specifically, pseudo-labels are obtained from unlabeled samples after weak image enhancement, and a temperature sharpening function is used to minimize entropy during pseudo-label acquisition. Furthermore, samples with significant differences are selected for manual labeling by calculating the JS divergence of the same unlabeled sample after weak and strong image enhancement. The model is trained using this new set of sample images, thereby enabling the classification model to fully utilize the features of unlabeled samples and improve classification accuracy. Attached Figure Description
[0045] Figure 1 This is a flowchart illustrating the steps of a semi-supervised active learning method for image classification provided in Embodiment 1 of the present invention.
[0046] Figure 2 This is a schematic diagram illustrating the steps of a semi-supervised active learning method for image classification provided in Embodiment 1 of the present invention.
[0047] Figure 3 Figure 2 shows a semi-supervised active learning device based on image enhancement provided in Embodiment 2 of the present invention.
[0048] Figure 4 Here is a schematic diagram of an image feature extraction network structure provided in Embodiment 2 of the present invention;
[0049] Figure 5 Figure 3 shows a semi-supervised active learning device based on image enhancement provided in Embodiment 3 of the present invention.
[0050] Figure 6 The figure shows the simulation results of a semi-supervised active learning method for image classification provided in Embodiment 1 of the present invention. Detailed Implementation
[0051] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0052] Combination Figure 1 Embodiment 1 of the present invention provides a semi-supervised active learning method for image classification, comprising:
[0053] Step S101: Obtain the first labeled sample image set and the first unlabeled sample image set;
[0054] It should be noted that the first labeled sample image set is the initial labeled sample image set, used for initial active learning; the first unlabeled sample image set is the unlabeled sample image set that needs to undergo weak and strong enhancement processing.
[0055] In one embodiment, during the construction of the image enhancement-based semi-supervised active learning model in this invention, a training set and a test set are first constructed. In this embodiment, the training set is formed by pre-selecting a certain number of unlabeled image training sample images and dividing the unlabeled image training sample image set into two parts. The first part is obtained by manually annotating a small number of images from the unlabeled image training sample image set to obtain a first labeled sample image set; the remaining unlabeled images are used as the first unlabeled sample image set.
[0056] In one specific embodiment, taking the CIFAR10 dataset as an example, the CIFAR10 training set contains 5000 images, and the test set contains 1000 images. The images are 32 rows × 32 columns in size, with 3 channels. n U represents the set of labeled sample images for the nth round of active learning. n Let U represent the set of unlabeled sample images from the nth round of active learning. The unlabeled sample image set U is... n 400 images were randomly selected and manually labeled. These labeled samples were used as the first labeled samples for the training set in the first round of active learning. A portion of the remaining unlabeled samples was then selected as the first unlabeled sample image set.
[0057] Step S102: Perform strong image enhancement processing on the first labeled sample image set to obtain the first processed sample image set;
[0058] In one embodiment, the first labeled sample image set is denoted as L = {(x b p b) Given the image set : b∈(1,...,B)}, perform strong image enhancement on the samples in the first labeled sample image set to obtain the first processed sample image set, denoted as . The specific processing method is shown in formula (1):
[0059]
[0060] StrongAugment refers to performing CTAugment on an image.
[0061] Step S103: Perform weak image enhancement processing on the first unlabeled sample image set to obtain the second processed sample image set;
[0062] In one embodiment, a subset of samples is selected from the unlabeled sample image set U to form a first unlabeled sample image set, denoted as U. Weak image enhancement is performed on the samples in the first labeled sample image set to obtain the second processed sample image set, denoted as . The specific processing method is shown in formula (2):
[0063]
[0064] Step S104: Perform strong image enhancement processing on the first unlabeled sample image set to obtain the third processed sample image set;
[0065] In one embodiment, after performing strong enhancement processing on the first unlabeled sample, a third processed sample image set is obtained, denoted as... The specific calculation method is as follows:
[0066]
[0067] in, This is the third set of sample images for processing.
[0068] Step S105: Predict the first processed sample image set to obtain the first label distribution probability;
[0069] In one embodiment, the first label distribution probability, denoted as p(y), is obtained by predicting the first labeled sample after strong image enhancement using a model.
[0070] Step S106: Predict the second processed sample image set to obtain the second label distribution probability;
[0071] In one embodiment, the second label distribution probability, denoted as q, is obtained by predicting the first unlabeled sample after weak enhancement using a model. pmodel (y|u;θ).
[0072] It should be noted that the present invention uses a prediction network to predict sample images, and the prediction result is the corresponding probability distribution.
[0073] Step S107: Predict the third processed sample image set to obtain the third label distribution probability;
[0074] Step S108: Align the label distribution of the second label distribution with the first label distribution probability to obtain the aligned distribution probability;
[0075] In one embodiment, the average value of the prediction results for the first unlabeled sample is calculated by the model and denoted as . And p(y), q and The distribution results of the labels of the first unlabeled sample and the distribution results of the labels of the first labeled sample are aligned using the processing method described in formula (4) to obtain the aligned distribution probability, denoted as . This ensures that the classification probability obtained after predicting the samples is consistent with the true distribution of the samples. The specific processing method is shown in formula (4):
[0076]
[0077] Step S109: Sharpen the aligned probability distribution to obtain pseudo-labels;
[0078] In one instance, a sharpening function is used to sharpen the aligned probability distribution, highlighting the main classification probabilities and reducing the probability of misclassification, thereby reducing the entropy of the label distribution. This allows the model to generate low-entropy predictions and makes the obtained data suitable for use in the model's calculations, ultimately yielding pseudo-labels, denoted as r. The specific processing method for obtaining pseudo-labels is shown in formula (5):
[0079]
[0080] Where T is a hyperparameter.
[0081] Step S1010: Calculate the first loss based on the probability distribution of the first label, and train the classification model based on the first loss. The first loss represents the fully supervised loss.
[0082] In one embodiment, during the training of the model using the first labeled sample after strong image enhancement, fully supervised learning is performed. Specifically, the first loss is calculated based on the probability distribution of the first label. This first loss is a pseudo-fully supervised loss, and the specific calculation method is shown in formula (6):
[0083]
[0084] in, This is a loss due to oversight.
[0085] Step S1011: Calculate the second loss based on the distribution probability of the pseudo-label and the third label, and train the consistency regularization term based on the second loss. The second loss represents the semi-supervised loss.
[0086] Optionally, calculating the second loss based on the probability distributions of the pseudo-labels and the third label includes:
[0087] The second loss is obtained by calculating the probability distribution of the third label and the cross-entropy of the pseudo-label.
[0088] In one embodiment, the model is trained using unlabeled samples after weak image enhancement. This process is semi-supervised learning. The semi-supervised loss is calculated based on the consistency regularization principle. By calculating the cross-entropy between the label distribution of the first labeled sample of the strongly enhanced image and the label distribution of the first unlabeled sample of the weakly enhanced image, the label of the first labeled sample of the strongly enhanced image is closer to the pseudo label. This further trains the model and improves the classification accuracy. The specific calculation method of the semi-supervised loss is as shown in formula (7):
[0089]
[0090] in, The cross-entropy is the probability distribution of the third label and the pseudo-label. This is a semi-supervised loss.
[0091] Optionally, the method further includes:
[0092] Calculate the JS divergence between the third label distribution result and the second label distribution probability.
[0093] The model is used to predict the third set of processed sample images to obtain the prediction results, and the JS divergence between the third label distribution result and the second label distribution probability is calculated using formula (8):
[0094]
[0095] Where x1 is the first unlabeled sample and x2 is the second unlabeled sample. The JS divergence of the first unlabeled sample. is the JS divergence of the second unlabeled sample, and IS(X1‖X2) is the JS divergence between the third label distribution result and the second label distribution probability.
[0096] Optionally, the method further includes:
[0097] Based on the JS divergence, a preset number of unlabeled samples are selected from the first unlabeled sample image;
[0098] The unlabeled samples were manually labeled to obtain a second set of labeled sample images.
[0099] In one embodiment, by calculating the JS divergence of the first unlabeled sample image set and the second unlabeled sample image set, the top-k unlabeled samples with the largest JS divergence are selected from the first unlabeled sample image set as the sample image set to be labeled in the next round, i.e., the second unlabeled sample image set, denoted as S. n .
[0100] This process utilizes a query strategy to unlabeled sample image set U n We selected a batch of samples with the most information and manually labeled them, which improved the efficiency of adding labels to the samples.
[0101] It should be noted that this invention introduces consistency regularization and entropy minimization, which are commonly used in semi-supervised learning, into active learning. It uses image enhancement to achieve semi-supervised learning and uses the difference between weak and strong image enhancement samples to select the samples that have the greatest effect on improving the model.
[0102] Optionally, the method further includes:
[0103] The second set of labeled sample images is appended to the first set of labeled sample images to form a new set of labeled sample images.
[0104] The classification model is trained based on the new set of labeled sample images.
[0105] In one embodiment, by using the labeled sample image set S n Added to the first labeled sample image set L n A new set of labeled sample images L is formed. n+1 The specific processing method is shown in formula (9):
[0106] L n+1 =L n ∪S n (9)
[0107] Among them, L n+1 For the new set of labeled sample images.
[0108] The model is then trained using the new labeled sample set.
[0109] Optionally, the total loss of the classification model is a weighted sum of the first loss and the second loss.
[0110] In one embodiment, after the model is initially trained using a first set of labeled sample images and a first set of unlabeled sample images, the model is further trained again using a newly formed set of labeled sample images. In the next round of model training, the model's loss value is continuously updated. The updated model's total loss is a weighted sum of the first loss and the second loss, as shown in formula (10).
[0111]
[0112] Here, λ is a hyperparameter representing the relative weights of the unlabeled loss, which is set according to the specific circumstances.
[0113] It should be noted that the classification model is repeatedly trained using the above method until the preset classification accuracy is achieved.
[0114] Embodiment 2 of the present invention provides a semi-supervised active learning classification device, such as... Figure 3 As shown, the device 30 includes:
[0115] Image acquisition module 301: used to acquire a first set of labeled sample images and a first set of unlabeled sample images;
[0116] First strong image enhancement module 302: used to perform strong image enhancement processing on the first labeled sample image set to obtain the first processed sample image set;
[0117] Weak image enhancement module 303: used to perform weak image enhancement processing on the first unlabeled sample image set to obtain a second processed sample image set;
[0118] The second strong image enhancement module 304 is used to perform strong image enhancement processing on the first labeled sample image set to obtain the third processed sample image set.
[0119] The first prediction module 305 is used to predict the first processed sample image set to obtain the first label distribution probability;
[0120] The second prediction module 306 is used to predict the second processed sample image set to obtain the second label distribution probability;
[0121] The third prediction module 307 is used to predict the third processed sample image set to obtain the third label distribution probability;
[0122] Alignment module 308: used to align the label distribution of the second label distribution probability with the first label distribution probability to obtain the aligned distribution probability;
[0123] Sharpening module 309: Used to sharpen the aligned probability distribution and obtain pseudo-labels;
[0124] First loss calculation module 3010: used to calculate the first loss based on the probability distribution of the first label, and to train the classification model based on the first loss. The first loss represents the fully supervised loss.
[0125] The second loss calculation module 3011 is used to calculate the second loss based on the distribution probability of the pseudo-label and the third label, and to train the consistency regularization term based on the second loss. The second loss represents the semi-supervised loss.
[0126] It should be noted that the semi-supervised active learning method for image classification implemented by the device 30 is consistent with the method steps provided in Embodiment 1 above, and will not be repeated here.
[0127] Embodiment 3 of the present invention provides a semi-supervised active learning classification device 400, such as... Figure 5 As shown, it includes a memory 410, a processor 420, and a computer program stored in the memory and executable on the processor 420. The feature is that when the processor executes the computer program, it implements a classification method based on image enhancement semi-supervised active learning as described in any of the above embodiments.
[0128] In one embodiment, the present invention provides an analysis of experimental simulation results based on the image enhancement-based semi-supervised active learning classification method provided in the embodiment.
[0129] It should be noted that the comparison experiment between this example and other classic active learning methods is as follows: Figure 6 As shown, this experiment selects three representative active learning methods—Lloss, CoreSet and VALL, UncertainGCN, TA-VAAL, and Random—to verify the performance of this example. All experiments were conducted using Python 3.7, with an Intel Core™ i9-10980XE CPU and an NVIDIA GeForce RTX 2080TI GPU.
[0130] from Figure 6 As can be seen, the highest line in the figure represents the simulation results of the image-enhanced semi-supervised active learning classification method provided by this invention. Compared with other active learning methods, the method in this example achieves the best accuracy in most cases. This demonstrates the effectiveness of the method of this invention. It can be observed that on the CIFAR10 dataset, using only 4000 labeled samples for training, the method of this invention achieves an accuracy of 85.99% on the test set. Compared to the random sampling method, the method of this invention improves accuracy by 10.70%; compared to the CoreSet method, it improves accuracy by 7.25%; compared to the Lloss method, it improves accuracy by 7.01%; compared to the VAAL method, it improves accuracy by 11.63%; compared to the UncertainGCN method, it improves accuracy by 8.52%; and compared to the TA-VAAL method, it improves accuracy by 5.90%. Meanwhile, it can be observed that on this dataset, the method of the present invention performs far better than other methods. This is because the method of the present invention introduces semi-supervised learning into active learning, and the query strategy can filter out samples that have a greater impact on the model. Therefore, the method of the present invention achieves better results.
Claims
1. A semi-supervised active learning method for image classification, characterized in that, include Obtain the first set of labeled sample images and the first set of unlabeled sample images; The first labeled sample image set is subjected to strong image enhancement processing to obtain the first processed sample image set; The first unlabeled sample image set is subjected to weak image enhancement processing to obtain the second processed sample image set; The first unlabeled sample image set is subjected to strong image enhancement processing to obtain the third processed sample image set; Predict the first label distribution probability from the first processed sample image set; Predict the second label distribution probability from the second processed sample image set; The third processed sample image set is used to predict the probability distribution of the third label; Based on the first label distribution probability, the second label distribution probability is aligned to obtain the aligned distribution probability; The aligned probability distribution is sharpened to obtain pseudo-labels for unlabeled samples; The first loss is calculated based on the probability distribution of the first label, and the classification model is trained based on the first loss. The first loss represents the fully supervised loss. The second loss is calculated based on the distribution probability of the pseudo-label and the third label, and the consistency regularization term is trained based on the second loss. The second loss represents the semi-supervised loss. Calculate the JS divergence between the probability distribution of the third label and the probability distribution of the second label; Based on the JS divergence, a preset number of unlabeled samples are selected from the first set of unlabeled sample images; The unlabeled samples are manually labeled to obtain a second set of labeled sample images; The second set of labeled sample images is appended to the first set of labeled sample images to form a new set of labeled sample images. The classification model is trained based on the new set of labeled sample images.
2. The semi-supervised active learning method for image classification of claim 1, wherein, The second loss is calculated based on the probability distributions of the pseudo-labels and the third label, including: The second loss is obtained by calculating the probability distribution of the third label and the cross-entropy of the pseudo-label.
3. The semi-supervised active learning method for image classification of claim 1, wherein, The total loss of the classification model is a weighted sum of the first loss and the second loss.
4. A semi-supervised active learning classification apparatus, characterized by, The device includes: Image acquisition module: used to acquire the first set of labeled sample images and the first set of unlabeled sample images; First strong image enhancement module: used to perform strong image enhancement processing on the first labeled sample image set to obtain the first processed sample image set; Weak image enhancement module: used to perform weak image enhancement processing on the first unlabeled sample image set to obtain a second processed sample image set; The second strong image enhancement module is used to perform strong image enhancement processing on the first unlabeled sample image set to obtain the third processed sample image set. The first prediction module is used to predict the first processed sample image set to obtain the first label distribution probability; The second prediction module is used to predict the second processed sample image set to obtain the second label distribution probability; The third prediction module is used to predict the third processed sample image set to obtain the third label distribution probability; Alignment module: used to align the label distribution of the second label distribution probability with the first label distribution probability to obtain the aligned distribution probability; Sharpening module: used to sharpen the aligned probability distribution and obtain pseudo-labels for unlabeled samples; First loss calculation module: used to calculate the first loss based on the probability distribution of the first label, and to train the classification model based on the first loss, wherein the first loss represents the fully supervised loss; The second loss calculation module is used to calculate the second loss based on the distribution probability of the pseudo-label and the third label, and to train the consistency regularization term based on the second loss. The second loss represents the semi-supervised loss. Calculate the JS divergence between the probability distribution of the third label and the probability distribution of the second label; Based on the JS divergence, a preset number of unlabeled samples are selected from the first set of unlabeled sample images; The unlabeled samples are manually labeled to obtain a second set of labeled sample images; The second set of labeled sample images is appended to the first set of labeled sample images to form a new set of labeled sample images. The classification model is trained based on the new set of labeled sample images. 5.A classification apparatus of semi-supervised active learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements a classification method based on image enhancement and semi-supervised active learning as described in any one of claims 1-3.