A method for semi-supervised learning using adaptive dual thresholds
By using adaptive dual-threshold filtering and similarity loss to optimize the utilization of unlabeled data, the problem of imbalance between noisy samples and learning states in semi-supervised learning is solved, achieving more efficient information mining and model performance improvement.
Patent Information
- Application Number
- CN202210461817.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-28
- Publication Date
- 2025-12-16
- Estimated Expiration
- 2042-04-28
AI Technical Summary
Existing semi-supervised learning methods suffer from problems such as the introduction of noisy samples, imbalance of learning states between different categories, and insufficient information utilization when using unlabeled data, especially failing to effectively extract information from similar unlabeled data.
An adaptive dual-threshold approach is used to filter unlabeled data. Different learning strategies are designed by combining fixed thresholds and class-adaptive thresholds. Information between unlabeled data is mined through similarity loss, and data augmentation and consistency regularization are used to optimize the model.
It improves the utilization rate of unlabeled data, reduces the impact of noisy samples, balances the model's learning state for different categories, makes full use of the effective information in unlabeled data, and improves the model's recognition performance.
Smart Images
Figure CN114842294B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of semi-supervised learning technology, and in particular to a method for semi-supervised learning using adaptive dual thresholds. Background Technology
[0002] Deep learning strategies have achieved tremendous success in fields such as image classification and image segmentation. However, this relies on supervised learning using large amounts of labeled data, which also means a significant need for human resources. This is especially true when expert evaluation is required in certain areas, consuming substantial financial resources. Semi-supervised learning aims to train models using both limited labeled data and a large amount of unlabeled data to achieve better performance.
[0003] A key challenge in semi-supervised learning is extracting sufficient information from unlabeled data. Current popular semi-supervised learning methods use a fixed confidence threshold to force the model to focus only on unlabeled data with high confidence, meaning that unlabeled data that fails to reach the threshold is ignored. Furthermore, since the model exhibits different learning states for different classes at different training stages, using the same high confidence threshold for all classes is unfair.
[0004] Existing technical solution 1: Sun Shuanzhu, Semi-supervised learning industrial data classification algorithm based on similarity, 2021.
[0005] This technology addresses the challenge of effectively utilizing a large amount of unlabeled data in real-world scenarios by proposing a semi-supervised learning algorithm based on data similarity matching. This method combines prior knowledge with unsupervised learning to calculate the similarity between unlabeled data and a small amount of labeled data, thereby expanding the minority class sample. The constructed dataset is then used for model training, improving the model's performance in identifying the minority class.
[0006] Existing technical solution 2: Gan Yuwan, Research on image classification algorithm based on semi-supervised learning, 2021.
[0007] In pseudo-label-based semi-supervised learning algorithms, model performance is limited in the early stages of training, making pseudo-label errors prone to occur. If these errors are not corrected promptly, they can negatively impact the model. To address these shortcomings and correct erroneous pseudo-labels, this paper proposes a semi-supervised classification algorithm based on weighted pseudo-label samples and mutual learning. This algorithm is based on an ensemble deep network model. First, labeled noisy data is used for diversity training, and unlabeled data is input into the trained model, with the output values serving as labels. Then, a screening and weighting strategy based on pseudo-label data is used to select high-confidence pseudo-label data, which, along with real labeled data, forms a new set for weighted training to update the model. Finally, a mutual learning strategy is used to enhance the predictive consistency among classifiers. By alternating between diversity fine-tuning and mutual learning, the optimal balance between diversity and consistency is found, improving the accuracy of pseudo-label predictions.
[0008] Existing technical solution 3: Shao Weizhi, Semi-supervised learning algorithm based on consistency regularization and entropy minimization, 2021.
[0009] This technique, based on the principles of entropy minimization and consistency regularization, designs a method to generate low-entropy pseudo-labels by integrating multi-perspective information through the combined action of multiple stages of the model, and utilizes consistency regularization to optimize the model's classification performance. This technique uses the combined action of multiple variants of the same model to generate pseudo-labels, enabling the generation of pseudo-labels to obtain complementary information from multiple perspectives, making the judgment of unlabeled data more reliable. This novel pseudo-label acquisition method ensures that the network can benefit from different angles and time periods, gradually improving the accuracy of the pseudo-labels.
[0010] Disadvantages of existing technology:
[0011] The shortcomings of the existing technical solution 1 are: 1) This solution only considers the information between different similar data, which cannot fully utilize unlabeled data. 2) Data augmentation technology is a crucial component in unsupervised learning, but this solution does not apply data augmentation technology when learning from unlabeled data;
[0012] The shortcomings of existing technical solution 2 are: 1) This solution does not consider the model's learning status for different categories when determining pseudo-labels for unlabeled data, which will result in poor prediction performance for classes that are difficult to identify. 2) This solution only learns from the filtered unlabeled data, which will cause the model to ignore the effective information in the unlabeled data that failed the filtering. Therefore, it cannot learn from unlabeled data sufficiently and effectively.
[0013] The drawbacks of the existing technical solution 3 are: 1) This solution adopts the same learning strategy for all unlabeled data, which introduces noisy samples into the model during the learning process of unlabeled data. 2) This solution only focuses on a single unlabeled data point and does not consider the information between different unlabeled data points. Summary of the Invention
[0014] (a) Technical problems to be solved
[0015] To address the shortcomings of existing technologies, this invention provides a semi-supervised learning method using adaptive dual thresholds. Besides using a fixed threshold to identify high-confidence unlabeled data, the adaptive dual thresholds also utilize unlabeled data whose predicted values are less than the fixed threshold but greater than the extracted class-adaptive threshold. Two learning strategies are designed to learn from these two types of unlabeled data respectively. For highly similar unlabeled data, a novel similarity loss is proposed to further mine information between similar unlabeled data.
[0016] (II) Technical Solution
[0017] This invention provides the following technical solution: a method for semi-supervised learning using adaptive dual thresholds, comprising the following steps:
[0018] S1. For labeled data, perform weak augmentation and then input it into the model. Calculate the cross-entropy loss between the prediction results and their respective labels.
[0019] S2. When training the model using unlabeled data, extract an adaptive threshold for each class.
[0020] S3. An adaptive dual threshold is formed by combining a fixed threshold and a class-adaptive threshold. The adaptive dual threshold is used to filter unlabeled data, and different learning strategies are adopted for different unlabeled data.
[0021] S4. Based on existing technologies, a new similarity loss is proposed to further mine information between similar unlabeled data.
[0022] First, to extract class-adaptive thresholds from the labeled data input to the model, perform the following operations:
[0023]
[0024]
[0025] Where c = argmax(y) represents the true class of x, and w(x) represents the weak augmentation operation performed on x. It is the class-adaptive threshold for category c and its initial value is set to 0.95.
[0026] Furthermore, in order to extract more effective information from unlabeled data and balance the model's learning progress for each category, a fixed threshold τ and a class-adaptive threshold are combined. An adaptive dual threshold is constructed, and while using the adaptive dual threshold to filter unlabeled data, different learning strategies are applied to different unlabeled data.
[0027] To minimize entropy, a sharpen operation is used during the generation of pseudo-labels, which can be specifically represented as:
[0028]
[0029] Where p represents the input distribution, and T is a predefined hyperparameter that can adjust the sharpening degree of the input distribution.
[0030] For the unlabeled data input to the model, a fixed threshold τ is first used to determine whether to use it to calculate the loss:
[0031]
[0032] in Representative model for w(u) b The output of ) for q b Performing an exponential moving average (EMA) operation yields... and This indicates the unlabeled data u b Perform strong augmentation, where H represents the calculation of cross-entropy loss for the two distributions.
[0033] For unlabeled data that fails to pass the fixed threshold, we then use a class-adaptive threshold for further filtering and employ a different training method than described above:
[0034]
[0035] here represent The class-adaptive threshold for the class to which the maximum value belongs. When the unlabeled data satisfies... At times, will be Applying a more moderate L2 loss to the target is because when using When used as an objective, it not only achieves the goal of minimizing entropy, but also carries more information to enhance the model's generalization ability. Most importantly, it is not as sensitive to noisy samples as one-hot labels.
[0036] Furthermore, in the semi-supervised learning process, the similarity loss is calculated using the relationships between different unlabeled data. Weakly augmented unlabeled data with sufficiently high confidence is used as the "anchor," and strongly augmented versions of unlabeled data with sufficiently high similarity are aligned with it. The similarity loss is defined as:
[0037]
[0038] This represents the confidence threshold for a single unlabeled data point. The Sim function calculates the similarity between two distributions using a similarity threshold between different unlabeled data.
[0039] Compared with existing technologies, this invention provides a method for semi-supervised learning using adaptive dual thresholds, which has the following advantages:
[0040] 1. Compared with the prior art 1, the advantages of the present invention are that it not only filters out the introduction of noise samples when pairing similar unlabeled data, but also learns individual unlabeled data by combining adaptive dual thresholds, data augmentation technology and consistency regularization, thus ensuring full utilization of unlabeled data.
[0041] 2. Compared with the prior art 2, the advantage of the present invention is that by using adaptive dual thresholds to confirm pseudo-labels for unlabeled data, it not only fully extracts information from the unlabeled data, but also balances the learning state of the model for different classes.
[0042] 3. Compared with the prior art 3, the advantages of the present invention are that it reduces the introduction of noise samples by using adaptive dual thresholds and focuses on the information between different unlabeled data by using similarity loss, thus making full use of the effective information in the unlabeled data.
[0043] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description
[0044] Figure 1 This invention provides an overview of a semi-supervised learning method (ADT-SSL) using adaptive dual thresholds.
[0045] Figure 2 This is a schematic diagram illustrating the class adaptive threshold extraction method in the ADT-SSL of this invention.
[0046] Figure 3 This is a detailed schematic diagram illustrating the similarity loss in the ADT-SSL of this invention. Detailed Implementation
[0047] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0048] The overall algorithm of this invention is attached. Figure 1 Algorithm 2. A method for semi-supervised learning using adaptive dual thresholds, comprising the following steps:
[0049] S1. For labeled data, perform weak augmentation and then input it into the model. Calculate the cross-entropy loss between the prediction results and their respective labels.
[0050] S2. When training the model using unlabeled data, extract an adaptive threshold for each class, see Appendix. Figure 2 ;
[0051] S3. An adaptive dual threshold is formed by combining a fixed threshold and a class-adaptive threshold. The adaptive dual threshold is used to filter unlabeled data and different learning strategies are adopted for different unlabeled data. Specifically, when the maximum value in the predicted distribution of a weakly augmented version of an unlabeled data is greater than the fixed threshold, the distribution is converted into a "one-hot" label and used as an "anchor". The cross-entropy loss is calculated with the model's prediction of its strongly augmented version. If the prediction cannot pass the fixed threshold, the class-adaptive threshold is used to filter it again. If the maximum value in the predicted distribution is greater than the adaptive threshold of the class, a sharpening operation is performed on it and used as an "anchor". The L2 loss is calculated with the model's prediction of its strongly augmented version.
[0052] S4. Based on existing technologies, a novel similarity loss is proposed to improve the consistency of the model's output on similar unlabeled data. See Appendix for details. Figure 3 .
[0053] Extraction class adaptive threshold
[0054] For the labeled data input to the model, perform the following operations:
[0055]
[0056]
[0057] Where c = argmax(y) represents the true class of x, and w(x) represents the weak augmentation operation performed on x. The class-adaptive threshold for category c is set to 0.95 initially. The process first determines whether the model's assessment of w(x) is correct. Then execute To check whether the maximum value in the model's prediction vector for w(x) is less than Only when both of the above conditions are met will it be... Assign to See attached Figure 2 .
[0058] Since semi-supervised learning utilizes both labeled and unlabeled data to train the model, this process exists throughout the entire training process. The detailed process of extracting the class-adaptive threshold can be found in Algorithm 1. It is worth noting that... It doesn't always show an upward or downward trend; in each cycle, each class is recalculated. And when Greater than When will Assign to
[0059] Adaptive dual threshold specific application process
[0060] To extract more effective information from unlabeled data and balance the model's learning progress for each class, a fixed threshold τ and a class-adaptive threshold are combined. An adaptive dual threshold is constructed, and while using the adaptive dual threshold to filter unlabeled data, different learning strategies are applied to different unlabeled data.
[0061] Inspired by entropy minimization, the Sharpen operation is used in the process of generating pseudo-labels. Specifically, it can be represented as:
[0062]
[0063] Here, p represents the input distribution. T is a predefined hyperparameter that adjusts the sharpening effect on the input distribution. As T→0, the function's output gets closer to the one-hot distribution, using this technique to encourage the model to output low-entropy predictions.
[0064] For the unlabeled data input to the model, a fixed threshold τ is first used to determine whether to use it to calculate the loss:
[0065]
[0066] in Representative model for w(u) b The output of q. b Performing an exponential moving average (EMA) operation yields... and This indicates the unlabeled data u b Perform strong augmentation, where H represents the calculation of cross-entropy loss for the two distributions.
[0067] τ is a relatively high fixed threshold, so a class-adaptive threshold is used. To re-filter the unlabeled data that did not pass τ, a different learning strategy was adopted than that described above:
[0068]
[0069] here represent The class-adaptive threshold for the class to which the maximum value belongs. When the unlabeled data satisfies... At times, will be A more moderate L2 loss is applied to the target. Because when... When used as an objective, it not only achieves the goal of minimizing entropy, but also carries more information to enhance the model's generalization ability. Most importantly, it is not as sensitive to noisy samples as one-hot labels.
[0070] Similar losses
[0071] In semi-supervised learning, to fully utilize unlabeled data, the similarity loss is calculated based on the relationships between different unlabeled data sets. Weakly augmented unlabeled data with sufficiently high confidence is used as an "anchor," and strongly augmented versions of unlabeled data with sufficiently high similarity are aligned with it. The similarity loss is defined as:
[0072]
[0073] This represents the confidence threshold for a single unlabeled data point. To establish a similarity threshold between different unlabeled data sets, the Sim function calculates the similarity between two distributions. For Sim, the Bach distance is used, and the output range is [0,1]. The calculation process can be expressed as follows:
[0074]
[0075] set up At this point, the unlabeled data that meets the confidence threshold has a distribution close to one-hot encoding. Other unlabeled data that are sufficiently similar also have a sufficiently high confidence level. Transform into a one-hot distribution and use as The objective is to calculate the cross-entropy loss; detailed steps can be found in the appendix. Figure 3 Given a distribution It is the model's prediction result for weakly augmented unlabeled data. First, regarding... Executing the sharpen operation yields Next, we test its confidence level. If the maximum value in the dataset exceeds the confidence threshold τ, it is converted into a one-hot label and used as the "anchor". The similarity to another unlabeled data tuple is then evaluated. If the similarity exceeds a similarity threshold... At that time, with To target p m (y|v r Calculate the cross-entropy loss to bring v closer. r The distance v from its cluster center r This represents a strongly augmented version of the unlabeled data, and v is used when both thresholds cannot be satisfied simultaneously. r They will not be included in the training.
[0076] In summary, the overall loss consists of four parts, as seen in Algorithm 2. These are the supervised losses. Unsupervised loss and Finally, similar losses
[0077]
[0078]
[0079]
[0080]
[0081] here It calculates the cross-entropy loss for weakly augmented labeled data. and Pay attention to the consistency between different augmentation forms of the same unlabeled data.
[0082] This invention's approach is validated by comparing it with existing semi-supervised learning techniques. ADT-SSL (a method for semi-supervised learning using adaptive dual thresholds) was tested on CIFAR-10, SHVN, and CIFAR-100. ADT-SSL demonstrated the best performance on CIFAR-100, and its performance on CIFAR-10 and SHVN was almost identical to that of fully supervised learning. The results are shown in Appendix 1 and Appendix 2. Specifically, on CIFAR-100, with 10,000 labels and a smaller model, ADT-SSL outperformed popular methods by nearly 3.5%. When using a larger model, the improvement was 1.2%. Experimental results show that ADT-SSL performs exceptionally well with complex datasets and adapts well to smaller network models.
[0083]
[0084]
[0085]
[0086] 1: Input: Labeled samples and one-hot labels Unlabeled samples Sharpening level T, weak augmentation amount K, strong augmentation amount K strong Confidence threshold τ, similarity threshold τ s Adaptive threshold
[0087]
[0088]
[0089] Appendix 1: Test accuracy on CIFAR-100 using Wide ResNet 28-2 and Wide ResNet 28-8 respectively.
[0090]
[0091] Appendix 2: Test accuracy for CIFAR-10, SHVN, and CIFAR-100. Wide ResNet 28-2 was used as the network model for CIFAR-10 and SHVN. Wide ResNet 28-8 was used as the network model for CIFAR-100.
[0092]
[0093] Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
[0094] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0095] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.
Claims
1. A method for semi-supervised learning using adaptive dual thresholds for image data labeling and classification, comprising the following steps: S1. For labeled data, perform weak augmentation and then input it into the model. Calculate the cross-entropy loss between the prediction results and their respective labels. S2. When training the model using unlabeled data, extract an adaptive threshold for each category. S3. An adaptive dual threshold is formed by combining a fixed threshold and a class-adaptive threshold. The adaptive dual threshold is used to filter unlabeled data, and different learning strategies are adopted for different unlabeled data. S4. Propose a new similarity loss; For S3, specifically, when the maximum value in the model's prediction of a weakly augmented version of unlabeled data is greater than a fixed threshold, the prediction distribution of the weakly augmented version of the unlabeled data is converted into a one-hot label and used as an "anchor". The cross-entropy loss is calculated with the model's prediction of its strongly augmented version. If the prediction cannot pass the fixed threshold, it will be filtered again using a class-adaptive threshold. If the maximum value in the prediction distribution is greater than the class-adaptive threshold, a sharpen operation is performed on it and used as an "anchor". The L2 loss is calculated with the model's prediction of its strongly augmented version. For S4, the new similarity loss is defined as: Where τ represents the confidence threshold for a single unlabeled data point. The Sim function calculates the similarity between two distributions using a similarity threshold between different unlabeled data.
2. The method for semi-supervised learning using adaptive dual thresholds according to claim 1, characterized in that, During the execution of S2, for the labeled data input to the model, the following operations are performed: Where c = argmax(y) represents the true class of x, and w(x) represents the weak augmentation operation performed on x. It is the class-adaptive threshold for category c and its initial value is set to 0.
95.
3. The method for semi-supervised learning using adaptive dual thresholds according to claim 2, characterized in that, For the unlabeled data input to the model, a fixed threshold τ is first used to determine whether to use it to calculate the loss: in Representative model for w(u) b The output of ) is used to perform an exponential moving average (EMA) operation on qb to obtain and H represents performing strong augmentation on the unlabeled data ub, and H represents calculating the cross-entropy loss over the two distributions; Then, for unlabeled data that fails to pass the fixed threshold, a class-adaptive threshold is used for further filtering, and a different training method is adopted than that described above: here represent The class-adaptive threshold of the class to which the maximum value belongs is used when the unlabeled data satisfies... At times, will be Apply a moderate L2 loss to the target.