Model training using teacher-student learning model

By employing a teacher-student model and an adaptive incremental threshold adjustment method, the problems of increased noise and insufficient stopping criteria in semi-supervised learning are addressed, thereby improving the accuracy and stability of model training.

CN112132170BActive Publication Date: 2024-11-26INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010527266.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-06-25
Filing Date
2020-06-11
Publication Date
2024-11-26
Estimated Expiration
2040-06-11

AI Technical Summary

Technical Problem

Existing semi-supervised learning techniques suffer from problems such as increased noise and lack of effective stopping criteria during model training, resulting in poor model performance.

Method used

The model is trained using a teacher-student model. An initial classifier is trained using a labeled dataset, and a pseudo-labeled dataset is generated by selecting an unlabeled dataset using a bootstrap sampling technique. Roles are assigned to rotate the teaching sample dataset for retraining, and an adaptive incremental adjustment threshold is used to achieve the stopping criterion.

Benefits of technology

This improved the accuracy and stability of model training, reduced noise, and ensured the effectiveness and accuracy of the training process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112132170B_ABST
    Figure CN112132170B_ABST
Patent Text Reader

Abstract

The present disclosure relates to model training using a teacher-student learning model. A method, system, and computer-readable storage medium for model training are provided. The method may include training a first classifier, a second classifier, and a third classifier using a subset of a labeled data set. The method may also include predicting a pseudo-labeled data set from an unlabeled data set using the first classifier, the second classifier, and the third classifier. The method also includes assigning roles to the first classifier, the second classifier, and the third classifier. The method may also include selecting a teaching sample data set from the pseudo-labeled data set based on the role assigned to the third classifier, wherein the third classifier is assigned a student role. The method may also include retraining the third classifier using the teaching sample data set and a subset of the labeled data set.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] The present disclosure relates to model training, and more particularly, to model training based on a semi-supervised training method using a teacher-student paradigm technique.

[0002] Typical semi-supervised training methods include self-training, co-training, tri-training, and tri-training with disagreement. Co-training involves two classifiers learning from each other to achieve better performance. Tri-training involves adding training samples to a third classifier if the other two classifiers reach a consensus on the predicted label. In tri-training with disagreement, a condition is imposed where the two base classifiers must agree on the prediction exceeding a static confidence threshold. The third classifier must disagree with the other two base classifiers. Once the conditions are met, training iterations can be performed. Summary of the invention

[0003] Various embodiments adopt a teacher-student learning mode for model training. The method may include training a first classifier, a second classifier, and a third classifier using a subset of a labeled data set. The subset may be determined using a bootstrap sampling technique or other various methods to increase the diversity of the trained classifiers. The method may also include predicting a pseudo-labeled data set from an unlabeled data set using the first classifier, the second classifier, and the third classifier. The pseudo-labeled data set may include a suggested label and a predicted probability of a sample in the pseudo-labeled data set. The method also includes assigning roles to the first classifier, the second classifier, and the third classifier. Each role assigned to the classifier is based on a prediction threshold and a labeling protocol. The method may also include selecting a teaching sample data set from the pseudo-labeled data set based on the role assigned to the third classifier, wherein the third classifier is assigned to a student role. The method may also include retraining the third classifier using a subset of the labeled data set and the teaching sample data set. After retraining, the method may include updating the prediction threshold of the role assignment and reaching a stopping criterion.

[0004] Further embodiments are directed to a computer-readable medium for model training using a teacher-student learning paradigm on a system, which may include program instructions that can be executed by a processor to cause the system to execute instructions. These instructions can cause the system to train a first classifier, a second classifier, and a third classifier using a subset of a labeled data set. The subset can be determined using a bootstrap sampling technique or other various methods to increase the diversity of the trained classifiers. The instructions can also cause the system to predict a pseudo-labeled data set from an unlabeled data set using the first classifier, the second classifier, and the third classifier. The pseudo-labeled data set can include a suggested label and a predicted probability of a sample in the pseudo-labeled data set. The instructions can also cause the system to assign roles to the first classifier, the second classifier, and the third classifier. Each role to which a classifier is assigned is based on a prediction threshold and a labeling protocol. The instructions can also cause the system to select a teaching sample data set from the pseudo-labeled data set based on the role assigned to the third classifier, wherein the third classifier is assigned to a student role. The instructions can also cause the system to retrain the third classifier using a subset of the labeled data set and the teaching sample data set. After retraining, the instructions can cause the system to update the prediction threshold of the role assignment and reach a stopping criterion.

[0005] Other embodiments are directed to a system that may include at least one processing component and at least one memory component. The system may also include a labeled data set, an unlabeled data set, a classifier for training and prediction, a pseudo-labeled data set, a role assigner configured to assign roles to various classifiers in the system, a teaching sample data set, and a stopping criterion component. The memory component may also include program instructions that cause the system to train a first classifier, a second classifier, and a third classifier using a subset of the labeled data set. The subset may be determined using a bootstrap sampling technique or various other methods to increase the diversity of the trained classifiers. These instructions may further cause the system to predict a pseudo-labeled data set from an unlabeled data set using the first classifier, the second classifier, and the third classifier. The pseudo-labeled data set may include a suggested label and a predicted probability for a sample in the pseudo-labeled data set. These instructions may also cause the system to assign roles to the first classifier, the second classifier, and the third classifier. Each role to which a classifier is assigned is based on a prediction threshold and a labeling protocol. These instructions may also cause the system to select a teaching sample data set from the pseudo-labeled data set based on the role assigned to the third classifier, wherein the third classifier is assigned to a student role. These instructions may also cause the system to retrain the third classifier using a subset of the labeled data set and the teaching sample data set. After retraining, the instructions enable the system to update the predicted thresholds for role assignments and reach a stopping criterion. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] These and other features, aspects and advantages of embodiments of the present disclosure will become better understood in conjunction with the following description, appended claims and accompanying drawings, in which:

[0007] Figure 1 is a block diagram illustrating a model training system according to an embodiment of the present disclosure.

[0008] Figure 2 is a flowchart illustrating a process of training a model based on a teacher-student mode according to an embodiment of the present disclosure.

[0009] Figure 3 is a flow chart illustrating a process of assigning roles to classifiers according to an embodiment of the present disclosure.

[0010] Figure 4 is a high-level block diagram illustrating an example computer system according to an embodiment of the present disclosure, which can be used to implement one or more of the methods, tools, and modules described herein and any related functionality.

[0011] Figure 5 A cloud computing environment according to an embodiment of the present disclosure is depicted.

[0012] Figure 6 Abstract model layers according to an embodiment of the present disclosure are depicted.

[0013] Although the present disclosure can be modified into various revisions and alternative forms, the details thereof have been shown in the drawings by way of example and will be described in detail. However, it should be understood that the purpose is not to limit the specific embodiments described. On the contrary, it is intended to cover all revisions, equivalents and alternative forms that fall within the scope of the present disclosure. In the accompanying drawings, the same reference numerals are used to represent the same parts. DETAILED DESCRIPTION

[0014] Machine learning algorithms or classifiers are typically trained using techniques such as supervised learning, semi-supervised learning, unsupervised learning, and reinforcement learning. The goal of these machine learning techniques is to build a model that can take an input and produce a desired result. The desired result produced by the model can be measured by the accuracy of the given result. In order to improve the results, the model needs to be trained. Based on the type of learning technique used, machine learning algorithms are trained using various forms of data to at least partially improve the accuracy of their results. These forms of data can include labeled data, unlabeled data, or a combination of labeled and unlabeled data. However, data, such as labeled data, can be difficult and expensive to obtain.

[0015] To alleviate some of the expense and availability of labeled data, semi-supervised learning combines unlabeled data with labeled data to train classifiers. Typical semi-supervised learning techniques include self-training, co-training, tri-training, and tri-training with disagreement. In self-training, an initial classifier is trained using a labeled dataset. Then, the classifier is used to make predictions on an unlabeled dataset. Samples with the highest prediction probability or confidence are added to the training set and used to retrain the classifier. In co-training, two base classifiers are initially trained using a labeled dataset. In each iteration, each classifier selects a sample dataset that most reliably predicts the unlabeled samples. These sample datasets are added to the training set of the other classifier and used to train each other. In tri-training, three classifiers are used. After initially training the classifiers using a labeled dataset, two classifiers are used to make predictions on an unlabeled dataset. If the two classifiers agree, only samples from the pseudo-labeled dataset are added to the training dataset. The third classifier is then retrained with the training dataset.

[0016] Typically, self-labeling algorithms, such as those using semi-supervised learning techniques or unsupervised learning techniques, provide inaccurate metrics during the iterative process. This can lead to an increase in noise in the model. Noise is a distortion in the data that is undesirable to the perceiver of the data. Noise can be anything that is artificial or unrelated to the original data, which is undesirable but is introduced due to a faulty process. The increase in noise can be attributed, at least in part, to labeling methods that rely on individual or aggregate prediction confidences derived from the underlying model used to select and label the unlabeled samples.

[0017] Another possible reason for the performance degradation of self-labeling algorithms is the lack of an effective stopping criterion. A stopping criterion is a decision point in the iterative training process where the algorithm stops adding instances to the labeled dataset. This is critical to the ability of the algorithm to work properly. A common type of stopping criterion used by existing self-labeling techniques is to set a threshold for the number of labels the algorithm is willing to generate. Another common type of stopping criterion is to stop the self-labeling algorithm when there is little improvement in accuracy during the training iterations. Stopping criteria that are too conservative or too liberal may produce incorrectly labeled samples during the self-labeling process, resulting in poor model performance.

[0018] In order to improve the above self-labeling technology, the present disclosure provides a semi-supervised learning technology that incorporates a student-teacher model into the learning process.

[0019] Disclosed herein is a method and system for model training using a teacher-student model. The method may include training a first classifier, a second classifier, and a third classifier using a subset of a labeled data set. The subset may be determined using a bootstrap sampling technique or various other methods to increase the diversity of the trained classifiers. The method may also include predicting a pseudo-labeled data set from an unlabeled data set using the first classifier, the second classifier, and the third classifier. The pseudo-labeled data set may include a suggested label and a predicted probability of a sample in the pseudo-labeled data set. The method also includes assigning roles to the first classifier, the second classifier, and the third classifier. Each role assigned to the classifier is based on a prediction threshold and a labeling protocol. The method may also select a teaching sample data set from the pseudo-labeled data set based on the role assigned to the third classifier, wherein the third classifier is assigned the role of a student. The method may also include retraining the third classifier using the teaching sample data set together with a subset of the labeled data set. After retraining, the method may include updating the prediction threshold of the role assignment and reaching a stopping criterion.

[0020] Figure 1 1 is a block diagram illustrating a model training system 100 according to an embodiment of the present disclosure. The model training system 100 includes a labeled data set 120, an unlabeled data set 130, a classifier 140, a pseudo-labeled data set 150, a role assigner 160, a teaching sample data set 170, and a stopping criterion component 180. The classifier 140 may include, but is not limited to, a first classifier 143, a second classifier 146, and a third classifier 149.

[0021] The labeled dataset 120 is a dataset that the model training system 100 uses to initially train the classifier 140. The labeled dataset 120 includes a set of data samples or samples that have been labeled with one or more tags. Tags can generally include one or more meaningful labels that are informative about the sample. For example, a tag may indicate that a photo contains a car, is on a road, and is next to a stop sign. In another example, a sample may be an audio recording with a tag indicating the words spoken in the recording.

[0022] The unlabeled dataset 130 is a dataset used by the model training system 100 to predict the pseudo-labeled dataset 150. The unlabeled dataset 130 includes a set of samples that are not labeled with any labels. For example, the unlabeled dataset 130 may include photos, video recordings, audio recordings, or any other type of unlabeled data. The unlabeled dataset 130 can be applied to a trained classifier to find and label structures in this data.

[0023] Classifier 140 is a component of model training system 100 that is configured to assign labels to data samples. For example, classifier 140 can perform predictive analysis, spam detection, pattern detection, image classification, and classification of other types of categories. Classifier 140 can use different algorithms to map and label the input data. For example, classifier 140 can be a perceptron, naive Bayes, decision tree, logistic regression, k-nearest neighbor, neural network, support vector machine, or any other type of algorithm capable of classification. Classifier 140 is further configured to provide a predicted probability for each label predicted. For example, if classifier 140 predicts an email as spam, the prediction is accompanied by the predicted probability or confidence that classifier 140 has when providing the prediction. The predicted probability can be a percentage ranging from 0 to 100%, depending on the confidence of the classifier. It should be noted that other forms of predicted probabilities can also be displayed by a given classifier. The confidence level of the predicted label. When classifiers 140 are trained, their predicted probabilities can also increase with each training iteration. The classifier 140 may be trained iteratively until the classifier 140 reaches a prediction probability threshold that is satisfactory for the task given to the model.

[0024] In some embodiments, the classifier 140 includes a first classifier 143, a second classifier 146, and a third classifier 149. Embodiments of the present disclosure require at least three classifiers to apply the teacher-student learning model. The embodiments discussed herein use only three classifiers to demonstrate the present disclosure, but should not be considered as limiting the present disclosure. Various other embodiments can utilize more than three classifiers to publicly perform various arrangements of the teacher-student model.

[0025] The pseudo-labeled dataset 150 is a dataset used by the model training system 100, and the pseudo-labeled dataset 150 contains samples including approximate labels and accompanying prediction probabilities provided by the classifier 140. When the unlabeled dataset 130 is applied to the classifier 140 trained by the labeled dataset 120, the pseudo-labeled dataset 150 is generated by the classifier 140.

[0026] The role allocator 160 is a component of the model training system 100, and the role allocator 160 is configured to assign roles to the classifiers 140. The role allocator 160 may include a teacher threshold and a student threshold. According to each threshold of the pseudo-labeled data set, the role allocator 160 may assign a classifier to a teacher role or a student role in the classifier 140. In each training iteration, if its predicted probability exceeds or reaches the teacher threshold, the first classifier 143 and the second classifier 146 are assigned the role of the teacher. If the predicted probability of the third classifier 149 is less than the student threshold, a student role is assigned to it. The role allocator 160 is also configured to ensure that at least two teacher classifiers are identified. This helps prevent bias in any model from affecting the quality of information imparted to the student classifier.

[0027] In some embodiments, the role allocator 160 rotates the role assignments after each training iteration. For example, the first classifier 143 may be assigned as a student classifier, while the second classifier 146 and the third classifier 149 may be assigned as teacher classifiers. By rotating the assigned roles, each classifier can learn from the experience of other classifiers.

[0028] The teaching sample data set 170 is a data set containing teaching samples used by the model training system 100. The teaching samples are selected from the pseudo-labeled data set 150, which is determined to be teachable to the classifier 140 assigned as the student classifier. The teaching samples are samples in which the predicted labels of the teaching samples in the teacher classifier are consistent with each other. In addition, the predicted probabilities of the teaching samples of the two teacher classifiers exceed the teacher threshold. The predicted probability of the student classifier must also be lower than the student threshold. Therefore, when a sample meets the required teaching sample criteria, it can be added to the teaching sample data set 170. The teaching sample data set 170 can be used to retrain the student classifier to improve its prediction probability.

[0029] The stopping criterion component 180 is a component of the model training system 100, and the stopping criterion component 180 is configured to determine whether the classifier 140 meets the stopping criterion. The stopping criterion component 180 can compare the student threshold with the teacher threshold during each training iteration. When the student threshold reaches the teacher threshold, the stopping criterion is met, and the training cycle can be ended.

[0030] Figure 2 2 is a flow chart showing a model training process 200 based on a teacher-student model according to an embodiment of the present disclosure. Figure 1 The model training system 100 is described in the context of Figure 2 .

[0031] The first classifier 143, the second classifier 146, and the third classifier 149 are trained from subsets of the labeled data set 120, respectively. This is illustrated in step 210. In some embodiments, the subset is the result of bootstrapping the labeled data. Bootstrapping labeling occurs when the classifier 140 is initially trained with a limited amount of labeled data and retrained with positive samples obtained from the classifier. In some other embodiments, the subset can be determined by independent random feature segmentation. Independent random feature segmentation can be achieved by randomly selecting samples from the labeled data set 120. The sampling order does not depend on any other samples or data. By providing random sampling, the diversity of the classifier 140 can be increased. The classifier 140 can also be trained using various methods. For example, these methods can include generative models, low-density separation methods, graph-based methods, and various other heuristic methods.

[0032] Once trained, the unlabeled data set 130 is applied to each classifier to predict and generate a pseudo-labeled data set. This is illustrated in step 220. In some embodiments, a subset of the unlabeled data set should be used for each classifier 140. For example, a subset of the unlabeled data set that is large enough can be generated for training iterations. Sufficient samples can be generated so that the student classifier can be retrained with appropriate samples. Depending on the size of the unlabeled data set, multiple subsets can be generated and applied during the training iteration. Once the classifier 140 predicts and labels the samples from the unlabeled data set 130, a pseudo-labeled data set 150 can be generated. The pseudo-labeled data set can include the predicted labels from each classifier 140 and the predicted probability of each predicted label.

[0033] The role assigner 160 may assign a role to each classifier 140. This is shown in step 230. In some embodiments, the role assigner 160 assigns at least two teacher roles and one student role based on the predicted probability compared to a predetermined threshold for each classifier 140 in the pseudo-labeled dataset 150. For example, if the predicted probability of the predicted labeling of the sample from the pseudo-labeled dataset 150 by the first classifier 143 and the second classifier 146 exceeds the teacher threshold, then the teacher role may be assigned to them. For the same sample, if the third classifier 149 has a predicted probability below the student threshold, then the third classifier 149 may be assigned to the student role. For example, if the teacher threshold is set to a predicted probability of 95%, the confidence of the classifier will need to exceed that 95% to be assigned to the teacher role. For the same reason, if the student threshold is set to a predicted probability of 50%, the confidence of the classifier must be below 50% to be assigned to the student role.

[0034] Select the teaching sample data set 170 from the pseudo-labeled data set 150. This is explained in step 240. The teaching sample data set 170 can be selected from the pseudo-labeled data set 150 based on the predicted probability of the classifier 140 and the labeling agreement of the teacher classifier. If the teacher classifier agrees with the labeling of the sample, the sample from the pseudo-labeled data set 150 can be added to the teaching sample data set 170. Similarly, the predicted probability of the teacher classifier must exceed the teacher threshold. In addition, the predicted probability of the student threshold must also be lower than the student threshold. If the standard is met, the sample can be placed in the teaching sample data set 170. For example, the first classifier 143 and the second classifier 146 are teacher classifiers, and the third classifier 149 is a student classifier. For a given sample, when the label predictions of the first classifier 143 and the second classifier 146 are consistent and the confidence of each of those identical labels is greater than the teacher threshold, and the sample confidence of the third classifier 149 is lower than the student threshold, the sample is added to the teaching sample data set. The label predictions made by the third classifier 149 or the student classifier do not need to be consistent or inconsistent with the teacher classifier. The only required condition is that the predicted probability of the sample is below the student threshold. In some embodiments, the teacher classifier only needs to reach the teacher threshold to add the sample to the teaching sample data set 170.

[0035] In some embodiments, the teaching sample data set 170 determines the role assigned to the classifier 140. The teaching sample data set 170 for each classifier 140 can be generated from the pseudo-labeled data set, where two classifiers are teacher classifiers and one classifier is a student classifier. Given three classifiers, a teaching sample data set can be generated for each of the three classifiers in one iteration of the training cycle.

[0036] The student classifier is retrained using the teaching sample data set 170 and the labeled data set. This is illustrated in step 250. In some embodiments, a subset of the labeled data set 120 used to train the student classifier is combined with the teaching sample data set 170 to retrain the student classifier. In some embodiments, the teaching sample data set 170 is added to the bootstrapped labeled data set for training the classifier 140 in step 210. In some embodiments, each classifier of the classifier 140 having the teaching sample data set 170 generated from the pseudo-labeled data set 150 is trained using its respective generated teaching sample data set 170. For example, if some samples 150 of the pseudo-labeled data set meet the criteria in such a way that the first classifier 143 is a student classifier and other samples meet the criteria for the third classifier 149 to be determined as a student classifier, the first classifier 143 can be trained using its samples and the third classifier 149 can be trained using its own samples.

[0037] After retraining, the teacher threshold and the student threshold are updated. This is illustrated in step 260. In some embodiments, the teacher threshold is decreased while the student threshold is increased by a predetermined increment. To adjust the teacher and student thresholds, static or adaptive increments can be used to change the thresholds in each iteration. For example, the thresholds can be updated by a static number such as 3%. If the teacher threshold is determined to be 90%, the threshold will drop to 87%; if the student threshold is determined to be 50%, the threshold will increase to 53%.

[0038] In some embodiments, the teacher threshold and the student threshold are adjusted based on separate increments. A predetermined teacher increment can be set to reduce the teacher threshold after each iteration, and a predetermined student increment can be set to increase the student threshold after each iteration. By having independent increments, the present disclosure can be adjusted to improve the performance of the classifier in each iteration.

[0039] In some embodiments, an adaptive increment is used to lower the teacher threshold and increase the student threshold. The adaptive increment can be adjusted after each iteration to optimize the training cycle of the classifier. The adaptive increment can be adjusted based on the size of the teaching sample data set, the increase in the classifier prediction probability, or other factors that the operator deems appropriate for use.

[0040] Determine whether the student threshold reaches the teacher threshold. This is illustrated in step 270. During each training iteration, the student threshold and the teacher threshold are adjusted and updated. After adjustment, process 200 can determine whether the student threshold reaches the teacher threshold. Generally, when the student threshold reaches the same confidence level as the teacher threshold, the training can be considered to be sufficiently complete. At this point, adding samples to the teaching sample data set 170 may not necessarily help improve the prediction probability of the classifier. In addition, if too many iterations are performed on the classifier 140, it may cause increased noise. If the student threshold does not reach the confidence level of the teacher threshold, process 200 returns to step 220. However, if the student threshold reaches the confidence level of the teacher threshold, the stopping criteria are reached and the training cycle is completed. This is illustrated in step 280.

[0041] In some embodiments, a final prediction may be provided based on a majority vote of the trained classifiers 140. The first classifier 143, the second classifier 146, and the third classifier 149 may provide predictions for the data set. The prediction agreed upon by the majority of the classifiers 140 may be provided as the final result.

[0042] Figure 3 It is shown that various embodiments according to the present disclosure are Figure 2 Flowchart of process step 230 of assigning roles to classifiers in FIG. 230. To illustrate process step 230, Figure 3 is Figure 1The model training system 100 is described in the context of.

[0043] A first label with a corresponding predicted probability predicted by the first classifier 143 is retrieved from the sample. This is illustrated in step 310. A second label with a corresponding predicted probability predicted by the second classifier 146 is also retrieved from the same sample. This is illustrated in step 320. A third label with a corresponding predicted probability predicted by the third classifier 149 is also retrieved from the same sample. This is illustrated in step 330.

[0044] Based on the retrieved labels and predicted probabilities, the first classifier 143 and the second classifier 146 are assigned as teacher roles. This is illustrated in step 340. To assign a teacher role, the predicted probability of the first classifier 143 and the predicted probability of the second classifier 146 exceed or meet a predetermined teacher threshold. In addition, the labels predicted by the first classifier 143 and the labels predicted by the second classifier 146 must be consistent. If the criteria are met, the first classifier 143 and the second classifier 146 can be assigned as teachers for the sample.

[0045] Based on the retrieved predicted probabilities, the third classifier is assigned the role of a student. This is illustrated in step 350. To be assigned the student role, the predicted probability of the third classifier 149 must be below or meet a predetermined student threshold.

[0046] If a sample cannot be assigned at least two teacher classifiers and one student classifier, the sample is not considered an ideal training sample and will not be placed in the teaching sample dataset.

[0047] Reference now Figure 4 , shows a high-level block diagram of an example computer system 400 (e.g., model training system 100) according to an embodiment of the present disclosure, which can be used to implement one or more of the methods, tools, and modules described herein and any related functions (e.g., using one or more processor circuits or computer processors of a computer). In some embodiments, the main components of the computer system 400 may include one or more processors 402, memory 404, terminal interface 412, I / O (input / output) device interface 414, memory interface 416, and network interface 418, all of which may be directly or indirectly communicatively coupled to communicate between components via memory bus 403, I / O bus 408, and I / O bus interface 410.

[0048] Computer system 400 may include one or more general purpose programmable central processing units (CPUs) 402-1, 402-2, 402-3, and 402-N, collectively referred to herein as processors 402. In some embodiments, computer system 400 may include multiple processors typical of relatively larger systems; however, in other embodiments, computer system 400 may instead be a single CPU system. Each processor 402 may execute instructions stored in memory 404 and may include one or more onboard levels of cache. In some embodiments, processor 402 performs the processes included herein (e.g., processes 200, 230).

[0049] The memory 404 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) 422 or a cache memory 424. The computer system 400 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a memory system 426 may be provided for reading and writing to a non-removable non-volatile magnetic medium such as a "hard drive". Although not shown, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), or an optical drive for reading or writing a removable non-volatile optical disk such as a CD-ROM, DVD-ROM, or other optical media may be provided. In addition, the memory 404 may include a flash memory, such as a flash stick drive or a flash drive. The memory device may be connected to the memory bus 403 via one or more data medium interfaces. The memory 404 may include at least one program product having a set of program modules (e.g., at least one) configured to perform the functions of various embodiments.

[0050] Despite Figure 4 In the embodiment, the memory bus 403 is shown as a single bus structure that provides a direct communication path between the processor 402, the memory 404 and the I / O bus interface 410. In some embodiments, the memory bus 403 may include multiple different buses or communication paths, which may be arranged in any different forms, such as point-to-point links in a hierarchical, star or Web configuration, multiple hierarchical buses, parallel and redundant paths, or any other suitable type of configuration. In addition, although the I / O bus interface 410 and the I / O bus 408 are shown as a single corresponding unit, in some embodiments, the computer system 400 may include multiple I / O bus interface units, multiple I / O buses, or both. In addition, although multiple I / O interface units are shown that separate the I / O bus 408 from the various communication paths running to various I / O devices, in other embodiments, some or all of the I / O devices may be directly connected to one or more system I / O buses.

[0051] In some embodiments, computer system 400 may be a multi-user host system, a single-user system, or a server computer or similar device that has little or no direct user interface but receives requests from other computer systems (clients). In addition, in some embodiments, computer system 400 may be implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a phone, a smart phone, a network switch or router, or any other suitable type of electronic device.

[0052] Notice, Figure 4 The main components of the exemplary computer system 400 are intended to be depicted. However, in some embodiments, the various components may have more Figure 4 Greater or lesser complexity than shown may also exist in addition to Figure 4 Components other than those shown or Figure 4 components other than those shown, and the quantity, type, and configuration of such components may vary.

[0053] One or more programs / utilities 428, each with at least one set of program modules 430, may be stored in memory 404. Programs / utilities 428 may include a hypervisor (also referred to as a virtual machine monitor), one or more operating systems, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Programs 428 and / or program modules 430 generally perform the functions or methods of various embodiments.

[0054] It should be understood that although the present disclosure includes a detailed description about cloud computing, the implementation of the technical solutions recorded therein is not limited to a cloud computing environment, but can be implemented in combination with any other type of computing environment now known or later developed.

[0055] Cloud computing is a service delivery model for convenient, on-demand network access to a shared pool of configurable computing resources. Configurable computing resources are resources that can be quickly deployed and released with minimal management cost or interaction with the service provider, such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0056] Features include:

[0057] On-demand self-service: Cloud consumers can unilaterally and automatically deploy computing capabilities such as server time and network storage on demand without human interaction with the service provider.

[0058] Broad network access: Computing power can be accessed over the network through standard mechanisms that facilitate the use of the cloud through different types of thin-client or thick-client platforms (e.g., mobile phones, laptops, personal digital assistants (PDAs)).

[0059] Resource pool: The computing resources of the provider are grouped into resource pools and serve multiple consumers through a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated on demand. Generally, consumers cannot control or even know the exact location of the provided resources, but can specify the location (such as country, state or data center) at a higher level of abstraction, so it is location-independent.

[0060] Rapid elasticity: The ability to quickly and elastically (sometimes automatically) deploy computing power to scale up quickly, and quickly release it to scale down quickly. To consumers, the computing power available for deployment often appears to be unlimited, and any amount of computing power can be accessed at any time.

[0061] Measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0062] The service model is as follows:

[0063] Software as a Service (SaaS): The capability provided to consumers is to use the provider's applications running on a cloud infrastructure. Applications can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., web-based email). Aside from limited user-specific application configuration settings, consumers neither manage nor control the underlying cloud infrastructure including networks, servers, operating systems, storage, or even individual application capabilities.

[0064] Platform as a Service (PaaS): The capability provided to consumers is to deploy applications created or acquired by consumers on the cloud infrastructure, which are created using programming languages ​​and tools supported by the provider. Consumers neither manage nor control the underlying cloud infrastructure including networks, servers, operating systems or storage, but have control over the applications they deploy and may also have control over the configuration of the application hosting environment.

[0065] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, network, and other basic computing resources on which consumers can deploy and run arbitrary software including operating systems and applications. Consumers neither manage nor control the underlying cloud infrastructure, but have control over the operating system, storage, and applications they deploy, and may have limited control over selected network components (such as host firewalls).

[0066] The deployment model is as follows:

[0067] Private Cloud: Cloud infrastructure is run solely for an organization. The cloud infrastructure can be managed by the organization or a third party and can exist inside or outside the organization.

[0068] Community Cloud: A cloud infrastructure is shared by several organizations and supports a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). A community cloud can be managed by multiple organizations within the community or by a third party and can exist inside or outside the community.

[0069] Public cloud: Cloud infrastructure is provided to the public or to large industry groups and is owned by an organization that sells cloud services.

[0070] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community or public) in a deployment model that remain distinct entities but are bound together by standardized or proprietary technologies (such as cloud burst sharing for load balancing between clouds) that enable data and application portability.

[0071] The cloud computing environment is service-oriented, with characteristics centered on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is the infrastructure that includes a network of interconnected nodes.

[0072] Reference now Figure 5 , in which an exemplary cloud computing environment 550 is shown. As shown, the cloud computing environment 550 includes one or more cloud computing nodes 510 with which a local computing device used by a cloud computing consumer can communicate, and the local computing device can be, for example, a personal digital assistant (PDA) or a mobile phone 554A, a desktop computer 554B, a laptop computer 554C and / or an automobile computer system 554N. The cloud computing nodes 510 can communicate with each other. The cloud computing nodes 510 can be physically or virtually grouped in one or more networks including but not limited to private clouds, community clouds, public clouds or hybrid clouds, or a combination thereof as described above (not shown in the figure). In this way, cloud consumers can request infrastructure as a service (IaaS), platform as a service (PaaS) and / or software as a service (SaaS) provided by the cloud computing environment 550 without having to maintain resources on local computing devices. It should be understood that Figure 5 The various types of computing devices 554A-N shown are merely illustrative, and cloud computing node 510 and cloud computing environment 550 can communicate with any type of computing device on any type of network and / or network addressable connection (eg, using a web browser).

[0073] Reference now Figure 6 , which shows a cloud computing environment 550 ( Figure 5 ) provides a set of functional abstraction layers. First of all, it should be understood that Figure 6 The components, layers, and functions shown are only exemplary and the embodiments of the present invention are not limited thereto. Figure 6 As shown, the following layers and corresponding functions are provided:

[0074] The hardware and software layer 660 includes hardware and software components. Examples of hardware components include: host 661; server 662 based on RISC (Reduced Instruction Set Computer) architecture; server 663; blade server 664; storage device 665; network and network components 666. Examples of software components include: network application server software 667 and database software 668.

[0075] The virtualization layer 670 provides an abstraction layer that can provide examples of the following virtual entities: virtual servers 671 , virtual storage 672 , virtual networks 673 (including virtual private networks), virtual applications and operating systems 674 , and virtual clients 675 .

[0076] In one example, the management layer 680 may provide the following functions: Resource provisioning function 681: Provide dynamic acquisition of computing resources and other resources for performing tasks in a cloud computing environment; Metering and pricing function 682: Track the cost of resource usage within the cloud computing environment and provide bills and invoices for this purpose. In one example, the resource may include an application software license. Security function: Provide identity authentication for cloud consumers and tasks, and provide protection for data and other resources. User portal function 683: Provide access to the cloud computing environment for consumers and system administrators. Service level management function 684: Provide allocation and management of cloud computing resources to meet the required service levels. Service level agreement (SLA) planning and fulfillment function 685: Provide pre-arrangement and provision for future demand for cloud computing resources predicted according to the SLA.

[0077] The workload layer 690 provides examples of functions that may be implemented in a cloud computing environment. In this layer, examples of workloads or functions that may be provided include: mapping and navigation 691; software development and lifecycle management 692; virtual classroom teaching provision 693; data analysis processing 694; transaction processing 695; and model training 696 using a teacher-student learning model.

[0078] At any possible technical detail combination level, the present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0079] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0080] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0081] The computer program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0082] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0083] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0084] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0085] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes displayed continuously can actually be completed as a step, executed simultaneously in a partially or fully overlapping manner, executed substantially simultaneously, executed in a partially or fully temporarily overlapping manner, or these boxes can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0086] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A system comprising: A first classifier and a second classifier configured to be trained using a subset of a labeled dataset, wherein the first classifier and the second classifier have a prediction probability exceeding a teacher threshold when predicting a pseudo-labeled dataset from an unlabeled dataset, the labeled dataset comprising photos or audio data that have been labeled with a label, and the unlabeled dataset comprising photos or audio data that have not been labeled; a third classifier configured to be trained using a subset of the labeled dataset, wherein the third classifier has a prediction probability below a student threshold when predicting the pseudo-labeled dataset; a role assigner configured to assign a teacher role and a student role to the first classifier, the second classifier, and the third classifier based on a predicted probability of samples from the unlabeled dataset; and A teaching sample dataset from the pseudo-labeled dataset, wherein the teaching sample dataset includes at least one sample from the pseudo-labeled dataset, wherein the first classifier and the second classifier agree on the labeling of the sample, and the predicted probabilities of the first classifier and the second classifier exceed the teacher threshold, and wherein the predicted probability of the third classifier is lower than the student threshold.

2. The system of claim 1, wherein the subset is an independent feature set determined by random feature segmentation.

3. The system of claim 1, wherein the subset is determined by a bootstrapped labeled dataset of the labeled dataset.

4. The system of claim 1, wherein the teacher threshold and the student threshold are adjustable after training iterations.

5. The system of claim 1, further comprising a stopping criteria component configured to determine whether the system reaches a stopping criteria.

6. The system of claim 5, wherein the stopping criterion is determined when the student threshold is at least as high as the teacher threshold.

7. A method comprising: training a first classifier, a second classifier, and a third classifier using a subset of a labeled dataset, the labeled dataset comprising photos or audio data that have been tagged with a label; Predicting a pseudo-labeled dataset from an unlabeled dataset using the first classifier, the second classifier, and the third classifier, wherein the unlabeled dataset includes unlabeled photos or audio data; assigning roles to the first classifier, the second classifier, and the third classifier based on a prediction threshold and a labeling convention; selecting a teaching sample data set from the pseudo-labeled data set based on the role assigned to the third classifier; Retrain the third classifier using the teaching sample data set and the labeled data set; updating the prediction threshold; as well as The stopping criterion is reached.

8. The method of claim 7, wherein assigning the roles comprises: Retrieve a first label and a corresponding first prediction probability for a sample of the pseudo-labeled dataset performed by the first classifier; Retrieve a second label and a corresponding second prediction probability for the sample of the pseudo-labeled dataset performed by the second classifier; Retrieve a third label and a corresponding third prediction probability for the sample of the pseudo-labeled dataset performed by the third classifier; assigning a teacher role to the first classifier and the second classifier based on the first predicted probability and the second predicted probability exceeding a teaching threshold and based on the first label being consistent with the second label; and Based on the third predicted probability not exceeding the student threshold, the third classifier is assigned a student role.

9. The method of claim 7, wherein the subsets are independent feature sets determined by independent random feature segmentation.

10. The method of claim 7, wherein the subset is determined by a bootstrapped labeled dataset of the labeled dataset.

11. The method of claim 7, wherein updating the prediction threshold comprises: reducing a teacher threshold of the prediction threshold by a teaching increment; and The student threshold of the prediction threshold is increased by a learning increment.

12. The method of claim 11, wherein: The teaching increment and the learning increment are static increments.

13. The method of claim 11, wherein: The teaching increment and the learning increment are predetermined amounts that are adjustable after each iteration.

14. The method of claim 7, further comprising: A final prediction is provided based on a majority vote of the first classifier, the second classifier, and the third classifier.

15. The method of claim 7, wherein reaching the stopping criterion comprises: determining whether a student threshold of the prediction threshold is at least as high as a teacher threshold of the prediction threshold; as well as The stopping criterion is reached upon determining that the student threshold is at least as high as the teacher threshold.

16. A computer-readable storage medium for model training using a teacher-student model, the computer-readable storage medium having program instructions contained therein, which can be executed by a processor to enable the processor to perform the steps of the method described in one of claims 7-15.

17. A system for model training using a teacher-student model, comprising a device for performing the steps of the method described in one of claims 7-15.

18. A system comprising: a memory having computer readable instructions, and A processor for executing computer readable instructions configured to perform the steps of the method of any one of claims 7-15.