Semi-supervised training of machine learning models using label guess

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a machine learning model. One of the methods includes receiving an unlabeled batch; generating a processed unlabeled batch and a processed labeled batch from the unlabeled batch and the labeled batch, the generating including, for each unlabeled training input of a plurality of unlabeled training inputs: generating a plurality of augmented unlabeled training inputs from the unlabeled training input; processing each of the augmented unlabeled training inputs using a machine learning model to generate a respective model output for each augmented unlabeled training input; generating a guessed model output from the model outputs of the augmented unlabeled training inputs; and associating the guessed model output with each of the augmented unlabeled training inputs; and training the machine learning model on the processed labeled batch and the processed unlabeled batch.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 843,806, filed May 6, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This manual pertains to training machine learning models. Background Technology

[0004] Machine learning models receive input and generate outputs, such as predicted outputs, based on the received inputs and the model's parameter values.

[0005] A neural network is a machine learning model that uses one or more non-linear units to predict the output of a received input. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0006] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which trains a machine learning model to perform a machine learning task through semi-supervised learning, i.e., by training the machine learning model on training data including unlabeled training inputs and labeled training inputs. Labeled training inputs are ground truth outputs for which ground truth outputs are obtainable; that is, inputs to the outputs that the machine learning model should generate by performing a specific machine learning task on the labeled training inputs. Unlabeled training inputs are training inputs for which ground truth outputs are not obtainable.

[0007] The system trains a machine learning model by partially generating guessing model outputs from unlabeled training inputs in the training data. To generate the guessing model outputs, the system generates multiple augmented unlabeled training inputs from the unlabeled training inputs. The system then uses the machine learning model to process the multiple augmented unlabeled training inputs to generate a corresponding model output for each augmented unlabeled training input. The system then generates a guessing model output from the corresponding model output for each augmented unlabeled training input.

[0008] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages.

[0009] The described system enables the training of machine learning models to perform well on machine learning tasks using limited labeled data. Specifically, by leveraging "label guessing"—that is, guessing the model output for unlabeled training input—the system trains machine learning models to achieve high performance compared to conventional techniques with a lower ratio of labeled to unlabeled data. Given the same amount of labeled and unlabeled data, the system trains machine learning models to achieve better accuracy compared to using conventional techniques. As a specific example, the described technique can be used to train machine learning models to achieve state-of-the-art performance on various image classification tasks.

[0010] Furthermore, the system can train machine learning models to be robust to input variability, for example, to effectively handle variability in the input. For instance, a trained machine learning model, trained according to the described techniques, will be able to effectively classify input images, even when these images have occlusion or blurring, different degrees of skew distortion, different degrees of rotation, etc.

[0011] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0012] Figure 1 An example machine learning model training system is shown.

[0013] Figure 2 This is a flowchart of an example process for training a machine learning model.

[0014] Figure 3A This is a flowchart of an example process for training a machine learning model on batches of unlabeled training input and batches of labeled training input.

[0015] Figure 3B This is a schematic diagram illustrating the generation of unlabeled batches during the initial processing.

[0016] Figure 4 The performance of the described technique relative to other semi-supervised learning techniques is shown.

[0017] The same reference numerals and labels in each figure indicate the same elements. Detailed Implementation

[0018] Figure 1 An example machine learning model training system 100 is shown. The machine learning model training system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, on which the following systems, components and techniques can be implemented.

[0019] The machine learning model training system 100 is a system for training a machine learning model 110 on training data including labeled training data 140 and unlabeled training data 150 to determine the training values ​​of the parameters of the machine learning model 110, referred to herein as model parameters, based on the initial values ​​of the model parameters.

[0020] Machine learning model 110 is a machine learning model configured to receive model input 102 and process the model input to map the model input 102 to model output 112 in order to perform a specific machine learning task according to the model parameters.

[0021] The machine learning model 110 can be configured to perform any of a variety of machine learning tasks, that is, to receive any kind of numerical data input as input and generate model output from the input. Typically, the model output is a probability distribution over the set of possible classes.

[0022] For example, if the task is image classification, the input to model 110 is an image, and the model output for a given image can be the probability of each of the set of object categories, where each probability represents an estimated likelihood that the image contains an object belonging to that category.

[0023] For example, if the task is video classification, the input to model 110 is one or more video frames, and the model output is a probability distribution over a set of object classes or a probability distribution over a set of topics.

[0024] As another example, if the machine learning task is document classification, the input to machine learning model 110 is text from an internet resource (e.g., a web page) or a document, and the model output for a given internet resource, document, or part of a document can be a score for each of a set of topics, where each score represents the probability that the internet resource, document, or part of a document is an estimate of the topic.

[0025] As another example, if the task is a natural language understanding task, then the input to the machine learning model 110 is a sequence of text, and the model output for a given sequence of text can be a probability distribution suitable for the natural language understanding task, such as a distribution on language acceptability category, language sentiment category, language paraphrase category, sentence similarity category, text entailment category, question answering category, etc.

[0026] As another example, if the task is a health prediction task, the input to the machine learning model 110 is the patient's electronic health record data, and the model output for a given sequence can be a probability distribution of the patient's health-related categories, such as possible diagnoses for the patient, possible future health events associated with the patient, etc.

[0027] As another example, if the task is a speech processing task, the input to the machine learning model 110 can be an audio signal representing spoken utterances, such as raw audio or acoustic features, i.e., audio data, and the model output can be a probability distribution over a set of speech classification categories, such as a probability distribution over possible languages, a probability distribution over natural language text, such as a probability distribution over a set of possible hot words, etc.

[0028] The machine learning model 110 can have any architecture suitable for the type of model input processed by the machine learning model 110. For example, when the model input is image or audio data, the machine learning model 110 can be a convolutional neural network. When the model input is a sequence of text or other features, such as electronic health record features or a sequence of audio features, the machine learning model 110 can be a self-attention-based neural network, such as a Transformer, or a recurrent neural network, such as a Long Short-Term Memory (LSTM) neural network. When the model input includes multiple modalities, such as both image and text input, the model 110 can include different types of neural networks, such as both convolutional layers and self-attention or recurrent layers.

[0029] The labeled training data 140 used by system 100 to train machine learning model 110 includes multiple batches of labeled training inputs. The training inputs are called "labeled" training inputs because the labeled training data 140 also includes ground truth outputs for each labeled training input; that is, the outputs that the machine learning model should generate by performing a specific machine learning task on the labeled training inputs. In other words, the ground truth outputs are the actual outputs when the machine learning task is performed on the corresponding labeled training inputs.

[0030] The unlabeled training data 150 used by system 100 to train machine learning model 110 includes multiple batches of unlabeled training inputs. The training inputs are called "unlabeled" training inputs because ground truth outputs for the unlabeled training inputs are unavailable; that is, system 100 cannot access any ground truth output for any of the unlabeled training inputs or cannot use ground truth outputs for the unlabeled training inputs used to train model 110 for some other reason.

[0031] Typically, system 100 trains machine learning model 110 by performing an iterative training process.

[0032] At each iteration of the training process, system 100 trains model 110 on batches of unlabeled training data and batches of labeled training data. In order to train model 110 on both batches at any given training iteration, system 100 generates processed batches of labeled data and processed batches of unlabeled data, and then trains the machine learning model on the processed labeled batches and processed unlabeled batches to adjust the current values ​​of the model parameters, i.e., the values ​​of the model parameters since the training iteration.

[0033] See below for reference Figure 2 , Figure 3A and Figure 3B The training iterations performed during the training of Model 110 are described in more detail.

[0034] Once model 110 has been trained, system 100 is able to provide data for the specified trained model to be used when processing new network inputs. That is, system 100 is able to output training values ​​of model parameters, for example, by outputting to a user device or by storing them in memory accessible to system 100, for later use when processing inputs using the trained model.

[0035] Alternatively, in addition to outputting trained model data, system 100 may also instantiate an instance of a machine learning model with trained values ​​of model parameters, for example, by receiving input to be processed through an application programming interface (API) provided by the system, using the trained model to process the received input to generate model output, and then providing the generated model output, classification output, or both in response to the received input.

[0036] Figure 2 This is a flowchart of an example process 200 for training a machine learning model on batches of unlabeled training input and batches of labeled training input. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a suitably programmed machine learning model training system, such as... Figure 1 The machine learning model training system 100 is capable of executing process 200.

[0037] The system can execute process 200 times over multiple different labeled-unlabeled batch combinations to determine the training values ​​of the model parameters based on the initial values ​​of the model parameters. That is, it can repeatedly execute process 200 to train the machine learning model during different training iterations of the iterative training process. For example, the system can continue to execute process 200 for a specified number of iterations, a specified amount of time, or until the parameter values ​​decrease below a threshold.

[0038] The system obtains a labeled batch, that is, a batch of labeled training inputs, and for each labeled training input, obtains the ground truth output that should be generated by the machine learning model by performing a specific machine learning task on the labeled training input (step 202).

[0039] The system obtains the unlabeled batch, that is, the batch of unlabeled training input (step 204).

[0040] The system generates processed unlabeled batches and processed labeled batches from the unlabeled batches and labeled batches (step 206).

[0041] After the processed labeled batch and processed unlabeled batch are generated, each input in the processed labeled batch and each input in the processed unlabeled batch are associated with the corresponding target model output.

[0042] Typically, and as will be described in more detail below, each input in the processed labeled batch corresponds to a corresponding one in the labeled training input, and the target output for the input is (i) the ground truth output for the corresponding labeled training input or (ii) derived from the ground truth output for the corresponding labeled training input.

[0043] Typically, and as will be described in more detail below, each input in the processed unlabeled batch corresponds to a corresponding one of the unlabeled training inputs, and the target output for each input is either (i) a guessing model output for the corresponding unlabeled training input or (ii) derived from the guessing model output for the corresponding unlabeled training input. The guessing model output is a guessing model output generated based on the model output of the machine learning model, i.e., and not generated from any ground truth information provided to the system as input.

[0044] The generation of the labeled batch and the unlabeled batch after processing are described in more detail below with reference to Figure 3.

[0045] The system trains the machine learning model on the processed labeled batch and the processed unlabeled batch to adjust the current values ​​of the model parameters (step 208).

[0046] Specifically, the system determines the update of the current values ​​of the model parameters by calculating the gradient of the self-supervised learning loss function, which includes both labeled and unlabeled loss terms. For example, the loss function can be the sum or weighted sum of labeled and unlabeled loss terms.

[0047] The label loss term measures the error between: (i) the model output generated by the machine learning model for each input in the processed label batch according to the current values ​​of the parameters, and (ii) the target output for the input in the processed label batch.

[0048] For example, the label loss term could be the cross-entropy loss between: (i) the model output generated by the machine learning model with the current values ​​of the parameters as input and (ii) the target output for the input in the processed label batch.

[0049] The unlabeled loss term measures the error between the following: (i) the model output generated by the machine learning model with the current values ​​of the parameters as inputs for each input in the processed unlabeled batch, and (ii) the target output for the inputs in the processed unlabeled batch.

[0050] The unlabeled loss term can be the same loss as the labeled loss term or a different type of machine learning loss, such as cross-entropy loss. For example, when the labeled loss term is cross-entropy loss, the unlabeled loss could be the squared L2 loss between: (i) the model output generated by the machine learning model with the current values ​​of the parameters as input and (ii) the target output for the input in the processed unlabeled batch. Using squared L2 loss can be beneficial because, unlike cross-entropy loss, it is bounded and less sensitive to mispredictions.

[0051] More specifically, the system updates the current values ​​of the model parameters by computing the gradient of the loss function relative to the model parameters and over both the processed labeled batch and the processed unlabeled batch, and then using this gradient to update the current values ​​of the model parameters. In particular, the system can apply an update rule, such as the learning rate, the Adam optimizer update rule, or the rmsProp update rule, to the gradient to generate an update, and then apply the update—that is, subtract or add to the current value of the gradient—to determine the updated values ​​of the model parameters.

[0052] Figure 3A This is a flowchart of an example process 300 for generating processed labeled batches and processed unlabeled batches. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a suitably programmed machine learning model training system, such as... Figure 1 The machine learning model training system 100 is capable of executing process 300.

[0053] The system generates initially processed labeled batches from the labeled batches (step 302).

[0054] In some implementations, the initial processed label batch is the same as the label batch, that is, the system does not modify the label training input in the label batch.

[0055] In some other implementations, in order to generate an initially processed batch of labeled inputs, the system generates a corresponding augmented labeled training input from each labeled training input and associates the augmented labeled training input with the ground truth output for the labeled training input.

[0056] The system uses data augmentation techniques to generate amplified labeled training inputs; any conventional augmentation technique suitable for the type of input the model is configured to process can be used. See below for reference. Figure 3B Examples describing data augmentation techniques.

[0057] Therefore, in these embodiments, the initial processed batch of labels includes a set of amplified labeled training inputs, each associated with a corresponding ground truth output.

[0058] The system generates an initial unlabeled batch from the unlabeled batch (step 304).

[0059] For each unlabeled training input in an unlabeled batch, the initially processed unlabeled batch includes K amplified unlabeled training inputs, each associated with the same guessing model output. To ensure diversity, K is set to a fixed positive integer greater than 1, such as 2, 4, or 5. Therefore, instead of each unlabeled training input in an unlabeled batch, the unlabeled batch includes multiple amplified unlabeled training inputs generated from the unlabeled training input.

[0060] Figure 3B This is a schematic diagram illustrating the generation of unlabeled batches after initial processing.

[0061] Specifically, in order to generate augmented unlabeled training inputs for a given unlabeled training input 320, the system generates K augmented unlabeled training inputs 330 from the unlabeled training input.

[0062] Specifically, the system applies data augmentation technology K times to the unlabeled training input to generate K augmented unlabeled training inputs 330 for the unlabeled training input 320.

[0063] Data augmentation techniques can be adapted to the type of input the model is configured to process and are random, i.e., any conventional augmentation technique that applies the same technique multiple times to the same input will typically result in multiple different augmented outputs.

[0064] As an example, when the input is an image, augmentation techniques can be applied to each input image using one or more of the following techniques: random horizontal flipping, random vertical flipping, random cropping, or random rotation.

[0065] As another example, when the input is an image, augmentation techniques can be techniques that add perturbations from a distribution, such as a Gaussian distribution, to each input.

[0066] As another example, when the input is audio data, amplification techniques can be techniques that add perturbations from a distribution, such as a Gaussian distribution, to each input.

[0067] As another example, when the input includes text data, augmentation techniques can be techniques that assign a probability to each word in the text data, select a fixed number of words according to the probability, and then replace each selected word with different words, such as words sampled from a vocabulary of possible words.

[0068] The system then uses a machine learning model to process each of the K amplified unlabeled training inputs 330 according to the current value of the parameters, so as to generate a corresponding model output 340 for each amplified unlabeled training input, that is, to generate K model outputs 340.

[0069] The system then generates a single guessing model output from the model output for the K amplified unlabeled training inputs and associates the guessing model output with each of the K amplified unlabeled training inputs 330, that is, associates each amplified unlabeled training input 330 with the same guessing model output.

[0070] More specifically, the system calculates the average value of the model output, 350, for K amplified unlabeled training inputs.

[0071] In some implementations, the system uses the average value as the output of the guessing model.

[0072] However, in some other implementations, the system applies a sharpening function 360 to the average value of the model output to reduce uncertainty in the average value, and then uses the output of the sharpening function as the guessing model output.

[0073] Specifically, when the model output includes L probabilities, the sharpening function satisfies the following for the i-th probability in the model output:

[0074]

[0075] Where p i T is the value of the i-th probability in the average of the model output, and T is a hyperparameter between 0 and 1, excluding values ​​such as .25, .5, or .75. By setting T between 0 and 1, the system sharpens the average probability distribution to reduce its entropy.

[0076] Therefore, as can be seen from the above description, the guessing model output for a given unlabeled training input is generated based on the model output generated by an augmented version of the model for the training input, i.e., and is not generated from any external ground truth data.

[0077] In some implementations, the system uses the initially processed labeled batch and the initially processed unlabeled batch as the final processed batch for a given iteration of training, i.e., as a reference on which the above calculation is performed. Figure 2 The batch of gradients described.

[0078] However, in some other implementations, the system further processes the initially processed labeled batches, the initially processed unlabeled batches, or both to generate the final processed batches. For example, the system can perform this further processing to regularize the training process and improve the model's generalization once trained.

[0079] In particular, in some implementations, the system generates the processed final batch by generating a processed labeled input associated with the processed ground truth output for each specific amplified labeled input and associated ground truth output (step 306).

[0080] Specifically, for each given amplified labeled input, the system selects an input-output pair from a set that includes at least the amplified labeled input and the associated ground truth output.

[0081] In other words, in some cases, the set may only include the amplified labeled inputs and the associated ground truth outputs.

[0082] However, in other cases, this set includes (i) amplified labeled inputs and associated ground truth outputs, and (ii) amplified unlabeled inputs and associated guessed outputs. In some cases, including both (i) and (ii) in this set can provide improved regularization for model training.

[0083] For example, the system can select pairs by randomly sampling from the set without replacement, i.e., such that the same pair is not selected for the labeled input of more than one specific amplification.

[0084] The system then performs a convex combination of the amplified labeled input and the input from the selected pair to generate the processed input. To perform this convex combination, the system is able to sample weights λ from a predetermined distribution and then compute a weighted sum between the amplified labeled input and the input from the selected pair, wherein the amplified labeled input is assigned weight λ and the input from the selected pair is assigned weight (1-λ).

[0085] The system also performs a convex combination of the ground truth output associated with the amplified labeled input and the output of the selected pair to generate the processed model output. To perform this convex combination, the system is able to compute a weighted sum between the ground truth output and the output of the selected pair, where the ground truth is assigned a weight λ and the output of the selected pair is assigned a weight (1-λ).

[0086] The system then correlates the processed input with the processed output.

[0087] In addition to performing step 306, in some implementations, the system generates a processed unlabeled input associated with the processed guessed output for each specific amplified unlabeled input and associated guessed output (step 308).

[0088] Specifically, for each unlabeled input for a given specific amplification, the system selects input-output pairs from the set including (i) and (ii) above. For example, the system can select pairs by randomly sampling from the set without replacement, i.e., such that the same pair is not selected for labeled inputs for more than one specific amplification. When step 306 is also performed, before sampling pairs of unlabeled inputs for amplification, the system removes any pairs sampled when generating the final processed labeled batch from the set.

[0089] The system then performs a convex combination of the amplified unlabeled input and the inputs from the selected pair to generate the processed input. To perform this convex combination, the system is able to sample the weights λ from a predetermined distribution and then compute the weighted sum as described above with reference to step 306.

[0090] The system also performs a convex combination of the guessed output associated with the amplified unlabeled input and the output of the selected pair to generate the processed model output, i.e., by calculating a weighted sum using λ as described above.

[0091] The system then correlates the processed input with the processed output.

[0092] In some implementations, instead of using values ​​directly sampled from the distribution as λ values, the system can use modified values ​​as λ values ​​for use in convex combinations. Specifically, when the distribution is in a range of values ​​between 0 and 1 (excluding or including 0 and 1), the system can sample values ​​from the distribution and then set the λ value to the maximum of (i) the sampled value and (ii) 1 minus the sampled value. This ensures that the processed input of a particular amplification, i.e., labeled or unlabeled input, is closer to the original amplified input than the input sampled from the set.

[0093] Figure 4 The performance of the described technique relative to other semi-supervised learning techniques is shown.

[0094] In particular, Figure 4 The diagram shows a comparison of the described technique (“MixMatch”) with several competing baselines on two datasets using different numbers of labeled inputs. Specifically, Figure 410 shows the performance of the described technique and baseline set on the CIFAR-10 dataset, while Figure 420 shows the performance of the described technique and baseline set on the SVHN dataset.

[0095] If from Figure 4 As observed, the trained model using the described technique consistently exhibits a lower error rate than the baseline, regardless of the size of the labeled data. In other words, the described technique consistently outperforms the baseline given different sizes of labeled data.

[0096] Therefore, even compared to other semi-supervised learning techniques, i.e., other techniques that use both labeled and unlabeled data, the described technique produces more efficient model training.

[0097] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that the system has software, firmware, hardware, or a combination of software, firmware, and hardware installed thereon that causes the system to perform those operations or actions in operation. For one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0098] The embodiments of the subject matter and functional operation described in this specification can be implemented as digital electronic circuits, as tangibly embodied computer software or firmware, as computer hardware including the structures disclosed in this specification and their structural equivalents, or as a combination of one or more of these. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of these. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus.

[0099] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0100] A computer program, also referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program can be stored as a portion of a file holding other programs or data, for example, as one or more scripts stored in a markup language document; in a single file dedicated to the program involved; or in multiple coordinating files, such as a file storing one or more modules, subroutines, or portions of code. A computer program can be deployed to be executed on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0101] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or not at all, and that can be stored on storage devices in one or more locations. Thus, for example, an indexed database can comprise multiple collections of data, each of which can be organized and accessed differently.

[0102] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines can be installed and run on the same computer or multiple computers.

[0103] The processes and logic described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. These processes and logic can also be executed by dedicated logic circuitry, such as FPGAs or ASICs, or by a combination of dedicated logic circuitry and one or more programmable computers.

[0104] A computer suitable for executing computer programs can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The essential components of a computer are the central processing unit for executing or carrying out instructions and one or more storage devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to said one or more mass storage devices, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game controller, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0105] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0106] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from a device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and in turn receiving response messages from the user.

[0107] The data processing apparatus for implementing machine learning models may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0108] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0109] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server; or middleware components, such as an application server; or frontend components, such as a client computer having a graphical user interface, web browser, or app through which a user can interact with embodiments of the subject matter described in this specification; or any combination including one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication through any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0110] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data, such as HTML pages, to a user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, can be received from the device at the server.

[0111] While this specification contains numerous details of specific implementation, these should not be construed as limiting the scope of any invention or potentially claimed claims, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments within this specification can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination can be removed from that combination in some cases, and the claimed combination can be directed as a sub-combination or a variation of a sub-combination.

[0112] Similarly, although the operations are depicted in a specific order in the accompanying drawings and described in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential manner, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing can be beneficial. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0113] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired result. In some cases, multitasking and parallel processing can be beneficial.

Claims

1. A method for training a machine learning model with multiple parameters to perform a machine learning task, wherein, The machine learning model is configured to receive input and process the input according to the parameters to generate a model output, the method comprising: Receive unlabeled batches that include multiple unlabeled training inputs; Receive a batch of labeled training inputs comprising multiple labeled training inputs, and for each labeled training input, receive a ground truth output that should be generated by the machine learning model by performing a specific machine learning task on the labeled training input; Generating processed unlabeled batches and processed labeled batches from the unlabeled batches and the labeled batches, the generation includes: For each of the plurality of unlabeled training inputs: Multiple amplified unlabeled training inputs are generated from the unlabeled training inputs; The machine learning model is used to process each of the amplified unlabeled training inputs according to the current value of the parameter, so as to generate a corresponding model output for each amplified unlabeled training input; A single corresponding guessing model output is generated from the model output of the amplified unlabeled training input, wherein the corresponding guessing model output is based on the average of the model outputs of the amplified unlabeled training input generated from the same unlabeled training input; and The corresponding guessing model output is associated with each of the amplified unlabeled training inputs, wherein each amplified unlabeled training input generated from the same unlabeled training input is associated with the same corresponding guessing model output; and the machine learning model is trained on the processed labeled batch and the processed unlabeled batch to adjust the current values ​​of the parameters. The machine learning model takes an image as input and outputs a probability distribution over a set of object classes. The machine learning model takes one or more video frames as input and outputs a probability distribution over a set of object classes or a probability distribution over a set of topics, or... The machine learning model takes text as input and outputs a probability distribution over a set of topics. The machine learning model takes an audio signal as input and outputs a probability distribution over a set of natural language text.

2. The method according to claim 1, wherein, Generating a guessing model output from the model output of the amplified unlabeled training input includes: Calculate the average value of the model output based on the amplified unlabeled training input.

3. The method according to claim 2, wherein, Generating a guessing model output from the model output of the amplified unlabeled training input further includes: A sharpening function is applied to the average value of the model output to reduce the uncertainty of the average value.

4. The method according to claim 1, wherein, Generating processed unlabeled batches and processed labeled batches from the unlabeled batches and the labeled batches further includes: For each of the plurality of labeled training inputs: Generate amplified labeled training input from the labeled training input; and The amplified labeled training input is associated with the ground truth output of the labeled training input.

5. The method according to claim 4, wherein, Generating processed unlabeled batches and processed labeled batches from the unlabeled batches and the labeled batches further includes: For each specific amplified labeled input and associated ground truth output, a processed labeled input is generated that is associated with the processed ground truth output, including: Choose input-output pairs from the following sets: (i) amplified labeled inputs and associated ground truth outputs and (ii) amplified unlabeled inputs and associated guessed outputs; The amplified labeled input is combined with the input in the input selection pair to generate the processed input; Perform a convex combination of the ground truth output associated with the amplified labeled input and the output from the selected pair to generate the processed output; and Associate the processed input with the processed output.

6. The method according to any one of claims 4 or 5, wherein, Generating processed unlabeled batches and processed labeled batches from the unlabeled batches and the labeled batches further includes: For each specific amplified unlabeled input and associated guess output, generate a processed unlabeled input associated with the processed guess output, including: Choose input-output pairs from the following sets: (i) amplified labeled inputs and associated ground truth outputs and (ii) amplified unlabeled inputs and associated guessed outputs; Perform a convex combination of the amplified unlabeled input and the input from the selected pair to generate the processed input; Perform a convex combination of the guessed output associated with the amplified unlabeled input and the selected output to generate the processed output; and Associate the processed input with the processed output.

7. A system comprising one or more computers and one or more storage devices and storage instructions, wherein the instructions, when executed by the one or more computers, cause the one or more computers to perform the operation of a corresponding method according to any one of claims 1-6.

8. A computer storage medium for storing one or more instructions, said instructions, when executed by one or more computers, causing said one or more computers to perform the operation of a corresponding method according to any one of claims 1-6.

Images