Training machine learning models using unsupervised data augmentation

Abstract

Methods, systems, and apparatus for training machine learning models include computer programs encoded on a computer storage medium. One method includes receiving training data comprising a plurality of unlabeled training inputs and a plurality of labeled training inputs; generating augmented training data, including generating corresponding augmented training inputs for each of the plurality of unlabeled training inputs by applying data augmentation techniques to the unlabeled training inputs; and training a machine learning model on the augmented training data. In particular, but not exclusively, the model can be trained for a perceptual task (e.g., a task related to vision or speech).

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 62 / 838,932, filed April 25, 2019, the entire contents of which are incorporated herein by reference. Background Technology

[0003] This specification relates to training machine learning models. Specifically, but not exclusively, this specification relates to training machine learning models for perception tasks (e.g., tasks related to vision or speech).

[0004] Machine learning models receive input and generate outputs, such as predicting outputs, based on the received input and the values ​​of the model's parameters.

[0005] A neural network is a machine learning model that uses one or more non-linear units to predict the output of a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer serves as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding parameter set.

[0006] Some neural networks are recurrent neural networks (RNNs). A RNN is a neural network that receives an input sequence and generates an output sequence from it. Specifically, a RNN may use some or all of the network's internal states from previous time steps when computing the output at the current time step. An example of a RNN is a Long Short-Term Memory (LSTM) neural network, which includes one or more LSTM memory blocks. Each LSTM memory block may include one or more units, each unit including an input gate, a forget gate, and an output gate, which allows the unit to store its previous state, for example, used to generate the current activation, or provided to other components of the LSTM neural network. Summary of the Invention

[0007] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which trains a machine learning model to perform a machine learning task through semi-supervised learning, i.e., by training the machine learning model on training data including unlabeled training inputs and labeled training inputs. Labeled training inputs are ground truth outputs—that is, the outputs that the machine learning model should generate by performing a specific machine learning task on the labeled training inputs—that are available inputs. Unlabeled training inputs are training inputs where ground truth outputs are not available.

[0008] The system trains a machine learning model by generating augmented training data from the training data and then training the machine learning model on the augmented training data. Generating the augmented training data involves generating a corresponding augmented training input for each of the plurality of unlabeled training inputs by applying data augmentation techniques to the unlabeled training inputs.

[0009] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. By training a machine learning model as described in this specification, the machine learning model can be trained to generate accurate model outputs. Specifically, by training a machine learning model as described in this specification, unlabeled training data can be effectively incorporated into the training, thereby improving the performance of the machine learning model. That is, the performance of the trained model can be improved without additional labeled training data. Because unlabeled training data is generally more readily available than labeled training data, the effectiveness of training can be improved without significantly increasing the time and computational cost of obtaining or generating additional training data. More specifically, enhancement techniques, such as those traditionally applied to labeled training data, can be effectively applied to unlabeled data to improve the training of machine learning models without requiring additional labeled or unlabeled training data. Furthermore, even if only a relatively small amount of labeled training data is available, the machine learning model can be trained efficiently by effectively incorporating readily available unlabeled training data into the training as described in this specification. Therefore, training the model to generate accurate model outputs becomes less dependent on the availability of accurate labeled training data. As a result, the model can be trained in an efficient manner and has far fewer labeled examples compared to a model trained on a full set of supervision.

[0010] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0011] Figure 1 An example machine learning model training system is shown.

[0012] Figure 2 This is a flowchart of an example process for training a machine learning model.

[0013] Figure 3 This is a flowchart of an example process for training a machine learning model on unlabeled training input batches and labeled training input batches.

[0014] Figure 4 The performance of the described technique relative to other semi-supervised learning techniques is shown.

[0015] In the various figures, the same reference numerals and designations indicate the same elements. Detailed Implementation

[0016] Figure 1 An example machine learning model training system 100 is shown. The machine learning model training system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, wherein the systems, components and techniques described below can be implemented.

[0017] The machine learning model training system 100 is a system for training a machine learning model 110 on training data including labeled training data 140 and unlabeled training data 150 to determine the training values ​​of the parameters of the machine learning model 110—referred to as model parameters in this specification—from the initial values ​​of the model parameters.

[0018] Machine learning model 110 is a machine learning model configured to receive model input 102 and process the model input to map the model input 102 to model output 112 according to the model parameters in order to perform a specific machine learning task.

[0019] The machine learning model 110 can be configured to perform any of a variety of machine learning tasks, that is, to receive any kind of numerical data input as input and generate a model output from that input. Typically, the model output is a probability distribution over a set of possible classifications.

[0020] For example, if the task is a computer vision task, such as image classification, then the input of model 110 is an image, and the model output given an image may be the probability of each object category in the set of object categories, where each probability represents the estimated likelihood that the image contains an object belonging to that category.

[0021] As another example, if the machine learning task is document classification, the input to machine learning model 110 is text from an internet resource (e.g., a webpage) or a document, and the model output, given an internet resource, document, or part of a document, can be a score for each topic in a set of topics, where each score represents the estimated likelihood that the internet resource, document, or part of a document is about that topic.

[0022] As another example, if the task is a natural language understanding task, then the input to the machine learning model 110 is a text sequence, and the model output given the text sequence may be a probability distribution suitable for the natural language understanding task, such as a distribution on language acceptability category, language sentiment category, language paraphrase category, sentence similarity category, text implication category, question answering category, etc.

[0023] As another example, if the task is a health prediction task, the input to the machine learning model 110 is the patient's electronic health record data, and the model output for a given sequence can be a probability distribution on the patient's health-related categories, such as the patient's possible diagnosis, possible future health events associated with the patient, etc.

[0024] As another example, if the task is a speech processing task, the input to the machine learning model 110 may be audio data representing spoken speech, such as raw audio or acoustic features, and the model output may be a probability distribution over a set of speech classification categories, such as a probability distribution over possible languages, a probability distribution over a set of possible hot words, and so on.

[0025] The machine learning model 110 may have any architecture suitable for the type of model input processed by the machine learning model 110. For example, when the model input is an image, the machine learning model 110 may be a convolutional neural network. When the model input is a sequence of text or a sequence of other features (e.g., electronic health record features), the machine learning model 110 may be a self-attention-based neural network—e.g., a transformer—or a recurrent neural network—e.g., a long short-term memory (LSTM) neural network. When the model input includes multiple modalities, such as both images and text, the model 110 may include different types of neural networks, such as both convolutional layers and self-attention layers or recurrent layers.

[0026] The labeled training data 140 used by system 100 to train machine learning model 110 includes multiple batches of labeled training inputs. The training inputs are referred to as “labeled” training inputs because the labeled training data 140 also includes ground truth outputs for each labeled training input; that is, the outputs that the machine learning model should generate by performing a specific machine learning task on the labeled training inputs. In other words, the ground truth outputs are the actual outputs of the machine learning task when performed on the corresponding labeled training inputs.

[0027] The unlabeled training data 150 used by system 100 to train machine learning model 110 includes multiple batches of unlabeled training inputs. The training inputs are referred to as “unlabeled” training inputs because the ground truth outputs of the unlabeled training inputs are not available; that is, system 100 cannot access any ground truth outputs of any unlabeled training inputs, or for some other reason, cannot use any ground truth outputs of the unlabeled training inputs to train model 110.

[0028] Typically, an unlabeled training input batch may include the same number of training inputs as a labeled training input batch, or it may include a different number of training inputs. For example, in the common case where there are significantly more unlabeled training items available than labeled training items, an unlabeled training item batch may include more training items than a labeled training item batch.

[0029] Typically, system 100 trains machine learning model 110 by generating augmented training data from training data and then training machine learning model 110 on the augmented training data. To generate augmented training data, system 100 generates corresponding augmented training inputs for each of a plurality of unlabeled training inputs by applying data augmentation techniques to that unlabeled training input.

[0030] The following is for reference. Figure 2 and Figure 3 The generation of augmented training data and the training of machine learning models on the augmented training data are described in more detail 110.

[0031] Once model 110 has been trained, system 100 can provide data specifying the training model for processing new network inputs. That is, system 100 can output training values ​​of model parameters, for example, by outputting to a user device or by storing them in memory accessible to system 100, for later use when processing inputs using the trained model.

[0032] As a replacement or supplement to the output of trained machine learning model data, system 100 may instantiate an instance of a machine learning model with training values ​​of model parameters, for example, by receiving input to be processed through an application programming interface (API) provided by the system, using the trained model to process the received input to generate model output, and then providing the generated model output, classification output, or both in response to the received input.

[0033] Figure 2 This is a flowchart of an example process 200 for training a machine learning model on training data including unlabeled and labeled training inputs. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a machine learning model training system, such as one that is appropriately programmed... Figure 1 The machine learning model training system 100 has an executable process 200.

[0034] The system receives training data for training a machine learning model to map model inputs to model outputs in order to perform a specific machine learning task (step 202). As described above, the training data includes both unlabeled training inputs and labeled training inputs. For each labeled training input, the training data also includes ground truth outputs that should be generated by the machine learning model by performing a specific machine learning task on the labeled training inputs.

[0035] The system generates augmented training data from the training data (step 204). Specifically, the system generates corresponding augmented unlabeled training inputs for each of a plurality of unlabeled training inputs by applying data augmentation techniques to that unlabeled training input.

[0036] The data augmentation techniques employed depend on what type of input the machine learning model is operating on.

[0037] For example, when the model input is an image, data augmentation techniques can be learned, data-dependent techniques, such as those learned using automatic augmentation methods. In other words, the system can use data-dependent automatic augmentation techniques to search from a set of image processing transformations for a combination of learned image processing transformations on labeled inputs to use as augmentation techniques for unlabeled inputs. Automatic augmentation is described in more detail in Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data.arXiv preprintarXiv:1805.09501,2018.

[0038] As another example, when the model input is an image, data augmentation techniques can be determined through hyperparameter search, i.e., before model training. For instance, a data augmentation technique could be RandAdugment, which applies grid search or other hyperparameter optimization techniques to identify the values ​​of hyperparameters that define the augmentation strategy: for example, defining multiple transformations applied to each image and defining the magnitude of each applied transformation. RandAdugment is described in more detail in Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. RandAdugment: Practical data augmentation with no separate search.arXiv preprint, 2019.

[0039] As another example, when the model input is natural language text, data augmentation techniques can be reverse translation techniques. Reverse translation refers to the process of translating an existing example in language A into another language B and then translating it back into A to obtain an augmented example. For example, the system can randomly select one or more words from the model input and reverse translate the randomly selected words to generate augmented unlabeled input.

[0040] As another example, when the model input is natural language text, data augmentation techniques can include TF-IDF-based word substitution. TF-IDF-based word substitution is a technique that replaces uninformative words with lower TF-IDF scores while preserving those with higher TF-IDF values. For example, the system can assign a probability based on the word's TF-IDF value to each word in the input, where words with higher TF-IDF values ​​have higher probabilities. The system can then sample a predetermined number of words based on the assigned probabilities and replace each sampled word with, for example, words sampled from a possible vocabulary of the input words.

[0041] Therefore, the augmented training data includes (i) labeled training inputs, (ii) unlabeled training inputs, and (iii) augmented unlabeled training inputs, wherein each unlabeled training input corresponds to one of the unlabeled training inputs.

[0042] Although (i) and (ii) will be described in this specification as training inputs that have not yet been augmented, in some cases, the system may apply simple augmentations—e.g., cropping and flipping when the input is an image—to the training examples in (i) and (ii) to improve the robustness of the training process. These simple augmentations are typically different from the augmentation strategies used to generate (iii) from (ii); that is, the augmentation strategies are usually much more complex than simple augmentations. In other words, in some cases, the labeled and unlabeled training inputs in the augmented training data already have simple augmentations applied using different augmentation strategies.

[0043] The system then trains the machine learning model on the augmented training data (step 206). Specifically, the system trains the model to optimize an objective, such as minimizing a loss, which is a combination of an unsupervised objective and a supervised objective, such as a sum, an average, or a weighted sum.

[0044] The unsupervised objective measures the difference between (i) the model output generated by the machine learning model for the given unlabeled training input and (ii) the model output generated by the machine learning model for the corresponding augmented unsupervised training input, i.e., the augmented training input generated from the unlabeled training input.

[0045] As a specific example, an unsupervised objective can be based on the Kullback-Leibler (KL) divergence between (i) the model output generated by a machine learning model for a given unlabeled training input and (ii) the model output generated by a machine learning model for a corresponding augmented unsupervised training input. In this example, the unsupervised objective can satisfy:

[0046]

[0047] Where λ is a normal value, such as 1.5, 1, or 0.5, E is the expectation operator, and x is the unlabeled training input from the unlabeled training data U. It is the corresponding augmented unlabeled training input generated from x by applying the augmentation policy q. It is (i) the model output generated by machine learning model p for unlabeled training input x and (ii) the output generated by machine learning model p for augmented training input x. The KL divergence between the generated model outputs Represents model parameters, and This represents a fixed copy of the current model parameters θ, thus indicating that gradients do not propagate through the model when calculating model parameter updates.

[0048] The supervised objective, for a given labeled training input, measures (i) the difference between the model output generated by the machine learning model for the given labeled training input and (ii) the ground truth output for the given labeled training input. As a specific example, the supervised objective could be based on the negative log-likelihood of the given ground truth output according to the probability distribution generated for the given labeled training input. In other words, the supervised objective could satisfy:

[0049]

[0050] Where x is the labeled training input from the labeled training data L, and y * This is the ground truth output for x.

[0051] In some implementations, there can be a large difference between the amount of unlabeled data and the amount of labeled data available for training a machine learning model; that is, there may be a much larger amount of unlabeled data in the available training data than labeled data. This can cause the model to quickly overfit to the finite amount of labeled data while still not adequately fitting the unlabeled data. To mitigate this difficulty, some implementations use a technique of gradually releasing the “training signal” of labeled examples as training progresses.

[0052] At a higher level, this technique allows the system to utilize only a labeled example if the model's confidence in it falls below a predefined threshold, which is increased during training based on scheduling.

[0053] More specifically, instead of using negative log-likelihood as the supervision objective as described above, the system uses a modified supervision objective.

[0054] Similar to the above objective, the modified supervision objective is based on the negative log-likelihood of a given ground truth output according to the probability distribution generated for a given labeled training input, but only if the probability assigned to the given ground truth output by the probability distribution is less than a confidence threshold.

[0055] The system removes a given training input from the loss function when the probability assigned to a given ground truth output by the probability distribution is equal to or greater than the confidence threshold. This removal can be achieved by setting the supervision target to zero.

[0056] Typically, the system increases this confidence threshold as training progresses. For example, if the number of classes on which the probability distribution is generated is equal to K, the system can gradually increase the confidence threshold from 1 / K to 1 as training progresses. As a specific example, the system can increase the confidence threshold after each training step, that is, after each iteration of the process 300 described below.

[0057] The system can increase the confidence threshold based on any schedule in various schedules.

[0058] For example, the system can increment the schedule based on logarithmic scheduling. In this example, the threshold value at training step t satisfies:

[0059] in And T is the total number of training steps.

[0060] As another example, the system can increment the schedule based on a linear schedule. In this example,

[0061] As another example, the system can increment the schedule based on exponential scheduling. In this example,

[0062] Intuitively, exponential scheduling is most suitable when the model is prone to overfitting, for example, when the task is relatively easy or the number of labeled examples is very limited, because most of the supervision signal is released at the end of training. Conversely, logarithmic scheduling can be used when the model is less likely to overfit (e.g., when there are a large number of labeled examples or when the model is effectively regularized).

[0063] Specifically, during training, the system trains the machine learning model by executing multiple iterations of a machine learning training procedure, where each iteration is performed on both labeled and unlabeled training input batches. During each iteration, the system determines an update to the current values ​​of the model parameters from the iteration's current values ​​based on both supervised and unsupervised objectives, and applies the update to the current values ​​to generate updated values ​​for the model parameters.

[0064] Figure 3 This is a flowchart of an example process 300 for training a machine learning model on unlabeled training input batches and labeled training input batches. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a machine learning model training system, appropriately programmed, such as... Figure 1 The machine learning model training system 100, executable process 300.

[0065] The system can execute process 300 multiple times for various combinations of labeled and unlabeled batches to determine the training values ​​of the model parameters based on their initial values. For example, the system can continue executing process 300 for a specified number of iterations, a specified amount of time, or until the change in parameter values ​​falls below a threshold.

[0066] The system obtains the labeled training input batch (step 302).

[0067] The system uses a machine learning model and processes each labeled training input based on the current values ​​of the model parameters to generate a corresponding model output for each labeled training input (step 304).

[0068] The system obtains unlabeled batches of unlabeled training inputs and augmented unlabeled training inputs for each unlabeled training input in the unlabeled batches (step 306).

[0069] The system uses a machine learning model and processes each unlabeled training input and each augmented unlabeled training input according to the current values ​​of the model parameters to generate the corresponding model output for each unlabeled training input (step 308).

[0070] The system updates the current values ​​of the model parameters based on the model output used for labeled training input, unlabeled training input, and unlabeled training input (step 310).

[0071] Specifically, the system determines the gradient of the model parameters with respect to the supervised objective for each labeled training input and the gradient of the model parameters with respect to the unsupervised objective for each unlabeled training input.

[0072] The system then combines the gradients to generate a combined gradient, for example by averaging or summing the corresponding gradients. This combined gradient is then used to update the current values ​​of the model parameters, for example by applying an update rule (e.g., learning rate), an Adam optimizer update rule, or an rmsProp update rule to the combined gradient to generate an update. This update is then applied to the current value, either by subtracting or adding.

[0073] Figure 4 The performance of the described technique relative to other semi-supervised learning techniques is shown.

[0074] In particular, Figure 4 The described techniques (“UDA” and “Randauggment” enhancement strategies) are shown in comparison with the following two highly competitive baselines: (1) Virtual Adversarial Training (VAT) (Miyato et al., 2018), an algorithm that generates adversarial Gaussian noise on the input, and (2) MixMatch, a technique that combines previous advances in semi-supervised learning (Berthelot et al., 2019).

[0075] As from Figure 4 As can be seen, the training model trained using the described technique consistently has a low error rate, meaning it outperforms the two baselines on different sizes of labeled data. In other words, given different sizes of labeled data, the described technique consistently outperforms the two baselines with a clear margin.

[0076] Therefore, even compared to other semi-supervised learning techniques—that is, other techniques that use both labeled and unlabeled data—the described technique leads to more efficient model training.

[0077] This specification uses the term "configured" together with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that the system has software, firmware, hardware, or a combination of software, firmware, and hardware installed thereon that causes the system to perform those operations or actions in operation. For one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0078] Embodiments of the subject matter and functional operation described in this specification may be implemented using digital electronic circuits, computer software or firmware tangibly implemented, computer hardware including the structures disclosed in this specification and their equivalents, or a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, which are one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of these. Alternatively or further, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus.

[0079] The term "data processing apparatus" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0080] A computer program, also referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; and it may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file holding other programs or data, such as one or more scripts stored in a markup language document; in a single file dedicated to said program; or in multiple coordinating files, such as a file storing one or more modules, subroutines, or portions of code. A computer program may be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a data communication network.

[0081] In this specification, the term "database" is used broadly to refer to any dataset: data that does not need to be structured in any particular way, or not structured at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may include multiple datasets, each of which may be organized and accessed differently.

[0082] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines may be installed and run on the same computer or on multiple computers.

[0083] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry, such as FPGAs or ASICs, or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0084] A computer suitable for executing computer programs may be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The essential components of a computer are the central processing unit for executing or carrying out instructions and one or more storage devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices for storing data, or operatively coupled to receive data from or transfer data to, or both of, said mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game controller, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0085] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0086] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to a user and a keyboard and pointing device that the user can use to provide input to the computer. The display device is, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and the pointing device is, for example, a mouse or trackball. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents to and from a device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application.

[0087] The data processing apparatus for implementing machine learning models may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production—i.e., inference, workloads.

[0088] Machine learning frameworks can be used to implement and deploy machine learning models. These frameworks include, for example, TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0089] Embodiments of the subject matter described in this specification can be implemented in computing systems that include backend components, such as data servers; or middleware components, such as application servers; or frontend components, such as client computers having a graphical user interface, web browser, or app that a user can use to interact with implementations of the subject matter described in this specification; or any combination of one or more such backend, middleware, or frontend components. Components of the system can be interconnected via digital data communication through any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0090] A computing system may include clients and servers. Clients and servers are generally geographically distant and typically interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data, such as HTML pages, to a user device for purposes such as displaying data to a user interacting with the device as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0091] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or potentially claimed claims, but rather as descriptions of features that may be specific to particular embodiments of the invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0092] Similarly, although operations are depicted in the drawings and described in the claims in a specific order, this should not be construed as requiring the operations to be performed in the specific order shown or in sequential order, or requiring the execution of all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0093] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require a specific order or sequence to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for training a machine learning model, said machine learning model being configured to map model inputs to model outputs to perform a specific machine learning task, wherein: The machine learning model is trained to map model inputs, including visual data, to model outputs, thereby performing computer vision tasks, or The machine learning model is trained to map input, including audio data representing spoken utterances, to output to perform a speech classification task, or The model input includes the patient's electronic health record data, and the corresponding model output includes a probability distribution of possible diagnoses for the patient, or The specific machine learning task is document classification, the model input is text from internet resources or documents, and the model output, given an internet resource, document, or document portion, is a score for each topic in a topic set, where each score represents the estimated likelihood of the internet resource, document, or document portion regarding that topic. The specific machine learning task is a natural language understanding task, the model input is a text sequence, and the model output given the text sequence is a probability distribution suitable for the natural language understanding task. The method includes: Receive training data for training the machine learning model, the training data including: Multiple unlabeled training inputs; and Multiple labeled training inputs, and ground truth outputs for each labeled training input that should be generated by the machine learning model by performing the specific machine learning task on the labeled training input; Generating augmented training data includes generating corresponding augmented unlabeled training inputs for each of the plurality of unlabeled training inputs by applying data augmentation techniques to that unlabeled training input; Training the machine learning model on the enhanced training data includes: The machine learning model is trained on the plurality of unlabeled training inputs and the corresponding plurality of augmented unlabeled training inputs to optimize an unsupervised objective, the unsupervised objective being measured as (i) the difference between the model output generated by the machine learning model for a given unlabeled training input and (ii) the difference between the model output generated by the machine learning model for the augmented unlabeled training input generated from the given unlabeled training input, and The machine learning model is trained on the plurality of labeled training inputs to optimize a supervised objective, the supervised objective measuring (i) the difference between the model output generated by the machine learning model for a given labeled training input and (ii) the ground truth output for the given labeled training input.

2. The method according to claim 1, wherein, The model output is a probability distribution, and the unsupervised objective is based on (i) the KL divergence between the model output generated by the machine learning model for the given unlabeled training input and (ii) the model output generated by the machine learning model for the enhanced unlabeled training input generated from the given unlabeled training input.

3. The method according to claim 1, wherein, The model output is a probability distribution, and the supervision objective is based on the negative log-likelihood of the ground truth output for the given labeled training input, according to the probability distribution generated for the given labeled training input.

4. The method according to claim 1, wherein, The labeled training input and the unlabeled training input in the enhanced training data have been enhanced by applying a data augmentation technique different from the data augmentation technique used to generate the enhanced unlabeled training input.

5. The method according to claim 1, wherein, The model output is a probability distribution, where: When the probability assigned to the ground truth output by the probability distribution is less than a confidence threshold, the supervision objective is based on the negative log-likelihood of the ground truth output according to the probability distribution generated for the given labeled training input, and The supervised target is zero when the probability assigned to the ground truth output by the probability distribution is equal to or greater than the confidence threshold.

6. The method according to claim 5, wherein, The training includes: The confidence threshold is increased as training progresses.

7. The method according to claim 6, wherein, Increasing the confidence threshold includes increasing the confidence threshold after each training step.

8. The method according to claim 1, wherein, The computer vision task is image classification, the model input is an image, and the model output for a given image is the probability of each object category in the set of object categories, each probability representing the estimated likelihood that the image contains an object belonging to that category.

9. A system for training a machine learning model, the system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform the operations of the method according to any one of claims 1-8.

10. A computer storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the method according to any one of claims 1-8.

Images