Method, system and computer storage medium for training machine learning models

By dynamically adjusting the task selection strategy and using more accurate learning progress signals, combined with adversarial robber algorithms, the problems of low training efficiency and inaccurate learning progress in the existing technology are solved, and more efficient machine learning model training and optimization are achieved.

CN117709426BActive Publication Date: 2024-11-15GDM HOLDINGS LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311499522.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-02-24
Filing Date
2018-02-19
Publication Date
2024-11-15
Estimated Expiration
2038-02-19

AI Technical Summary

Technical Problem

The existing machine learning model training methods are difficult to dynamically adjust the task selection strategy, resulting in inefficient training and inaccurate traditional learning progress signals, which affects the rapid training and optimization of the model.

Method used

By dynamically modifying the task selection strategy during training, combining the learning progress signal to measure based on the ratio of model complexity, the adversarial robber algorithm update strategy is used to optimize the adjustment and selection of model parameters.

Benefits of technology

Faster model training is realized, the consumption of computing resources is reduced, the task performance of the model under the same number of iterations is improved, and the learning progress signal is more accurate, so that good performance can be achieved faster.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117709426B_ABST
    Figure CN117709426B_ABST
Patent Text Reader

Abstract

Methods, systems, and apparatus for training a machine learning model, including a computer program encoded on a computer storage medium. In one aspect, the method includes receiving training data for training a machine learning model on a plurality of tasks, wherein each task includes a plurality of batches of training data. Selecting a task according to a current task selection policy. Selecting a batch of training data from the selected task. Training the machine learning model on the selected batch of training data to determine updated values ​​for model parameters. Determining a learning progress measure representing the progress of training the machine learning model as a result of training the machine learning model on the selected batch of training data. Updating the current task selection policy using the learning progress measure.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application with application date of February 19, 2018, application number 201880013612.0, and invention name “Training machine learning models using learning progress measurement”. Technical Field

[0002] This specification relates to training a machine learning model. Background Art

[0003] A machine learning model receives input and generates an output, such as a predicted output, based on the received input and parameter values ​​of the model.

[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input of the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input according to the current values ​​of the corresponding set of parameters.

[0005] Some neural networks are recursive neural networks. A recursive neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recursive neural network can use some or all of the network's internal state from previous time steps to compute the output for the current time step. An example of a recursive neural network is a long short term (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block may include one or more cells, each cell including an input gate, a forget gate, and an output gate that allow the cell to store the previous state of the cell, e.g., for generating a current activation or being provided to other components of the LSTM neural network. Summary of the invention

[0006] This specification describes a system implemented as a computer program on one or more computers in one or more locations that performs training of a machine learning model.

[0007] According to a first aspect, a method is provided for training a machine learning model having multiple model parameters, for example on multiple partitions, to determine training values ​​of the model parameters from initial values ​​of the model parameters. The method may include receiving training data for training the machine learning model. For example, the training data may be divided into multiple partitions or tasks. The method may also include identifying multiple tasks. Each task / partition may include corresponding batches of training data.

[0008] The method may involve training a machine learning model on training data by repeatedly performing the following operations: selecting a task / partition from a plurality of tasks / partitions according to a current partition selection strategy; selecting a batch of training data from a plurality of batches of training data for the selected task / partition; training the machine learning model on the selected batch of training data to determine updated values ​​of model parameters from current values ​​of the model parameters; determining a learning progress measure representing the training progress of the machine learning model as a result of training the machine learning model on the selected batch of training data; and updating the current partition selection strategy using the learning progress measure.

[0009] The training data for the machine learning model can be divided into multiple partitions, each of which includes corresponding batches of training data. The partitions can correspond to tasks, and vice versa. For example, the training data in each of the partitions can be training data for tasks of different natures, where the model is trained to perform all tasks in the task well. As another example, one of the partitions can be a target partition, where the model is trained to perform well on inputs similar to the inputs in the training data in the target partition. In this example, the training data in other partitions can be data that is expected to be easier to train the model, for example.

[0010] According to a second aspect, a method for training a machine learning model having multiple model parameters to determine training values ​​of the model parameters from initial values ​​of the model parameters is provided, the method comprising: receiving training data for training the machine learning model on multiple tasks, wherein each task includes corresponding multiple batches of training data; and training the machine learning model on the training data by repeatedly performing the following operations: selecting a task from the multiple tasks according to a current task selection strategy; selecting a batch of training data from the multiple batches of training data for the selected task; training the machine learning model on the selected batch of training data to determine updated values ​​of the model parameters from current values ​​of the model parameters; determining a learning progress measurement representing the training progress of the machine learning model as a result of training the machine learning model on the selected batch of training data; and using the learning progress measurement to update the current task selection strategy.

[0011] In some implementations, training the machine learning model on the selected batch includes training the machine learning model to reduce a loss on the selected batch as measured by a loss function, and the learning progress measure is based on a reduction in the loss as a result of training the machine learning model on the selected batch of training data.

[0012] In some implementations, determining a learning progress measure includes: determining a first loss on the selected batch based on current values ​​of the model parameters; determining a second loss on the selected batch based on updated values ​​of the model parameters. The learning progress measure includes a difference between the first loss and the second loss.

[0013] In some embodiments, determining a learning progress measure includes: sampling a new batch from a plurality of batches in a selected task; determining a first loss on the new batch based on current values ​​of the model parameters; determining a second loss on the new batch based on updated values ​​of the model parameters, and the learning progress measure includes a difference between the first loss and the second loss.

[0014] In some embodiments, one of the tasks is identified as a target task, wherein the target task includes training inputs that are most similar to inputs to be processed by the machine learning model after the machine learning model is trained on the training data, and determining a learning progress measure includes: sampling a new batch from multiple batches in the target task; determining a first loss for the new batch based on current values ​​of the model parameters; determining a second loss for the new batch based on updated values ​​of the model parameters, and the learning progress measure includes a difference between the first loss and the second loss.

[0015] In some embodiments, determining a learning progress measure includes: randomly sampling tasks from a plurality of tasks; sampling a new batch from a plurality of batches in the sampled tasks; determining a first loss on the new batch based on current values ​​of model parameters; determining a second loss on the new batch based on updated values ​​of the model parameters, and the learning progress measure includes a difference between the first loss and the second loss.

[0016] In some implementations, the learning progress measure includes the norm of a gradient vector of a gradient of a loss function with respect to model parameters generated by training the machine learning model on the selected batch.

[0017] In some implementations, the learning progress measure is based on an increase in model complexity of the machine learning model as a result of training the machine learning model on a selected batch of training data.

[0018] In some embodiments, values ​​of model parameters are defined by a posterior distribution of possible values ​​of the model parameters, wherein during training, posterior distribution parameters of the parameterized posterior distribution are optimized such that training values ​​of the model parameters are defined by training values ​​of the posterior distribution parameters, and wherein training the machine learning model on a selected batch of training data includes training the machine learning model on the selected batch of training data to determine adjusted values ​​of the posterior distribution parameters from current values ​​of the posterior distribution parameters.

[0019] In some implementations, determining a learning progress measure includes: determining a first Kullback-Leibler (KL) divergence between (i) a posterior distribution defined by updated values ​​of parameters of the posterior distribution and (ii) a prior distribution of possible values ​​of the model parameters; and determining a second KL divergence between (i) the posterior distribution defined by current values ​​of the parameters of the posterior distribution and (ii) the prior distribution of possible values ​​of the model parameters, and the learning progress measure includes a difference between the first KL divergence and the second KL divergence.

[0020] In some embodiments, the prior distribution is defined by prior distribution parameters, and training the machine learning model on a selected batch of training data includes determining adjusted values ​​of the prior distribution parameters from current values ​​of the prior distribution parameters, wherein the first KL divergence is the KL divergence between (i) the posterior distribution defined by the updated values ​​of the posterior distribution parameters and (ii) the prior distribution defined by the updated values ​​of the prior distribution parameters, and wherein the second KL divergence is the KL divergence between (i) the posterior distribution defined by the current values ​​of the posterior distribution parameters and (ii) the prior distribution defined by the current values ​​of the prior distribution parameters.

[0021] In some embodiments, training a machine learning model on a selected batch of training data includes determining adjusted values ​​of prior distribution parameters from current values ​​of the prior distribution parameters, wherein the prior distribution parameters parameterize a prior distribution of possible values ​​of the model parameters, and a learning progress measure is based on a) a gradient of the prior distribution parameters and the posterior distribution parameters with respect to a KL divergence between (i) a posterior distribution defined by the current values ​​of the posterior distribution parameters and (ii) a prior distribution defined by the current values ​​of the prior distribution parameters; and b) a gradient of the posterior distribution parameters with respect to an expected loss on the selected batch as measured by a loss function.

[0022] In some implementations, the learning progress measure is based on the difference between a first norm of a vector of updated values ​​of the model parameters and a second norm of a vector of current values ​​of the model parameters.

[0023] In some implementations, the learning progress measure is based on the product between (i) the transpose of the vector of current values ​​of the model parameters and (ii) a gradient vector of the gradients of the loss function with respect to the model parameters generated as part of training the machine learning model on the selected batch.

[0024] In some implementations, the task selection strategy is a strategy that assigns a respective probability to each of the plurality of tasks, and selecting a task from the plurality of tasks according to the current task selection strategy includes randomly selecting a task according to the probabilities assigned by the current task selection strategy.

[0025] In some implementations, the task selection policy is an adversarial bandits policy, wherein updating the current task selection policy using the learning progress measurement comprises: determining a benefit from the learning progress measurement; and updating the current task selection policy based on the benefit using an adversarial bandits algorithm.

[0026] In some implementations, determining the benefit includes: normalizing the learning progress measure by dividing the learning progress measure by a normalization factor for the selected task to generate a normalized learning progress measure; and generating the benefit by scaling the normalized learning progress measure to fall within a benefit range.

[0027] In some implementations, the normalization factor for a selected task is the time required or an approximation of the time required for a machine learning model to generate an output for a training input in the selected task.

[0028] In some implementations, the approximation is the number of time steps required for the machine learning model to generate outputs for the training inputs in the selected batch, or the number of separate inputs in each of the training inputs in the selected batch.

[0029] In some implementations, selecting a batch of training data from multiple batches of training data in the selected task includes: randomly selecting a batch of training data from multiple batches of training data in the selected task.

[0030] In some embodiments, the machine learning model is a recurrent neural network.

[0031] According to a third aspect, a method for training a machine learning model having multiple model parameters to determine training values ​​of the model parameters from initial values ​​of the model parameters is provided, the method comprising: receiving training data for training the machine learning model, wherein the training data is partitioned into multiple partitions, and wherein each partition includes corresponding multiple batches of training data; and training the machine learning model on the training data by repeatedly performing the following operations: selecting a partition from the multiple partitions according to a current partition selection strategy; selecting a batch of training data from the multiple batches of training data of the selected partition; training the machine learning model on the selected batch of training data to determine updated values ​​of the model parameters from current values ​​of the model parameters; determining a learning progress measurement representing the training progress of the machine learning model as a result of training the machine learning model on the selected batch of training data; and using the learning progress measurement to update the current partition selection strategy.

[0032] According to another aspect, a method for training a machine learning model having multiple model parameters to determine training values ​​of the model parameters from initial values ​​of the model parameters is provided, wherein the values ​​of the model parameters are defined by a posterior distribution of possible values ​​of the model parameters, the method comprising: receiving training data for training the machine learning model on multiple tasks, wherein each task comprises corresponding multiple batches of training data; and training the machine learning model on the training data, wherein during training, posterior distribution parameters that parameterize the posterior distribution are optimized so that the training values ​​of the model parameters are defined by the training values ​​of the posterior distribution parameters, wherein the training comprises, in each of multiple training iterations: selecting a task from the multiple tasks according to a current task selection strategy; ... The invention relates to a method for selecting a task by selecting a batch of training data from multiple batches of training data; training a machine learning model on the selected batch of training data to determine updated values ​​of the model parameters from current values ​​of the model parameters, including training the machine learning model on the selected batch of training data to determine adjusted values ​​of the posterior distribution parameters from current values ​​of the posterior distribution parameters; determining a learning progress measurement representing the training progress of the machine learning model as a result of training the machine learning model on the selected batch of training data; and updating the current task selection strategy based on the learning progress measurement, including: determining a benefit achieved in the training iteration from the learning progress measurement; and using the benefit to update the current task selection strategy to promote the selection of a task that maximizes the cumulative measure of the benefit achieved over multiple training iterations.

[0033] According to another aspect, a system is provided, comprising one or more computers and one or more storage devices storing instructions, wherein the instructions, when executed by the one or more computers, cause the one or more computers to perform operations for training a machine learning model having multiple model parameters to determine training values ​​of the model parameters from initial values ​​of the model parameters, wherein the values ​​of the model parameters are defined by a posterior distribution of possible values ​​of the model parameters, the operations comprising: receiving training data for training the machine learning model on multiple tasks, wherein each task comprises corresponding multiple batches of training data; and training the machine learning model on the training data, wherein during training, posterior distribution parameters that parameterize the posterior distribution are optimized so that the training values ​​of the model parameters are defined by the training values ​​of the posterior distribution parameters, wherein the training comprises, in multiple training iterations, Each training iteration: selecting a task from the multiple tasks according to a current task selection strategy; selecting a batch of training data from multiple batches of training data for the selected task; training a machine learning model on the selected batch of training data to determine updated values ​​of the model parameters from current values ​​of the model parameters, including training the machine learning model on the selected batch of training data to determine adjusted values ​​of the posterior distribution parameters from current values ​​of the posterior distribution parameters; determining a learning progress measurement representing the training progress of the machine learning model as a result of training the machine learning model on the selected batch of training data; and updating the current task selection strategy based on the learning progress measurement, including: determining a benefit achieved in the training iteration from the learning progress measurement; and using the benefit to update the current task selection strategy to prompt the selection of a task that maximizes the cumulative measure of benefits achieved over multiple training iterations.

[0034] According to another aspect, one or more computer storage media are provided that store instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a machine learning model having a plurality of model parameters to determine training values ​​of the model parameters from initial values ​​of the model parameters, wherein the values ​​of the model parameters are defined by a posterior distribution of possible values ​​of the model parameters, the operations comprising: receiving training data for training the machine learning model on a plurality of tasks, wherein each task comprises a corresponding plurality of batches of training data; and training the machine learning model on the training data, wherein during training, posterior distribution parameters that parameterize the posterior distribution are optimized such that the training values ​​of the model parameters are defined by the training values ​​of the posterior distribution parameters, wherein the training comprises, at each training iteration in a plurality of training iterations: Selecting a task from the multiple tasks according to a current task selection strategy; selecting a batch of training data from multiple batches of training data for the selected task; training a machine learning model on the selected batch of training data to determine updated values ​​of the model parameters from current values ​​of the model parameters, including training the machine learning model on the selected batch of training data to determine adjusted values ​​of the posterior distribution parameters from current values ​​of the posterior distribution parameters; determining a learning progress measurement representing the training progress of the machine learning model as a result of training the machine learning model on the selected batch of training data; and updating the current task selection strategy based on the learning progress measurement, including: determining the benefit achieved in the training iteration from the learning progress measurement; and using the benefit to update the current task selection strategy to prompt the selection of a task that maximizes the cumulative measure of the benefit achieved over multiple training iterations.

[0035] The above aspects may be implemented in any convenient form. For example, aspects and embodiments may be implemented by a suitable computer program, which may be carried on a suitable carrier medium, which may be a tangible carrier medium (e.g., a disk) or an intangible carrier medium (e.g., a communication signal). Aspects may also be implemented using a suitable device, which may take the form of a programmable computer running a computer program.

[0036] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. By dynamically modifying the task selection strategy during training, a training system as described in this specification can adapt the strategy to the capabilities developed by the model during training. In addition, a training system as described in this specification can measure learning progress based on the rate at which the complexity of the machine learning model increases using a novel learning progress signal. By combining these (and other) improvements, a training system as described in this specification can train a machine learning model faster (e.g., on fewer training iterations) than a traditional training system that does not include these improvements. Because a training system as described in this specification can train a machine learning model faster than a traditional training system, the training system described in this specification can consume fewer computing resources (e.g., memory and computing power) than a traditional training system.

[0037] For example, by adaptively modifying the task selection strategy during training to account for the changing capabilities of the model, a training system as described in this specification can train a machine learning model on a task progress that maximizes the learning progress of the machine learning model. By maximizing the learning progress of the machine learning model during training, a training system as described in this specification can require fewer training iterations to train the machine learning model relative to a training system with a static task selection strategy.

[0038] As another example, the rate at which the complexity of the machine learning model increases can be a more accurate learning progress signal (i.e., can better reflect the actual learning progress of the machine learning model) than a traditional learning progress signal. By measuring the learning progress based on a more accurate learning progress signal, a training system as described in this specification maximizes the learning progress of the machine learning model during training, and thus can require fewer iterations to train the machine learning model.

[0039] In addition, a training system as described in this specification can determine trained machine learning model parameters, wherein the trained machine learning model parameters result in better task performance (e.g., higher prediction accuracy) of the trained machine learning model relative to a machine learning model trained by a traditional training system. In particular, because the training system as described in this specification achieves greater learning progress of the machine learning model during training relative to a traditional training system (as described above), the machine learning model trained by the training system described in this specification at the same fixed number of iterations can have better task performance relative to the machine learning model trained by the traditional training system at the same fixed number of iterations.

[0040] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1A An example training system is shown.

[0042] Figure 1B Examples of training data, tasks, batches, and training examples are shown.

[0043] Figure 2 is a flowchart of an example process for training a machine learning model.

[0044] Figure 3 is a flow chart of an example process for updating a task selection policy.

[0045] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0046] Figure 1A An example of a training system 100 is shown. Training system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0047] The training system 100 is configured to train the machine learning model 118 based on the training data 102 .

[0048] The model 118 is configured to receive an input and process the input according to the current values ​​of the set of machine learning model parameters 116 to generate an output for the input. For example, if the model 118 is a neural network, the model parameters 116 may include weight values ​​of neuron connections of the neural network. The current values ​​of the set of model parameters 116 may be sampled from a model parameter distribution 124 that defines a probability distribution of a set of possible values ​​of the model parameters 116. For example, the model parameter distribution 124 may characterize each of the model parameters by a normal distribution with a different mean and standard deviation.

[0049] The model 118 can be any of a variety of supervised machine learning models, such as a neural network model (e.g., a recursive neural network), a random forest model, a support vector machine model, or a linear regression model. In general, the model 118 can be configured to receive any type of digital data input, including but not limited to image, video, sound, and text data, and generate any type of score, prediction, classification, or regression output based on the input. The output data can be of the same or a different type or modality than the input data. Some applications of the model are described below; other applications include use in machine control systems.

[0050] For example, if the input to model 118 is an image or features that have been extracted from an image, the output generated by model 118 for a given image may be a score for each of a set of object categories, where each score represents an estimated likelihood that the image depicts an object belonging to that category.

[0051] As another example, if the input to model 118 is an internet resource (e.g., a web page), a document or a portion of a document, or features extracted from an internet resource, a document or a portion of a document, the output generated by model 118 for a given web resource, document or portion of a document can be a score for each of a set of topics, where each score represents an estimated likelihood that the internet resource, document or portion of a document is about the topic.

[0052] As another example, if the input to model 118 is features of the impression context for a particular advertisement, the output generated by model 118 may be a score representing the estimated likelihood that the particular advertisement will be clicked.

[0053] As another example, if the input to model 118 is features of personalized recommendations for a user, e.g., features that characterize the context of the recommendations, e.g., features that characterize previous actions taken by the user, the output generated by model 118 can be a score for each of a set of content items, where each score represents an estimated likelihood that the user will respond favorably to the recommended content item.

[0054] As another example, if the input to model 118 is a sequence of text in one language, the output generated by model 118 may be a score for each of a set of text segments in another language, where each score represents an estimated likelihood that the text segment in the other language is a correct translation of the input text into the other language.

[0055] As another example, if the input to model 118 is a sequence representing a spoken utterance, the output generated by model 118 may be a score for each of a set of text fragments, where each score represents an estimated likelihood that the text fragment is a correct transcript of the utterance.

[0056] The training data 102 includes a plurality of training examples. Each training example includes an input to a model 118 and a corresponding target output. The target output corresponding to the input refers to the preferred output of the model 118 in response to processing the input. The training data 102 includes data specifying a plurality of tasks 120. A task is a collection of batches of training examples from the training data 102. A batch of training examples refers to a collection of one or more training examples. Figure 1B, depicts an example diagram of training data, tasks, batches, and training examples.

[0057] Typically, a particular training example may be included in multiple batches, a particular batch of training data may be included in multiple tasks, and different batches may include different numbers of training examples.

[0058] The system 100 trains the model 118 to perform well (i.e., make accurate predictions) for the training examples in each of the different tasks. A model is said to make accurate predictions for a training example if the model, by processing the input of the training example, generates an output that is similar to the target output for the training example. For example, the system 100 may train the model 118 with the goal of minimizing the objective function:

[0059]

[0060] where N is the number of different tasks and is the loss of model 118 on task k, where the loss of model 118 on task k can be given by:

[0061]

[0062] in refers to the expected value with respect to the batch included in task k and L(x) refers to the loss function 114 evaluated on the output generated by the model 118 for the input of the training examples in the batch x. In general, the loss function 114 measures the accuracy of the output generated by the model 118 (as described in more detail below with reference to the description of 210).

[0063] Each task includes batches of training examples of different properties. Generally, a training model 118 that is to be performed well for training examples in a specific task can make the model perform better for training examples in other tasks (i.e., make more accurate predictions). In some cases, training examples in different tasks may have different semantics. For example, if the system 100 trains the model 118 to perform machine translation, the training examples in different batches can correspond to the translation of text sequences between different languages. In some cases, the training data 102 includes data that one or more of the specified tasks are target tasks, and the target task is a task that includes training examples with inputs that are most similar to the inputs to be processed by the model 118 after the training model 118. In these cases, the system 100 can train the model 118 to perform well for the training examples in the target task. In these cases, tasks other than the target task (referred to as auxiliary tasks in this specification) can include batches of training examples, and it is expected that the model 118 is easier to train on these training examples. Training the model 118 on the auxiliary task can help improve the performance of the model 118 on the target task. For example, system 100 can train model 118 to minimize the Given an objective function where task N is the target task (and the other tasks are auxiliary tasks) and refers to the loss of model 118 on task N (as described above).

[0064] The system 100 trains the model 118 in multiple iterations of an iterative training process.

[0065] The system 100 includes a task selection engine 104 configured to determine a selected task 106 from tasks 120 in each iteration of a plurality of iterations. The task selection engine 104 determines the selected task 106 for the iteration based on a task selection strategy. The task selection engine 104 updates the task selection strategy at each iteration using a learning progress measure 126 determined for the iteration. For example, the task selection strategy may be an adversarial bandit algorithm. Figure 3 , describes an example process for updating a task selection policy.

[0066] The system 100 includes a batch selection engine 108 configured to determine, at each iteration, a selected batch 110 of training examples from the selected tasks 106 for the iteration. The batch selection engine 108 determines the selected batch 110 by randomly sampling batches from the selected tasks 106.

[0067] The system 100 includes a training engine 112 configured to train a model 118 at each iteration based on a selected batch 110 and a loss function 114 for the iteration.

[0068] In some implementations, the training engine 112 trains the model 118 by determining updated values ​​for the model parameters 116 from the current values ​​of the model parameters 116 so as to reduce the loss on the selected batch as measured by the loss function 114 .

[0069] In some other implementations, the training engine trains the model 118 by determining updated parameter values ​​for the model parameter distribution 124 from current parameter values ​​for the model parameter distribution 124 so as to reduce the loss on the selected batch as measured by the loss function 114 .

[0070] Training engine 112 determines a learning progress measure 126 for each iteration. In general, learning progress measure 126 for an iteration is a scalar value indicating the progress of training model 118 as a result of training model 118 on training examples from the selected batch 110 for the iteration.

[0071] In some embodiments, when the system 100 determines that the training of the model 118 is complete, the system 100 stores the model 118 and the trained model parameters 116. In some embodiments, when the system 100 determines that the training of the model 118 is complete, the system 100 provides the model 118 and the model parameters 116 to a third party. In some embodiments, when the system 100 determines that the training of the model 118 is complete, the system 100 processes the input (e.g., from the user device) using the model 118 according to the trained model parameters 116 and provides the corresponding output of the model 118 to, for example, the user device.

[0072] Figure 1B 1 is a diagram of examples of training data, tasks, batches, and training examples. Training data 142 includes multiple training examples (e.g., 144). Each training example includes an input (e.g., 146) and a corresponding target output (e.g., 148). Each batch (e.g., 130, 132, 134, 136, 138, 140, ...) includes multiple training examples from training data 142. Each of the tasks (e.g., 126, 128, ...) is a collection of multiple batches of training examples. For example, task 126 is associated with batches 130, 134, 138, etc., while task 128 is associated with batches 132, 136, 140, etc. As described above, different batches of training data may include different numbers of training examples, and different tasks may be associated with different numbers of batches.

[0073] Figure 2 2 is a flow chart of an example process for training a machine learning model. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., training system 100 of FIG. 1 ) appropriately programmed according to the present specification can perform process 200.

[0074] The system receives training data (202). The training data includes a plurality of training examples. Each training example includes an input to a machine learning model and a corresponding target output. The input can be any type of digital data and the target output can be any type of score, classification, or regression output based on the input. The training data specifies a plurality of tasks. Each task is a collection of a plurality of batches of training examples from the training data.

[0075] The system selects tasks according to a task selection strategy (206). Specifically, the task selection strategy assigns a corresponding probability to each of the tasks, and the system randomly selects tasks according to the corresponding probability assigned to each of the tasks. Typically, the corresponding probability assigned to each of the tasks varies from iteration to iteration.

[0076] The system selects a batch of training examples from the selected task for iteration (208). The system selects the batch by randomly sampling batches from the selected task.

[0077] The system trains the model based on the selected batch of training examples for iteration (210). The system can train the model based on the selected batch of training examples for iteration using any suitable training technique. For example, if the model is a neural network, the system can use gradient descent to train the model.

[0078] In some embodiments, the system trains the model by determining updated values ​​of the model parameters from the current values ​​of the model parameters so as to reduce the loss on the selected batch as measured by the loss function. For example, the loss function may be given by:

[0079]

[0080] where B is the number of training examples in batch x, i is the index of a training example in batch x, θ refers to the parameters of the model, and p θ (b i |a i ) is obtained by comparing with input a i The corresponding target output b i The conditional probability generated by the model.

[0081] In some other embodiments, the system trains the model by determining updated parameter values ​​of a model parameter distribution at possible values ​​of the model parameter from current parameter values ​​of the model parameter distribution at possible values ​​of the model parameter so as to reduce the loss on a selected batch as measured by a loss function. The model parameter distribution includes a distribution called a prior distribution and a distribution called a posterior distribution. The posterior distribution is the distribution that is modified by training. In some cases, the prior distribution can be fixed, in which case the parameter values ​​of the prior distribution are not updated by the system during training. The loss function can be given by:

[0082]

[0083] where S is the number of training examples in the training data, KL(·|·) is the Kullback-Leibler (KL) divergence measure (i.e., the distance measure between probability distributions), φ refers to the posterior distribution parameters, and P φ is the posterior distribution of the possible values ​​of the model parameters, ψ refers to the prior distribution parameters, Q ψ is the prior distribution of possible values ​​of the model parameters, and is the expected value of the loss L(x,θ) (e.g., as described above) on the chosen batch when the model parameters are given by θ, where θ is drawn from the posterior distribution P φ Reference is made to, for example, Graves, A., “Practical variational inference for neural networks,” Advances in Neural Information Processing Systems (NIPS), 2348-2356, 2011., which describes an example process of training a model by determining updated parameter values ​​for the model parameter distribution.

[0084] The system determines a learning progress measure for the iteration (212).

[0085] In some implementations, the learning progress measure is based on the change in model output caused by training the model on the selected batches.

[0086] For example, the learning progress measure can be given by:

[0087] L(x,θ)-L(x,θ′),

[0088] where L(x,θ) refers to the value of the loss function evaluated on the outputs generated by the model for the inputs in the selected batch x according to the current values ​​θ of the model parameters, and L(x,θ′) refers to the value of the loss function evaluated on the outputs generated by the model for the inputs in the selected batch x according to the updated values ​​θ′ of the model parameters.

[0089] As another example, a learning progress measure may be given by:

[0090]

[0091] in refers to the gradient vector of the gradient of the loss function L(x,θ) with respect to the current value of the model parameter θ generated as part of training the model on the selected batch x, and |·| refers to the norm. For example, the norm can be the squared L2 norm.

[0092] As another example, a learning progress measure may be given by:

[0093] L(x′,θ)-L(x′,θ′),

[0094] where L(x′, θ) refers to the value of the loss function evaluated on the output generated by the model for the inputs in the new batch x′ according to the current values ​​θ of the model parameters, and L(x′, θ′) refers to the value of the loss function evaluated on the output generated by the model for the inputs in the new batch x′ according to the updated values ​​θ′ of the model parameters. In some cases, the system selects the new batch from the selected task. In some other cases, the system selects the new batch from the target task. In some other cases, the system randomly selects the new task and selects the new batch from the randomly selected new task. Given a specific task from which the system selects a new batch, the system randomly selects the new batch from the batch of the specific task.

[0095] In some implementations, the learning progress measure is based on the change in model complexity caused by training the model on the selected batches.

[0096] For example, the learning progress measure can be given by:

[0097] |θ′|-|θ|,

[0098] where |·| refers to the norm (e.g., the squared L2 norm), θ′ refers to the vector of updated values ​​of the model parameters, and θ refers to the vector of current values ​​of the model parameters.

[0099] As another example, a learning progress measure may be given by:

[0100]

[0101] where θ T refers to the transpose of the vector of current values ​​of the model parameters and refers to the gradient of the loss function with respect to the model parameters θ on the selected batch x.

[0102] As another example, a learning progress measure may be given by:

[0103] KL(P φ′ |Q ψ′ )-KL(P φ |Q ψ ),

[0104] where KL(·|·) refers to the KL divergence measure, P φ′ refers to the posterior distribution defined by the updated values ​​of the posterior distribution parameters, P φ refers to the posterior distribution defined by the current values ​​of the posterior distribution parameters, and Qψ refers to the prior distribution of possible values ​​of the model parameters defined by the current values ​​of the prior distribution parameters. If the system updates the values ​​of the prior distribution parameters during training, then Q ψ′ refers to the prior distribution defined by the updated values ​​of the prior distribution parameters. If the system does not update the values ​​of the prior distribution parameters during training, then Q ψ′ Refers to Q ψ Same prior distribution.

[0105] As another example, a learning progress measure may be given by:

[0106]

[0107] in refers to the posterior distribution P defined by the current values ​​of the posterior distribution parameters φ and the prior distribution Q defined by the current values ​​of the prior distribution parameters ψ The KL divergence between the gradient of the posterior distribution parameters φ and the prior distribution parameters ψ, [·] R refers to the transpose operation, and refers to the gradient of the expected posterior distribution parameters of the loss L(x,θ) on the selected batch, where φ The resulting model parameter θ takes this expectation.

[0108] The system determines whether training is complete (214). In some cases, the system determines that training is complete if the system has trained the model for a predetermined number of iterations. In some cases, the system determines that training is complete if a predetermined number of iterations of a measure of learning progress falls below a predetermined threshold.

[0109] In response to determining that training is not complete, the system updates the task selection strategy based on (ie, dependent on) the determined learning progress measure for the iteration (218), returns to 206, and repeats the previous steps. Figure 3 , describes an example process for updating a task selection policy. In response to determining that training is complete (218), the system terminates the training.

[0110] Figure 3 300 is a flow chart of an example process for updating a task selection policy. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., training system 100 of FIG. 1 ) appropriately programmed according to the present specification can perform process 300.

[0111] The system determines a normalized measure of learning progress for the iteration (302).The system determines the normalized measure of learning progress for the iteration by dividing the measure of learning progress for the iteration by a normalization factor.

[0112] In some embodiments, the normalization factor is the time required for the model to generate an output for an input of a training example in the selected task. This time can be measured in time steps.

[0113] In some embodiments, particularly when the model is configured to process inputs over a series of time steps (e.g., if the model is a recurrent neural network), the normalization factor is the number of time steps required for the model to generate outputs for the inputs of the selected batch of training examples.

[0114] In some embodiments, the normalization factor is the number of separate inputs in each input of the input of the training examples in the selected batch. For example, if the input is a sequence of words in a natural language (e.g., English), the number of separate inputs in the input can be the number of words in the input.

[0115] The system determines a payoff for an iteration using the learning progress measure, such as by scaling the normalized learning progress measure so that it falls within a payoff range (304), which can be a predetermined range. The system determines a lower percentile and an upper percentile (e.g., the 20th percentile and the 80th percentile, or any other appropriate percentile) of the normalized learning progress measure determined for a previous iteration. In some cases, the system determines the corresponding percentile based on a representative sample of previous iterations rather than the normalized learning progress measures of all previous iterations. For example, if the index of an iteration is divisible by a fixed factor (e.g., 10), the iteration can be included in the representative sample of previous iterations. The system determines a payoff for an iteration by mapping the normalized learning progress measure for the iteration to fall within a predetermined range, for example, based on a mapping:

[0116]

[0117] where r is the normalized learning progress measure for the iteration, and p low is the lower percentile, and p high is the upper percentile and has a predetermined range of [-1,1].

[0118] The system uses an adversarial bandit algorithm to update the task selection policy based on the iterated benefits. For example, the adversarial bandit algorithm can be the Exp3.S algorithm, as described in Auer, P., Cesa-Bianchi, N., Freund, Y., Schapire, RE, "The nonstochastic multiarmed bandit problem," SIAM Journal on Computing, 32(1): 48-77. As another example, the adversarial bandit algorithm can be the Exp3.M algorithm, as described in Uchiya, T., Nakamura, A., Kudo, M., "Algorithms for adversarial bandit problems with multiple plays," Proceedings of the International Conference on Algorithmic Learning Theory, 2010. The adversarial bandit algorithm can be described as an algorithm that selects actions with the goal of minimizing regret, where regret can be defined as the expected benefit loss compared to the best action. Using the adversarial bandit algorithm, the system updates the task selection strategy by updating the corresponding selection probabilities of the tasks, aiming to maximize the sum of the gains achieved over multiple training iterations.

[0119] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when run, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0120] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangibly implementable computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more computer program instruction modules encoded on a tangible non-temporary storage medium, for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0121] The term "data processing apparatus" refers to data processing hardware and includes all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also or further include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to the hardware, the apparatus may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0122] A computer program (which may also be referred to or described as a program, software, software application, application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute the computer program on one computer or on multiple computers located on one site or distributed on multiple sites and interconnected by a data communications network.

[0123] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same computer or multiple computers.

[0124] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit (e.g., FPGA or ASIC) or by a combination of a special purpose logic circuit and one or more programmed computers.

[0125] A computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by or incorporated into a special logic circuit. Typically, a computer will also include one or more mass storage devices (such as magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to one or more mass storage devices to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device (e.g., a mobile phone, a personal digital assistant (personal digital assistant, PDA), a mobile audio or video player, a game console, a global positioning system (Global Positioning System, GPS) receiver or a portable storage device (e.g., a universal serial bus (universal serial bus, USB) flash drive), just to name a few).

[0126] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0127] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound input, voice input, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. Also, the computer may interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smart phone running a messaging application), and receiving a response message from the user as a reply.

[0128] The data processing apparatus for implementing a machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts (i.e., inference, workloads) of machine learning training or production.

[0129] The machine learning model can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).

[0130] Embodiments of the subject matter described in this specification may be implemented in a computing system including a back-end component (e.g., as a data server), or in a computing system including a middleware component (e.g., an application server), or in a computing system including a front-end-terminal component (e.g., a client computer with a graphical user interface, a web browser, or an application, through which a user can interact with an implementation of the subject matter described in this specification), or in a computing system including any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0131] A computing system may include a client and a server. The client and the server are usually remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by means of computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated at the user device, such as the result of a user interaction, can be received from the device at the server.

[0132] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination. In addition, although features may be described above as acting in a certain combination and even initially claimed as such, one or more features from the claimed combination may be cut out of the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0133] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in sequence, or that all of the illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

Claims

1. A method for training a machine learning model having a plurality of model parameters to determine training values ​​of the model parameters from initial values ​​of the model parameters, in, The values ​​of the model parameters are defined by a posterior distribution of possible values ​​of the model parameters, the method comprising: Receiving training data for training a machine learning model on a plurality of tasks, wherein each task comprises a corresponding plurality of batches of training data, wherein the training data comprises image, video, sound or text data; and training a machine learning model on the training data, wherein during the training, a posterior distribution parameter that parameterizes the posterior distribution is optimized such that training values ​​of the model parameters are defined by the training values ​​of the posterior distribution parameter, wherein the training comprises, at each training iteration in a plurality of training iterations: selecting a task from the plurality of tasks according to a current task selection strategy; Select a batch of training data from multiple batches of training data for the selected task; training a machine learning model on the selected batch of training data to determine updated values ​​of the model parameters from current values ​​of the model parameters, including training the machine learning model on the selected batch of training data to determine adjusted values ​​of the posterior distribution parameters from current values ​​of the posterior distribution parameters; determining a learning progress measure representing the progress of training of the machine learning model as a result of training the machine learning model on the selected batch of training data; and Updating the current task selection strategy based on the learning progress measurement includes: determining the gain achieved at that training iteration from the learning progress measure; and The gain is used to update the current task selection policy to encourage the selection of tasks that maximize the cumulative measure of gain achieved over multiple training iterations.

2. The method according to claim 1, in, Training the machine learning model on the selected batch includes training the machine learning model to reduce a loss on the selected batch measured by a loss function, and wherein the learning progress measure is based on a reduction in the loss as a result of training the machine learning model on the selected batch of training data.

3. The method according to claim 2, in, Determining the learning progress measure includes: determining a first loss on the selected batch based on current values ​​of the model parameters; and determining a second loss on the selected batch based on the updated values ​​of the model parameters, and Wherein the learning progress measure comprises a difference between the first loss and the second loss.

4. The method according to claim 1, in, Determining the learning progress measure includes: Sample new batches from multiple batches in the selected task; determining a first loss on the new batch based on the current values ​​of the model parameters; and determining a second loss on the new batch based on the updated values ​​of the model parameters, and Wherein the learning progress measure comprises a difference between the first loss and the second loss.

5. The method according to claim 1, in, One of the tasks is identified as a target task, the target task comprising a training input that is most similar to an input to be processed by the machine learning model after training the machine learning model on the training data, wherein determining the learning progress measure comprises: Sampling a new batch from a plurality of batches in the target task; determining a first loss on the new batch based on the current values ​​of the model parameters; and determining a second loss on the new batch based on the updated values ​​of the model parameters, and Wherein the learning progress measure comprises a difference between the first loss and the second loss.

6. The method according to claim 1, in, Determining the learning progress measure includes: randomly sampling tasks from the plurality of tasks; sampling new batches from a plurality of batches in the sampled task; determining a first loss on the new batch based on the current values ​​of the model parameters; and determining a second loss on the new batch based on the updated values ​​of the model parameters, and Wherein the learning progress measure comprises a difference between the first loss and the second loss.

7. The method according to claim 1, in, The learning progress measure comprises a norm of a gradient vector of a gradient of a loss function with respect to the model parameters generated by training the machine learning model on the selected batch.

8. The method according to claim 1, in, The learning progress measure is based on an increase in model complexity of the machine learning model as a result of training the machine learning model on a selected batch of training data.

9. The method according to claim 1, in, Determining the learning progress measure includes: determining a first Kullback-Leibler KL divergence between (i) a posterior distribution defined by updated values ​​of the posterior distribution parameters and (ii) a prior distribution of possible values ​​of the model parameters; and determining a second KL divergence between (i) a posterior distribution defined by the current value of the posterior distribution parameter and (ii) a prior distribution of possible values ​​of the model parameter, and Wherein, the learning progress measure comprises a difference between the first KL divergence and the second KL divergence.

10. The method according to claim 9, in, The prior distribution is defined by prior distribution parameters, wherein training the machine learning model on a selected batch of training data includes determining adjusted values ​​of the prior distribution parameters from current values ​​of the prior distribution parameters, wherein a first KL divergence is a KL divergence between (i) a posterior distribution defined by the updated values ​​of the posterior distribution parameters and (ii) a prior distribution defined by the updated values ​​of the prior distribution parameters, and wherein a second KL divergence is a KL divergence between (i) a posterior distribution defined by the current values ​​of the posterior distribution parameters and (ii) a prior distribution defined by the current values ​​of the prior distribution parameters.

11. The method according to claim 1, in, Training the machine learning model on the selected batch of training data includes determining an adjusted value of a prior distribution parameter from a current value of the prior distribution parameter, wherein the prior distribution parameter parameterizes a prior distribution of possible values ​​of the model parameter, and wherein the learning progress measure is based on a) the gradients of the posterior distribution parameters and the prior distribution parameters with respect to the KL divergence between (i) a posterior distribution defined by current values ​​of the posterior distribution parameters and (ii) a prior distribution defined by current values ​​of the prior distribution parameters, and b) the gradients of the posterior distribution parameters with respect to an expected loss on a selected batch as measured by a loss function.

12. The method of claim 1, in, The learning progress measure is based on a difference between a first norm of a vector of updated values ​​of the model parameters and a second norm of a vector of current values ​​of the model parameters.

13. The method of claim 1, in, The learning progress measure is based on the product between (i) the transpose of the vector of current values ​​of the model parameters and (ii) a gradient vector of the gradients of the loss function with respect to the model parameters generated as part of training the machine learning model on the selected batch.

14. The method of claim 1, in, The task selection strategy is a strategy that assigns a respective likelihood to each of the plurality of tasks, and wherein selecting a task from the plurality of tasks according to the current task selection strategy includes randomly selecting a task according to the likelihood assigned by the current task selection strategy.

15. The method of claim 14, in, The task selection strategy is an adversarial bandit strategy, where using the payoff to update the current task selection strategy includes: The current task selection policy is updated using the payoff via an adversarial bandit algorithm.

16. The method of claim 1, in, Determining the benefits includes: normalizing the learning progress measure by dividing the learning progress measure by a normalization factor for a selected task to generate a normalized learning progress measure; and The benefit is generated by scaling the normalized learning progress measure to fall within a benefit range.

17. In the method of claim 16, in, The normalization factor for the selected task is an approximation of the time it takes or the time it takes for the machine learning model to generate the output for the training inputs in the selected task.

18. The method of claim 17, in, The approximation is the number of time steps required for the machine learning model to generate outputs for the training inputs in the selected batch, or the number of separate inputs in each of the training inputs in the selected batch.

19. The method of claim 1, in, Selecting a batch of training data from multiple batches of training data in the selected task includes: Randomly select a batch of training data from multiple batches of training data in the selected task.

20. The method of claim 1, in, The machine learning model is a recurrent neural network.

21. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a machine learning model having a plurality of model parameters to determine training values ​​of the model parameters from initial values ​​of the model parameters, in, The value of the model parameter is defined by a posterior distribution of possible values ​​of the model parameter, and the operation comprises: Receiving training data for training a machine learning model on a plurality of tasks, wherein each task comprises a corresponding plurality of batches of training data, wherein the training data comprises image, video, sound or text data; and training a machine learning model on the training data, wherein during the training, a posterior distribution parameter that parameterizes the posterior distribution is optimized such that training values ​​of the model parameters are defined by the training values ​​of the posterior distribution parameter, wherein the training comprises, at each training iteration in a plurality of training iterations: selecting a task from the plurality of tasks according to a current task selection strategy; Select a batch of training data from multiple batches of training data for the selected task; training a machine learning model on the selected batch of training data to determine updated values ​​of the model parameters from current values ​​of the model parameters, including training the machine learning model on the selected batch of training data to determine adjusted values ​​of the posterior distribution parameters from current values ​​of the posterior distribution parameters; determining a learning progress measure representing the progress of training of the machine learning model as a result of training the machine learning model on the selected batch of training data; and Updating the current task selection strategy based on the learning progress measurement includes: determining the gain achieved at that training iteration from the learning progress measure; and The gain is used to update the current task selection policy to encourage the selection of tasks that maximize the cumulative measure of gain achieved over multiple training iterations.

22. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a machine learning model having a plurality of model parameters to determine training values ​​of the model parameters from initial values ​​of the model parameters, in, The value of the model parameter is defined by a posterior distribution of possible values ​​of the model parameter, and the operation comprises: Receiving training data for training a machine learning model on a plurality of tasks, wherein each task comprises a corresponding plurality of batches of training data, wherein the training data comprises image, video, sound or text data; and training a machine learning model on the training data, wherein during the training, a posterior distribution parameter that parameterizes the posterior distribution is optimized such that training values ​​of the model parameters are defined by the training values ​​of the posterior distribution parameter, wherein the training comprises, at each training iteration in a plurality of training iterations: selecting a task from the plurality of tasks according to a current task selection strategy; Select a batch of training data from multiple batches of training data for the selected task; training a machine learning model on the selected batch of training data to determine updated values ​​of the model parameters from current values ​​of the model parameters, including training the machine learning model on the selected batch of training data to determine adjusted values ​​of the posterior distribution parameters from current values ​​of the posterior distribution parameters; determining a learning progress measure representing the progress of training of the machine learning model as a result of training the machine learning model on the selected batch of training data; and Updating the current task selection strategy based on the learning progress measurement includes: determining the gain achieved at that training iteration from the learning progress measure; and The gain is used to update the current task selection policy to encourage the selection of tasks that maximize the cumulative measure of gain achieved over multiple training iterations.