Train machine learning models on multiple machine learning tasks
By adjusting model parameters on training data and using penalties, the problem of 'catastrophic forgetting' of machine learning models in multi-task training is solved, achieving the effect of maintaining performance on multiple tasks, reducing system complexity and storage need.
Patent Information
- Application Number
- CN201780054998.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2016-07-18
- Filing Date
- 2017-07-18
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2037-07-18
AI Technical Summary
Machine learning models are prone to 'catastrophic forgetting' when trained on multiple tasks, that is, losing knowledge of previous tasks and making it difficult to maintain the performance of previous tasks on new tasks.
By adjusting the model parameters on the training data, the objective function is optimized, combined with the penalty items for parameter importance, ensuring that the model maintains the performance of previous tasks on the new task while learning new tasks. Specific methods include using elastic weight merge and Fischer information matrix to calculate parameter importance and forming penalty terms to constrain the parameters not to deviate from the original value.
Maintain acceptable performance on multiple tasks, reduces storage space requirements, reduces system complexity, and allows the model to effectively learn new tasks continuously while protecting knowledge of previous tasks.
Smart Images

Figure CN109690576B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Provisional Application Serial No. 62 / 363,652, filed on July 18, 2016. The disclosure of that prior application is considered part of the disclosure of the present application and is incorporated by reference into the disclosure of the present application. Technical Field
[0003] This specification relates to training a machine learning model. Background Art
[0004] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate output based on the received input and the model's parameter values.
[0005] Some machine learning models are deep models that utilize multiple layers of the model to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, where each hidden layer applies a nonlinear transformation to the received input to generate an output. However, machine learning models may suffer from "catastrophic forgetting" when trained on multiple tasks, i.e., losing knowledge of previous tasks when learning a new task.
[0006] Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that takes an input sequence and generates an output sequence based on the input sequence. Specifically, a recurrent neural network generates an output based on the current input in the input sequence, using some or all of the network's internal state after processing previous inputs in the input sequence. Summary of the invention
[0007] This specification describes how a system implemented as a computer program on one or more computers at one or more locations can train a machine learning model on multiple machine learning tasks.
[0008] In general, one innovative aspect may be embodied in a method for training a machine learning model having a plurality of parameters. The machine learning model has been trained on a first machine learning task to determine a first value of a parameter of the machine learning model. The method includes: for each of the plurality of parameters, determining a corresponding measure of the importance of the parameter to the machine learning model achieving acceptable performance on the first machine learning task; obtaining training data for training the machine learning model on a second, different machine learning task; and, by training the machine learning model on the training data, training the machine learning model on the second machine learning task to adjust the first value of the parameter so that the machine learning model achieves an acceptable level of performance on the second machine learning task while maintaining an acceptable level of performance on the first machine learning task, wherein, during the training of the machine learning model on the second machine learning task, the value of the parameter that is more important to the machine learning model achieving acceptable performance on the first machine learning task is more strongly constrained to not deviate from the first value than the value of the parameter that is less important to the machine learning model achieving acceptable performance on the first machine learning task.
[0009] Training the machine learning model on the training data may include adjusting first values of parameters to optimize an objective function, more specifically to minimize an objective function, the objective function including: (i) a first term that measures the performance of the machine learning model on a second machine learning task, and (ii) a second term that imposes a penalty on parameter values that deviate from the first parameter value, wherein the second term penalizes deviations from the first value of parameters that are more important to achieving acceptable performance on the first machine learning task more heavily than parameters that are less important to achieving acceptable performance on the first machine learning task. For each of the plurality of parameters, the second term may depend on the product of a corresponding measure of the importance of the parameter and a difference between a current value of the parameter and the first value of the parameter.
[0010] In some embodiments, training may implement "elastic weight consolidation (EWC)", in which during training of the second task, parameters are anchored to their first values via an elastic penalty, which is a penalty on the tuning parameter that increases with distance from the first value of the parameter. The stiffness or degree of the elastic penalty may depend on a measure of the importance of the parameter to the first task, or more generally, on a measure of the importance of the parameter to any previously learned task. Thus, elastic weight consolidation may be implemented as a soft constraint, such as quadratic with increasing distance, so that each weight is pulled back to the old value by an amount that depends on (e.g., proportional to) a measure of its importance to one or more previously performed tasks. Broadly speaking, the parameters are tempered by a prior, which is a posterior distribution over the parameters derived from (multiple) previous tasks.
[0011] Typically, the machine learning model includes a neural network such as a convolutional neural network or a recurrent neural network, and the parameters include weights of the neural network. As further described below, the importance (weight) of each parameter can be determined in a variety of different ways. Optionally, the importance (weight) of each parameter can be recalculated before starting training for a new task.
[0012] For each training example in the training data, training the machine learning model on the training data may include: processing the training example using the machine learning model to determine a model output according to current values of parameters of the machine learning model; determining a gradient of an objective function using the model output, a target output for the training example, current values of the parameters of the machine learning model, and a first value of the parameters of the machine learning model; and, adjusting the current values of the parameters using the gradient to optimize the objective function.
[0013] For each of a plurality of parameters, determining a corresponding measure of the importance of the parameter to the machine learning model achieving acceptable performance on a first machine learning task may include, for each of the plurality of parameters, determining an approximation of a probability that a current value of the parameter is a correct value of the parameter given first training data used to train the machine learning model on the first task.
[0014] For each of a plurality of parameters, one way to determine a corresponding measure of the importance of the parameter to the machine learning model achieving acceptable performance on a first machine learning task may include determining a Fisher Information Matrix (FIM) of the plurality of parameters of the machine learning model relative to the first machine learning task, wherein for each of the plurality of parameters, the corresponding measure of the importance of the parameter is the corresponding value on the diagonal of the FIM. This is computationally convenient because the FIM can be calculated from the first derivative. For example, the FIM can be determined based on the covariance of the model log probability relative to its parameters.
[0015] Using the diagonal values of FIM as important metrics can reduce computational complexity while effectively calculating point estimates of parameter variance, i.e., uncertainty of weights. In some embodiments, the mean and standard deviation (or variance) of weights can be estimated by utilizing the Bayes by Backprop process, as described in "Weight Uncertainty in Neural Networks" by Blundell et al., ICML (2015). This can be accomplished using changes in the backpropagation process.
[0016] In some implementations, the first machine learning task and the second machine learning task are different supervised learning tasks.
[0017] In some other embodiments, the first machine learning task and the second machine learning task are different reinforcement learning tasks. In the reinforcement learning task, the objective function may include a discounted reward term that depends on the expected reward for taking an action in a certain state. The machine learning model can be based on, for example, a Deep Q-Network (DQN), a Double-DQN, an Advantage Actor Critic (A3C) network, or other architectures.
[0018] In some embodiments, particularly but not limited to in reinforcement learning (RL) systems, machine learning tasks can be identified. For example, a model can be used to infer tasks, explicitly label them or automatically identify them. Then, one or more penalty terms of the objective function can be selected based on the identified tasks. When a task switch is identified, one or more penalty terms can be selected to constrain parameter learning to be close to the values learned for one or more previous tasks (based on their importance to (multiple) previous tasks, as described above). The switch may be to a new, previously unseen task, or back to a previous task. The penalty term may include constraints on all previously seen tasks except the current task. The constraints may be quadratic constraints. Optionally, the importance (weights) of each parameter can be recalculated when switching from one learning task to another.
[0019] The method may also include: after the machine learning model is trained on the second machine learning task to determine a second value of a parameter of the machine learning model: obtaining third training data for training the machine learning model on a third, different machine learning task; and, by training the machine learning model on the third training data, training the machine learning model on the third machine learning task to adjust the second value of the parameter so that the machine learning model achieves an acceptable performance level on the third machine learning task while maintaining an acceptable performance level on the first machine learning task and the second machine learning task, wherein, during the training of the machine learning model on the third machine learning task, the values of the parameters that are more important to the machine learning model achieving acceptable performance on the first machine learning task and the second machine learning task are more strongly constrained to not deviate from the second value than the values of the parameters that are less important to the machine learning model achieving acceptable performance on the first machine learning task and the second machine learning task.
[0020] The method may then further include: for each of the plurality of parameters, determining a corresponding measure of the importance of the parameter to the machine learning model achieving acceptable performance on the second machine learning task. Training the machine learning model on the third training data may then include: adjusting the second value of the parameter to optimize an objective function comprising a first term that measures the performance of the machine learning model on the third machine learning task and a second term that imposes a penalty on parameter values that deviate from the second parameter value. Similar to the method described above, the second term penalizes the deviation of a parameter from the second value that is more important for achieving acceptable performance on the first machine learning task and the second machine learning task more heavily than a parameter that is less important for achieving acceptable performance on the first machine learning task and the second machine learning task. The second term of the objective function may include two separate penalty terms, one of which is used for the previous task, or a combined penalty term. For example, when the penalty terms each include a quadratic penalty, such as the square of the difference between the parameter and its previous value, the sum of two such penalties can itself be written as a quadratic penalty.
[0021] The above aspects may be implemented in any convenient form. For example, various aspects and embodiments may be implemented by a suitable computer program, which may be carried on a suitable carrier medium, which may be a tangible carrier medium (e.g., a disk) or an intangible carrier medium (e.g., a communication signal). Various aspects may also be implemented using a suitable device, which may take the form of a programmable computer running a computer program.
[0022] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages. As described in this specification, by training the same machine learning model on multiple tasks, once the model is trained, the model can be used for each of the multiple tasks at an acceptable performance level. Therefore, systems that need to be able to achieve acceptable performance on multiple tasks can do so while using less storage capacity and reducing system complexity. For example, by maintaining a single instance of a model rather than multiple different instances of the model (each with different parameter values), only one set of parameters need to be stored rather than multiple different sets of parameters, thereby reducing the amount of storage space required while maintaining acceptable performance on each task. In addition, by adjusting the values of the model parameters to train the model on a new task to optimize an objective function that depends in part on how important the parameters are to the task (s) previously learned, the model can effectively continuously learn new tasks while protecting knowledge about previous tasks.
[0023] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 is a block diagram of an example of a machine learning system for training a machine learning model.
[0025] Figure 2 is a flowchart of an example process for training a machine learning model with multiple parameters on multiple tasks.
[0026] Figure 3 is a flowchart of an example process for training a machine learning model on a third machine learning task after training the machine learning model on the first and second machine learning tasks.
[0027] Like reference numbers in the various drawings represent like elements. DETAILED DESCRIPTION
[0028] This specification describes how a system (e.g., a machine learning system) implemented as a computer program on one or more computers at one or more locations can train machine learning models on multiple machine learning tasks.
[0029] In some cases, multiple machine learning tasks are different supervised learning tasks. For example, supervised learning tasks may include different classification tasks, such as image processing tasks, speech recognition tasks, natural language processing tasks, or optical character recognition tasks. For example, image processing tasks may include different image recognition tasks, each of which requires recognition of different objects or patterns in an image. As another example, a speech recognition task may include multiple hot word detection tasks, each of which requires recognition of different hot words or hot word sequences.
[0030] In some other cases, the multiple machine learning tasks are different reinforcement learning tasks. For example, a reinforcement learning task can include multiple tasks in which an agent interacts with different environments or interacts with the same environment to achieve different goals. For example, a reinforcement learning task can include multiple tasks in which a computerized agent interacts with multiple different simulated or virtualized environments. As another example, a reinforcement learning task can include multiple tasks in which a robotic agent interacts with a real-world environment to attempt to achieve different goals. Such a robotic agent can be embodied in a static or mobile machine or vehicle.
[0031] Figure 1 An example machine learning system 100 is shown. The machine learning system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.
[0032] The machine learning system 110 is configured to continuously train the machine learning model 110 on multiple machine learning tasks. The machine learning model 110 can receive input and generate output, such as a predicted output, based on the received input.
[0033] In some cases, the machine learning model 110 is a parametric model having multiple parameters. In these cases, the machine learning model 110 generates an output based on the received input and the values of the parameters of the model 110.
[0034] In some other cases, the machine learning model 110 is a deep machine learning model that utilizes multiple layers of the model to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, where each hidden layer applies a nonlinear transformation to the received input to generate an output.
[0035] Typically, the machine learning system 100 trains the machine learning model 110 on a specific task (i.e., learns the specific task) by adjusting the values of the parameters of the machine learning model 110 so as to optimize the performance of the model 110 on the specific task (e.g., by optimizing the objective function 118 of the model 110).
[0036] The system 100 is capable of training the model 110 to learn a plurality of machine learning task sequences. Typically, in order to allow the machine learning model 110 to learn new tasks without forgetting previous tasks, the system 100 trains the model 110 to optimize the performance of the model 110 on the new tasks, while protecting the performance of the model 110 on the previous tasks by constraining the parameters based on information about the previous tasks so that the parameters are in an acceptable performance region (e.g., a low error region) of the previous tasks.
[0037] The system 100 determines information about previous tasks using the importance weight calculation engine 112. Specifically, for each task that the model 110 has been previously trained on, the engine 112 determines a set of importance weights corresponding to those tasks. The set of importance weights for a given task typically includes a corresponding weight for each parameter of the model 110, the corresponding weight representing a measure of the importance of the parameter to the model 110 achieving acceptable performance on the task. The system 100 then trains the model 110 on the new task using the set of importance weights corresponding to the previous tasks, so that the model 110 achieves an acceptable level of performance on the new task while maintaining an acceptable level of performance on the previous tasks.
[0038] like Figure 1As shown, assuming that the model 110 has been trained on a first machine learning task (e.g., task A) using first training data to determine first values of parameters of the model 110, the importance weight calculation engine 112 determines a set of importance weights 120 corresponding to task A. Specifically, for each parameter of the model 110, the engine 112 determines a corresponding importance weight that represents a measure of the importance of the parameter to the model 110 achieving acceptable performance on task A. Determining the corresponding importance weight for each parameter includes: for each parameter, given the first training data used to train the machine learning model 110 on task A, determining an approximation of the probability that the current value of the parameter is the correct value of the parameter.
[0039] For example, after the model 110 has been trained on prior training data from prior machine learning task(s), the engine 112 may determine a posterior distribution of possible values for the parameters of the model 110. For each parameter, the posterior distribution specifies a value for the current value of the parameter, where the value represents the probability that the current value is the correct value for the parameter.
[0040] In some embodiments, the engine 112 can use an approximation method (e.g., using a Fisher information matrix (FIM)) to approximate the posterior distribution. The engine 112 can determine the FIM of the parameters of the model 110 relative to task A, where for each parameter, the corresponding importance weight of the parameter is the corresponding value on the diagonal of the FIM. That is, each value on the diagonal of the FIM corresponds to a different parameter of the machine learning model 110.
[0041] The engine 112 can determine the FIM by computing the second derivative of the objective function 118 at the parameter values that optimize the objective function 118 with respect to task A. The FIM can also be computed from first-order derivatives only, so the FIM is easy to compute even for large machine learning models. The FIM is guaranteed to be positive semi-definite. The computation of the FIM is described in more detail in Pascanu R, Bengio Y (2013), "Revisiting natural gradient for deep networks", arXiv:1301.3584.
[0042] After the engine 112 has determined the set of importance weights 120 corresponding to Task A, the system 100 is able to train the model 110 on new training data 114 corresponding to a new machine learning task (e.g., Task B).
[0043] In order to allow the model 118 to learn task B without forgetting task A, during the training of the model 110 on task B, the system 100 uses the set of importance weights 120 corresponding to task A to form a penalty term in the objective function 118, the penalty term being intended to maintain acceptable performance on task A. That is, the model 110 is trained to determine the post-training parameter values 116 that optimize the objective function 118 with respect to task B, and because the objective function 118 includes the penalty term, the model 110 will still maintain acceptable performance on task A even after training on task B. Figure 2 The process of training model 110 and objective function 118 are described in more detail.
[0044] When there are more than two tasks in a machine learning task sequence, for example, when the model 110 still needs to be trained on a third task (e.g., task C) after being trained on task B, after determining the trained parameter value 116 of task B, the machine learning system 100 provides the trained parameter value 116 to the engine 112 so that the engine 112 can determine a new set of importance weights corresponding to task B.
[0045] When training the model 110 on Task C, the system 100 can train the model 110 to use the set of importance weights corresponding to Task A and the new set of importance weights corresponding to Task B, forming a new penalty term in the objective function 118 to be optimized by the model 110 with respect to Task C. Figure 3 The process of training the machine learning model 110 on task C is described in more detail. The training process can be repeated until the model 110 has learned all tasks in the machine learning task sequence.
[0046] Figure 2 is a flow chart of an example process 200 for continuously training a machine learning model with multiple parameters on multiple tasks.
[0047] For convenience, process 200 is described as being performed by a system of one or more computers located at one or more locations. For example, a machine learning system (e.g., Figure 1 The machine learning system 100 is capable of performing process 200.
[0048] The machine learning model is a model that has been trained on a first machine learning task to determine first values for parameters of the machine learning model.
[0049] To train a machine learning model on another task, for each of a plurality of parameters, the system first determines a corresponding measure of the importance of the parameter to the machine learning model achieving acceptable performance on the first machine learning task (e.g., an importance weight of the parameter) (step 202).
[0050] Determining a corresponding metric for the importance of a parameter includes determining an approximation of the probability that a current value of the parameter is a correct value for the parameter given first training data for training a machine learning model on a first machine learning task.
[0051] For example, after the model has been trained on previous training data from (multiple) previous machine learning tasks, the system can determine a posterior distribution of possible values for the parameters of the model. For each parameter, the posterior distribution specifies a value for the current value of the parameter, where the value represents the probability that the current value is the correct value for the parameter.
[0052] In some embodiments, the system can use an approximation method (e.g., using a Fisher Information Matrix (FIM)) to approximate the posterior distribution. Specifically, the system determines the Fisher Information Matrix (FIM) of the parameters of the machine learning model relative to the first machine learning task, where for each parameter, the corresponding measure of the importance of the parameter is the corresponding value on the diagonal of the FIM. That is, each value on the diagonal of the FIM corresponds to a different parameter of the machine learning model.
[0053] Next, the system obtains new training data for training the machine learning model on a second, different machine learning task (step 204). The new training data includes multiple training examples. Each training example includes a pair of input examples and a target output of the input example.
[0054] In some embodiments, the first machine learning task and the second machine learning task are different supervised learning tasks. In some other embodiments, the first machine learning task and the second machine learning task are different reinforcement learning tasks.
[0055] The system then trains the machine learning model on the second machine learning task by training the machine learning model on new training data (step 206).
[0056] To allow the machine learning model to learn the second task without forgetting the first task (e.g., by retaining memory about the first task), the system trains the machine learning model on new training data by adjusting first values of the parameters to optimize an objective function that depends in part on a penalty term, where the penalty term is based on a measure of the determined importance of the parameter to the machine learning model relative to the first task.
[0057] The penalty term penalizes deviations from the first value of parameters that are more important to achieving acceptable performance on the first machine learning task more heavily than parameters that are less important to achieving acceptable performance on the first machine learning task. By adjusting the first values of the parameters to optimize (e.g., minimize) the objective function, the machine learning model ensures that the values of the parameters that are more important to the machine learning model achieving acceptable performance on the first machine learning task are more strongly constrained to not deviate from the first values than the values of the parameters that are less important to the machine learning model achieving acceptable performance on the first machine learning task.
[0058] In some embodiments, the objective function can be expressed as:
[0059]
[0060] The objective function shown in Equation 1 consists of two terms. The first term L B (θ) measures the performance of the machine learning model on the second machine learning task. The first term can be any objective function suitable for the second machine learning task, such as cross entropy loss, mean squared error, maximum likelihood, deep Q network (DQN) objective, etc. The second term is a penalty term that imposes a penalty on parameter values that deviate from the first parameter value. Specifically, for each parameter i of the plurality of parameters of the machine learning model, the penalty term depends on the product of: (i) a corresponding measure F of the importance of the parameter to the machine learning model in achieving an acceptable level of performance on the first machine learning task; i , and (ii) the current value of the parameter θ i and the first value of the parameter The second term also depends on λ, which sets how important the old task (e.g., the first machine learning task) is compared to the new task (e.g., the second machine learning task). i The value may represent uncertainty in the neural network weights and may be derived from the FIM diagonal values or other values.
[0061] In one method of adjusting a first value of a parameter, the system performs the following iterations for each of a plurality of training examples in new training data.
[0062] For each training example, the system processes the input example with the machine learning model, following the current values of the machine learning model's parameters, to determine the model output.
[0063] For the first iteration, the current value of the parameter is equal to the first value of the parameter, where the first value of the parameter is determined after the machine learning model is trained on the first machine learning task.
[0064] Next, the system determines the gradient of the objective function using the model output, the target output for the input example, the current values of the parameters of the machine learning model, and the first values of the parameters of the machine learning model.
[0065] The system then uses the determined gradient to adjust the current value of the parameter to optimize the objective function. The adjusted parameter value is then used as the current value of the parameter in the next iteration.
[0066] In some cases, the system processes multiple training examples in batches. In these cases, for each batch, the current value is fixed for each training example in the batch. More specifically, the system processes the input examples of the training examples in each batch using the machine learning model according to the current values of the parameters of the machine learning model to determine the model output. The system then uses the model output, the target output of the input examples in the batch, the current value of the current value of the parameter of the machine learning model, and the first value of the parameter of the machine learning model to determine the gradient of the objective function. The system then uses the determined gradient to adjust the current value of the parameter to optimize the objective function. The adjusted parameter value is then used as the current value of the parameter in the next batch.
[0067] After the system performs the above iterations for all training examples in the new training data, the system completes training the machine learning model on the new training data for the second machine learning task. The current values of the parameters obtained at the final iteration are determined as the trained parameters of the machine learning model with respect to the second machine learning task. Training the model in this manner produces trained parameter values that allow the machine learning model to achieve an acceptable performance level on the second machine learning task while maintaining an acceptable performance level on the first machine learning task.
[0068] Once the system has trained the machine learning model on a second machine learning task, the system can continue to train the machine learning model to achieve an acceptable level of performance on the new machine learning task while maintaining acceptable performance on the task for which the model has been trained.
[0069] For example, after the machine learning model is trained on a second machine learning task to determine second values for a parameter of the machine learning model, the system can continue to train the machine learning model on a third, different machine learning task.
[0070] Figure 3 is a flowchart of an example process 300 for training a machine learning model on a third machine learning task after the machine learning model is trained on first and second machine learning tasks.
[0071] For convenience, process 300 will be described as being performed by a system of one or more computers located at one or more locations. For example, a machine learning system (e.g., Figure 1 The machine learning system 100) can perform process 300.
[0072] For each of the plurality of parameters, the system may selectively determine a corresponding measure of the importance of the parameter to the machine learning model achieving acceptable performance on a second machine learning task (step 302).
[0073] The system obtains third training data for training the machine learning model on a third machine learning task (step 304).
[0074] The system then trains the machine learning model on the third machine learning task to adjust the second value of the parameter by training the machine learning model on the third training data, so that the machine learning model achieves an acceptable performance level on the third machine learning task while maintaining an acceptable performance level on the first machine learning task and the second machine learning task (step 306). During the training of the machine learning model on the third machine learning task, the value of the parameter that is more important for the machine learning model to achieve acceptable performance on the first machine learning task and the second machine learning task is more strongly constrained to not deviate from the second value than the value of the parameter that is less important for the machine learning model to achieve acceptable performance on the first machine learning task and the second machine learning task.
[0075] In some cases, the system trains the machine learning model to adjust the second value of the parameter to obtain a third parameter value that optimizes a new objective function that depends in part on a penalty term, where the penalty term is based on a measure of the importance of the parameter to the machine learning model achieving acceptable performance on the first machine learning task and based on a measure of the importance of the parameter to the machine learning model achieving acceptable performance on the second machine learning task.
[0076] For example, the new objective function includes: (i) a first term that measures the performance of the machine learning model on a third machine learning task, (ii) a second term that penalizes parameter values that deviate from the first parameter value, wherein the second term penalizes deviations from the first value of parameters that are more important to achieving acceptable performance on the first machine learning task more heavily than for parameters that are less important to achieving acceptable performance on the first machine learning task, and (iii) a third term that penalizes parameter values that deviate from the second parameter value, wherein the third term penalizes deviations from the second value of parameters that are more important to achieving acceptable performance on the second machine learning task more heavily than for parameters that are less important to achieving acceptable performance on the second machine learning task.
[0077] For each parameter in the plurality of parameters, a second term of the new objective function may depend on the product of: (i) a corresponding measure of the importance of the parameter to the machine learning model achieving acceptable performance on the first machine learning task, and (ii) a difference between the current value of the parameter and the first value of the parameter.
[0078] For each parameter in the multiple parameters, the third term of the new objective function can depend on the product of: (i) a corresponding measure of the importance of the parameter to the machine learning model achieving acceptable performance on the second machine learning task, and (ii) the difference between the current value of the parameter and the second value of the parameter.
[0079] The obtained third parameter values allow the machine learning model to achieve an acceptable performance level on the third machine learning task while maintaining an acceptable performance level on the first machine learning task and the second machine learning task.
[0080] In some embodiments, the objective function can be expressed as:
[0081]
[0082] in, is a penalty term that imposes a penalty on parameter values that deviate from the first parameter value, and for each parameter i of the plurality of parameters of the machine learning model, the penalty term depends on the product of: (i) a corresponding measure F of the importance of the parameter to the machine learning model in achieving an acceptable level of performance on the first machine learning task; A,i , and (ii) the current value of the parameter θ i with the first value of the parameter The difference between. is another penalty term that imposes a penalty on parameter values that deviate from the second parameter value, and for each parameter i of the plurality of parameters of the machine learning model, the penalty term depends on the product of: (i) a corresponding measure F of the importance of the parameter to the machine learning model in achieving an acceptable level of performance on the second machine learning task B,i , and (ii) the current value of the parameter θ i with the second value of the parameter The difference between.
[0083] A machine learning system (e.g., an RL system) implementing the above method may include a system that automatically discerns switching between tasks. This can implement an online clustering algorithm trained without supervision. For example, this can be done by modeling the current task as a categorical context c, which is considered to be a hidden variable of a Hidden Markov Model that explains the current observation. The task context c can adjust the generative model that predicts the probability of the observation, and a new generative model can be added if the new generative model explains the recent data better than the existing generative model pool. For example, at the end of each continuous time window, the model that best corresponds to the current task can be selected, and an uninitialized (uniformly distributed) model can be selected to create a new generative model and task context.
[0084] An RL system implementing the above method can operate on-policy or off-policy; where off-policy operation can maintain a separate experience buffer for each task of discrimination or inference. Optionally, the gains and biases of the neural network may be task-specific.
[0085] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.
[0086] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-temporary storage medium, for operation by a data processing device or for controlling the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal) that is generated to encode information to transmit the information to a suitable receiver device for operation by a data processing device.
[0087] The term "data processing apparatus" refers to data processing hardware and includes various devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to the hardware, the apparatus may optionally include code that creates an operating environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0088] A computer program may also be referred to or described as a program, software, software application, application (app), module, software module, script or code, and may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, a single file dedicated to the program in question, or in multiple coordinated files, such as files that store one or more modules, subroutines or code portions. A computer program may be deployed to run on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.
[0089] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.
[0090] The processes and logic flows described in this specification can be performed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. These processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs or ASICs, or by a combination of special purpose logic circuits and one or more programmed computers.
[0091] A computer suitable for running a computer program may be based on a general or special purpose microprocessor or both, or any other type of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by or incorporated into a special purpose logic circuit. Typically, a computer will also include or be operably coupled to receive data from one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or to transfer data to one or more mass storage devices or both for storing data. However, a computer need not have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.
[0092] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
[0093] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending text messages or other forms of messages to a personal device (such as a smartphone running a messaging application), and in return receive a response message from the user.
[0094] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, i.e., inference, workloads.
[0095] Machine learning models can be implemented and deployed using machine learning frameworks such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.
[0096] Embodiments of the subject matter described in this specification may be implemented in a computing system including a back-end component, such as a data server, or including a middleware component, such as an application server, or including a front-end component, such as a client computer with a graphical user interface, a web browser, or an application, through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.
[0097] A computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data to the user device, such as an HTML page, for example, for displaying data to the device acting as a client and receiving user input from the user interacting with the device. Data generated at the user device, such as the result of the user interaction, can be received from the device at the server.
[0098] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Some features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as acting in some combinations, or even initially claimed, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may point to a sub-combination or a variant of a sub-combination.
[0099] Similarly, although operations are described in the drawings and recited in a particular order in the claims, this should not be understood as requiring that the operations be performed in the particular order shown or in the sequential order, or that all of the operations shown be performed, to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.
[0100] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes described in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.
Claims
1. A method for training a machine learning model with multiple parameters, in, The machine learning model is configured to be trained on a plurality of machine learning tasks, and for each machine learning task, the machine learning model receives inputs representing different environments or different objects in the environments, and generates outputs for interacting with the different environments or different objects in the environments based on the received inputs and values of parameters of the machine learning model, wherein the environments are real-world environments, or wherein the objects are voices or patterns and the interaction is recognition of the voices or patterns, wherein the machine learning model has been trained on a first machine learning task to determine a first value of a parameter of the machine learning model, is configured to receive an input and generate an output for interaction based on the received input and the determined first value of the parameter, and Wherein, the method comprises: For each of the plurality of parameters, determining a corresponding measure of importance of the parameter to achieving acceptable performance of the machine learning model on the first machine learning task; Obtaining training data for training the machine learning model on a second machine learning task; and By training the machine learning model on the training data, training the machine learning model on the second machine learning task to adjust the first value of the parameter so that the machine learning model achieves an acceptable performance level on the second machine learning task while maintaining an acceptable performance level on the first machine learning task, wherein, during training of the machine learning model on the second machine learning task, values of parameters that are more important to the machine learning model achieving acceptable performance on the first machine learning task are more strongly constrained to not deviate from the first values than values of parameters that are less important to the machine learning model achieving acceptable performance on the first machine learning task, and After training the machine learning model on the second machine learning task, only one set of parameter values among the multiple parameters needs to be stored instead of multiple sets of parameter values.
2. The method according to claim 1, wherein: The first machine learning task and the second machine learning task are different supervised learning tasks.
3. The method according to claim 1, wherein: The first machine learning task and the second machine learning task are different reinforcement learning tasks.
4. The method according to any one of claims 1 to 3, wherein: Training the machine learning model on the training data includes: The first value of the parameter is adjusted to optimize an objective function, wherein the objective function comprises: (i) first, measuring the performance of the machine learning model on the second machine learning task, and (ii) a second term that penalizes parameter values that deviate from the first parameter value, wherein the second term penalizes deviations from the first value more heavily for parameters that are more important to achieving acceptable performance on the first machine learning task than for parameters that are less important to achieving acceptable performance on the first machine learning task.
5. The method according to claim 4, wherein: For each training example in the training data, training the machine learning model on the training data comprises: processing the training examples using the machine learning model according to current values of parameters of the machine learning model to determine a model output; determining a gradient of the objective function using the model output, the target output of the training example, current values of parameters of the machine learning model, and first values of parameters of the machine learning model; and The gradient is used to adjust the current value of the parameter to optimize the objective function.
6. The method according to claim 4, wherein: For each parameter of the plurality of parameters, the second term depends on the product of a respective measure of importance of the parameter and a difference between a current value of the parameter and a first value of the parameter.
7. The method according to any one of claims 1 to 3, wherein: For each of the plurality of parameters, determining a corresponding metric of importance of the parameter to achieving acceptable performance of the machine learning model on the first machine learning task comprises: For each of the multiple parameters, determine an approximation of the probability that a current value of the parameter is a correct value of the parameter given first training data used to train the machine learning model on the first machine learning task.
8. The method according to any one of claims 1 to 3, wherein: For each of the plurality of parameters, determining a corresponding metric of importance of the parameter to achieving acceptable performance of the machine learning model on the first machine learning task comprises: Determine a Fisher information matrix FIM of multiple parameters of the machine learning model relative to the first machine learning task, wherein, for each of the multiple parameters, a corresponding measure of the importance of the parameter is a corresponding value on the diagonal of the FIM.
9. The method according to any one of claims 1 to 3, further comprising: After the machine learning model is trained on the second machine learning task to determine second values of the parameters of the machine learning model: Obtaining third training data for training the machine learning model on a third machine learning task; as well as By training the machine learning model on the third training data, the machine learning model is trained on the third machine learning task to adjust the second value of the parameter so that the machine learning model achieves an acceptable performance level on the third machine learning task while maintaining an acceptable performance level on the first machine learning task and the second machine learning task, During training of the machine learning model on the third machine learning task, values of parameters that are more important for the machine learning model to achieve acceptable performance on the first machine learning task and the second machine learning task are more strongly constrained not to deviate from the second value than values of parameters that are less important for the machine learning model to achieve acceptable performance on the first machine learning task and the second machine learning task.
10. The method according to claim 9, further comprising: For each of the plurality of parameters, determining a corresponding measure of importance of the parameter to achieving acceptable performance of the machine learning model on the second machine learning task; and Wherein, training the machine learning model on the third training data comprises adjusting the second value of the parameter to optimize an objective function, wherein the objective function comprises: (i) first, measuring the performance of the machine learning model on the third machine learning task, (ii) a second term that penalizes parameter values that deviate from the first parameter value, wherein the second term penalizes deviations from the first value more heavily for parameters that are more important to achieving acceptable performance on the first machine learning task than for parameters that are less important to achieving acceptable performance on the first machine learning task, and (iii) a third item that penalizes parameter values that deviate from the second parameter value, wherein the third item penalizes deviations from the second value of parameters that are more important to achieving acceptable performance on the second machine learning task more heavily than deviations from the second value of parameters that are less important to achieving acceptable performance on the second machine learning task.
11. The method according to claim 10, wherein: For each of the multiple parameters, the second term depends on the product of: (i) a corresponding measure of the importance of the parameter to the machine learning model achieving acceptable performance on the first machine learning task, and (ii) a difference between a current value of the parameter and the first value of the parameter.
12. The method according to claim 10, wherein: For each of the multiple parameters, the third term depends on the product of: (i) a corresponding measure of the importance of the parameter to the machine learning model achieving acceptable performance on the second machine learning task, and (ii) a difference between a current value of the parameter and the second value of the parameter.
13. The method of claim 4 further includes identifying when to switch from one machine learning task to another machine learning task and updating a second term of the objective function in response thereto.
14. The method according to claim 13, wherein: Recognizing when to switch from one machine learning task to another machine learning task includes inferring which task is being performed based on one or more models.
15. A system for training a machine learning model with multiple parameters, comprising one or more computers and one or more storage devices storing operable instructions, which, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 14.
16. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 14.
Citation Information
Patent Citations
Training distilled machine learning models
CN105160397A
Techniques for evaluation, building and / or retraining of a classification model
US20130254153A1
