Neural Networks for Scalable Continuous Learning in Domains with Sequential Learning Tasks

Abstract

A method, system and apparatus, including a computer program encoded on a computer storage medium, for scalable continuous learning using a neural network. One of the methods includes: receiving new training data for a new machine learning task; training an active subnet on the new training data to determine training values ​​of active network parameters from initial values ​​of the active network parameters while keeping current values ​​of knowledge parameters fixed; training a knowledge subnet on the new training data to generate knowledge outputs for the new training input that match the active outputs generated by the trained active subnet for the new training input, and determining updated values ​​of the knowledge parameters from the current values ​​of the knowledge parameters.

Description

Technical Field

[0001] This specification relates to multi-tasking neural networks, that is, using the same neural network to perform multiple machine learning tasks. Background Art

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to one or more other layers in the network, i.e., one or more other hidden layers, the output layer, or both. Each layer of the network generates an output from a received input based on the current values ​​of a corresponding set of parameters. Summary of the invention

[0003] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, that trains a neural network to perform a plurality of machine learning tasks.

[0004] Neural networks are capable of continuous learning in a scalable manner due to their architecture and the way they are trained on various machine learning tasks. Continuous learning refers to the ability to learn new machine learning tasks without forgetting earlier tasks. In other words, continuous learning refers to training a neural network to achieve acceptable performance on new machine learning tasks while maintaining acceptable performance on earlier tasks.

[0005] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0006] Many systems may be needed to perform multiple different machine learning tasks. For example, some systems may be needed to perform multiple different image classification tasks, e.g., multiple different tasks of classifying an input image into different categories. As another example, some systems may be needed to perform multiple different speech recognition tasks, e.g., multiple different hotword detection tasks for different users of the same mobile device. As yet another example, a system controlling a mechanical agent may be needed to control the same agent to perform multiple different tasks.

[0007] Some conventional methods for multi-task learning require adding new parameters for each task. For example, some methods use a separate neural network for each task. As another example, some methods add a new subnetwork to an existing neural network for each task. Because new parameters are added for each task, these systems quickly become too computationally intensive to effectively use them for multiple tasks. That is, because new parameters are added for each task, the systems consume more computational resources, such as memory, processing power, and processing time, and they need to perform each new task.

[0008] Other conventional approaches to multi-task learning use the same neural network for each task and adjust the neural network's parameter values ​​for each new task the system needs to perform. These systems can fail when multiple tasks need to be performed. In some cases, when parameter values ​​are changed to enable high-quality performance on a new task, performance on earlier tasks degrades because the new values ​​are no longer useful for the earlier tasks. In other cases, such as when parameter values ​​are constrained to be close to earlier values, it is impossible to train the neural network to achieve high-quality performance on the new task.

[0009] In contrast, a system as described herein can generate high-quality output for multiple different tasks in a resource-efficient manner, ie, without consuming excessive computing resources due to an increase in the number of tasks that the system needs to perform.

[0010] In summary, this document describes a neural network that includes a knowledge subnet and an active subnet. For example, each subnet can be a multi-layer system with an output layer that generates the output of the subnet. During training, the two subnets will typically receive the same input (e.g., when the neural network is performing a given task, the inputs to both subnets are examples of input data that the neural network will transform into output data according to the task), but some or all layers of the active subnet receive additional input generated by the knowledge subnet (e.g., the output of a layer of the output network that is not the output layer of the knowledge subnet).

[0011] The neural networks are continuously trained to learn multiple tasks. For each task, when corresponding examples of the task are input to both networks, the active subnet is first trained (e.g., starting from an initial state) to learn the task, and then the knowledge subnet is trained to generate outputs that match the outputs of the active subnet. For each new task (e.g., each task except the first task), the training of the knowledge subnet for the new task uses an objective function that includes one or more terms that penalize modifying parameters of the knowledge network compared to the values ​​of the parameters before training the knowledge subnet for the new task.

[0012] The described system is able to maintain effective performance on earlier tasks even after training on new tasks, even though the number of parameters of the neural network does not increase as the neural network is trained on the new task. This is because the trained active subnet is used to "extract" the knowledge learned from the new task into the knowledge subnet, while preventing the knowledge subnet from "forgetting" the earlier tasks. That is, the knowledge subnet is trained to match the output generated by the trained active subnet on the new task, while still maintaining effective performance on the earlier task.

[0013] When training a new task, because of the connection between the knowledge subnetwork and the active subnetwork, and because the knowledge parameters are fixed when training the active subnetwork, the described system effectively leverages knowledge extracted from earlier tasks to speed up training on the new task (and reduce the amount of computational resources required for training), improve performance on the new task, or both.

[0014] Because of the way the knowledge subnetwork is trained, the described system can be trained on a large number of new tasks without causing excessive degradation in performance on earlier tasks and, as mentioned above, without increasing the amount of computational resources consumed by the neural network.

[0015] Additionally, the described system is capable of forward shifting. That is, because of the way the knowledge subnet is trained on a given task, after the system trains a neural network on a new task, if the new task is similar or related to the earlier task, the performance of the knowledge subnet on the earlier task can improve. Thus, the system can improve performance on the earlier task without having to look at new training data for that task, allowing training to be more resource-efficient.

[0016] This makes the described system suitable for applications with limited computing resources. A specific example of this is a neural network implemented on a consumer device (e.g., a mobile device (e.g., a smart phone)) with limited computing power and memory resources. For example, the described system can be trained to solve multiple user-defined tasks, such as recognizing specific hot words (e.g., commands) spoken by a user, for example, different hot words are relevant in different situations where the user uses the mobile device. Another specific example is a neural network embedded in a robot control system, where the neural network is used to control the robot to perform multiple different tasks.

[0017] Furthermore, because after training the same knowledge subnetwork is used to generate outputs for all tasks, the system does not need to label received inputs as to which tasks they correspond to. This allows the described system to be used even when the system does not have access to information specifying which input corresponds to which task at test time.

[0018] In some cases, at least one task, and preferably more than one of a series of tasks, such as all of a series of tasks, that the neural network learns is:

[0019] (i) The task of converting input sensor data representing the real world (e.g., images or videos captured by a camera; or sound signals captured by a microphone) into an output that classifies the sensor data into one of multiple categories;

[0020] (ii) the task of converting input data (e.g. textual data) into control data for a unit that acts on the real world (e.g. an actuator, an image generating device (screen or projector) or a speaker unit), or

[0021] (iii) The task of transforming input data representing an environment (e.g., the real world) with which an agent (e.g., a robot or autonomous vehicle) interacts into output data defining the agent's actions (e.g., the output data is control data for the agent or is used to generate control data for the agent).

[0022] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1A and Figure 1B An example multi-task neural network system is shown.

[0024] Figure 2 is a flow chart of an example process for training a neural network for a new task.

[0025] Figure 3 is a flow chart of an example process for training a neural network for additional tasks after the neural network has been trained for one or more tasks.

[0026] Figure 4 is a flow chart of an example process for determining updates to knowledge parameters during a compression phase of training.

[0027] Reference numerals and designations in the various drawings indicate similar elements. DETAILED DESCRIPTION

[0028] Figure 1A and Figure 1B An example multi-task neural network system 100 is shown. Multi-task neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0029] The multi-task neural network system 100 trains a neural network to perform multiple machine learning tasks. Due to the architecture of the neural network and the way the neural network is trained on various machine learning tasks, the neural network is able to continuously learn in a scalable manner. Continuous learning refers to the ability to learn new machine learning tasks without forgetting earlier tasks. In other words, continuous learning refers to training the neural network to achieve acceptable performance on new machine learning tasks while maintaining acceptable performance on earlier tasks. "Acceptable" can be defined as an error rate on a task below a threshold. The definition of error rate and the corresponding threshold may be different for different tasks. Optionally, after learning a new task, there may be a step of checking whether the error rate of one or more earlier tasks is below the corresponding threshold, and if it is determined that the error rate for one or more earlier tasks is not the case, then those earlier (or multiple) tasks can be learned again in succession (e.g., until the error rate of those tasks is acceptable again).

[0030] Typically, the system 100 trains a neural network on multiple tasks from a domain of tasks learned sequentially (i.e., the domain of tasks learned in sequence). That is, the system 100 trains a neural network on multiple machine learning tasks sequentially, i.e., one after the other, rather than training a neural network on multiple tasks in parallel. In some cases, the system "revisits" a particular task during training, i.e., the same task may appear in multiple positions in the sequence.

[0031] As system 100 trains neural networks on tasks sequentially, once a neural network has been trained on a given task, system 100 can receive and process input for any task on which the neural network has been trained.

[0032] In some cases, many machine learning tasks are supervised learning tasks.

[0033] For example, the plurality of tasks may include a plurality of different image classification tasks. Each image classification task requires an output that is a distribution of scores over a set of object categories. In some cases, the images to be classified may be drawn from different distributions for different image classification tasks, e.g., one task may require classifying images of real-world objects, while another task may require classifying paintings or other artificially generated images. Alternatively or in addition, the sets of object categories are different for different image classification tasks. For example, one task may require classifying images depicting objects from a subset of a larger set of categories, while another task may require classifying images depicting objects from a different subset of a larger set of categories.

[0034] As another example, the plurality of tasks may include a plurality of different speech recognition tasks. For example, the tasks may include a plurality of hot word detection tasks, each of which requires recognition of a different hot word or hot word sequence.

[0035] As another example, the plurality of tasks may include a plurality of different natural language processing tasks. For example, the tasks may include a plurality of machine translation tasks, each task needing to be translated into a different language.

[0036] In other cases, the multiple machine learning tasks are multiple different reinforcement learning tasks.

[0037] For example, a task may include a plurality of different agent control tasks, i.e., a task that includes controlling a mechanical agent so that the agent performs a specific task. The agent may be, for example, a robot or an autonomous or semi-autonomous vehicle. Each input to the neural network may include an observation that characterizes the state of the environment with which the agent interacts, and one of the outputs of the neural network may be a policy output that defines an action to be performed by the agent in response to the observation, for example, an output that defines a probability distribution of possible control inputs to the agent, an output that predicts the return of one or more possible control inputs, or an output that directly identifies the best control input for the agent in response to the observation. The neural network may also generate a value output that estimates the value to complete the current task for the environment in the current state. A plurality of different tasks may include tasks that require the same agent to be controlled to complete two different tasks, tasks that require different agents to be controlled to complete the same task, or both. In different tasks, the environment may be the same or different.

[0038] The neural network trained to perform multiple tasks includes a knowledge subnet 140 and an activity subnet 170. Both the knowledge subnet 140 and the activity subnet 170 are configured to generate the same type of output, i.e., the type of output required for the multiple tasks for which the neural network is being trained. For example, the two subnets can generate a probability distribution over a set of possible classifications for each received input of a supervised learning task, or can generate a policy output, a value output, or both for a reinforcement learning task. The output generated by the knowledge subnet 140 will be referred to as a knowledge output in this specification, and the output generated by the activity subnet 170 will be referred to as an activity output in this specification.

[0039] Once a neural network has been trained for a particular task, the knowledge output, ie, the output of the knowledge subnetwork 140, can be used as the output of the neural network for input corresponding to that task (and any earlier tasks in the training sequence for which the neural network has been trained).

[0040] Although the activity subnet 170 and the knowledge subnet 140 generate the same type of outputs, the two subnets typically have different neural network architectures.

[0041] More specifically, both the active subnet 170 and the knowledge subnet 140 include corresponding multiple neural network layers (referred to as “active layers” and “knowledge layers”, respectively) and have corresponding parameters (referred to as “active parameters” and “knowledge parameters”, respectively).

[0042] However, at least one of the active layers in the active subnet 170 is configured to receive as input a layer output generated by one of the knowledge layers in the knowledge subnet 140 during the processing of the inputs of both subnets. As will be seen in the following description, configuring the neural network in this manner allows the active subnet 170 to use information learned by the knowledge subnet 140 for an earlier task while training for a given task.

[0043] The active subnet 170 and the knowledge subnet 140 may have any of a variety of neural network architectures, wherein some or all of the active neural network layers have corresponding counterparts in the knowledge subnet and receive as inputs inputs derived from the outputs of the corresponding knowledge layers.

[0044] In one form, each subnetwork comprises a sequence of layers, where the first layer of the sequence receives the input to the subnetwork, the last layer of the sequence is an output layer that generates the output of the subnetwork, and each layer (except the last layer) generates an output that is fed into the input of the next layer of the sequence. One or both of the two networks may be feed-forward systems, where output data from one layer is not fed back to earlier layers in the sequence. Depending on the input to the neural network, the layers in the sequence may include one or more of the following: fully connected layers, convolutional layers, or recurrent layers.

[0045] As a specific example, the knowledge subnet and a portion of the active subnet can have the same architecture (e.g., each can include the same number of layers), but the active subnet can also include multiple hierarchical adapter layers. Each hierarchical adapter layer corresponds to a corresponding knowledge layer and a corresponding active layer, and is configured to receive as input a layer output generated by one of the corresponding knowledge layers, process the layer output to generate a transformation output, and provide the transformation output as input to one of the corresponding active neural network layers. Thus, the corresponding active neural network layer receives input from both the hierarchical adapter layer and another active layer in the active subnet 170.

[0046] In the case where the active subnet 170 and the knowledge subnet 140 respectively include a sequence of layers, sequentially, a given hierarchical adapter layer may connect the output of the knowledge layer at position i in the sequence to the input of the active layer at position i+1 in the sequence. When the knowledge layer is a fully connected layer or a recurrent layer, the hierarchical adapter layer may be a multilayer perceptron. When the knowledge layer is a convolutional layer, the hierarchical adapter layer may be a 1x1 convolutional layer.

[0047] Examples of hierarchical adapter layers and architectures for activity and knowledge subnetworks that can be used for various tasks are described in more detail in Progressive Neural Networks, Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, Raia Hadsell, available at https: / / arxiv.org / abs / 1606.04671, the contents of which are incorporated herein by reference.

[0048] To train a neural network for a given new task, the system first trains the active subnet 170 on the training data for the task, while keeping the values ​​of the active parameters 140 of the knowledge subnet fixed. This training of the active subnet 170 will be referred to as the "progress" phase of training, and Figure 1A Described in.

[0049] If the given new task is not the first task for which the neural network is being trained, the system reinitializes the values ​​of the activity parameters of the activity subnetwork 170 and then trains the activity subnetwork 170 to determine the training values ​​of the activity parameters from the reinitialized values. Figure 3 Describes reinitialization activity parameters in more detail.

[0050] The system 100 then trains the knowledge subnet 140 while keeping the trained values ​​of the activity parameters of the activity subnet 170 fixed. In particular, the system trains the knowledge subnet 140 to generate knowledge outputs that match the activity outputs generated by the trained activity subnet for training inputs for the new task. When the neural network has been trained on one or more earlier tasks, the system 100 trains the knowledge subnet to achieve acceptable performance in matching the activity outputs while maintaining acceptable performance for the earlier machine learning tasks. Thus, once trained, the knowledge subnet 140 is able to generate high quality outputs for both new and earlier tasks. This training of the knowledge subnet 140 will be referred to as the "compression" phase of training, and is performed during the training. Figure 1B Described in.

[0051] In particular, if Figure 1A As shown, system 100 performs a progression phase of training by repeatedly performing phases A through C on different sets of training inputs 110 for new tasks.

[0052] For example, for a supervised learning task, different sets of training inputs may be different micro-batches of training inputs (and corresponding target outputs) for a new task. As another example, for a reinforcement learning task, different sets of training inputs may be different sets of experience data, i.e., different micro-batches of trajectories or experience tuples generated as a result of the agent interacting with the environment to perform a new task.

[0053] In stage A of the progression phase of training, the system 100 processes each received new task network input 110 using the knowledge subnet 140 and the activity subnet 170 .

[0054] Because some or all of the active layers receive inputs generated by or derived from outputs generated by the knowledge layers, as described above, when the neural network has been trained on one or more earlier tasks, this allows the outputs generated by the active sub-network 170 to take advantage of information learned by the knowledge sub-network 140 during training on the earlier tasks, which may also be relevant to the current task. In other words, the system 100 effectively exploits similarities between the earlier tasks and the new task to improve the training of the active sub-network 170 on the new task.

[0055] At stage B of the progression phase of training, the training engine 180 determines the gradients with respect to the activity parameters using a machine learning technique appropriate to the machine learning task. For example, for a supervised learning task, the system may determine the gradient of a supervised learning objective function (e.g., a cross entropy loss or a log likelihood loss) that measures the quality of the activity output relative to the ground truth or target output of the new input 110. As another example, for a reinforcement learning task, the system may determine the gradient using a reinforcement learning technique appropriate to the type of output generated by the subnetwork, such as a policy gradient technique, an actor-critic technique, or a Q-learning technique.

[0056] In stage C of the progression phase of training, the system 100 updates the current values ​​of the activity parameters according to the update rules employed by the machine learning technique while keeping the knowledge parameters fixed.

[0057] Thus, during the progression of training, both the knowledge sub-network 140 and the active sub-network 170 process each received training input, but only the values ​​of the active parameters are updated, while the values ​​of the knowledge parameters are fixed.

[0058] like Figure 1B As shown, system 100 performs the compression phase of training by repeatedly performing stages D through F on different sets of training inputs 210 for a new task.

[0059] In some cases, the training input 210 used in the compression phase is the same as the training input 110 used in the progression phase.

[0060] In some other cases, the training inputs used in the compression phase are different from the training inputs used in the advancement phase. For example, the training inputs used in the compression phase can be a superset of the training inputs used in the advancement phase, e.g., in a supervised learning task where additional unlabeled inputs are available.

[0061] In stage D of the compression phase of training, the system 100 processes each received new task network input 210 using the knowledge subnet 140 to generate a knowledge output for each task network input 210 , and processes each received new task network input 210 using the activity subnet 170 to generate an activity output for each task network input 210 .

[0062] Specifically, the system 100 uses the active subnet 170 to process the network input according to the training values ​​of the active parameters determined during the progress phase of training, i.e., the values ​​of the activity parameters are fixed after the progress phase and are not adjusted during the compression phase, while using the knowledge subnet 140 to process the network input according to the values ​​of the knowledge parameters after the previous iteration of phase DF.

[0063] At stage E of the compression phase of training, the training engine 180 determines the gradient of the knowledge parameters with respect to the objective function that encourages the knowledge outputs for new training inputs to match the activity outputs generated by the training activity subnetwork for the new training inputs.

[0064] More specifically, when the knowledge sub-network 140 has been trained on one or more earlier tasks, the objective function encourages the knowledge sub-network to achieve acceptable performance in matching the activity outputs while maintaining acceptable performance on the earlier machine learning tasks.

[0065] Refer to the following Figure 4 Examples of objective functions that may be used in the compression phase of training are described in more detail.

[0066] In stage F of the compression phase of training, the training engine 180 uses the determined gradients to update the current values ​​of the knowledge parameters, i.e., by determining updates to the current values ​​of the knowledge parameters from the gradients using an update rule (e.g., using an rmsProp or Adam optimizer update rule), and then adding the updates to the current values ​​of the parameters.

[0067] By repeatedly performing phase DF in this manner, the system 100 trains the knowledge subnet 140 to achieve acceptable performance on new tasks while maintaining acceptable performance on any earlier tasks on which the knowledge subnet has been trained.

[0068] After training the knowledge sub-network 140 and the activity sub-network 170 for a new task, the system 100 can use the knowledge output generated by the knowledge sub-network 140 for the new input as the output of the neural network for the new task.

[0069] Thus, the system 100 trains the knowledge subnet 140 to perform well on multiple tasks in a scalable manner, i.e., without increasing the number of parameters of the knowledge subnet 140 after training any given task. In particular, by increasing the "knowledge" of the neural network during the progression phase, and then "refining" or "compressing" the knowledge back into the knowledge subnet 140 during the compression phase, the system 100 effectively enables the knowledge subnet 140 to perform well on both the new task and any earlier tasks without increasing the size of the knowledge subnet.

[0070] Figure 2 1 is a flow chart of an example process 200 for training a neural network for a new task. For convenience, process 200 is described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed multi-task neural network system, such as multi-task neural network system 100 of FIG. 1 , can perform process 200.

[0071] The system receives new training data for a new machine learning task (step 202). The new training data includes new training inputs. When the new task is a supervised learning task, the new training data also includes target outputs for the new training inputs. When the new task is a reinforcement learning task and the training inputs are or each include observations, the new training data also includes data specifying the actions performed by the agent in response to the observations for some or all of the observations and the reward values ​​received by the agent after performing the actions.

[0072] The system trains the active subnet on the new training data to determine the training values ​​of the active network parameters from the initial values ​​of the active network parameters while keeping the current values ​​of the knowledge parameters fixed (step 204). As described above, if the neural network has been trained for any earlier task, the current values ​​of the knowledge parameters are the values ​​of the knowledge parameters after training for the earlier task. In addition, as described above, due to the connection between the knowledge layer and the active layer, even if the values ​​of the active parameters are reset after the earlier training, the active subnet utilizes the information learned by the knowledge subnet during the earlier training.

[0073] The system trains the knowledge subnet on the new training data to generate knowledge outputs for the new training input by training the knowledge subnet, thereby determining updated values ​​of the knowledge parameters from the current values ​​of the knowledge parameters, and the knowledge outputs for the new training input match the activity outputs generated by the trained activity subnet for the new training input (step 206).

[0074] As mentioned above, since the system uses the activity output as the training target for the knowledge subnet, the system can use additional training data for new tasks (if available) when training the knowledge subnet, even if the additional training data is unlabeled.

[0075] Additionally, as described above, when the knowledge subnet has been trained on one or more earlier machine learning tasks to determine the current values ​​of the knowledge parameters, the system trains the knowledge subnet to achieve acceptable performance in matching the activity outputs while maintaining acceptable performance of the earlier machine learning tasks. Figure 4 Updating knowledge parameters during training is described in more detail.

[0076] Figure 3 1 is a flow chart of an example process 300 for training a neural network for an additional task after the neural network has been trained for one or more earlier tasks. For convenience, process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed multi-task neural network system, such as multi-task neural network system 100 of FIG. 1 , can perform process 300.

[0077] The system receives training data for an additional task (step 302).

[0078] The system resets the values ​​of the activity parameters (step 304). For example, the system can reset the values ​​to predetermined initial values, such as zero or a fixed small positive integer. As another example, the system can reset the values ​​by generating initial values ​​for the activity parameters according to conventional machine learning parameter initialization techniques, for example, by sampling values ​​for each activity parameter from a predetermined distribution. Importantly, the system does not reset the values ​​of the knowledge parameters, and retains the values ​​of those parameters from earlier training.

[0079] The system then trains the neural network on the additional task, as described above.

[0080] In particular, the system first trains the active subnet on the additional training data to determine the training values ​​of the active parameters from the reset values ​​(step 306), while keeping the already trained values ​​of the knowledge parameters fixed. During this training, even if the values ​​of the parameters are reset, the active subnet can use information learned in earlier training, which may be relevant to the additional task, due to the connection between the active layer and the knowledge layer.

[0081] The system then trains the knowledge subnetwork on additional training data to determine further training values ​​for the knowledge parameters while keeping the training values ​​for the activity parameters fixed (step 308).

[0082] Figure 44 is a flow chart of an example process 400 for determining updates to knowledge parameters during a compression phase of training. For convenience, process 400 is described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed multi-task neural network system, such as multi-task neural network system 100 of FIG. 1 , can perform process 400.

[0083] The system can repeat process 400 for each new network input in the set of network inputs to determine a corresponding update for each network input, and then combine (eg, average) the updates for each network input to determine a final update to the knowledge parameters of the set of network inputs.

[0084] The system processes the new network input using the active subnet based on the trained values ​​of the active parameters from the progress phase to generate an active output for the new network input (step 402).

[0085] The system processes the new network input using the knowledge subnet based on the current values ​​of the knowledge parameters to generate a knowledge output for the new network input (step 404).

[0086] In some implementations, the output of the subnetwork is or includes a probability distribution, and the output layer of the subnetwork includes a softmax layer. A softmax layer is a layer that performs a softmax function defined by temperature. For example, with z k represents the input of the softmax layer (where the integer k represents the input), and the output of the softmax layer can be σ k ,in where temperature is the inverse of β. During training of the knowledge subnet, activity outputs and knowledge outputs may be generated at elevated temperatures relative to outputs generated by the knowledge subnet after training. That is, the system sets the temperature of the softmax output layer to a value higher than the value that will be used after training, e.g., a value greater than 1, so that the probability distribution generated by the softmax output layer is more uniform than the distribution generated by the trained neural network.

[0087] The system determines updates to the current values ​​of the knowledge parameters (step 406).

[0088] In particular, the system determines a gradient of a knowledge parameter with respect to an objective function that encourages the knowledge outputs of a new training input to match the activity outputs generated by the trained activity subnetwork for the new training input. If the knowledge subnetwork has been trained on one or more earlier tasks, the objective function also encourages the knowledge subnetwork to achieve acceptable performance in matching the activity outputs while maintaining acceptable performance on the earlier machine learning tasks.

[0089] More specifically, the system can minimize a loss function comprising: (i) a first term that measures the similarity between the knowledge output and the activity output given a new training input; and (ii) one or more second terms that penalize the knowledge subnetwork if the value of its knowledge parameters deviates from the value of its knowledge parameters after a previous round of training.

[0090] For example, the first term may be based on the KL divergence between the activity output and the knowledge output, ie, may be the KL divergence or may be the KL divergence adjusted by a weight.

[0091] In some cases, the second term is a penalty term from the elastic weight consolidation (EWC) loss function. The penalty term of the EWC loss slows down learning, that is, it encourages small updates to certain parameters, depending on how important those parameters are for earlier tasks. In particular, the penalty term caused by the EWC loss is the sum of the penalties for each earlier task. For a given new task i, the penalty term (i.e., the second term in the loss function) satisfies:

[0092]

[0093] where θ is the value of the knowledge parameter when evaluating the loss, is the maximum a posteriori (MAP) parameter for learning task j, F j is the Fisher information matrix of the knowledge subnetwork of task j evaluated in MAP parameters for task j, and its norm is the Mahalonobis norm.

[0094] The MAP parameter is the mode of the posterior distribution of the knowledge parameters when learning a given task i, i.e., p(θ|T 1:i ), where θ is the knowledge parameter and T represents the task. The MAP parameter can be calculated using a standard stochastic optimizer applied to the following loss:

[0095]

[0096] Alternatively, the second term can be a modified online EWC penalty term that maintains a single overall Fisher information matrix for all earlier tasks, rather than maintaining a separate Fisher for each earlier task. In particular, the second term can satisfy:

[0097]

[0098] where θ is the value of the knowledge network parameters at the time of evaluating the loss, is the MAP parameter after training the knowledge subnet for the earlier task, that is, the MAP parameter for task i-1, is the overall Fisher matrix after training the knowledge subnetwork for all earlier tasks evaluated on the MAP parameter, γ is a hyperparameter less than 1, and the norm is the Mahalonobis norm.

[0099] In these cases, once the system has trained the neural network for task i, the system updates the MAP parameters, i.e., determines the new MAP parameters after training the knowledge subnetwork for task i using the above stochastic optimizer, and updates the overall Fisher matrix. For example, after training the knowledge subnetwork for task i, the overall Fisher matrix Can satisfy:

[0100]

[0101] Among them, F i is the Fisher matrix of task i evaluated under the MAP parameters after training task i. In some cases, the system first uses the Fisher matrix F i Normalize it before using it to compute the overall Fisher matrix. This can cause the loss function to treat each task equally when updating the Fisher, for example, rather than giving more weight to tasks with Fisher matrices with larger norms.

[0102] Once a neural network has been trained for a given task, the system can provide data specifying the trained knowledge subnet (e.g., trained values ​​for the knowledge parameters) and data defining the knowledge subnet architecture to another system for use in performing the new task as well as any earlier tasks on which the knowledge subnet was trained. Alternatively or in addition, the system can begin performing the new task using the trained knowledge subnet on newly received inputs.

[0103] This specification uses the term "configured to" in relation to system and computer program components. For a system of one or more computers configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0104] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium to be executed or control operations by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0105] The term "data processing apparatus" refers to data processing hardware and includes various devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof.

[0106] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple coordinated files (e.g., files storing one or more modules, subroutines or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0107] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way or at all, and the data may be stored on storage devices in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed differently.

[0108] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0109] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits (e.g., FPGAs or ASICs) or by a combination of special purpose logic circuits and one or more programmed computers.

[0110] Computers suitable for executing computer programs can be based on general or special microprocessors or both, or on any other type of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory can be supplemented by a dedicated logic circuit or incorporated into the dedicated logic circuit. Typically, a computer will also include or be operably coupled to receive data from one or more large-capacity storage devices (e.g., magnetic, magneto-optical disks or optical disks) for storing data or to transfer data to one or more large-capacity storage devices or both. However, a computer does not have to have such a device. In addition, a computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, for example, a universal serial bus (USB) flash drive, to name a few.

[0111] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0112] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse or trackball, through which the user can provide input to the computer). Other types of devices may also be used to provide interaction with a user, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, a computer may interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving a response message from the user.

[0113] Data processing devices used to implement machine learning models may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0114] The machine learning model can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).

[0115] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification) or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0116] The computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data such as HTML pages to the user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input therefrom. Data generated at the user device, such as the result of user interaction, can be received from the device on the server.

[0117] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on any invention or the scope of the claimed protection, but should be interpreted as descriptions of features specific to a particular embodiment of a particular invention. Specific features described in this specification in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments or in any suitable sub-combination, respectively. In addition, although the features may be described above as working in a specific combination and even initially claimed as such, in some cases, one or more features in the claimed combination may be deleted from the combination, and the claimed combination may be used for a sub-combination or a variation of the sub-combination.

[0118] Similarly, although operations are depicted in a particular order in the drawings and recited in the claims, this should not be construed as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0119] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

Claims

1. A method for training multi-task neural networks for new machine learning tasks, in, Select a new machine learning task from the following group: The task of converting input sensor data representing the real world into an output that classifies the sensor data into one of a plurality of categories, wherein the input sensor data is one of images, videos, and audio; The task of converting input data into control data for actuators that act on the real world; or The task of transforming input data representing an environment with which an agent interacts into output data defining actions performed by the agent on the environment; wherein the agent is one of a robot and an autonomous vehicle; wherein the multi-task neural network comprises (i) a knowledge subnet and (ii) an activity subnet, The knowledge subnet includes multiple knowledge neural network layers and has multiple knowledge parameters. The knowledge subnet is configured to receive network input and process the network input to generate knowledge output of the network input. The active subnetwork includes a plurality of active neural network layers and has a plurality of active parameters. The active subnet is configured to receive network input and process the network input to generate an active output of the network input, and wherein at least one of the activity neural network layers is configured to receive as input an output generated by one of the knowledge neural network layers while the multi-task neural network generates an activity output and a knowledge output from a network input, the method comprising: receiving new training data for a new machine learning task, the new training data comprising a plurality of new training inputs; training the active subnetwork on the new training data to determine training values ​​of the active network parameters from initial values ​​of the active network parameters while keeping current values ​​of the knowledge parameters fixed; and Training a knowledge subnet on new training data to determine updated values ​​of knowledge parameters from current values ​​of the knowledge parameters by training the knowledge subnet on new training inputs to minimize a loss function, the knowledge subnet being trained on new training inputs to minimize the loss function comprising: (i) a first term measuring the similarity between a knowledge output and an activity output given the new training input, and (ii) one or more second terms penalizing a knowledge subnet whose value of a knowledge parameter of the knowledge subnet deviates from the current value of the knowledge parameter.

2. The method according to claim 1, wherein: The knowledge subnetwork has been trained on one or more earlier machine learning tasks to determine the current values ​​of the knowledge parameters.

3. The method according to claim 1, wherein: The first term is based on the KL divergence between the activity output and the knowledge output.

4. The method according to claim 1, wherein: The one or more second items include items satisfying the following: where θ is the value of the knowledge network parameters at the time of evaluating the loss function, is the MAP parameter after training the knowledge subnet for the earlier task, is the overall Fisher matrix after training the knowledge subnetwork on the earlier task, γ is a hyperparameter, and the norm is the Mahalonobis norm.

5. The method according to claim 4, further comprising: After training the knowledge subnetwork for the new task, update the MAP parameters; as well as After training the knowledge subnetwork for the new task, the overall Fisher matrix is ​​updated.

6. The method according to claim 1, further comprising: receiving second new training data for a second new machine learning task; Reset the values ​​of active network parameters to initial values; as well as The active subnetwork is trained on the second new training data to adjust the values ​​of the active network parameters starting from the initial values ​​while keeping the trained values ​​of the knowledge parameters fixed.

7. The method according to claim 1, wherein: The active neural network layer comprises one or more hierarchical adapter layers, and wherein each hierarchical adapter layer is configured to: receiving as input a layer output generated by a corresponding one of the knowledge layers; Processing layer outputs to generate transformed outputs; and Provide the output of the transformation as input to the corresponding one of the active neural network layers.

8. The method according to claim 1, wherein: The knowledge subnet and the activity subnet each include a softmax layer that performs a softmax function defined by a temperature; After training the knowledge subnet, use the neural network to perform the task; as well as During training of the knowledge subnet, activity outputs and knowledge outputs are generated at an elevated temperature compared to a temperature used after training of the knowledge subnet.

9. The method according to claim 2, further comprising: After training the knowledge subnetwork on new training data, it receives input for one of the earlier machine learning tasks, Processing the input using the knowledge subnetwork to generate a knowledge output for the input, and Provides knowledge output as the output of the network input.

10. A system comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to implement a multi-tasking neural network: in, The multi-task neural network includes a knowledge subnet, The knowledge subnet includes multiple knowledge neural network layers and has multiple knowledge parameters. wherein the knowledge subnet is configured to receive a network input and process the network input to generate a knowledge output for the network input, and Wherein, using the operation of the corresponding method of any one of claims 1-9, the knowledge subnet and the activity subnet are jointly trained to perform multiple machine learning tasks.

11. A system comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, are operable to cause the one or more computers to perform the operations of the corresponding method of any one of claims 1-9.

12. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1-9.

Images