Training of Agent Neural Network through Open-Ended Learning

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, for training an agent neural network for use in controlling an agent to perform a plurality of tasks. One of the methods includes maintaining population data specifying a population of one or more candidate agent neural networks, and training each candidate agent neural network with a respective set of one or more tasks to update parameter values ​​of parameters of the candidate agent neural network in the population data, the training step including, for each candidate agent neural network, obtaining data identifying candidate tasks, obtaining data specifying a control strategy for the candidate tasks, determining whether to train the candidate agent neural network with the candidate tasks, and, in response to determining to train the candidate agent neural network with the candidate tasks, training the candidate agent neural network with the candidate tasks.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 226,124, filed on July 27, 2021. The disclosure of the prior application is considered a part of the disclosure of this application and is incorporated herein by reference.

[0002] This specification relates to the processing of data using machine learning models.

Background Art

[0003] A machine learning model receives an input and generates an output, for example, a predicted output, based on the received input. Some machine learning models are parametric models that generate an output based on the received input and the values of the model's parameters.

[0004] Some machine learning models are deep models that utilize multiple layers of models to generate an output for the received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non - linear transformation to the received input to generate the output.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non - Patent Documents

[0006]

Non - Patent Document 1

Non-Patent Document 2

Summary of the Invention

Means for Solving the Problems

[0007] This specification generally describes a system implemented as a computer program on one or more computers in one or more locations for training an agent neural network used to select actions to be performed by an agent that interacts with an environment. In particular, the system trains the agent neural network such that the agent neural network can be used to control the agent to perform any of a plurality of tasks. Each task can include, for example, one or more of navigating to a specified location in the environment, identifying a particular object in the environment, manipulating a particular object in a specified manner, and the like.

[0008] In one aspect, a computer-implemented method for training an agent neural network for use in controlling an agent to perform a plurality of tasks is described. The method includes maintaining population data that specifies a population of one or more candidate agent neural networks, the population data specifying, for each candidate agent neural network in the population, (i) respective parameter values of the parameters of the candidate agent neural network, and (ii) respective hyperparameter values for training the candidate agent neural network. The method involves training each candidate agent neural network with each respective set of one or more of a plurality of common tasks so as to update the parameter values of the parameters of the candidate agent neural network in the population data. In an implementation, the step of training involves, for each candidate agent neural network, obtaining data identifying a candidate task (e.g., one of the plurality of tasks) for training the candidate agent neural network, obtaining data specifying a control policy for the candidate task, and determining whether to train (or not to train) the candidate agent neural network on the candidate task based on (i) the performance of the candidate agent neural network in the candidate task, (ii) the performance of the control policy for the candidate task in the candidate task, and (iii) one or more of the hyperparameter values for the candidate agent neural network. In response to a determination to train the candidate agent neural network on the candidate task, the method includes training the candidate agent neural network on the candidate task so as to update the parameter values of the parameters of the agent neural network, for example using reinforcement learning. The training may be repeatedly performed each time data identifying a candidate task is obtained, i.e., each time one of the plurality of tasks is used as the candidate task.

[0009] In an implementation of such a method, the training tasks consumed by the agent are dynamically generated in response to the agent's performance. The effect is that the task allocation can be changed throughout the training, and the task allocation itself can be optimized to improve the agent's performance. More specifically, whether to train (or not train) the candidate agent neural network is determined based on a combination of three factors as listed above. For example, generally, if the performance is (significantly) better than the control policy, the agent should be trained only on tasks, and by doing so, the agent will perform significant actions, i.e., actions that affect the return. The performance of the agent can generally be judged, for example, from the return from the tasks. The control policy can be any action selection policy (for controlling the agent to perform candidate tasks), such as a uniform random action policy or the agent's past policy.

[0010] The trained agent neural network can be used to control the agent to perform multiple tasks in one or more environments. The environment may be a real-world environment, but part or all of the training may be within a simulation of the real-world environment. In some implementations, especially when part or all of the training is within a simulation, the "proposed" tasks for training the candidate agent neural network are automatically generated procedurally, i.e., based on task parameters, and these tasks can then be used to determine whether to train the candidate agent neural network.

[0011] The above method can be used to obtain a population comprising one or more candidate agent neural networks. When the population comprises two or more candidate agent neural networks, the population of the plurality of candidate agent neural networks is trained. Then, one or more can be selected as the trained agent neural networks. Among the population of the plurality of candidate agent neural networks, they may have the same architecture as each other, but do not have to, and generally, they have different respective parameter values, such as weights, and may have different respective hyperparameter values.

[0012] In some implementations, the population of candidate agent neural networks is used for population-based training. That is, the population of candidate agent neural networks is trained, and an adaptation measure is determined for each of the agents, for example, to compare two (or more) of the agents. This may be done, for example, by evaluating each of the agents using a fitness function, and there are many suitable adaptation measures, and generally, the fitness can be determined from the return from the task.

[0013] As described above, each candidate agent neural network is controlled by its respective hyperparameter values. The adaptation measure can be used to modify the population of candidate agent neural networks, particularly by adjusting the hyperparameter values for one or more of the candidate agent neural networks, with the aim of improving the performance of the population. The adjustment can be carried out in many ways. As an example, the hyperparameter values may be perturbed randomly.

[0014] As another example, if an adaptation measure for a first candidate agent neural network (across multiple tasks) is determined to dominate an adaptation measure of a second candidate agent neural network (across multiple tasks), the hyperparameter values for the second candidate agent neural network may be adjusted to be equal to a mutated (modified) version of the hyperparameter values for the first candidate agent neural network. Pareto dominance can be considered achieved if the adaptation measure is good at least across multiple tasks and better for at least one of the tasks. The tasks may be candidate tasks, or other tasks, such as verification tasks.

[0015] In some implementations, a less preferred candidate agent neural network, i.e., one with a smaller adaptation measure (e.g., when comparing two agents), can have its respective parameter values (e.g., weights), and optionally its hyperparameter values as well, set to the respective parameter values, and optionally hyperparameter values, of another candidate agent neural network, e.g., one with a larger adaptation measure (e.g., when comparing two agents).

[0016] As described above, the fitness or performance of an agent can generally be determined from the return from a task. However, when there are many candidate tasks, the return can vary widely. Thus, in some implementations, calculating an adaptation (or performance) measure for a task may include, for example, determining the k-th, e.g., 50th percentile score achieved by the agent in the task, thereby determining each normalized percentile metric, where the score is normalized (and thus divided) using a normalization constant that is the score achieved by a mixture or set of one or more agents (candidate agent neural networks) that achieve the best score in the task. Here, the score is based on the reward or return achieved by the agent in the task. This approach avoids the need to compare performance, e.g., to an optimal policy for a useful task, as it may not be straightforward to determine an optimal policy, etc.

[0017] The candidate agent neural network may be trained on the candidate task to optimize the predicted return from the task while being distilled from the candidate agent neural network that performs the best in the population, as determined, for example, at the end of the previous training generation that directly precedes the current training generation in the sequence. Such training during distillation may include training using a distillation loss that encourages the action selection output of the candidate agent neural network being trained towards the action selection output of the candidate agent neural network that performs the best, e.g., a loss based on the difference between these outputs.

[0018] Training can be for self-reward play purposes, i.e., the candidate agent neural network can be sequentially trained to achieve goal g and to achieve not(g) (where not(g) is the negation of the goal). This can encourage exploration and make the agent appear to be competing with itself. The goal can be represented as a set of options (disjunctions) for each set of predicates, for example, representing one or more requirements for each option. The goal can be represented in natural language, but it need not be.

[0019] Generally, an agent neural network (and each of the candidate agent neural networks) is configured for use in controlling an agent to perform multiple tasks in one or more environments. Thus, generally, an agent neural network (and each of the candidate agent neural networks) is configured to receive and process an observation record characterizing the current state of the environment and, in some implementations, to receive and process data identifying a (specific) task to be performed, such as goal data representing a goal to be satisfied, in order to perform a task in the environment (alternatively, for example, the task may be inferred from the environment).

[0020] Generally, an agent neural network (and each of the candidate agent neural networks) is configured to generate an action selection output that characterizes an action to be performed by an agent in response to an observation in order to perform one or more of the tasks. In an implementation, a trained agent neural network is used to select an action to be performed by an agent in a real-world environment, and the selected action relates to an action to be performed by the agent in the real-world environment (and the observation record relates to an observation record of the real-world environment). As an example, the agent may include a robotic or mechanical agent such as an autonomous or semi-autonomous vehicle, and other examples of agents will be listed later.

[0021] The subject matter described herein may be implemented in certain implementations to realize one or more of the following advantages.

[0022] The described techniques can implement an open-ended reinforcement learning training process, during which the training task assignment and training objectives are such that the agent controlled by one or more of the trained network instances is robust and generally achieves qualified performance across a vast space of different tasks and environments. To effectively train the network instances continuously so that multiple instances of the agent neural network rarely stop learning, they are dynamically adapted. The described techniques for training an agent neural network are universally applicable to any type of complex environment in which an agent may be deployed to perform any type of technically difficult task.

[0023] By using the techniques described, a generally capable agent neural network is thus provided that not only outperforms state-of-the-art but also human-level performance in a wide range of agent control tasks, is further generalizable, and can be readily adapted, for example, by fine-tuning to new tasks including novel tasks that are of course distinct from existing tasks for which the network may have been trained. In addition to achieving improved agent performance, by training a population of agent neural network instances over a very large number of tasks, e.g., an infinite multitask continuum, the training process also consumes fewer computational resources, e.g., memory and processing power, than conventional approaches that require training one neural network model from scratch for each new task or new environment.

[0024] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7

DETAILED DESCRIPTION OF THE INVENTION

[0026] Like reference numbers and names in the various drawings indicate like elements.

[0027] FIG. 1 shows an exemplary training system 100. The training system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations where the systems, components, and techniques described below can be implemented.

[0028] Training system 100 is a system that implements an open-ended reinforcement learning training process for training multiple instances of an agent neural network, each of which can be used to select actions to be performed by an agent, such as agent 102A, in environment 104 to control the agent to perform a wide range of machine learning tasks. In some cases, the machine learning task is a single agent task that the agent itself can perform, and in other cases, the machine learning task is a multi-agent task that requires the agent to interact with one or more other reinforcement learning agents, such as agents 102B - L, in environment 104, for example, to compete or cooperate. That is, training system 100 acquires (i.e., generates or receives) observation records, each of which characterizes a respective state of environment 104, and in response to each observation, selects an action from a predetermined set of actions to be performed by reinforcement learning agent 102A in response to the observation. In response to some or all of the actions performed by agent 102A, training system 100 acquires a reward. Each reward is a numerical value received from environment 104 as a result of agent 102A performing an action, and in particular, the reward varies depending on the state to which environment 104 transitions as a result of agent 102A performing the action. In some cases, the predetermined set of actions can define an individual action space, and in other cases, the predetermined set of actions can alternatively define a continuous action space, i.e., all of the action values in an individual action are selected from a continuous range of possible values, or, further alternatively, a hybrid action space can be defined, i.e., one or more of the action values in an individual action are selected from a continuous range of possible values.

[0029] In particular, by implementing the described open-ended reinforcement learning training process, it is possible to train a single but generally capable agent neural network that can be used to control an agent to achieve, or exceed, state-of-the-art and even human-level performance across thousands or millions of tasks in different environments.

[0030] In some implementations, the environment is a real-world environment and the agent is a machine agent that interacts with the real-world environment, such as a robot that operates within the environment or navigates through the environment, or an autonomous or semi-autonomous land, air, or sea vehicle, and the action is an action taken by the machine agent within the real-world environment to perform a task. For example, the agent may be a robot that interacts with the environment to perform a particular task, such as to locate an object of interest within the environment, or to move the object of interest to a designated location within the environment, or to navigate to a designated destination within the environment.

[0031] In these implementations, the observation record may include, for example, images, object position data, and sensor data, such as one or more of image, distance, or position sensors, or sensor data from an actuator, for capturing the observation record when the agent interacts with the environment. For example, in the case of a robot, the observation record may include one or more of data characterizing the current state of the robot, such as joint position, joint velocity, joint force, torque or acceleration, such as gravity compensation torque feedback, and the overall or relative pose of a member held by the robot. In the case of a robot or other mechanical agent or vehicle, the observation record may similarly include one or more of position, linear or angular velocity, force, torque or acceleration, and the overall or relative pose of one or more parts of the agent. The observation record may be defined in one, two or three dimensions and may be absolute and / or relative observation records. The observation record may include, for example, detected electronic signals, such as motor current or temperature signals, and / or, for example, image or video data from a camera or LIDAR sensor, such as data from the agent's sensors or from sensors placed separately from the agent in the environment.

[0032] In these implementations, an action may be a control signal or high-level control command for controlling torque about a joint of a robot or other machine agent, or for a self- or semi-autonomous land, air, or marine vehicle, such as a control surface or other control element of the vehicle, such as a steering control element, or torque or high-level control command thereto. The control signal may include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or components of another machine agent. The control signal may also or instead include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within the environment whose control affects the state of the observed environment. For example, in the case of a self- or semi-autonomous land, air, or marine vehicle, the control signal may define actions for controlling navigation, such as steering, and movement, such as braking and / or accelerating the vehicle.

[0033] In some implementations, the environment is a simulation of the above-described real-world environment, and the agent is implemented as one or more computers that interact with the simulated environment. For example, the simulated environment may be a simulation of a robot or vehicle, and the reinforcement learning system is trained in the simulation and then, once trained, may be used in the real world.

[0034] In some implementations, the environment is a real-world manufacturing environment for manufacturing products such as chemicals, biologicals, or mechanical products, or food. As used herein, "manufacturing" a product includes purifying starting materials to create the product or, for example, processing the starting materials to remove contaminants to produce a washed or recycled product. A manufacturing plant may include a plurality of manufacturing units such as containers for chemical or biological substances or machines for processing solids or other materials, such as robots. The manufacturing units are configured such that intermediate versions or components of the product are movable between the manufacturing units during the manufacture of the product, for example via pipes or mechanical conveyance. As used herein, the manufacture of a product also includes the manufacture of food by a kitchen robot.

[0035] An agent may include an electronic agent configured to control a manufacturing unit or a machine such as a robot that operates to manufacture a product. That is, the agent may comprise a control system configured to control the manufacture of chemicals, biologicals, or mechanical products. For example, the control system may be configured to control one or more of the manufacturing units or machines or to control the movement of intermediate versions or components of the product between manufacturing units or machines.

[0036] As an example, tasks performed by the agent may include tasks for manufacturing a product or an intermediate version or component thereof. As another example, tasks performed by the agent may include tasks for controlling the use of resources, such as minimizing the use of power, water, or any material or consumable used in the manufacturing process.

[0037] An action can include controlling the use of a machine or manufacturing unit to process a solid or liquid material to produce a product, or an intermediate or component thereof, or controlling the movement of an intermediate version of a product or a component within a manufacturing environment, e.g., between manufacturing units or machines. Generally, an action can be any action that is effective on the observed state of the environment, e.g., an action configured to adjust any of the detected parameters described below. These can include actions for adjusting the physical or chemical conditions of a manufacturing unit, or actions for controlling the movement of mechanical parts of a machine or joints of a robot. An action can include imposing operating conditions on a manufacturing unit or machine, or an action that adjusts, controls, or changes settings to turn on, off, or otherwise adjust the operation of a manufacturing unit or machine.

[0038] A reward or return can be related to a metric of the performance of a task. For example, in the case of the task of manufacturing a product, the metric can include the quantity of the product being manufactured, the quality of the product, a metric of the speed of production of the product, or, for the physical cost of performing the manufacturing task, e.g., a metric of the amount of energy, materials, or other resources used to perform the task. In the case of the task of controlling the use of a resource, the metric can include any metric of the use of the resource.

[0039] Generally, the observational record of the state of the environment can include any electronic signals representing the functionality of the electronic and / or mechanical components of the device. For example, the representation of the state of the environment may be derived from observations made by a sensor detecting the state of the manufacturing environment, for example, the sensor detecting the state or configuration of a manufacturing unit or machine, or the sensor detecting the movement of substances between manufacturing units or machines. As some examples, such sensors may be configured to detect mechanical movement or force, pressure, temperature, electric current, voltage, frequency, impedance and other electrical conditions, the amount, level, flow rate / movement rate or flow path / movement path of one or more materials, physical or chemical conditions, for example, physical state, shape or configuration or chemical state such as pH, the mechanical configuration of a unit or machine, etc., the configuration of a unit or machine, or a valve configuration, an image or video sensor for capturing an image or video observational record of a manufacturing unit or machine or movement, or any other suitable type of sensor. In the case of a machine such as a robot, the observational record from the sensor may include observational records of the position, linear or angular velocity, force, torque or acceleration, or posture of one or more parts of the machine, for example, data characterizing the current state of the machine or robot, or a member held or processed by the machine or robot. The observational record may include, for example, detected electronic signals such as motor current or temperature signals, or image or video data from, for example, a camera or LIDAR sensor. Such sensors may be part of an agent in the environment or may be located separately therefrom.

[0040] In some implementations, the environment is a service facility that includes multiple components of electronic devices, such as a server farm or data center for storing or processing data, e.g., a telecommunications data center or a computer data center, or the real-world environment of any service facility. The service facility may also include auxiliary control devices for controlling the operating environment of the components of the devices, such as environmental control devices for temperature control, e.g., cooling devices, or airflow control or air conditioning devices. The task may include tasks for controlling the use of resources, such as tasks for controlling power consumption or water consumption, e.g., minimizing them. The agent may include an electronic agent configured to control the operation of the components of the devices or to control the operation of auxiliary control devices, e.g., environmental ones.

[0041] Generally, an action may be any action that has an effect on the observed state of the environment, e.g., an action configured to adjust any of the detected parameters described below. These may include actions for controlling the components of the devices or the auxiliary control devices or for imposing operating conditions on them, e.g., actions that result in a change in the settings for adjusting, controlling, or turning on or off the operation of the components of the devices or the components of the auxiliary control devices.

[0042] Generally, the observed record of the state of the environment may include any electronic signal representing the functionality of the facility or the devices within the facility. For example, the representation of the state of the environment may be derived from observations made by any sensor for detecting the state of the physical environment of the facility or observations made by any sensor for detecting the state of one or more of the components of the devices or one or more of the components of the auxiliary control devices. These include sensors configured to detect electrical conditions such as current, voltage, power, or energy, the temperature of the facility, fluid flow, temperature, or pressure within the facility or within the cooling system of the facility, or the physical facility configuration such as whether a vent is open or not.

[0043] A reward or return may be related to a metric of task performance. For example, in the case of a task for controlling the use of a resource, such as minimizing the use of a resource, like a task for controlling the use of power or water, the metric may include any metric of resource use.

[0044] In some implementations, the environment is the real-world environment of a power generation facility, such as a renewable power generation facility like a solar power plant or a wind power plant. The task may include, for example, a control task for controlling the power generated by the facility to meet demand or to reduce the risk of a mismatch between elements of the power distribution network, such as controlling the supply of power to the power distribution network or maximizing the power generated by the facility. The agent may include an electronic agent configured to control the generation of power by the facility or the connection of the generated power to the power distribution network. The action may include, for example, an action for controlling the electrical or mechanical configuration of a generator, such as the configuration of a wind turbine, or the configuration of one or more solar panels or mirrors, or an action for controlling the electrical or mechanical configuration of a rotating electrical machine, such as controlling the conversion from energy input to electrical energy output, for example, controlling the efficiency of the conversion or the degree of coupling between the energy input and the electrical energy output. The electrical control action may include, for example, an action for controlling one or more of the voltage, current, frequency or phase of the generated power.

[0045] The reward or return may be related to a metric of the performance of the task. For example, in the case of a task for controlling the supply of power to a power distribution network, the metric may be related to a measure of the power transferred, or to a measure of the electrical mismatch between the power generation facility and the power distribution network, such as voltage, current, frequency or phase mismatch, or to a measure of the power or energy losses in the power generation facility. In the case of a task for maximizing the supply of power to the power distribution network, the metric may be related to a measure of the power or energy transferred to the power distribution network, or to a measure of the power or energy losses in the power generation facility.

[0046] Generally, the observational record of the state of the environment may include any electronic signals representing the electrical or mechanical functionality of the power generation equipment in the power generation facility. For example, the representation of the state of the environment may be derived from observations made by any sensors that detect the physical or electrical state of the equipment in the power generation facility that is generating power, or the physical environment of such equipment, or the condition of the auxiliary equipment supporting the power generation equipment. Such sensors may include, for example, sensors configured to detect observational records of the electrical conditions of the equipment, such as current, voltage, power or energy, the temperature or cooling of the physical environment, fluid flow, or the physical configuration of the equipment, and the electrical conditions of the power distribution network, from local or remote sensors. The observational record of the state of the environment may also include one or more predictions regarding future conditions of the operation of the power generation equipment, such as predictions of future wind levels or solar irradiance, or predictions of future electrical conditions of the power distribution network.

[0047] As another example, the environment may be a chemical synthesis or protein folding environment, and by so being, each state is a state of a protein chain or of each of one or more intermediates or chemical precursors, and the agent is a computer system for determining how to fold a protein chain or synthesize a chemical substance. In this example, the action is a possible folding action for folding the protein chain or an action for gathering chemical precursors / intermediates, and the result to be achieved may include, for example, folding the protein so that it is stable and achieves a particular biological function, or providing an effective synthesis route for a chemical substance. As another example, the agent may be a machine agent that performs or controls protein folding actions or chemical synthesis steps automatically selected by the system without human interaction. The observation record may include direct or indirect observation records of the state of the protein or chemical substance / intermediate / precursor and / or may be derived from a simulation.

[0048] Similarly, the environment may be a drug design environment such that each state is a state of a potential pharmaceutical chemical and the agent is a computer system for determining elements of the pharmaceutical chemical and / or a synthesis route for the pharmaceutical chemical. The drug / synthesis may be designed, for example in a simulation, based on a reward derived from the target of the drug. As another example, the agent may be a machine agent that performs or controls the synthesis of a drug.

[0049] In some further applications, the environment is the real-world environment and the agent manages, for example, the distribution of tasks across computing resources on a mobile device and / or in a data center. In these implementations, the action may include assigning a task to a particular computing resource.

[0050] As a further example, the action may include presenting an advertisement, the observational record may include an advertisement impression or click-through count or rate, and the reward may be characterized by previous selections of items or content made by one or more users.

[0051] In some cases, the observational record may include text or spoken commands provided to the agent by a third party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and the user of the autonomous vehicle may provide text or spoken commands to the agent (e.g., to navigate to a particular location).

[0052] As another example, the environment may be an electrical, mechanical, or electro-mechanical design environment, e.g., an environment in which the design of an electrical, mechanical, or electro-mechanical entity is simulated. The simulated environment may be a simulation of the real-world environment in which the entity is intended to operate. The task may be to design the entity. The observation record may include an observation record characterizing the entity, i.e., an observation record of the mechanical shape of the entity, or of the electrical, mechanical, or electro-mechanical configuration of the entity, or an observation record of the parameters or properties of the entity. The action may include an action to modify the entity, e.g., modify one or more of the observation records. The reward or return may include one or more metrics of the performance of the design of the entity. For example, the reward or return may be related to one or more physical characteristics of the entity, such as weight or strength, or to one or more electrical characteristics of the entity, such as a measure of the efficiency in performing a particular function for which the entity is designed. The design process may include outputting a design for manufacturing, e.g., in the form of computer-executable instructions for manufacturing the entity. The process may include making the entity according to the design. Thus, the design of the entity may be optimized, e.g., by reinforcement learning, and then the optimized design may be output as, e.g., computer-executable instructions for manufacturing the entity, and an entity having the optimized design may then be manufactured.

[0053] As described above, the environment may be a simulated environment. Generally, in the case of a simulated environment, the observation record may include one or more simulated versions of the aforementioned observation records or observation record types, and the action may include one or more simulated versions of the aforementioned actions or action types. For example, the simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle that navigates through the motion simulation. In these implementations, the action may be a control input for controlling the simulated user or the simulated vehicle. Generally, the agent may be implemented as one or more computers that interact with the simulated environment.

[0054] The simulated environment may be a simulation of a specific real-world environment and agents. For example, the system may be used to select actions in the simulated environment during training or evaluation of the system, and after training and / or evaluation is completed, it may be deployed to control real-world agents in the specific real-world environment that was the subject of the simulation. This can avoid unnecessary wear and tear on the real-world environment or real-world agents, and enable the control neural network to be trained and evaluated based on situations that rarely occur, are difficult to reproduce, or are unsafe in the real-world environment. For example, the system may be partially trained using a simulation of a machine agent in a simulation of a specific real-world environment and then deployed to control a real machine agent in the specific real-world environment. Thus, in such cases, the observation records of the simulated environment are related to the real-world environment, and the selected actions in the simulated environment are related to the actions to be performed by the machine agent in the real-world environment.

[0055] Optionally, in any of the above implementations, the observation record at any given time step may include data from previous time steps that may be useful in characterizing the environment, such as actions performed at the previous time step, rewards received at the previous time step, or both.

[0056] Generally, the agent neural network can have any suitable neural network architecture that enables it to process data that identifies its described functionality, such as the task to be performed and the observations for generating an action selection output that characterizes the actions to be taken by the agent in response to the observations. For example, the agent neural network can have any suitable number of layers (e.g., 5 layers, 10 layers, or 25 layers) of any suitable type (e.g., fully connected layers, convolutional layers, attention layers, transformer layers, etc.) connected in any suitable configuration (e.g., as a linear sequence of layers with or without residual connections).

[0057] In one example, the action selection output may include respective numerical probability values for each action in the set of possible actions that the agent can perform. The system can select the action to be taken by the agent, for example, by sampling actions according to the probability values of the actions or by selecting the action with the highest probability value.

[0058] In another example, the action selection output can directly define the action to be taken by the agent, for example, by defining the value of the torque to be applied to the joints of a robotic agent.

[0059] In another example, the action selection output may include respective Q-values for each action in the set of possible actions that the agent can perform. The system can process the Q-values (e.g., using a softmax function) to generate respective probability values for each possible action, which can be used to select the action (as previously described) to be performed by the agent. The system can also select the action with the highest Q-value as the action to be performed by the agent.

[0060] The Q-value for an action is an estimated value of the "return" to the current observation record that results from the agent performing the action and then selecting future actions that the agent 102 will perform according to the current values of the parameters of the agent neural network.

[0061] Return refers to the cumulative measure of the "reward" that the agent receives, e.g., the time-discounted sum of the rewards. The agent can receive respective rewards at each time step, and the rewards are specified by scalar numerical values, e.g., characterizing the progress of the task assigned to the agent until completion.

[0062] In some cases, the system can select the action to be performed by the agent according to an exploration policy. For example, the exploration policy may be an ε-greedy exploration policy, and the system selects the action to be performed by the agent according to the action selection output with a probability of 1 - ε, and randomly selects an action with a probability of ε. In this example, ε is a scalar value between 0 and 1.

[0063] One specific example of the architecture for the agent neural network is further described with reference to FIG. 4.

[0064] The training system 100 maintains a population repository 140 that stores population data specifying a population of one or more candidate agent neural networks 142A - M. The population repository 140 is implemented as one or more logical storage devices at one or more physical locations, or as logical storage spaces allocated to one or more storage devices at one or more physical locations. At any time during training, the population repository 140 stores data specifying the current population of candidate agent neural networks 142A - M.

[0065] Specifically, the population repository 140 stores, for each candidate agent neural network 142A - M in the current population, a set of maintenance values that define the respective candidate agent neural network. The set of maintenance values includes network parameters, hyperparameters, and in an implementation, is also convenient for maintaining performance measures (or simply "performance") for each candidate agent neural network 142A - M in different tasks. For example, in the case of candidate agent neural network A 142A, the set of maintenance values includes network parameter A 144A, hyperparameter A 146A, and performance A 148A (which may include respective performances for different tasks). If there are multiple candidate agent neural networks in the population, each candidate agent neural network will generally have the same architecture, but different respective parameter values, and in some cases, different respective hyperparameter values from other candidate agent neural networks in the population.

[0066] Hyperparameters for a candidate agent neural network are values that affect how network parameters, such as weights, are updated by training. Hyperparameters can include discount factors, learning rates, objective function values, or weights, etc., which are assigned to various terms of the objective function. Further, hyperparameters can include one or more specified values for use in determining whether to train a candidate agent neural network on a given candidate task. Thus, generally, in the described system, hyperparameters control the learning of parameters (which can be done, for example, by backpropagation of the gradients of one or more objective functions).

[0067] To train the agent neural networks, the training system 100 also maintains a training data repository 130 that stores training data for training candidate agent neural networks 142A - M. The training data repository 130 is implemented as one or more logical storage devices at one or more physical locations, or as logical storage space allocated to one or more storage devices at one or more physical locations. The training data repository 130 stores data defining a set of candidate tasks 132A - N on which the candidate agent neural networks can be trained, and corresponding control strategies 133A - N that can be used to control an agent to perform the candidate tasks, e.g., control strategy 133A for task A 132A. Optionally, the training data repository 130 stores data defining a set of validation tasks for evaluating the performance of the candidate agent neural networks 142A - M in a validation task.

[0068] Training system 100 can receive data in any of a variety of ways. For example, system 100 can receive data defining a set of candidate tasks and / or verification tasks as an upload from a remote user of the system via a data communication network (e.g., using an application programming interface (API) made available by training system 100). As another example, training system 100 can receive an input from a user specifying which of any data already maintained by training system 100 should be used as data defining a set of candidate tasks and / or verification tasks.

[0069] In some implementations, the training data repository 130 remains fixed throughout the training process, while in other implementations, the training data repository 130 expands (e.g., to infinity) or contracts as the training process is iterated, e.g., new candidate tasks can be dynamically added to the repository and existing tasks for which all of a predetermined number of candidate agent neural networks have achieved a threshold performance can be discarded. In these other implementations, the training system 100 can use the task generation engine 124 to generate new tasks from the space of candidate tasks, e.g., to generate a continuum of multi-tasks, repeatedly, i.e., in each of a plurality of training iterations over the course of training. This space of candidate tasks can be parameterized by a set of task parameters each associated with one or more values, e.g., scores, individual values, or continuous values. For example, the task generation engine 124 can generate tasks randomly, e.g., by randomly selecting different task parameter values from a set of task parameters. Next, to actually generate the tasks according to these selected task parameter values, the training system 100 can use any of the exemplary techniques described in Shaker, Noor, et al., "Procedural content generation in games. Switzerland: Springer International Publishing", 2016, in addition to or instead of other known digital content generation techniques.

[0070] Generally, a set of task parameters can include any of a variety of adjustable parameters that collectively define the space of candidate tasks. For example, a set of task parameters can include a first plurality of environmental parameters that define different properties or characteristics of the environment with which the agents interact. More specifically, the environmental parameters can include the static topology of the environment (e.g., the layout and structure of the topological building blocks of the environment), the lighting of the environment, the type or format of the observational records that characterize the state of the environment that can be provided to the agents, the properties of the dynamic objects present in the environment (e.g., the location and physical properties of target objects or obstacles), a predetermined set of actions that the agents can perform in response to receiving the observational records, the number of other agents present in the environment (in the case of competitive or cooperative tasks), or combinations thereof, and in some cases can define or otherwise specify more or others.

[0071] As another example, a set of task parameters can include a second plurality of goal parameters that define or otherwise specify different goals for each of one or more agents to achieve when interacting with the environment. The goal parameters can include, for example, various terms of a reward function that are used to calculate a reward (represented as a numerical value) to be issued to an agent in response to the agent performing different actions when interacting with the environment. The reward generally reflects the progress of the agent towards achieving the goal specified for the candidate task that the agent is configured to perform.

[0072] The training system 100 can use a population-based training (PBT) engine 120 to train a population of one or more candidate agent neural networks 142A - M using the population-based training techniques described in more detail by PCT patent application No. PCT / EP2018 / 082162, which is incorporated herein by reference. As part of the training, the PBT engine 120 trains each candidate agent neural network in each respective set of one or more tasks to update the parameter values of the parameters of the candidate agent neural networks in the population repository 140. However, unlike conventional population-based training, before training a given candidate agent neural network on any given candidate task, the PBT engine 120 obtains data identifying the candidate tasks for training the candidate agent neural network and obtains data specifying the control policies for the candidate tasks.

[0073] In particular, the PBT engine 120 determines whether to train the candidate agent neural network on the candidate task based on (i) the performance of the candidate agent neural network in the candidate task (i.e., when used according to the parameter values currently stored in the population data for the candidate agent neural network), (ii) the performance of the control policy for the candidate task in the candidate task, and (iii) one or more of the hyperparameter values for the candidate agent neural network. Generally, given the current learning stage, i.e., given the current values of the candidate parameters stored in the population data, if the PBT engine determines that the task is "useful" for the candidate, the PBT engine 120 simply trains the candidate agent neural network on the given task. The PBT engine 120 can make this determination based in part on the performance of the control policy relative to the performance of the candidate.

[0074] In response to a determination to train a candidate agent neural network on a candidate task, the PBT engine 120 trains the candidate agent neural network on the candidate task to update the parameter values of the parameters of the agent neural network. In response to a determination not to train the candidate agent neural network on the candidate task, the system refrains from training the candidate agent neural network on the candidate task.

[0075] By using control policies, in this way, for example, the PBT engine 120 is made to train the candidate agent neural network on the candidate task only if the performance of the candidate agent neural network is significantly better than the performance of the control policy, thereby improving the effectiveness and efficiency of the training process. In some implementations, for each candidate task, such as task A 132A, the corresponding control policy, such as control policy 133A, may be a fixed policy, such as a random action policy that randomly selects actions, and in other implementations, the corresponding control policy may be a policy that is controlled by candidates from a previous training generation, i.e., according to the historical network parameter values of candidates from an early point in time during training. In these other implementations, by using control policies, the PBT engine 120 can further determine whether the performance of the candidate agent neural network has recently improved or deteriorated with respect to previous training iterations, and accordingly, can take appropriate measures, such as adjusting hyperparameters or simply removing from the current population.

[0076] In some implementations, the PBT engine 120 keeps the hyperparameters for candidates fixed during training. In some other implementations, the PBT engine 120 adjusts the hyperparameters for candidates during training so that the dynamic task selection described above always selects only those tasks that are useful for learning when the candidate parameter values change over the course of training.

[0077] Some or all of the tasks for training candidates may be multi-agent tasks, i.e., tasks that require an agent controlled using a candidate to interact with one or more other agents controlled using a different policy. For these tasks, the system may use any of a variety of policies to control one or more other agents. For example, the system may use a fixed policy, such as a random action policy that randomly selects actions, a no-op action policy where other agents do not perform any action, an expert policy that represents the behavior of an expert agent, or a policy controlled by high-performance candidates from a previous training generation.

[0078] After training, the training system 100 may select one of the candidates in the population for use as an agent neural network, i.e., for use in controlling the agent to perform new tasks. For example, the training system 100 may select the highest-performance candidate after training is complete. Alternatively, the training system 100 may use an ensemble of multiple candidates in the population as the final agent neural network.

[0079] Figure 2 is a flowchart of an exemplary process 200 for training a population of candidate agent neural networks. For convenience, process 200 is described as being implemented by a system comprising one or more computers located at one or more locations. For example, a population-based neural network training system appropriately programmed in accordance with this specification, such as training system 100 of FIG. 1, can implement process 200.

[0080] As described above, the system maintains population data that specifies a population of one or more candidate agent neural networks. The population data specifies, for each candidate agent neural network in the population, (i) respective parameter values of the parameters of the candidate agent neural network, (ii) respective hyperparameter values for training the candidate agent neural network, and (iii) performance for the candidate agent neural network (which may include respective performance for different tasks).

[0081] The system repeatedly implements process 200 for each candidate neural network in the population. In some implementations, the system repeatedly implements process 200 in parallel for each candidate neural network and asynchronously with respect to implementing the process for each other candidate neural network in the population.

[0082] The system obtains data identifying candidate tasks for training the candidate agent neural network (step 202). Generally, the system obtains different candidate tasks in different iterations. Different tasks can involve different environments in which agents controlled by the candidate agent neural network interact, different goals to be achieved when the agents interact with the environment, different numbers of other agents present in the environment (in the case of competitive or cooperative tasks), and so on.

[0083] In some implementations, the candidate tasks may be new tasks automatically and dynamically (i.e., over the course of training) generated by the system according to different sets of task parameters sampled from the candidate task space. In other implementations, the candidate tasks may be obtained in each iteration by sampling a fixed number of candidate tasks from a set of candidate tasks included in the training data repository. In some of these implementations, the system can use the same task sampling strategy for all candidate agent neural networks in the population, and in other ones of these implementations, the system can use different candidate-specific task sampling strategies for the population. Further, in implementations where the system maintains data specifying a list of previously trained historical candidate tasks for a candidate agent neural network, the system can update the list to include the obtained candidate tasks.

[0084] The system obtains data specifying a control policy for a candidate task (step 204). Generally, the control policy can be any action selection policy that can be used to control an agent (which may include other agents in the environment) to interact with the environment. For example, the control policy for a candidate task is a uniform random action policy that performs an action uniformly randomly selected from a predetermined set of actions when controlling the agent to perform the candidate task. As another example, the control policy for a candidate task is a single-task policy that uses a single candidate agent neural network trained only on the candidate task when controlling the agent to perform the candidate task. As another example, the control policy for a candidate task is a policy that uses an instance of the candidate agent neural network but uses historical parameter values for the network parameters of the candidate agent neural network from an early point during the training of the candidate agent neural network when controlling the agent to perform the candidate task.

[0085] The system determines whether to train the candidate agent neural network on the candidate task based on (i) the performance of the candidate agent neural network in the candidate task, (ii) the performance of the control policy in the candidate task, and (iii) one or more of the hyperparameter values for the candidate agent neural network (step 206).

[0086] Specifically, to make this determination, the system can use a planning algorithm, such as a Monte Carlo Tree Search (MCTS) algorithm or another look-ahead planning algorithm, to predict multiple future states after the initial state of the environment at a given time step. This is referred to as a task episode. A task episode represents a rollout of the environment at a time after a given time step, assuming that the agent performs a specific action selected by using a candidate agent neural network. The system can run the planning algorithm to repeatedly generate multiple task episodes for a candidate task, for example, 5, 10, 20, or more task episodes.

[0087] A task episode refers to a sequence of time steps during which the agent interacts with the environment. A task episode can end, for example, when the agent has interacted with the environment for a predefined number of time steps or when the agent has completed the task.

[0088] In some implementations, each task episode can include a sequence of multiple trajectories, where each trajectory can include each current observation characterizing the respective current state of the environment, each current action performed by the agent in response to the current observation, each next state characterizing the respective next state of the environment, and the reward received by the agent in response to performing the current action.

[0089] For each of the multiple task episodes of the candidate task, the system determines each candidate agent return received by controlling the agent to execute the task episode of the candidate task using a candidate agent neural network. The system also determines each control policy return received by controlling the agent to execute the task episode of the candidate task using a control policy. Both the candidate agent return and the control policy return may be task-specific returns calculated according to a reward function defined as a cumulative measure of the reward received by the agent in response to performing one or more actions when interacting with the environment, for example, according to the selected task parameters.

[0090] In some implementations, the system determines to proceed with training the candidate agent neural network on the candidate task only when the performance of the candidate agent neural network, measured by each candidate agent return, does not exceed a performance threshold level defined by hyperparameter values for the candidate agent neural network.

[0091] In these implementations, the system may determine to train the candidate agent neural network on the candidate task only when the proportion of task episodes for which each candidate agent return exceeds a first specified value is less than a second specified value. For example, the first specified value may be an integer or floating-point value within the possible range of the total reward that the agent can receive, and the second specified value may be a decimal value between zero and one, where the first specified value, the second specified value, or both are hyperparameter values for the candidate agent neural network.

[0092] An example of this criterion in mathematical notation is Pr[R π (x)>m s <m solved and may be, in the formula, R π (x) are respective candidate agent returns, and m s and m solved are the first and second specified values, respectively.

[0093] In some implementations, the system determines to train the candidate agent neural network on a candidate task only when the performance of the candidate agent neural network, measured by each candidate agent return, exceeds the performance of the control policy, measured by each control policy return, by more than a threshold level of performance defined by hyperparameter values for the candidate agent neural network.

[0094] In these implementations, the system may determine to train the candidate agent neural network on a candidate task only when the proportion of task episodes in which each candidate agent return exceeds each control policy return by at least a third specified value is greater than a fourth specified value. For example, the third specified value may be an integer or floating-point value within the possible range of the total reward that the agent can receive, and the fourth specified value may be a decimal value between zero and one, where the third specified value, the fourth specified value, or both are hyperparameter values for the candidate agent neural network. The third and fourth specified values may be the same as or different from the first and second specified values, respectively.

[0095] An example of this criterion in mathematical expression is

[0096]

Number

[0097] and may be, in the formula, R π (x) are respective candidate agent returns, and R πcont (x) are respective control policy returns, and m> and m >cont are the third and fourth specified values, respectively.

[0098] In some implementations, the system determines to train the candidate agent neural network on a candidate task only when the performance of the control policy measured by each control policy return is lower than a threshold level of performance defined by hyperparameter values for the candidate agent neural network. That is, the system determines to train the candidate agent neural network on a candidate task that requires a control policy that is, for example, more sophisticated or advanced than a uniform random action policy, in order to achieve the specified goal of the candidate task.

[0099] In these implementations, the system may determine to train the candidate agent neural network on a candidate task only when the average (or median, or other representative value) of each control policy return is less than a fifth specified value. For example, the fifth specified value may be an integer or floating point value within the possible range of total rewards that the agent can receive, where the fifth specified value is a hyperparameter value for the candidate agent neural network. The fifth specified value may be the same as or different from the first or third specified values.

[0100] An example of this criterion in mathematical notation is

[0101] [Number]

[0102] and may be, where V πcont (x) is the expected return of each control policy, and m cont is the fifth specified value.

[0103] In any of the above-described implementations, in response to a determination to train the candidate agent neural network on a candidate task, the system uses any suitable reinforcement learning technique to update the maintained values of the network parameters of the agent neural network, i.e., by optimizing an objective function determined by the reward received by the agent controlled using the candidate agent neural network from the environment, to train the candidate agent neural network on the candidate task (step 208). For example, the reinforcement learning technique may be an on-policy RL training technique, such as one of the RL algorithms described in more detail in Song et al., "V-MPO: On-policy maximum a posteriori policy optimization for discrete and continuous control," International Conference on Learning Representations, 2019, and the system may train the candidate agent neural network on the candidate task for a certain number of iterations or for a set time period.

[0104] Applied in parallel with reinforcement learning techniques is the population-based training technique. Specifically, the system uses the maintenance values for the hyperparameters and network parameters of the candidate agent neural network to train the candidate neural network on each set of one or more candidate tasks, iteratively generating updated network parameters for the candidate agent neural network until the PBT termination criterion is satisfied. The PBT termination criterion is one or more set conditions, and when satisfied by the candidate agent neural network, the system causes the repository for the candidate agent neural network to be updated with the new network parameters, new hyperparameters, and new performance metrics. Examples of when the PBT termination criterion is satisfied are when the candidate agent neural network has been trained for a certain number of iterations (e.g., 1e4, 1e6, 1e8, etc.) or a set time period (e.g., 1 hour, 2 hours, 10 hours, etc.) of the iterative training process on each set of one or more candidate tasks. Another example of when the PBT termination criterion is satisfied is when the candidate agent neural network falls below a certain performance threshold.

[0105] When the PBT termination criterion is satisfied, the system performs a population repository update process that includes determining the updated performance for the candidate agent neural network according to the updated values of the network parameters for the candidate agent neural network. The updated performance reflects the potential performance increase of the candidate agent neural network as a result of the updated network parameters.

[0106] The system also determines new values for hyperparameters and network parameters for the candidate neural network. In some implementations, the system determines new values for hyperparameters (i.e., adjusts the maintained hyperparameter values for the candidate agent neural network) and network parameters for the candidate agent neural network based at least on the maintained performance for the population of candidate agent neural networks in the population repository and the updated performance of the candidate agent neural network.

[0107] After determining new network parameters, new hyperparameters, and new performance for the candidate agent neural network, the iterative training process continues. That is, the system uses the new hyperparameters and new network parameters of the candidate neural network to train the candidate agent neural network through reinforcement learning on each set of one or more newly acquired candidate tasks to iteratively generate updated network parameters for the candidate agent neural network. The system continues the iterative training process for the candidate agent neural network until the following PBT termination criterion is satisfied (and the system also repeats the population repository update process for the candidate agent neural network). In some implementations, the system may end the iterative training process for the candidate agent neural network, for example, when the PBT termination criterion is satisfied a predetermined number of times or when a performance criterion is satisfied that instructs the system to stop training. In other implementations, the system may continue the iterative training process indefinitely.

[0108] Figure 3 shows an illustration of training a population of candidate agent neural networks. A training system, e.g., the training system 100 of FIG. 1, executes an iterative training process for candidate agent neural networks A - G 142A - G in the population (the "population 1") in parallel and until a PBT termination criterion is satisfied. When the training system executes the iterative training process for candidate agent neural networks A - G 142A - G, the network parameters A - G for the candidate agent neural networks are updated accordingly.

[0109] The system updates the performance of each candidate agent neural network based on an evaluation of the performance of the candidate agent neural network in each of a plurality of validation tasks. The plurality of validation tasks can be obtained by sampling from a set of validation tasks. The number of tasks in the validation set is generally much smaller than the total number of tasks in the set of candidate tasks. For example, the training data repository can store a set of millions, five million, or more candidate tasks, while the number of validation tasks is on the order of thousands. In some implementations, the sampling can be random sampling, while in other implementations, the sampling can alternatively be gradient sampling to ensure uniform coverage of the set of validation tasks. The updated performance for each validation task can be determined by a task - specific return calculated according to a reward function defined by the selected task parameters, e.g., as a cumulative measure of the reward received by the agent in response to performing one or more actions selected using the candidate agent neural network when interacting with the environment.

[0110] Unlike conventional population-based training where performance is directly used to update hyperparameters and network parameters for each candidate agent neural network, the evaluation task can vary depending on the complexity of the task, the scale of the return, or both. Thus, the system further calculates a respective PBT adaptation measure (or, abbreviated, "adaptation measure") 310 for each candidate agent neural network, which is then used to update the hyperparameters and network parameters for the candidate agent neural network. The adaptation measure 310 enables better characterization of network performance and robustness across a set of evaluation tasks.

[0111] Specifically, each adaptation measure, which can be regarded as a multi-dimensional measure of the performance of each candidate agent neural network across multiple validation tasks, is calculated by determining a respective normalized percentile metric for the candidate agent neural network in each of the multiple validation tasks.

[0112] Figure 7 shows an illustration for determining the normalized percentile metric. As shown, for each candidate agent neural network, to determine the normalized percentile metric for the population of candidate agent neural networks for each of the validation tasks 711, 712, 713, 714, and 715, the system uses the best performance of the candidate in the validation task (e.g., the best performance 701B for task 711) as the normalization constant, and then uses the normalization constant to normalize the performance of the candidate agent neural network (e.g., the candidate agent neural network performance 701A for task 711). For each candidate agent neural network, the normalized performance of the candidate across multiple validation tasks (e.g., the normalized candidate agent neural network performance 724C for task 714) is then sorted, for example, in monotonically increasing order (as shown by the normalized percentile curve shown in Figure 7), from which the respective normalized percentile metric for the candidate in each validation task can be determined.

[0113] In the example of Figure 7, the best performance of the candidate in any given validation task may be the best performance achieved by the candidate agent neural network, measured by each candidate agent return received by the candidate in the validation task (in the case of a single-agent task), or alternatively, the best performance (Nash equilibrium) achieved by one of the groups of multiple candidate agent neural networks, measured by each candidate agent return received by the candidate in the validation task (in the case of a multi-agent task).

[0114] The system then applies group-based training techniques to each fitness measure to adjust the respective hyperparameter values for each of one or more candidate agent neural networks 142A - G in the group. For example, the respective fitness measures of the candidate agent neural networks are compared to each other, and in response to a determination that the fitness measure for candidate agent neural network 142D is better than the respective fitness measures for another candidate agent neural network in the group, e.g., candidate agent neural network 142E, the system identifies candidate agent neural network 142D as a candidate for "better" performance within the current group. For example, the comparison results can be based on Pareto dominance. As an example here, if the fitness measures for a plurality of tasks, e.g., fitness measures for a plurality of validation tasks, e.g., normalized percentile metrics, are all at least as good as those for another candidate for the plurality of tasks, e.g., the plurality of validation tasks, but are strictly better in at least one of the tasks, e.g., the fitness measure, e.g., the normalized percentile metric, for a validation task, then the performance of the candidate can be considered to Pareto dominate the performance of another candidate.

[0115] In this example, the system can then "exploit" the network hyperparameters for candidate agent neural network 142D, i.e., by adjusting the hyperparameter values for other candidate agent neural networks to be equal to the hyperparameter values for candidate agent neural network 142D, new values for the network hyperparameters for other candidate agent neural networks can be determined. In this example, the system can alternatively "explore" the network hyperparameters for candidate agent neural network 142D, i.e., by adjusting the hyperparameter values for other candidate agent neural networks to be equal to a mutated (e.g., randomly perturbed) version of the hyperparameter values for candidate agent neural network 142D, new values for the network hyperparameters for other candidate agent neural networks can be determined.

[0116] The system similarly applies population-based training techniques to each fitness measure to adjust the respective parameter values for each candidate agent neural network 142A - G in the population. Continuing the above example where candidate agent neural network 142D is identified as a candidate performing "better" in the current population, the system can set the respective parameter values for another candidate agent neural network, e.g., candidate agent neural network 142E, to be equal to the respective parameter values (or a mutated version of the respective parameter values) for candidate agent neural network 142D.

[0117] As shown in FIG. 3, in some implementations, the system further incorporates generational training techniques into the population-based training technique to further improve the effectiveness and speed of RL training. In these implementations, the training of the candidate agent neural network spans a sequence of training generations, where each candidate agent neural network trained in each training generation can be considered together as one particular population. For example, each training generation can include a fixed number of iterations (e.g., 1e8, 5e8, 10e8, etc.) of an iterative training process or a set time period (e.g., 12 hours, 24 hours, 48 hours, etc.).

[0118] FIG. 3 shows an exemplary implementation where each population of seven candidate agent neural networks is trained in each training generation during a sequence of four training generations, although there may be more or fewer training generations and more, and sometimes many times more, candidate agent neural networks being trained for more or fewer training generations.

[0119] Generational group-based training enables the use of policy distillation techniques during training, which generally allows a candidate agent neural network in the current training generation to bootstrap its behavior from another candidate in the immediately preceding training generation. Implementing the policy distillation technique involves distilling the candidate agent neural network from the best-performing candidate agent neural network while training the candidate agent neural network to optimize the expected return during the current training generation of the sequence of training generations during the training of the population of candidate agent neural networks, for example, from the best performance candidate at the end of the preceding training generation that directly precedes the current training generation in the sequence, in the same or a different candidate task in which a candidate agent neural network that performs "better" from a preceding training generation (e.g., from the perspective of task-specific performance or fitness measure) has already been trained.

[0120] Specifically, the system can incorporate a preliminary policy distillation loss term

[0121]

Number

[0122] into the objective function used in the RL training of the candidate agent neural network, where

[0123]

Number

[0124] and in the formula, π teacherAnd π are, respectively, the action selection outputs of the "better" performance candidate agent neural network and the "learning" candidate agent neural network. In the above exemplary equation, the policy loss term is masked over the environmental state and the reward is obtained, i.e., the preliminary loss (in terms of the Kullback-Leibler information measure) only affects the RL training at specific time steps in various training task episodes where the reward cannot be received.

[0125] Furthermore, in these implementations, the candidate agent neural network that performs "better" performance from the previous training generation (e.g., from the perspective of task-specific performance or adaptation measure) can be used to control other agents in the multi-agent candidate task so as to increase the diversity of the RL training.

[0126] In some implementations, at least some of the candidate agent neural networks in each of one or more of the training generations in a sequence can be trained for self-reward play purposes. This can be used to assist RL training by making it so that the candidate agent neural networks cannot achieve only the corresponding goals in a smaller number of tasks. Specifically, the self-reward play purpose rewards the candidate agent neural network for satisfying goal g, and after success, the candidate is rewarded for satisfying not(g) in the same environment, and this flip in the goal is repeated after each satisfaction. For example, assume that goal g is to move an object towards a target location, and then not(g) is to move the object away from the target location. This can be considered as two agents playing in competition with themselves, where one agent has to satisfy g and the other agent has to satisfy not(g), but the agents act sequentially and are controlled by the same candidate agent neural network.

[0127] FIG. 4A shows an exemplary reinforcement learning agent control system 400. The reinforcement learning agent control system 400 is an example of a system implemented as a computer program on one or more computers at one or more locations where the systems, components, and techniques described below can be implemented.

[0128] The reinforcement learning system 400 includes an agent neural network 420 obtained as a result of the open-ended reinforcement learning training process described herein.

[0129] The reinforcement learning agent control system 400 uses an agent neural network 420 to select an action 410 to be performed by an agent 408, and then controls the agent 408 that interacts with the environment 402 by causing the agent 408 to perform the selected action 410.

[0130] The performance of the selected action 410 by the agent 408 generally causes the environment 402 to transition to a new state. By repeatedly causing the agent 408 to act in the environment 402, the system 400 can control the agent 408 to complete each of a plurality of specified tasks.

[0131] At each of a plurality of time steps, the agent neural network 420 is configured to process an input including (i) a current observation record 404 characterizing the current state of the environment 402 and (ii) goal data 406 representing a goal to be satisfied in order to perform a task in the environment, according to the trained values of the network parameters to generate an action selection output. The goal may be represented as a set of options for each set of predicates, each predicate may map the current state of the environment to a corresponding reward value, and each option is, for example, a conjunction of one or more predicates such that a reward is received only if all predicates return a non-zero reward. In some implementations, the goal data 406 can be represented as one or more tensors of numerical values, for example, where each predicate is represented as a multi-hot encoded vector.

[0132] Specifically, in some implementations, the current observation record 404 characterizing the current state of the environment 402 can include information defining the positions, orientations, velocities, etc. of different entities (e.g., target objects, obstacles, and other agents) present in the environment. In these implementations, a set φ of atomic predicates in the form of physical relationships regarding some or all of the entities j may be defined, and the goal is a set φ of atomic predicatesj It can be defined as a Boolean expression for. These physical relationships may include, for example, being close, above, seeing, and holding, as well as their negations, and the entities are static topological building blocks such as target objects, obstacles, agents, or the ground of the environment. An exemplary predicate can thus be "close to (the purple ball, the opponent)", which represents the goal of approaching one of the opposing agents to the purple ball in the environment. With the set of possible predicates fixed, the goal of the agent can be represented by a set of options (disjunctions) for a set of relevant (or necessary) predicates for this option (conjunction). In other words, the goal is a mapping φ j : S → {0, 1} d which may be, which assigns a binary vector of d predicate truth values starting from each current state, where 0 indicates a predicate not related to the goal when the environment is in the state characterized by the observation record S, and 1 indicates a predicate related to the goal when the environment is in the state characterized by the observation record S. For example, the goal represented by the goal data 406 may be

[0133]

Number

[0134] as follows, which, for some exemplary predicates, means "while being close to the yellow ball (φ j2 ), hold the purple ball (φ j1 ), or while looking at an opposing agent (φ j4 ) that is not holding the yellow ball (φ j3 ), be close to the yellow ball (φ j2 )".

[0135] System 400 then uses the action selection output to control the agent, i.e., selects the action 410 to be performed by the agent at the current time step according to the action selection output, and then causes the agent to perform the action 410, for example, by directly sending a control signal to the agent or by sending data identifying the action 410 to a control system for the agent. Exemplary action selection outputs and how they can be used to select actions are described above with reference to FIG. 1.

[0136] The agent neural network 420 is implemented with a neural network architecture that enables the network to perform its described functions. As shown in FIG. 4A, the agent neural network 420 includes a state encoder neural network 430, an embedding neural network 440, an attention neural network 445, a value neural network (value neural network "V" head) 450, and a policy neural network (policy neural network "π" head) 460. That is, the value neural network may also be referred to as the value neural network head, and similarly, the policy neural network may be referred to as the policy neural network head. The term "head" may indicate that both the value neural network and the policy neural network receive input shared data generated by one or more layers of the attention neural network 445, but need not indicate so. Each of the neural networks 420, 430, 440, 445, and 450 includes a different subset of the plurality of neural network layers within the agent neural network 420.

[0137] The state encoder neural network 430 may include a stack of multiple convolutional layers, followed by one or more pooling layers (e.g., max pooling layer), and then one or more recurrent layers (e.g., long short-term memory (LSTM) layer). The state encoder neural network 430 is configured to, i.e., through training, receive the observation record 404 and update the hidden state of the state encoder neural network 430 by processing the received observation record 404, i.e., generate the current hidden state 432 representing the current state of the environment. Usually, embedding is an ordered set of numbers or other values with a fixed number of dimensions. The fixed number of dimensions may depend on the actual number of LSTM neurons included in each LSTM layer of the state encoder neural network 430, e.g., 128, 256, etc.

[0138] The embedding neural network 440 may be a fully connected neural network, i.e., including a plurality of fully connected layers, followed optionally by an activation layer (e.g., ReLU activation layer), and the activation layer is configured to process the goal data 406 and the current hidden state 432 to generate (i) a predicate prediction, (ii) a goal embedding of the goal, and (iii) a respective option embedding for each of the options represented by the goal data. The embedding neural network 440 may then include a predicate predictor neural network, a goal embedding neural network, and an option embedding neural network, each of which includes a respective subset of the plurality of fully connected layers of the embedding neural network 440.

[0139] The predicate predictor neural network is configured to process the goal data 406 and the current hidden state 432 to generate a predicate prediction. In some implementations, the predicate prediction is a set φ of atomic predicates assuming that the environment is in a state characterized by the current observation jA multi-label binary classification prediction that specifies which predicates from [[ID=]] are relevant to (e.g., contained within) the goal represented in the goal data 406 may be used.

[0140] The option embedding neural network is configured to process the goal data 406 and the current hidden state 432, data derived from the goal data 406 and the current hidden state 432, or both, to generate respective option embeddings for each of the options. Similarly, the goal embedding neural network is configured to process the goal data 406 and the current hidden state 432, data derived from the goal data 406 and the current hidden state 432, or both, to generate a goal embedding for the goal. Typically, an embedding is an ordered set of numbers or other values with a fixed number of dimensions. The fixed number of dimensions may depend on the actual number of hidden units contained within each fully connected layer of the embedding neural network 440, e.g., 128, 256, etc.

[0141] The attention neural network 445 is configured to process an input that includes (i) the current hidden state, (ii) the goal embedding, and (iii) a predicate prediction for generating a goal attention hidden state. As used herein, an attention neural network is a neural network that includes one or more attention layers, where each attention layer is a neural network layer that includes an attention mechanism, e.g., a scaled dot product attention mechanism. To generate the goal attention hidden state, the attention mechanism maps a query, a set of key-value pairs, and an output (the goal attention hidden state), where the query can be from (or otherwise derived from) the goal embedding, and the set of key-value pairs can be derived from the current hidden state 432 (e.g., a linear projection and / or reformatted version of the current hidden state).

[0142] Furthermore, the attention neural network 445 is configured to process an input including, for each option, (i) the current hidden state, (ii) each option embedding for the option, and (iii) the predicate prediction, to generate a respective option attention hidden state for the option. To generate a respective option attention hidden state for each option, the attention mechanism similarly maps a query, and a set of key-value pairs, to the output (option attention hidden state), where the query can be the option embedding for the option, and the set of key-value pairs can be derived from the current hidden state 432.

[0143] The value neural network 450 may be a fully-connected neural network, i.e., including one or more fully-connected layers, optionally followed by an activation layer (e.g., a ReLU activation layer), where the activation layer is configured to process the goal attention hidden state to generate a goal value estimate for the goal, representing an estimated return to be achieved, e.g., a cumulative measure of the reward the agent would receive if the agent were to attempt to satisfy the goal starting from the current state. The value neural network 450 is also configured to, for each option, use a value neural network head to process each respective option attention hidden state for the option to generate a respective option value estimate for the option, representing an estimated return to be achieved if the agent were to attempt to satisfy the option starting from the current state.

[0144] The goal attention hidden state and each option attention hidden state are then combined to generate a combined hidden state to be provided to the policy neural network 460, and the network 460 is configured to process the combined hidden state to generate an action selection output. In some implementations, the combined hidden state may be an unweighted combination, while in other implementations, the combined hidden state may be a weighted combination, i.e., one that combines the goal attention hidden state and each option attention hidden state according to respective weights.

[0145] In some of these implementations, the respective weights for the goal and each option from their respective option value estimates for goal value estimation and option value estimation can be calculated by applying the softmax function to the respective goal value estimation and each option value estimation for option value estimation to generate respective softmax scores for the goal and each option, which are then used as weights for calculating the weighted combination. Generally, the softmax function is a function that converts numerical values to probabilities. In other of these implementations, the respective weights can be calculated by assigning a weight of 1 to the highest value estimate among the respective goal value estimation and each option value estimation for option value estimation, and a weight of zero to all other value estimates of the respective goal value estimation and each option value estimation for option value estimation.

[0146] The policy neural network 460 may include one or more fully connected layers followed by one or more output layers. In some implementations, the one or more output layers may include a single softmax layer configured to process the output of the preceding fully connected layer to generate a probability distribution over a set of actions that the agent can perform. In some other implementations, the one or more output layers may include multiple softmax layers each corresponding to a different subset of the set of actions that the agent can perform, and each of these may be configured to process the output of the preceding fully connected layer to generate a respective probability distribution over the subset. In yet other implementations, the one or more output layers may include one or more additional fully connected layers configured to generate respective Q-values for each action in the set of actions that the agent can perform.

[0147] FIG. 4B shows details of one exemplary implementation of the agent neural network 420.

[0148] FIG. 5 is a flowchart of an exemplary process 500 for controlling an agent that interacts with an environment. For convenience, process 500 is described as being implemented by a system comprising one or more computers at one or more locations. For example, a reinforcement learning system, such as the reinforcement agent control system 400 of FIG. 4 appropriately programmed, can implement process 500.

[0149] The system receives an observation record characterizing the current state of the environment (step 502). For example, the observation record may include an audio data segment, an image, or a sentence in natural language. Optionally, the observation record may also include information derived from the immediately preceding time step, such as the immediately preceding action performed, the reward received at the immediately preceding time step, or both.

[0150] The system receives goal data representing goals to be satisfied (step 504) in order to perform tasks in an environment. Among the received goal data, the goals are represented as sets of options for each set of predicates. In the example of FIG. 4, the goals are represented as two sets of options, each including the single predicate "hold a purple ball" or "look at a yellow cube", but in other examples, the goals may be represented by more options, each consisting of more predicates, and the same predicate may be shared across multiple options.

[0151] The system processes inputs including observation records and goal data (step 506) using an agent neural network to generate an action selection output, as further described below with reference to FIG. 6. In some implementations, the action selection output can define a probability distribution over a set of actions that the agent can perform. In some implementations, the action selection output can include respective Q-values for each action in a set of actions that the agent can perform. In some implementations, the action selection output is an action from a continuous action space, i.e., all action values among the individual actions are selected from a continuous range of possible values.

[0152] The system selects an action to be performed by the agent using the action selection output (step 508). In implementations where the action selection output defines a probability distribution, the system can select an action by sampling actions according to the probability values for the actions or by selecting the action with the highest probability value. In implementations where the action selection output includes Q-values, the system can process the Q-values (e.g., using a softmax function) to generate respective probability values for each possible action, which can be used to select the action to be performed by the agent. The system can also select the action with the highest Q-value as the action to be performed by the agent.

[0153] The system causes the agent to perform the selected action (step 510), for example, by instructing the agent to perform the action or by passing a control signal to a control system for the agent.

[0154] FIG. 6 is a flowchart of an exemplary process 600 that uses an agent neural network to generate an action selection output. For convenience, process 600 is described as being performed by a system comprising one or more computers at one or more locations. For example, a reinforcement learning system, such as the reinforcement learning system 400 of FIG. 4 programmed appropriately, can perform process 600.

[0155] The system processes the observation records using the state encoder neural network of the agent neural network to generate a current hidden state representing the current state of the environment (step 602). The state encoder neural network includes one or more recurrent layers and is configured to generate the current hidden state by processing the received current observation record to modify the hidden state of the state encoder neural network generated by processing previous observation records.

[0156] The system processes the goal data and the current hidden state using the embedding neural network of the agent neural network to generate (i) a predicate prediction, (ii) a goal embedding of the goal, and (iii) a respective option embedding for each of the options represented by the goal data (step 604).

[0157] The system processes an input including (i) the current hidden state, (ii) the goal embedding, and (iii) the predicate prediction using the attention neural network to generate a goal attention hidden state (step 606). The goal attention hidden state is generated by applying an attention mechanism to at least (i) the current hidden state, (ii) the goal embedding, and (iii) the predicate prediction.

[0158] For each of the options, the system processes an input including (i) the current hidden state, (ii) the respective option embedding for the option, and (iii) the predicate prediction using the attention neural network to generate a respective option attention hidden state for the option (step 608). The option attention hidden state is generated by applying an attention mechanism to at least (i) the current hidden state, (ii) the respective option embedding for the option, and (iii) the predicate prediction.

[0159] The system uses the value neural network of the agent neural network to process the goal attention hidden state and generate a goal value estimate for the goal, representing the estimated return that would be achieved if the agent were to start from the current state and attempt to satisfy the goal (step 610).

[0160] For each of the options, the system uses the value neural network to process each option attention hidden state for the option and generate an option value estimate for the option, representing the estimated return that would be achieved if the agent were to start from the current state and attempt to satisfy the option (step 612).

[0161] The system generates respective weights for the goal and for each of the options from the goal value estimate and the respective option value estimates for the options. In some implementations, generating the respective weights may include applying softmax to the goal value estimate and the respective option value estimates for the options. In some other implementations, generating the respective weights may include assigning a weight of 1 to the highest value estimate among the goal value estimate and the respective option value estimates for the options, and a weight of zero to all other value estimates of the goal value estimate and the respective option value estimates for the options.

[0162] The system combines the goal attention hidden state and the respective option attention hidden states according to the respective weights to generate a combined hidden state (step 616).

[0163] The system uses the policy neural network of the agent neural network to process the combined hidden state and generate an action selection output (step 618).

[0164] This specification uses the term "configured" in connection with system and computer program components. A system consisting of one or more computers being configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that, when operating, causes the system to perform the operation or action. One or more computer programs being configured to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing apparatus, cause the apparatus to perform the operation or action.

[0165] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in any combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver apparatus for execution by a data processing apparatus.

[0166] The term "data processing apparatus" refers to data processing hardware and encompasses, by way of example, any kind of apparatus, device, and machine for processing data, including programmable processors, computers, or multiple processors or computers. The apparatus may be, or further include, special purpose logic circuit means, such as, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Optionally, the apparatus may include, in addition to the hardware, code for creating an execution environment for a computer program, such as processor firmware, a protocol stack, a database management system, an operating system, or code constituting one or more combinations thereof.

[0167] A computer program may be referred to as, or described as, a program, software, a software application, an app, a module, a software module, a script, or code, and may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program, or as a module, component, subroutine, or other unit suitable for use in a computing environment. The program may or may not correspond to a file in a file system. The program may be stored in a portion of a file that holds one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, such as files that hold one or more modules, subprograms, or portions of code. The computer program may be deployed to be executed on one computer or multiple computers located in one place, or may be distributed across multiple places and interconnected by a data communication network.

[0168] As used herein, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more particular functions. Generally, an engine will be implemented as one or more software modules or components and installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine, and in other cases, multiple engines may be installed and run on the same computer or multiple computers.

[0169] The processes and logical flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows may also be implemented by, for example, special purpose logic circuitry, such as an FPGA, or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0170] A computer suitable for the execution of a computer program may be based on a general purpose or special purpose microprocessor or both, or any other kind of central processing unit. Generally, the central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for carrying out or executing instructions, and one or more memory devices for storing the instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or is operatively coupled to a mass storage device for receiving data from, transferring data to, or both, a mass storage device. However, a computer need not have such a device. Moreover, a computer can be incorporated in another device, such as, by way of example only, a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable memory device, such as a universal serial bus (USB) flash drive.

[0171] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0172] To enable interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can be used to provide interaction with the user, for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic, voice, or tactile input. Further, the computer can interact with the user by sending documents to and receiving documents from the devices used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser, or by running a messaging application on a personal device, such as a smartphone, to send a text message or other form of message to the user and receiving a response message in return from the user.

[0173] A data processing apparatus for implementing a machine learning model can also include, for example, a common and computationally intensive part of machine learning training or generation, i.e., an inference, a special purpose hardware accelerator unit for processing the workload.

[0174] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework.

[0175] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, such as a data server, or middleware components, such as an application server, or front-end components, such as a graphical user interface, a web browser, or an app with which a user can interact with implementations of the subject matter described herein, or a client computer having one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0176] The computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact over a communication network. The relationship of client and server arises by virtue of computer programs running on respective computers and having a client-server relationship to each other. In some embodiments, the server is for the purpose of displaying data, such as HTML pages, to a user interacting with a device functioning as a client and receiving user input from the user, and sending it to the user device. Data generated at the user device, such as the result of user interaction, can be received at the server from the device.

[0177] This specification includes many detailed implementation forms, but these should not be construed as limitations within the scope of any invention or within the scope of what can be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Also, some of the features described herein in the context of separate embodiments can be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately in multiple embodiments, or in any suitable sub-combination. Furthermore, although features are described above as functioning in a particular combination and may even be claimed as such initially, one or more features from the claimed combination can, in some cases, be deleted from that combination, and the claimed combination can be directed to a sub-combination or a variant of a sub-combination.

[0178] Similarly, operations are shown in the drawings and recited in the claims in a particular order, but this should not be understood to require that such operations be performed in the particular order shown or sequentially, or that all of the shown operations be performed to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various system modules and components in the embodiments described above should not be understood as required in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0179] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. As one example, the processes illustrated in the accompanying figures do not necessarily require the particular order or sequence shown to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Explanation of Signs

[0180] 100 Training system, system 102 Agent, reinforcement learning agent 120 Population-based training (PBT) engine 124 Task generation engine 130 Training data repository 140 Population repository 142 Candidate agent neural network 400 Reinforcement learning agent control system, reinforcement learning system, system 408 Agent 420 Agent neural network, neural network 430 State encoder neural network, neural network 440 Embedding neural network, neural network 445 Attention neural network, neural network 450 Value neural network, value neural network "V" head, neural network 460 Policy neural network, policy neural network "π" head

Claims

1. A computer-implemented method for training an agent neural network for use in controlling an agent to perform a plurality of tasks, the method comprising: maintaining population data specifying a population of one or more candidate agent neural networks, the population data specifying, for each candidate agent neural network in the population, (i) respective parameter values of the parameters of the candidate agent neural network, and (ii) respective hyperparameter values for training the candidate agent neural network; training each candidate agent neural network with each respective set of one or more tasks to update the parameter values of the parameters of the candidate agent neural network in the population data, the training step comprising, for each candidate agent neural network: obtaining data identifying a candidate task for training the candidate agent neural network; obtaining data specifying a control policy for the candidate task; determining whether to train the candidate agent neural network on the candidate task based on (i) the performance of the candidate agent neural network on the candidate task, (ii) the performance of the control policy for the candidate task on the candidate task, and (iii) one or more of the hyperparameter values for the candidate agent neural network; and training the candidate agent neural network on the candidate task to update the parameter values of the parameters of the agent neural network in response to a determination to train the candidate agent neural network on the candidate task.

2. The step of training the candidate agent neural network with the candidate task includes the step of training the candidate agent neural network with the candidate task through reinforcement learning, according to the method of claim 1.

3. The control policy for the candidate task is a uniform random action policy that performs an action randomly and uniformly selected from a set of actions when controlling the agent to perform the candidate task, according to the method of claim 1.

4. The control policy for the candidate task is a single-task policy that uses a single candidate agent neural network trained only with the candidate task when controlling the agent to perform the candidate task, according to the method of claim 1.

5. The control policy for the candidate task is a policy that uses an instance of the candidate agent neural network but uses historical parameter values for the parameters of the candidate agent neural network from an early point during the training of the candidate agent neural network when controlling the agent to perform the candidate task, according to the method of claim 1.

6. The step of determining whether to train the candidate agent neural network with the candidate task based on (i) the performance of the candidate agent neural network in the candidate task, (ii) the performance of the control policy for the candidate task in the candidate task, and (iii) one or more of the hyperparameter values for the candidate agent neural network is For each of a plurality of task episodes of the candidate task, determining each candidate agent return received by controlling the agent to perform the task episode of the candidate task using the candidate agent neural network, and determining each control policy return received by controlling the agent to perform the task episode of the candidate task using the control policy, according to the method of claim 1.

7. (i) The performance of the candidate agent neural network in the candidate task, (ii) the performance of the control policy for the candidate task in the candidate task, and (iii) based on one or more of the hyperparameter values for the candidate agent neural network, the step of determining whether to train the candidate agent neural network on the candidate task is The method according to claim 6, including the step of determining to train the candidate agent neural network on the candidate task only when the performance of the candidate agent neural network measured by each respective candidate agent return does not exceed a threshold level of performance defined by the hyperparameter value for the candidate agent neural network. **Claim 8** The step of determining to train the candidate agent neural network on the candidate task only when the performance of the candidate agent neural network measured by each respective candidate agent return does not exceed a threshold level of performance defined by the hyperparameter value for the candidate agent neural network is including the step of determining to train the candidate agent neural network on the candidate task only when the proportion of task episodes in which each respective candidate agent return exceeds a first specified value is less than a second specified value, (i) the first specified value, (ii) the second specified value, or (iii) both are hyperparameter values for the candidate agent neural network, the method according to claim 7. **Claim 9** (i) The performance of the candidate agent neural network in the candidate task, (ii) the performance of the control policy for the candidate task in the candidate task, and (iii) based on one or more of the hyperparameter values for the candidate agent neural network, the step of determining whether to train the candidate agent neural network on the candidate task is The method according to claim 6, comprising the step of determining to train the candidate agent neural network on the candidate task only when the performance of the candidate agent neural network measured by each of the candidate agent returns exceeds the performance of the control policy measured by each of the control policy returns.

10. The step of determining to train the candidate agent neural network on the candidate task only when the performance of the candidate agent neural network measured by each of the candidate agent returns exceeds the performance of the control policy measured by each of the control policy returns, includes the step of determining to train the candidate agent neural network on the candidate task only when the ratio of task episodes in which each of the candidate agent returns exceeds each of the control policy returns by at least a third specified value is greater than a fourth specified value, where (i) the third specified value, (ii) the fourth specified value, or (iii) both are hyperparameter values for the candidate agent neural network, according to the method of claim 9.

11. (i) Based on one or more of (i) the performance of the candidate agent neural network in the candidate task, (ii) the performance of the control policy for the candidate task in the candidate task, and (iii) the hyperparameter values for the candidate agent neural network, the step of determining whether to train the candidate agent neural network on the candidate task includes the step of determining to train the candidate agent neural network on the candidate task only when the performance of the control policy measured by each of the control policy returns is lower than a threshold level of performance defined by the hyperparameter values for the candidate agent neural network, according to the method of claim 6.

12. The step of determining to train the candidate agent neural network on the candidate task only when the performance of the control policy measured by each of the control policy returns is lower than a threshold level of performance defined by the hyperparameter values for the candidate agent neural network is including the step of determining to train the candidate agent neural network on the candidate task only when the average of each of the control policy returns is less than a fifth specified value, The method according to claim 11, wherein the fifth specified value is a hyperparameter value for the candidate agent neural network. **Claim 13** After training each candidate agent neural network on each respective set of one or more tasks, The method according to claim 1, further comprising the step of adjusting the hyperparameter values for one or more of the candidate agent neural networks in the population. **Claim 14** The population of one or more candidate agent neural networks includes a plurality of neural networks, and the step of adjusting the hyperparameter values for one or more of the candidate agent neural networks in the population is measuring the respective performance of each of the plurality of candidate agent neural networks over a plurality of validation tasks and calculating a respective adaptation measure for each of the plurality of candidate agent neural networks; and applying a population-based training technique to the respective adaptation measures to adjust the hyperparameter values for one or more of the candidate agent neural networks. The method according to claim 13. **Claim 15** The population-based training technique also sets, for each of one or more of the candidate agent neural networks, the respective parameter values for the candidate agent neural network to be equal to the respective parameter values for another candidate agent neural network in the population. The method according to claim 14. **Claim 16** The step of calculating each of the adaptation measures includes the step of determining a respective normalization percentile metric for the candidate agent neural network in each of the plurality of verification tasks, the method according to claim 14.

17. The step of applying the population-based training technique includes the step of determining that each adaptation measure for a first candidate agent neural network Pareto dominates each adaptation measure for a second candidate agent neural network, and in response thereto, the step of adjusting the hyperparameter values for the second candidate agent neural network to be equal to a mutated version of the hyperparameter values for the first candidate agent neural network, the method according to claim 14.

18. For each set of one or more tasks, each of the training of the candidate agent neural network is performed during the current training generation of a sequence of training generations during the training of the agent neural network, and the step of training the candidate agent neural network on the candidate task to update the parameter values of the parameters of the agent neural network includes the step of training the candidate agent neural network on the candidate task to optimize the expected return while being distilled from the best performing candidate agent neural network in the population at the end of the previous training generation immediately preceding the current training generation in the sequence, the method according to claim 1.

19. During the previous training generation, the candidate agent neural networks in the population are trained for self-reward play purposes, the method according to claim 18.

20. A method for training an agent neural network for use in controlling an agent to perform a plurality of tasks, implemented by one or more computers, for controlling an agent that interacts with an environment to cause the agent to perform tasks in the environment, Receiving an observation record characterizing the current state of the environment; Receiving goal data representing a goal to be satisfied in order to perform the task in the environment, wherein the goal is represented as a set of options for each set of predicates; Processing the observation record and the goal data using an agent neural network to generate an action selection output; Processing the observation record using a state encoder neural network to generate a current hidden state representing the current state of the environment; Generating a goal embedding of the goal and a respective option embedding for each of the options from the goal data; Processing an input including the current hidden state and the goal embedding using an attention neural network to generate a goal attention hidden state; For each of the options, processing an input including the current hidden state and the respective option embedding for the option using the attention neural network to generate a respective option attention hidden state for the option; Processing the goal attention hidden state using a value neural network head to generate a goal value estimate for the goal, representing an estimated return that would be achieved if the agent were to attempt to satisfy the goal starting from the current state; For each of the options, processing the respective option attention hidden state for the option using the value neural network head to generate a respective option value estimate for the option, representing an estimated return that would be achieved if the agent were to attempt to satisfy the option starting from the current state; Generating respective weights for the goal and for each of the options from the goal value estimate and the respective option value estimates for the options; Combining the goal attention hidden state and the respective option attention hidden states according to the respective weights to generate a combined hidden state; Processing the combined hidden state using a policy neural network head to generate the action selection output, and steps; Selecting an action to be performed by the agent using the action selection output; The method according to claim 1, further comprising causing the agent to perform the selected action.

21. The method according to claim 20, wherein the state encoder neural network is a recurrent neural network.

22. The step of using an agent neural network to process the observation record and the goal data to generate an action selection output includes: Processing the goal data and the current hidden state using a predicate predictor neural network to generate a predicate prediction, wherein the input including the current hidden state and the goal embedding further includes the predicate prediction, and for each option, the input including the current hidden state and the respective option embedding further includes the predicate prediction. The method according to claim 20, further comprising steps.

23. The step of generating respective weights for the goal and for each of the options from the goal value estimate and the respective option value estimates for the options includes applying softmax to the goal value estimate and the respective option value estimates for the options. The method according to claim 20.

24. The step of generating respective weights for the goal and for each of the options from the goal value estimate and the respective option value estimates for the options includes assigning a weight of 1 to the highest value estimate among the goal value estimate and the respective option value estimates for the options, and a weight of zero to all other value estimates of the goal value estimate and the respective option value estimates for the options. The method according to claim 20.

25. The method according to claim 20, wherein the action selection output defines a probability distribution over a set of actions that the agent can perform.

26. The action selection output according to claim 20, which includes respective Q-values for each action in a set of actions that the agent can perform.

27. The method according to claim 20, wherein the action selection output is an action from a continuous action space.

28. One or more computers, A system comprising one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective methods according to any one of claims 1 to 27.

29. One or more non-transitory computer storage media that, when executed by one or more computers, store instructions that cause the one or more computers to perform the operations of the respective methods according to any one of claims 1 to 27.

30. The method according to any one of claims 1 to 27, wherein the agent neural network is used to control the agent in a real-world environment, and is configured to process an observation record regarding the state of the real-world environment to generate an action selection output regarding an action to be performed by the agent in the real-world environment.

31. The method according to claim 30, wherein the agent is a machine agent, and the agent neural network is used to control the machine agent in the real-world environment to perform the task or one of the tasks.

32. The system according to claim 28, wherein the agent neural network is used to control the agent in a real-world environment, and is configured to process an observation record regarding the state of the real-world environment to generate an action selection output regarding an action to be performed by the agent in the real-world environment.

33. The agent is a machine agent, and the agent neural network is used to control the machine agent in the real-world environment so as to perform the task or one of the tasks, according to the system of claim 32.

34. The agent neural network is used to control the agent in the real-world environment, and is configured to process an observation record regarding the state of the real-world environment to generate an action selection output regarding an action to be performed by the agent in the real-world environment, according to one or more non-transitory computer storage media of claim 29.

35. The agent is a machine agent, and the agent neural network is used to control the machine agent in the real-world environment so as to perform the task or one of the tasks, according to one or more non-transitory computer storage media of claim 34.

Images