Training agent neural network through open-ended learning

Abstract

To provide a method of training agent neural networks through open-ended learning.SOLUTION: A method includes the steps of: maintaining population data specifying a population of one or more candidate agent neural networks; and training each candidate agent neural network on a respective set of one or more tasks to update parameter values of the candidate agent neural networks in the population data. The training step comprises, for each candidate agent neural network: obtaining data identifying a candidate task; obtaining data specifying a control policy for the candidate task; determining whether to train the candidate agent neural network on the candidate task; and in response to determining to train the candidate agent neural network on the candidate task, training the candidate agent neural network on the candidate task.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 226,124, filed July 27, 2021. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated by reference.

[0002] This specification relates to processing data using machine learning models. [Background technology]

[0003] A machine learning model receives input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models, generating an output based on the received input and on the values ​​of the parameters of the model.

[0004] Some machine learning models are deep models that utilize multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to the received input to generate an output. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] PCT Patent Application No. PCT / EP2018 / 082162 [Non-patent literature]

[0006] [Non-Patent Document 1] Shaker, Noor et al., "Procedural content generation in games. Switzerland: Springer International Publishing," 2016 [Non-patent document 2] Song et al., “V-MPO: On-policy maximum a posteriori policy optimization for discrete and continuous control,” International Conference on Learning Representations, 2019. Summary of the Invention [Means for solving the problem]

[0007] This specification generally describes a system, implemented as a computer program on one or more computers at one or more locations, that trains an agent neural network used to select actions to be performed by an agent interacting with an environment. In particular, the system trains the agent neural network so that the agent neural network can be used to control the agent to perform any of a plurality of tasks. Each task may include, for example, one or more of navigating to a specified location in the environment, identifying a particular object in the environment, manipulating a particular object in a specified manner, etc.

[0008] In one aspect, a computer-implemented method for training an agent neural network for use in controlling agents to perform a plurality of tasks is described. The method includes maintaining population data specifying a population of one or more candidate agent neural networks, the population data specifying, for each candidate agent neural network in the population, (i) respective parameter values ​​for the parameters of the candidate agent neural network and (ii) respective hyperparameter values ​​for training the candidate agent neural network. The method involves training each candidate agent neural network on a respective set of one or more tasks among a plurality of general tasks to update the parameter values ​​for the parameters of the candidate agent neural network in the population data. In an implementation, the training step includes, for each candidate agent neural network, obtaining data identifying a candidate task (e.g., one of multiple tasks) for training the candidate agent neural network, obtaining data specifying a control strategy for the candidate task, and determining (or not) to train the candidate agent neural network on the candidate task based on one or more of (i) performance of the candidate agent neural network on the candidate task, (ii) performance of the control strategy for the candidate task on the candidate task, and (iii) hyperparameter values ​​for the candidate agent neural network. In response to determining that the candidate agent neural network should be trained on the candidate task, the method includes training the candidate agent neural network on the candidate task, for example, using reinforcement learning, to update parameter values ​​of parameters of the agent neural network. The training may be performed repeatedly each time data identifying a candidate task is obtained, i.e., each time one of the multiple tasks is used as the candidate task.

[0009] In an implementation of such a method, the training tasks consumed by the agent are dynamically generated in response to the agent's performance. The effect is that task distribution is varied throughout training, and the task distribution itself can be optimized to improve the agent's performance. More specifically, training (or not) a candidate agent neural network is determined based on a combination of three factors, as listed above. For example, generally, the agent should only be trained on tasks whose performance is (significantly) better than the control policy, so that the agent performs meaningful actions, i.e., actions that affect returns. The agent's performance can generally be determined, for example, from the returns from the tasks. The control policy can be any action selection policy (for controlling the agent to perform the candidate task), such as a uniform random action policy or the agent's past policies.

[0010] The trained agent neural network can be used for use in controlling agents to perform multiple tasks in one or more environments. The environments can be real-world environments, although some or all of the training can be in a simulation of a real-world environment. In some implementations, particularly when some or all of the training is in a simulation, "suggested" tasks for training candidate agent neural networks can be generated procedurally, i.e., automatically, based on task parameters, and these tasks can then be used to determine whether to train the candidate agent neural network.

[0011] The above method can be used to obtain a population including one or more candidate agent neural networks. When the population includes two or more candidate agent neural networks, a population of multiple candidate agent neural networks is trained. One or more can then be selected as the trained agent neural network. Among the population of multiple candidate agent neural networks, they can have the same architecture as each other, but do not need to, and generally they can have different respective parameter values, such as weights, and different respective hyperparameter values.

[0012] In some implementations, a population of candidate agent neural networks is used for population-based training. That is, a population of candidate agent neural networks is trained, and a fitness measure is determined for each of the agents, e.g., to compare two (or more) of the agents. This may be done, for example, by evaluating each of the agents using a fitness function; there are many suitable fitness measures; generally, fitness may be determined from returns from a task.

[0013] As described above, each candidate agent neural network is controlled by a respective hyperparameter value. An adaptive measure may be used to modify the population of candidate agent neural networks, particularly by adjusting the hyperparameter values ​​for one or more of the candidate agent neural networks, with the aim of improving the performance of the population. Adjustments may be implemented in many ways. As one example, the hyperparameter values ​​may be randomly perturbed.

[0014] As another example, if the fitness measure for a first candidate agent neural network Pareto (across multiple tasks) is determined to dominate the fitness measure of a second candidate agent neural network (across multiple tasks), the hyperparameter values ​​for the second candidate agent neural network may be adjusted to be equal to mutated (modified) versions of the hyperparameter values ​​for the first candidate agent neural network. Pareto domination may be considered achieved if the fitness measure is at least good across multiple tasks and better for at least one of the tasks. The tasks may be candidate tasks or other, e.g., validation tasks.

[0015] In some implementations, a less preferred candidate agent neural network, i.e., one with a smaller adaptation measure (e.g., when comparing two agents), may have its respective parameter values ​​(e.g., weights), and optionally also its hyperparameter values, set to the respective parameter values, and optionally hyperparameter values, of another candidate agent neural network, e.g., one with a larger adaptation measure (e.g., when comparing two agents).

[0016] As mentioned above, an agent's fitness or performance can generally be determined from the returns from a task. However, when there are many candidate tasks, the returns can vary widely. Thus, in some implementations, calculating a fitness (or performance) measure for a task includes determining a respective normalized percentile metric, e.g., by determining the kth, e.g., 50th, percentile score achieved by the agent in the task, where the score is normalized using (divided by) a normalization constant that is the score achieved by a mix or set of one or more agents (candidate agent neural networks) that achieve the best score in the task. Here, the score is based on the reward or return achieved by the agent in the task. This approach avoids the need to compare performance to, for example, an optimal policy for a useful task, since determining such an optimal policy may not be straightforward.

[0017] A candidate agent neural network may be trained on a candidate task to optimize expected returns from the task while being distilled from the best-performing candidate agent neural network in the population, e.g., as determined at the end of the previous training generation that immediately precedes the current training generation in the sequence. Such training while being distilled may include training using a distillation loss, e.g., a loss based on the difference between the outputs, that encourages the action selection output of the candidate agent neural network to be trained toward the action selection output of the best-performing candidate agent neural network.

[0018] Training may involve a self-reward play objective, i.e., the candidate agent neural network is sequentially trained to achieve goal g and to achieve not(g), where not(g) is the negation of the goal. This can encourage exploration, where the agent appears to be competing with itself. Goals can be expressed as a set of options (disjunctions), for each set of predicates, expressing, for example, one or more necessary conditions for each option. Goals may, but need not, be expressed as natural language.

[0019] Generally, the agent neural network (and each of the candidate agent neural networks) is configured for use in controlling agents to perform multiple tasks in one or more environments. Thus, generally, the agent neural network (and each of the candidate agent neural networks) is configured to receive and process observations characterizing the current state of the environment, and in some implementations may be configured to receive and process data identifying (particular) tasks to be performed, such as goal data representing goals to be satisfied in order to perform the tasks in the environment (alternatively, for example, the tasks may be inferred from the environment).

[0020] Generally, the agent neural network (and each of the candidate agent neural networks) is configured to generate an action selection output that characterizes an action to be performed by the agent in response to the observations to perform one or more of the tasks. In an implementation, the trained agent neural network is used to select an action to be performed by the agent in a real-world environment, the selected action relating to the action to be performed by the agent in the real-world environment (and the observation record relating to the observation record of the real-world environment). By way of example, the agent may include a mechanical agent such as a robot or an autonomous or semi-autonomous vehicle; other examples of agents are provided below.

[0021] The subject matter described herein can be implemented in particular implementations to realize one or more of the following advantages.

[0022] The described techniques can implement an open-ended reinforcement learning training process, during which training task distribution and training objectives are dynamically adapted so that multiple instances of an agent neural network rarely stop learning, to effectively train network instances successively so that agents controlled by one or more of the trained network instances achieve robust and generally competent performance across a large space of different tasks and environments. The described techniques for training agent neural networks are also universally applicable to any type of complex environment in which agents may be deployed to perform any type of technically challenging task.

[0023] Use of the described techniques therefore results in generally competent agent neural networks that are not only state-of-the-art but can surpass human-level performance in a wide range of agent control tasks, and that are also generalizable and easily adaptable by fine-tuning to new tasks, including, for example, novel tasks that are naturally distinct from existing tasks for which the network may have been trained. In addition to achieving improved agent performance, by training a ensemble of agent neural network instances across a very large number of tasks, e.g., an infinite multitask continuum, the training process also consumes fewer computational resources, e.g., memory and processing power, than traditional approaches that require training one neural network model from scratch for each single new task or new environment.

[0024] The details of one or more implementations of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 illustrates an exemplary training system. [Figure 2] 1 is a flowchart of an exemplary process for training a population of candidate agent neural networks. [Figure 3] FIG. 1 illustrates an illustration of training a population of candidate agent neural networks. [Figure 4A] FIG. 1 illustrates an exemplary reinforcement learning agent control system. [Figure 4B] FIG. 1 illustrates an exemplary implementation of an agent neural network. [Figure 5] 1 is a flowchart of an exemplary process for controlling an agent interacting with an environment. [Figure 6] 1 is a flowchart of an exemplary process for using an agent neural network to generate an action selection output. [Figure 7] FIG. 1 illustrates an illustration of determining a normalized percentile metric. DETAILED DESCRIPTION OF THE INVENTION

[0026] Like reference numbers and designations in the various drawings indicate like elements.

[0027] 1 illustrates an exemplary training system 100. Training system 100 is an example of a system in which the systems, components, and techniques described below may be implemented as a computer program on one or more computers at one or more locations.

[0028] Training system 100 is a system that implements an open-ended reinforcement learning training process for training multiple instances of an agent neural network, each of which can be used to select actions to be performed by an agent, e.g., agent 102A, in environment 104 to control the agent to perform a wide range of machine learning tasks. In some cases, the machine learning task is a single-agent task that the agent itself can perform, while in other cases, the machine learning task is a multi-agent task that requires the agent to interact, e.g., compete or cooperate, with one or more other reinforcement learning agents, e.g., agents 102B-L, in environment 104. That is, training system 100 acquires (i.e., generates or receives) observation records, each characterizing a respective state of environment 104, and, in response to each observation, selects an action from a predetermined set of actions to be performed by reinforcement learning agent 102A in response to the observation. In response to some or all of the actions performed by agent 102A, training system 100 obtains a reward. Each reward is a numerical value received from the environment 104 as a result of the agent 102A performing an action, and in particular, the reward varies depending on the state the environment 104 transitions to as a result of the agent 102A performing the action. In some cases, a given set of actions may define a discrete action space; in other cases, a given set of actions may alternatively define a continuous action space, i.e., all of the action values ​​in the individual actions are selected from a continuous range of possible values, or, even alternatively, may define a hybrid action space, i.e., one or more of the action values ​​in the individual actions are selected from a continuous range of possible values.

[0029] In particular, by implementing the described open-ended reinforcement learning training process, it becomes possible to train a single, but generally competent, agent neural network that can be used to control agents to achieve or exceed not only state-of-the-art but human-level performance across thousands or millions of tasks in different environments.

[0030] In some implementations, the environment is a real-world environment, the agent is a machine agent that interacts with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle that operates or navigates within the environment, and the action is an action taken by the machine agent in the real-world environment to perform a task. For example, the agent may be a robot that interacts with the environment to accomplish a particular task, e.g., to locate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment.

[0031] In these implementations, the observations may include, for example, one or more of images, object position data, and sensor data, e.g., from image, distance, or position sensors, or from actuators, to capture the observations as the agent interacts with the environment. For example, in the case of a robot, the observations may include data characterizing the current state of the robot, e.g., one or more of joint positions, joint velocities, joint forces, torques, or accelerations, e.g., gravity-compensated torque feedback, and global or relative poses of members held by the robot. In the case of a robot or other mechanical agent or vehicle, the observations may similarly include one or more of positions, linear or angular velocities, forces, torques, or accelerations, and global or relative poses of one or more parts of the agent. The observations may be defined in one, two, or three dimensions and may be absolute and / or relative observations. The observations may also include sensed electronic signals, such as motor current or temperature signals, and / or image or video data, for example from a camera or LIDAR sensor, e.g., data from sensors on the agent or from sensors located separately from the agent in the environment.

[0032] In these implementations, an action may be a control signal or advanced control command for controlling a robot or other mechanical agent, e.g., torque for a joint of the robot, or a torque or advanced control command to a control surface or other control element, e.g., a steering control element, of an autonomous or semi-autonomous land, air, or sea vehicle, e.g., a vehicle control surface or other control element of the vehicle. The control signal may include, e.g., position, velocity, or force / torque / acceleration data for one or more joints of a robot or part of another mechanical agent. The control signal may also or instead include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices in the environment, whose control affects observed environmental conditions. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, the control signal may define actions for controlling navigation, e.g., steering, and movement, e.g., braking and / or acceleration of the vehicle.

[0033] In some implementations, the environment is a simulation of the real-world environment described above, and the agent is implemented as one or more computers that interact with the simulated environment. For example, the simulated environment may be a simulation of a robot or vehicle, and the reinforcement learning system may be trained in the simulation and then, once trained, used in the real world.

[0034] In some implementations, the environment is a real-world manufacturing environment for producing products such as chemical, biological, or mechanical products, or food products. As used herein, "manufacturing" a product also includes purifying starting materials to create the product or processing starting materials, for example, to remove contaminants, to produce a cleaned or regenerated product. A manufacturing plant may include multiple production units, such as containers for chemical or biological substances, or machines, e.g., robots, for processing solids or other materials. The production units are configured such that intermediate versions or components of the product can be moved between them during the production of the product, for example, via pipes or mechanical conveyances. As used herein, producing a product also includes the production of food products by kitchen robots.

[0035] The agent may include an electronic agent configured to control a manufacturing unit or a machine, such as a robot, that operates to manufacture a product. That is, the agent may comprise a control system configured to control the manufacturing of a chemical, biological, or mechanical product. For example, the control system may be configured to control one or more of the manufacturing units or machines, or to control the movement of intermediate versions or components of a product between manufacturing units or machines.

[0036] As one example, the tasks performed by the agents may include tasks for manufacturing a product or intermediate versions or components thereof. As another example, the tasks performed by the agents may include tasks for controlling, e.g., minimizing, resource usage, such as tasks for controlling power consumption, or water consumption, or the consumption of any materials or consumables used in the manufacturing process.

[0037] Actions may include control actions for controlling the use of a machine or manufacturing unit to process solid or liquid materials to produce a product, or an intermediate or component thereof, or for controlling the movement of intermediate versions or components of a product within a manufacturing environment, for example, between manufacturing units or machines. Generally, actions may be any action that has an effect on observed environmental conditions, such as actions configured to adjust any of the sensed parameters described below. These may include actions for adjusting physical or chemical conditions of a manufacturing unit or for controlling the movement of mechanical parts of a machine or joints of a robot. Actions may include actions that impose operating conditions on a manufacturing unit or machine, or actions that effect changes to settings to adjust, control, or turn on or off the operation of a manufacturing unit or machine.

[0038] The reward or return may be related to a metric of task performance. For example, for the task of manufacturing a product, metrics may include metrics of the quantity of product manufactured, the quality of the product, the speed of production of the product, or for the physical cost of performing a manufacturing task, metrics of the amount of energy, material, or other resources used to perform the task. For the task of controlling resource usage, metrics may include any metric of resource usage.

[0039] In general, observations of the state of the environment can include any electronic signals representative of the functionality of electronic and / or mechanical components of equipment. For example, the representation of the state of the environment can be derived from observations made by sensors sensing a state of the manufacturing environment, e.g., sensors sensing the state or configuration of a manufacturing unit or machine, or sensors sensing the transfer of material between manufacturing units or machines. As some examples, such sensors may be configured to sense mechanical movement or force, pressure, temperature, electrical conditions such as current, voltage, frequency, impedance, etc., quantity, level, flow rate / movement rate, or flow path / path of one or more materials, physical or chemical conditions, e.g., physical state, shape, configuration, or chemical condition such as pH, mechanical configuration of a unit or machine, configuration of a unit or machine, or valve configuration, image or video sensors for capturing images or video observations of a manufacturing unit or machine or movement, or any other suitable type of sensor. In the case of a machine such as a robot, observations from sensors may include observations of position, linear or angular velocity, force, torque or acceleration, or pose of one or more parts of the machine, e.g., data characterizing the current state of the machine or robot, or of a member held or handled by the machine or robot. Observations may also include sensed electronic signals, e.g., motor current or temperature signals, or image or video data, e.g., from a camera or LIDAR sensor. Sensors such as these may be part of the agent or may be located separately from it in the environment.

[0040] In some implementations, the environment is a service facility including multiple pieces of electronic equipment, such as a server farm or data center, e.g., a telecommunications data center, or a computer data center, for storing or processing data, or any real-world environment of a service facility. The service facility may also include auxiliary controls that control the operating environment of the pieces of equipment, e.g., environmental controls such as temperature controls, e.g., refrigeration equipment, or airflow controls or air conditioning equipment. The tasks may include tasks for controlling, e.g., minimizing, resource usage, such as tasks for controlling power consumption or water consumption. The agents may include electronic agents configured to control the operation of the pieces of equipment or to control the operation of auxiliary, e.g., environmental, controls.

[0041] In general, the actions may be any action that has an effect on the observed environmental conditions, such as actions configured to adjust any of the sensed parameters described below. These may include actions to control or impose operating conditions on items of equipment or auxiliary controls, for example, actions that effect changes to settings to adjust, control, or turn on or off the operation of items of equipment or auxiliary controls.

[0042] In general, observations of the state of the environment can include any electronic signals representative of the functionality of the facility or of equipment within the facility. For example, a representation of the state of the environment may be derived from observations made by any sensors that sense the state of the facility's physical environment, or by any sensors that sense the state of one or more pieces of equipment or one or more pieces of auxiliary control equipment. These include sensors configured to sense electrical conditions such as current, voltage, power, or energy, facility temperature, fluid flow, temperature, or pressure within the facility or in the facility's cooling system, or physical facility configuration such as whether a vent is open.

[0043] The reward or return may be related to a metric of performance of the task. For example, in the case of a task to control, e.g., minimize, resource usage, such as a task to control power or water usage, the metric may include any metric of resource usage.

[0044] In some implementations, the environment is a real-world environment of a power generation facility, e.g., a renewable power generation facility such as a solar or wind power plant. The tasks may include, for example, a control task for controlling power generated by the facility to meet demand or reduce the risk of mismatches between elements of the power grid, e.g., for controlling the supply of power to the power grid, or for maximizing the power generated by the facility. The agents may include electronic agents configured to control the generation of power by the facility or the coupling of generated power to the power grid. The actions may include actions for controlling the electrical or mechanical configuration of a generator, such as the electrical or mechanical configuration of one or more renewable power generation elements, e.g., the configuration of a wind turbine, or the configuration of one or more solar panels or mirrors, or the electrical or mechanical configuration of a rotating power generation machine. The mechanical control actions may include, for example, actions for controlling the conversion of energy input to electrical energy output, e.g., the efficiency of the conversion or the degree of coupling of the energy input and electrical energy output. The electrical control actions may include, for example, actions for controlling one or more of the voltage, current, frequency, or phase of the generated power.

[0045] The reward or return may be related to a metric of the performance of the task. For example, in the case of a task to control the supply of power to an electrical grid, the metric may be related to a measure of power transferred, or to a measure of an electrical mismatch between a power generation facility and the electrical grid, such as a voltage, current, frequency, or phase mismatch, or to a measure of power or energy losses at the power generation facility. In the case of a task to maximize the supply of power to the electrical grid, the metric may be related to a measure of power or energy transferred to the electrical grid, or to a measure of power or energy losses at the power generation facility.

[0046] In general, observations of environmental conditions may include any electronic signals representative of the electrical or mechanical functionality of power generation equipment in a power generation facility. For example, a representation of the environmental condition may be derived from observations made by any sensors that sense the physical or electrical condition of equipment in a power generation facility generating electrical power, or the physical environment of such equipment, or the conditions of auxiliary equipment supporting the power generation equipment. Such sensors may include sensors configured to sense observations of the electrical conditions of the equipment, such as current, voltage, power, or energy, the temperature or cooling of the physical environment, fluid flow, or the physical configuration of the equipment, and the electrical conditions of the electrical grid, e.g., from local or remote sensors. Observations of environmental conditions may also include one or more predictions regarding future conditions of operation of the power generation equipment, such as predictions of future wind levels or solar irradiance or predictions of future electrical conditions of the electrical grid.

[0047] As another example, the environment may be a chemical synthesis or protein folding environment, whereby each state is a respective state of a protein chain or of one or more intermediates or chemical precursors, and the agent is a computer system for determining how to fold the protein chain or synthesize a chemical. In this example, the actions are possible folding actions or actions to assemble chemical precursors / intermediates to fold the protein chain, and the results to be achieved may include, for example, folding the protein so that it is stable and achieves a specific biological function, or providing a valid synthetic route for the chemical. As another example, the agent may be a machine agent that performs or controls protein folding actions or chemical synthesis steps selected by the system automatically without human interaction. The observations may include direct or indirect observations of the protein or chemical / intermediate / precursor states and / or may be derived from simulations.

[0048] Similarly, the environment may be a drug design environment where each state is a respective state of a potential pharmaceutical chemical and the agent is a computer system for determining elements of the pharmaceutical chemical and / or synthetic pathways for the pharmaceutical chemical. The drug / synthesis may be designed based on rewards derived from the drug target, for example, in a simulation. As another example, the agent may be a machine agent that performs or controls the synthesis of a drug.

[0049] In some further applications, the environment is a real-world environment and the agent manages the distribution of tasks across computational resources, for example, on mobile devices and / or in a data center. In these implementations, the action may include assigning a task to a particular computational resource.

[0050] As a further example, the action may include presenting an advertisement, the observation record may include an impression or click-through count or rate of the advertisement, and the reward may characterize a previous selection of an item or content made by one or more users.

[0051] In some cases, the observations may include text or spoken instructions provided to the agent by a third party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and a user of the autonomous vehicle may provide text or spoken instructions to the agent (e.g., to navigate to a particular location).

[0052] As another example, the environment may be an electrical, mechanical, or electromechanical design environment, e.g., an environment in which the design of an electrical, mechanical, or electromechanical entity is simulated. The simulated environment may be a simulation of a real-world environment in which the entity is intended to operate. The task may be to design the entity. The observations may include observations characterizing the entity, i.e., observations of the entity's mechanical shape, or of its electrical, mechanical, or electromechanical configuration, or observations of the entity's parameters or properties. The action may include an action that modifies the entity, e.g., modifying one or more of the observations. The reward or return may include one or more metrics of the performance of the entity's design. For example, the reward or return may be related to one or more physical characteristics of the entity, such as weight or strength, or to one or more electrical characteristics of the entity, such as a measure of efficiency in performing a particular function for which the entity is designed. The design process may include outputting a design for manufacturing, e.g., in the form of computer-executable instructions for manufacturing the entity. The process may include making the entity according to the design. Thus, the design of the entity may be optimized, for example by reinforcement learning, and the optimized design may then be output, for example as computer-executable instructions, for manufacturing the entity, and the entity having the optimized design may then be manufactured.

[0053] As mentioned above, the environment may be a simulated environment. Generally, in the case of a simulated environment, the observations may include simulated versions of one or more of the aforementioned observations or observation types, and the actions may include simulated versions of one or more of the aforementioned actions or action types. For example, the simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle that navigates through the motion simulation. In these implementations, the actions may be control inputs for controlling a simulated user or a simulated vehicle. Generally, the agent may be implemented as one or more computers that interact with the simulated environment.

[0054] The simulated environment may be a simulation of a particular real-world environment and agent. For example, the system may be used to select actions in the simulated environment during training or evaluation of the system, and after training or evaluation, or both, may be deployed to control a real-world agent in the particular real-world environment that was the subject of the simulation. This may avoid unnecessary wear and tear on the real-world environment or the real-world agent, and may allow the control neural network to be trained and evaluated based on situations that are rare, difficult to reproduce, or unsafe in the real-world environment. For example, the system may be partially trained using a simulation of a machine agent in a simulation of a particular real-world environment, and then deployed to control a real machine agent in the particular real-world environment. Thus, in such cases, the observations of the simulated environment relate to the real-world environment, and the selected actions in the simulated environment relate to actions to be performed by the machine agent in the real-world environment.

[0055] Optionally, in any of the above implementations, the observation record at any given time step may include data from previous time steps that may be useful in characterizing the environment, such as actions taken in the previous time step, rewards received in the previous time step, or both.

[0056] In general, the agent neural network may have any suitable neural network architecture that enables it to perform its described functions, e.g., processing data that identifies a task to be performed and an observation to generate an action selection output that characterizes an action to be performed by the agent in response to the observation. For example, the agent neural network may include any suitable number of layers (e.g., 5 layers, 10 layers, or 25 layers) of any suitable type (e.g., fully connected layers, convolutional layers, attention layers, transformer layers, etc.) connected in any suitable configuration (e.g., as a linear sequence of layers with or without residual connections).

[0057] In one example, the action selection output may include a respective numerical probability value for each action in a set of possible actions that the agent can perform. The system may select an action to be performed by the agent, for example, by sampling the actions according to their probability values ​​or by selecting the action with the highest probability value.

[0058] In another example, the action selection output may directly define the action to be performed by the agent, for example, by defining the value of the torque that should be applied to the joints of the robotic agent.

[0059] In another example, the action selection output may include a respective Q value for each action in a set of possible actions that the agent can perform. The system may process the Q values ​​(e.g., using a softmax function) to generate a respective probability value for each possible action, which may be used to select an action (described previously) to be performed by the agent. The system may also select the action with the highest Q value as the action to be performed by the agent.

[0060] The Q-value for an action is an estimate of the "return" to the current observations that result from the agent performing the action and then selecting future actions for the agent 102 to perform according to the current values ​​of the parameters of the agent neural network.

[0061] Return refers to a cumulative measure of the "reward" an agent receives, e.g., a time-discounted sum of rewards. An agent can receive a respective reward at each time step, where the reward is specified by a scalar numeric value, e.g., characterizing the agent's progress towards completing an assigned task.

[0062] In some cases, the system may select an action for the agent to perform according to a search strategy. For example, the search strategy may be a greedy search strategy, where the system selects an action for the agent to perform according to an action selection output with probability 1-∈, randomly selecting an action with probability ∈. In this example, ∈ is a scalar value between 0 and 1.

[0063] One example of an architecture for an agent neural network is further described with reference to FIG.

[0064] Training system 100 maintains population repository 140, which stores population data specifying a population of one or more candidate agent neural networks 142A-M. Population repository 140 is implemented as one or more logical storage devices in one or more physical locations, or as logical storage space allocated on one or more storage devices in one or more physical locations. At any time during training, population repository 140 stores data specifying the current population of candidate agent neural networks 142A-M.

[0065] In particular, the population repository 140 stores, for each candidate agent neural network 142A-M in the current population, a set of maintenance values ​​that define the respective candidate agent neural network. The set of maintenance values ​​includes network parameters, hyperparameters, and, in implementations, also advantageously maintains performance measures (or "performance," for short) for each candidate agent neural network 142A-M across different tasks. For example, for candidate agent neural network A 142A, the set of maintenance values ​​includes network parameters A 144A, hyperparameters A 146A, and performance A 148A (which may include respective performances for different tasks). When there are multiple candidate agent neural networks in the population, each candidate agent neural network will generally have the same architecture but different respective parameter values ​​and, in some cases, different respective hyperparameter values ​​from other candidate agent neural networks in the population.

[0066] Hyperparameters for a candidate agent neural network are values ​​that affect how the values ​​of network parameters, e.g., weights, are updated through training. Hyperparameters may include discount factors, learning rates, objective function values, or weights assigned to various terms in an objective function, etc. Additionally, hyperparameters may include one or more specified values ​​for use in determining whether to train a candidate agent neural network on a given candidate task. Thus, generally, in the described systems, hyperparameters control the learning of parameters (which may be done, for example, by backpropagation of gradients of one or more objective functions).

[0067] To train the agent neural networks, training system 100 also maintains training data repository 130, which stores training data for training candidate agent neural networks 142A-M. Training data repository 130 is implemented as one or more logical storage devices in one or more physical locations, or as logical storage space allocated to one or more storage devices in one or more physical locations. Training data repository 130 stores data defining a set of candidate tasks 132A-N on which candidate agent neural networks may be trained, and for each candidate task, a corresponding control strategy 133A-N that can be used to control an agent to perform the candidate task, e.g., control strategy 133A for task A 132A. Training data repository 130 optionally stores data defining a set of validation tasks for evaluating the performance of candidate agent neural networks 142A-M on validation tasks.

[0068] Training system 100 may receive data in any of a variety of ways. For example, system 100 may receive data defining the set of candidate tasks and / or validation tasks as an upload from a remote user of the system over a data communications network (e.g., using an application programming interface (API) made available by training system 100). As another example, training system 100 may receive input from a user specifying which data already maintained by training system 100 should be used as data defining the set of candidate tasks and / or validation tasks.

[0069] In some implementations, the training data repository 130 remains fixed over the course of the training process, while in other implementations, the training data repository 130 grows or shrinks (e.g., to infinity) as iterations of the training process are performed, e.g., new candidate tasks can be dynamically added to the repository and existing tasks can be discarded for which a predetermined number of candidate agent neural networks have all achieved threshold performance. In these other implementations, the training system 100 can use the task generation engine 124 to generate new tasks from a space of candidate tasks, e.g., to generate a continuum of multiple tasks, at each iteration, i.e., multiple training iterations, over the course of training. This space of candidate tasks can be parameterized by a set of task parameters, each associated with one or more values, e.g., a score, a discrete value, or a continuous value. For example, the task generation engine 124 can generate tasks randomly, e.g., by randomly selecting different task parameter values ​​from the set of task parameters. To actually generate the tasks according to these selected task parameter values, the training system 100 can then use any of the exemplary techniques described in Shaker, Noor et al., "Procedural content generation in games. Switzerland: Springer International Publishing," 2016, in addition to or instead of other known digital content generation techniques.

[0070] In general, the set of task parameters may include any of a variety of adjustable parameters that can collectively define a space of candidate tasks. For example, the set of task parameters may include a first plurality of environmental parameters that define different properties or characteristics of the environment with which the agent interacts. More specifically, the environmental parameters may define or otherwise specify the static topology of the environment (e.g., the layout and structure of the environment's topological building blocks), the lighting of the environment, the types or formats of observations characterizing the state of the environment that can be provided to the agent, the properties of dynamic objects present in the environment (e.g., the location and physical properties of target objects or obstacles), a predetermined set of actions that the agent can perform in response to receiving observations, the number of other agents present in the environment (in the case of competitive or cooperative tasks), or combinations thereof, and possibly more.

[0071] As another example, the set of task parameters may include a second plurality of goal parameters that define or otherwise specify different goals for each of one or more agents to achieve as they interact with the environment. The goal parameters may include, for example, various terms of a reward function used to calculate rewards (expressed as numerical values) to be issued to the agents in response to the agents performing different actions as they interact with the environment. The rewards generally reflect the progress of the agents toward achieving the goals specified for the candidate tasks that the agents are configured to perform.

[0072] Training system 100 may use a population-based training (PBT) engine 120 to train a population of one or more candidate agent neural networks 142A-M using population-based training techniques described in more detail in PCT Patent Application No. PCT / EP2018 / 082162, which is incorporated herein by reference. As part of the training, PBT engine 120 trains each candidate agent neural network on a respective set of one or more tasks to update parameter values ​​for the candidate agent neural network's parameters in population repository 140. However, unlike traditional population-based training, before training a given candidate agent neural network on any given candidate task, PBT engine 120 obtains data identifying the candidate task for training the candidate agent neural network and obtains data specifying a control strategy for the candidate task.

[0073] In particular, PBT engine 120 determines whether to train a candidate agent neural network on a candidate task based on one or more of: (i) the performance of the candidate agent neural network on the candidate task (i.e., when used according to parameter values ​​currently stored in the ensemble data for the candidate agent neural network), (ii) the performance of the control strategy for the candidate task on the candidate task, and (iii) the hyperparameter values ​​for the candidate agent neural network. Generally, PBT engine 120 will only train a candidate agent neural network on a given task if the PBT engine determines that the task is “useful” for the candidate given the current learning stage, i.e., given the current values ​​of the candidate's parameters stored in the ensemble data. PBT engine 120 may make this determination based in part on the performance of the control strategy relative to the candidate's performance.

[0074] In response to determining to train the candidate agent neural network with the candidate task, the PBT engine 120 trains the candidate agent neural network with the candidate task to update parameter values ​​of the parameters of the agent neural network. In response to determining not to train the candidate agent neural network with the candidate task, the system refrains from training the candidate agent neural network with the candidate task.

[0075] The use of a control strategy thus improves the effectiveness and efficiency of the training process, for example, by having PBT engine 120 train a candidate agent neural network on a candidate task only if the candidate agent neural network's performance is significantly better than that of the control strategy. In some implementations, for each candidate task, e.g., task A 132A, the corresponding control strategy, e.g., control strategy 133A, may be a fixed strategy, e.g., a random action strategy that randomly selects actions; in other implementations, the corresponding control strategy may be a strategy that is controlled by candidates from a prior training generation, i.e., according to the candidate's historical network parameter values ​​from an earlier point during training. In these other implementations, the use of a control strategy further enables PBT engine 120 to determine whether the candidate agent neural network's performance has recently improved or deteriorated relative to prior training iterations and take appropriate action accordingly, e.g., adjusting hyperparameters or simply removing it from the current population.

[0076] In some implementations, the PBT engine 120 fixes the hyperparameters for the candidates during training. In other implementations, the PBT engine 120 adjusts the hyperparameters for the candidates during training to ensure that the dynamic task selection described above continues to select only tasks that are useful for learning as the candidate's parameter values ​​change over the course of training.

[0077] Some or all of the tasks for training the candidate may be multi-agent tasks, i.e., tasks that require an agent controlled using the candidate to interact with one or more other agents controlled using a different policy. For these tasks, the system may use any of a variety of policies to control one or more other agents. For example, the system may use a fixed policy, e.g., a random action policy that randomly selects an action, a no-action action policy in which no other agent performs any action, an expert policy that represents the behavior of an expert agent, or a policy controlled by a high-performing candidate from a prior training generation.

[0078] After training, training system 100 may select one of the candidates in the population for use as the agent neural network, i.e., for use in controlling the agent to perform a new task. For example, training system 100 may select the best-performing candidate after training is complete. Alternatively, training system 100 may use an ensemble of multiple candidates in the population as the final agent neural network.

[0079] 2 is a flowchart of an exemplary process 200 for training a population of candidate agent neural networks. For convenience, process 200 is described as being performed by a system of one or more computers at one or more locations. For example, a population-based neural network training system suitably programmed in accordance with this specification, such as training system 100 of FIG. 1, can perform process 200.

[0080] As described above, the system maintains population data that specifies a population of one or more candidate agent neural networks, and for each candidate agent neural network in the population, the population data specifies (i) respective parameter values ​​for the parameters of the candidate agent neural network, (ii) respective hyperparameter values ​​for training the candidate agent neural network, and (iii) performance for the candidate agent neural network (which may include respective performance for different tasks).

[0081] The system repeatedly performs process 200 for each candidate neural network in the population. In some implementations, the system repeatedly performs process 200 for each candidate neural network in parallel and asynchronously with performing the process for each other candidate neural network in the population.

[0082] The system obtains data identifying candidate tasks for training a candidate agent neural network (step 202). The system generally obtains different candidate tasks in different iterations. Different tasks may involve different environments with which an agent controlled using the candidate agent neural network interacts, different goals to be achieved when the agent interacts with the environment, different numbers of other agents present in the environment (in the case of competitive or cooperative tasks), etc.

[0083] In some implementations, candidate tasks may be new tasks automatically and dynamically generated by the system (i.e., over the course of training) according to different sets of task parameters sampled from the space of candidate tasks. In other implementations, candidate tasks may be obtained by sampling a fixed number of candidate tasks from the set of candidate tasks contained in the training data repository at each iteration. In some of these implementations, the system may use the same task sampling strategy for all candidate agent neural networks in the population, while in others of these implementations, the system may use different candidate-specific task sampling strategies for the population. Furthermore, in implementations in which the system maintains data specifying a list of historical candidate tasks on which candidate agent neural networks were previously trained, the system may update the list to include the obtained candidate tasks.

[0084] The system obtains data specifying a control strategy for the candidate task (step 204). Generally, the control strategy may be any action selection strategy that can be used to control an agent (possibly including other agents in the environment) to interact with the environment. For example, a control strategy for the candidate task may be a uniformly random action strategy that takes an action uniformly randomly selected from a predetermined set of actions when controlling the agent to perform the candidate task. As another example, a control strategy for the candidate task may be a single-task strategy that uses a single candidate agent neural network trained only on the candidate task when controlling the agent to perform the candidate task. As another example, a control strategy for the candidate task may be a strategy that uses an instance of a candidate agent neural network but uses historical parameter values ​​for the network parameters of the candidate agent neural network from an earlier point in time during the training of the candidate agent neural network when controlling the agent to perform the candidate task.

[0085] The system determines whether to train the candidate agent neural network on the candidate task based on one or more of: (i) the performance of the candidate agent neural network on the candidate task; (ii) the performance of the control strategy on the candidate task; and (iii) hyperparameter values ​​for the candidate agent neural network (step 206).

[0086] Specifically, to make this determination, the system can use a planning algorithm, such as a Monte Carlo Tree Search (MCTS) algorithm or another look-ahead planning algorithm, to make predictions about multiple future states of the environment after an initial state at a given time step, called a task episode. A task episode represents the rollout of the environment at a time after a given time step, assuming that the agent performs a particular action selected using the candidate agent neural network. The system can run the planning algorithm to iteratively generate multiple task episodes for the candidate task, for example, 5, 10, 20, or more task episodes.

[0087] A task episode refers to a sequence of time steps in which an agent interacts with an environment. A task episode may end, for example, when the agent has interacted with the environment for a predefined number of time steps or when the agent has completed a task.

[0088] In some implementations, each task episode can include a sequence of multiple trajectories, and each trajectory can include a respective current observation characterizing a respective current state of the environment, a respective current action performed by the agent in response to the current observation, a respective next state characterizing a respective next state of the environment, and a reward received by the agent in response to performing the current action.

[0089] For each of a plurality of task episodes of the candidate task, the system uses the candidate agent neural network to determine a respective candidate agent return received by controlling the agent to perform the task episode of the candidate task. The system also uses the control strategy to determine a respective control policy return received by controlling the agent to perform the task episode of the candidate task. Both the candidate agent return and the control policy return may be task-specific returns calculated according to a reward function defined by selected task parameters, for example, as a cumulative measure of rewards received by the agent in response to performing one or more actions when interacting with an environment.

[0090] In some implementations, the system decides to proceed with training the candidate agent neural network on the candidate task only when the performance of the candidate agent neural network, as measured by each candidate agent return, does not exceed a threshold level of performance defined by hyperparameter values ​​for the candidate agent neural network.

[0091] In these implementations, the system may determine to train a candidate agent neural network on a candidate task only when the percentage of task episodes in which the respective candidate agent return exceeds a first specified value is less than a second specified value. For example, the first specified value may be an integer or floating-point value between the possible range of total rewards the agent can receive, and the second specified value may be a decimal value between zero and one, where the first specified value, the second specified value, or both are hyperparameter values ​​for the candidate agent neural network.

[0092] An example of this criterion in mathematical terms is: Pr[R π (x)>m s ] <m solved wherein R π (x) is the return of each candidate agent, and m s and m solved are the first and second specified values, respectively.

[0093] In some implementations, the system determines to train a candidate agent neural network on a candidate task only when the performance of the candidate agent neural network, as measured by each candidate agent return, exceeds the performance of the control strategy, as measured by each control strategy return, by more than a threshold level of performance defined by hyperparameter values ​​for the candidate agent neural network.

[0094] In these implementations, the system may determine to train a candidate agent neural network on a candidate task only if the proportion of task episodes in which the respective candidate agent return exceeds the respective control policy return by at least a third specified value is greater than a fourth specified value. For example, the third specified value may be an integer or floating-point value between the possible range of total rewards the agent can receive, and the fourth specified value may be a decimal value between zero and one, where the third specified value, the fourth specified value, or both, are hyperparameter values ​​for the candidate agent neural network. The third and fourth specified values ​​may be the same as or different from the first and second specified values, respectively.

[0095] An example of this criterion in mathematical terms is:

[0096]

number

[0097] wherein R π (x) is the return of each candidate agent, and R πcont (x) is the return of each control strategy, and m> and m >cont are the third and fourth specified values, respectively.

[0098] In some implementations, the system determines to train a candidate agent neural network on a candidate task only when the performance of the control strategies, as measured by their respective control strategy returns, is below a threshold level of performance defined by hyperparameter values ​​for the candidate agent neural network. That is, the system determines to train a candidate agent neural network on a candidate task that requires a control strategy that is more sophisticated or advanced than, for example, a uniform random action strategy, to achieve the specified goal of the candidate task.

[0099] In these implementations, the system may determine to train a candidate agent neural network on a candidate task only if the mean (or median, or other representative value) of the respective control policy returns is less than a fifth specified value. For example, the fifth specified value may be an integer or floating-point value between the possible range of total rewards that the agent can receive, where the fifth specified value is a hyperparameter value for the candidate agent neural network. The fifth specified value may be the same as or different from the first or third specified value.

[0100] An example of this criterion in mathematical terms is:

[0101]

number

[0102] wherein V πcont (x) is the expected return of each control strategy, and m cont is the fifth specified value.

[0103] In any of the above implementations, in response to determining to train the candidate agent neural network on the candidate task, the system trains the candidate agent neural network on the candidate task using any suitable reinforcement learning technique to update maintenance values ​​of the agent neural network's network parameters, i.e., by optimizing an objective function that depends on rewards received from the environment by an agent controlled using the candidate agent neural network (step 208). For example, the reinforcement learning technique may be an on-policy RL training technique, such as one of the RL algorithms described in more detail in Song et al., "V-MPO: On-policy maximum a posteriori policy optimization for discrete and continuous control," International Conference on Learning Representations, 2019, and the system may train the candidate agent neural network on the candidate task for a fixed number of iterations or a set period of time.

[0104] Applied in parallel with reinforcement learning techniques are population-based training techniques. Specifically, the system trains a candidate agent neural network on each set of one or more candidate tasks using maintained values ​​for the hyperparameters and network parameters of the candidate agent neural network to iteratively generate updated network parameters for the candidate agent neural network until PBT termination criteria are satisfied. The PBT termination criteria are one or more set conditions that, when satisfied by the candidate agent neural network, cause the system to update the repository for the candidate agent neural network with new network parameters, new hyperparameters, and new performance measures. An example of when the PBT termination criteria are satisfied is when the candidate agent neural network has trained on each set of one or more candidate tasks for a certain number of iterations (e.g., 1e4, 1e6, 1e8, etc.) of the iterative training process or a set time period (e.g., 1 hour, 2 hours, 10 hours, etc.). Another example of when the PBT termination criteria are satisfied is when the candidate agent neural network falls below a certain performance threshold.

[0105] Upon meeting the PBT termination criteria, the system executes a population repository update process that includes determining updated performance for the candidate agent neural network according to updated values ​​of the network parameters for the candidate agent neural network, where the updated performance reflects a potential performance increase for the candidate agent neural network as a result of the updated network parameters.

[0106] The system also determines new values ​​of hyperparameters and network parameters for the candidate agent neural network. In some implementations, the system determines new values ​​of hyperparameters (i.e., adjusts maintained hyperparameter values ​​for the candidate agent neural network) and network parameters for the candidate agent neural network based at least on the maintained performance for the population of candidate agent neural networks in the population repository and the updated performance of the candidate agent neural network.

[0107] After determining the new network parameters, new hyperparameters, and new performance for the candidate agent neural network, the iterative training process continues. That is, the system uses the new hyperparameters and new network parameters of the candidate neural network to train the candidate agent neural network through reinforcement learning on each set of one or more newly acquired candidate tasks to iteratively generate updated network parameters for the candidate agent neural network. The system continues the iterative training process for the candidate agent neural network (and repeats the population repository update process for the candidate agent neural network) until the next PBT termination criterion is satisfied. In some implementations, the system may terminate the iterative training process for the candidate agent neural network when, for example, the PBT termination criterion is satisfied a predetermined number of times or when a performance criterion is satisfied that instructs the system to stop training; in other implementations, the system may continue the iterative training process indefinitely.

[0108] 3 shows an illustration of training a population of candidate agent neural networks. A training system, for example, training system 100 of FIG. 1, performs an iterative training process for candidate agent neural networks A-G 142A-G in a population (“Population 1”) in parallel and until a PBT termination criterion is satisfied. As the training system performs the iterative training process for candidate agent neural networks A-G 142A-G, the network parameters A-G for the candidate agent neural networks are updated accordingly.

[0109] The system updates the performance of each candidate agent neural network based on an evaluation of the candidate agent neural network's performance on each of a plurality of validation tasks. The plurality of validation tasks can be obtained by sampling from a set of validation tasks. The number of tasks in the validation set is generally much smaller than the total number of tasks in the set of candidate tasks. For example, a training data repository may store a set of one million, five million, or more candidate tasks, while the number of validation tasks is on the order of a few thousand. In some implementations, the sampling can be random sampling, while in other implementations, the sampling can alternatively be gradient sampling to ensure uniform coverage of the set of validation tasks. The updated performance for each validation task can depend on a task-specific return calculated according to a reward function defined by the selected task parameters, for example, as a cumulative measure of rewards received by the agent in response to performing one or more selected actions using the candidate agent neural network when interacting with an environment.

[0110] Unlike traditional ensemble-based training, in which performance is used directly to update the hyperparameters and network parameters for each candidate agent neural network, because evaluation tasks may vary by task complexity, scale of return, or both, the system further computes a respective PBT adaptation measure (or "adaptation measure" for short) 310 for each candidate agent neural network, which is then used to update the hyperparameters and network parameters for the candidate agent neural network. The adaptation measures 310 allow for better characterization of network performance and robustness across a set of evaluation tasks.

[0111] In particular, each fitness measure, which can be viewed as a multidimensional measure of the performance of each candidate agent neural network across multiple validation tasks, is calculated by determining a respective normalized percentile metric for the candidate agent neural network in each of the multiple validation tasks.

[0112] 7 shows an illustration of determining normalized percentile metrics. As shown, to determine, for each candidate agent neural network, a normalized percentile metric for a population of candidate agent neural networks for each of validation tasks 711, 712, 713, 714, and 715, the system may use the candidate's best performance in the validation task (e.g., best performance 701B for task 711) as a normalization constant and then normalize the candidate agent neural network's performance (e.g., candidate agent neural network performance 701A for task 711) using the normalization constant. For each candidate agent neural network, the candidate's normalized performance across multiple validation tasks (e.g., normalized candidate agent neural network performance 724C for task 714) is then sorted, e.g., arranged in monotonically ascending order (as shown by the normalized percentile curves shown in FIG. 7), from which a respective normalized percentile metric for the candidate in each validation task can be determined.

[0113] In the example of FIG. 7, the best performance of a candidate in any given validation task may be the best performance reached by the candidate agent neural network as measured by the respective candidate agent returns received by the candidate in the validation task (for single-agent tasks), or alternatively, it may be the best performance reached by one of a group of multiple candidate agent neural networks as measured by the respective candidate agent returns received by the candidate in the validation task (Nash equilibrium) (for multi-agent tasks).

[0114] The system then applies population-based training techniques to the respective adaptation measures to adjust the respective hyperparameter values ​​for each of one or more candidate agent neural networks 142A-G in the population. For example, the adaptation measures of the candidate agent neural networks are compared with each other, and in response to determining that the adaptation measure for candidate agent neural network 142D is better than the respective adaptation measure for another candidate agent neural network in the population, e.g., candidate agent neural network 142E, the system identifies candidate agent neural network 142D as the “better” performing candidate in the current population. For example, the comparison results may be based on Pareto domination. As an example herein, a candidate's performance may be considered to Pareto dominate the performance of another candidate if the candidate's adaptation measures, e.g., normalized percentile metrics, for multiple tasks, e.g., multiple validation tasks, are all at least as good as those for another candidate for the multiple tasks, e.g., multiple validation tasks, but strictly better in the adaptation measure, e.g., normalized percentile metric, for at least one of the tasks, e.g., the validation task.

[0115] In this example, the system can then determine new values ​​for the network hyperparameters for the other candidate agent neural networks by "exploiting" the network hyperparameters for candidate agent neural network 142D, i.e., adjusting the hyperparameter values ​​for the other candidate agent neural networks to be equal to the hyperparameter values ​​for candidate agent neural network 142D. In this example, the system can alternatively determine new values ​​for the network hyperparameters for the other candidate agent neural networks by "exploring" the network hyperparameters for candidate agent neural network 142D, i.e., adjusting the hyperparameter values ​​for the other candidate agent neural networks to be equal to mutated (e.g., randomly perturbed) versions of the hyperparameter values ​​for candidate agent neural network 142D.

[0116] The system similarly applies population-based training techniques to the respective adaptive measures to adjust the respective parameter values ​​for each candidate agent neural network 142A-G in the population. Continuing with the example above in which candidate agent neural network 142D was identified as the "better" performing candidate in the current population, the system can set the respective parameter values ​​for another candidate agent neural network, e.g., candidate agent neural network 142E, to be equal to the respective parameter values ​​(or mutated versions of the respective parameter values) for candidate agent neural network 142D.

[0117] As shown in Figure 3, in some implementations, the system further incorporates generational training techniques into the population-based training techniques to further improve the effectiveness and speed of RL training. In these implementations, the training of candidate agent neural networks spans a sequence of training generations, where the respective candidate agent neural networks trained in each training generation can be collectively considered as one particular population. For example, each training generation can include a certain number of iterations (e.g., 1e8, 5e8, 10e8, etc.) or a set time period (e.g., 12 hours, 24 hours, 48 ​​hours, etc.) of an iterative training process.

[0118] Although FIG. 3 shows an example implementation in which a respective population of seven candidate agent neural networks is trained at each training generation in a sequence of four training generations, there may be many more, and sometimes many times more, candidate agent neural networks trained for more or fewer training generations.

[0119] Generational ensemble-based training enables the use of policy distillation techniques during training, which generally allow a candidate agent neural network in a current training generation to bootstrap its behavior from another candidate in a immediately preceding training generation. Implementing a policy distillation technique involves, during a current training generation in a sequence of training generations during the training of a population of candidate agent neural networks, distilling from the best-performing candidate agent neural network, e.g., distilling from the best-performing candidate at the end of the previous training generation that immediately precedes the current training generation, while training the candidate agent neural network to optimize expected returns on the same or different candidate task on which a "better" performing candidate agent neural network (e.g., in terms of task-specific performance or fitness measures) from a previous training generation has already been trained, e.g., during a current training generation in a sequence of training generations during the training of a population of candidate agent neural networks.

[0120] Specifically, the system adds a preliminary policy distillation loss term to the objective function used in the RL training of the candidate agent neural network.

[0121]

number

[0122] can be incorporated here

[0123]

number

[0124] where π teacherand π are the action selection outputs of the "better" performing candidate agent neural network and the "learning" candidate agent neural network, respectively. In the example equations above, the policy loss term is masked onto the environment state and rewards are obtained, i.e., the preliminary loss (in terms of Kullback-Leibler divergence) only affects RL training at specific time steps among various training task episodes where no rewards are received.

[0125] Furthermore, in these implementations, "better" performing candidate agent neural networks (e.g., in terms of task-specific performance or fitness measures) from prior training generations can be used to control other agents in a multi-agent candidate task to increase the diversity of the RL training.

[0126] In some implementations, at least some of the candidate agent neural networks in the population corresponding to one or more of the training generations in the sequence can be trained with a self-reward play objective. This can be used to aid RL training by preventing the candidate agent neural network from achieving only the corresponding goal in a smaller number of tasks. Specifically, the self-reward play objective rewards the candidate agent neural network for satisfying goal g. After success, the candidate is rewarded for satisfying not(g) in the same environment, and this flip in goals is repeated after each satisfaction. For example, suppose goal g is to move an object toward a target location, and then not(g) is to move the object away from the target location. This can be viewed as two agents playing against themselves, where one agent must satisfy g and the other must satisfy not(g), but the agents act sequentially and are controlled by the same candidate agent neural network.

[0127] 4A illustrates an exemplary reinforcement learning agent control system 400. Reinforcement learning agent control system 400 is an example of a system in which the systems, components, and techniques described below may be implemented, implemented as a computer program on one or more computers at one or more locations.

[0128] The reinforcement learning system 400 includes an agent neural network 420 obtained as a result of the open-ended reinforcement learning training process described herein.

[0129] The reinforcement learning agent control system 400 controls the agent 408 interacting with the environment 402 by using an agent neural network 420 to select an action 410 to be performed by the agent 408 and then causing the agent 408 to perform the selected action 410.

[0130] Performance of the selected action 410 by the agent 408 generally transitions the environment 402 to a new state. By repeatedly operating the agent 408 in the environment 402, the system 400 can control the agent 408 to complete each of a number of specified tasks.

[0131] At each of a plurality of time steps, the agent neural network 420 is configured to process inputs including (i) current observation records 404 characterizing the current state of the environment 402 and (ii) goal data 406 representing goals to be satisfied to perform a task in the environment, according to trained values ​​of the network parameters, to generate an action selection output. The goals may be represented as a set of options for a respective set of predicates, where each predicate may map the current state of the environment to a corresponding reward value, and each option is, for example, a concatenation of one or more predicates such that a reward is received only if all of the predicates return a non-zero reward. In some implementations, the goal data 406 may be represented as one or more tensors of numerical values, where, for example, each predicate is represented as a multi-hot coded vector.

[0132] Specifically, in some implementations, the current observations 404 characterizing the current state of the environment 402 may include information defining the positions, orientations, velocities, etc. of different entities (e.g., target objects, obstacles, and other agents) present in the environment. In these implementations, a set of atomic predicates φ, in the form of physical relations for some or all of the entities, may be included. j may be defined, and the goal is to define a set of atomic predicates φj , and the entities are static topological building blocks such as target objects, obstacles, agents, or the ground of the environment. An example predicate might thus be "near(purple sphere, opponent)," which represents the goal of bringing one of the opponent agents closer to the purple sphere in the environment. With the set of possible predicates fixed, the agent's goal can be represented by a set of options (disjunctions) for a set of associated (or necessary) predicates for this option (conjunction). In other words, a goal is a set of options (disjunctions) for a mapping φ j :S→{0,1} d which assigns from each current state to a binary vector of d predicate truth values, where 0 indicates a predicate that is not relevant to the goal when the environment is in the state characterized by observation record S, and 1 indicates a predicate that is relevant to the goal when the environment is in the state characterized by observation record S. For example, the goal represented by goal data 406 may be

[0133]

number

[0134] This means that for some example predicates, j2 ), the purple sphere (φ j1 ) or hold the yellow ball (φ j4 ) is not held by the opposing agent (φ j3 ), while looking at the yellow ball (φ j2 ) can mean "to be near."

[0135] The system 400 then uses the action selection output to control the agent, i.e., selects an action 410 to be performed by the agent at the current time step according to the action selection output, and then causes the agent to perform the action 410, for example, by sending a control signal directly to the agent or by sending data identifying the action 410 to a control system for the agent. Exemplary action selection outputs, as well as how they may be used to select an action, are described above with reference to FIG. 1.

[0136] Agent neural network 420 is implemented with a neural network architecture that enables the network to perform its described functions. As shown in FIG. 4A , agent neural network 420 includes a state encoder neural network 430, an embedded neural network 440, an attention neural network 445, a value neural network (value neural network “V” head) 450, and a policy neural network (policy neural network “π” head) 460. That is, the value neural network may also be referred to as a value neural network head, and similarly, the policy neural network may also be referred to as a policy neural network head, and the term “head” may, but need not, indicate that both the value neural network and the policy neural network receive input shared data generated by one or more layers of attention neural network 445. Each of neural networks 420, 430, 440, 445, and 450 includes a different subset of multiple neural network layers in agent neural network 420.

[0137] The state encoder neural network 430 may include a stack of multiple convolutional layers, followed by one or more pooling layers (e.g., max-pooling layers), and followed by one or more recurrent layers (e.g., long short-term memory (LSTM) layers). The state encoder neural network 430 is configured, i.e., through training, to receive observation records 404 and process the received observation records 404 to update the hidden state of the state encoder neural network 430, i.e., to generate a current hidden state 432 that represents the current state of the environment. Typically, an embedding is an ordered set of numeric or other values ​​having a fixed dimensionality. The fixed dimensionality may depend on the actual number of LSTM neurons, e.g., 128, 256, etc., included in each LSTM layer of the state encoder neural network 430.

[0138] The embedded neural network 440 may be a fully-connected neural network, i.e., one that includes multiple fully-connected layers, optionally followed by an activation layer (e.g., a ReLU activation layer) that is configured to process the goal data 406 and the current hidden state 432 to generate (i) a predicate prediction, (ii) a goal embedding of the goal, and (iii) a respective option embedding for each of the options represented by the goal data. The embedded neural network 440 may in turn include a predicate predictor neural network, a goal embedding neural network, and an option embedding neural network, each of which includes a respective subset of the multiple fully-connected layers of the embedded neural network 440.

[0139] The predicate predictor neural network is configured to process the goal data 406 and the current hidden state 432 to generate a predicate prediction. In some implementations, the predicate prediction is based on a set of atomic predicates φ given the environment is in a state characterized by the current observation. jmay be a multi-label binary classification prediction that specifies which predicates from are associated with (e.g., included in) the goal represented in goal data 406.

[0140] The option embedding neural network is configured to process the goal data 406 and the current hidden state 432, data derived from the goal data 406 and the current hidden state 432, or both, to generate a respective option embedding for each of the options. Similarly, the goal embedding neural network is configured to process the goal data 406 and the current hidden state 432, data derived from the goal data 406 and the current hidden state 432, or both, to generate a goal embedding for the goal. Typically, an embedding is an ordered set of numeric or other values ​​having a fixed dimensionality. The fixed dimensionality may depend on the actual number of hidden units included in each fully connected layer of the embedding neural network 440, e.g., 128, 256, etc.

[0141] The attention neural network 445 is configured to process inputs including (i) the current hidden state, (ii) the goal embedding, and (iii) the predicate prediction to generate a goal attention hidden state. As used herein, an attention neural network is a neural network that includes one or more attention layers, each of which is a neural network layer that includes an attention mechanism, e.g., a scaled dot-product attention mechanism. To generate the goal attention hidden state, the attention mechanism maps a query and a set of key-value pairs to an output (the goal attention hidden state), where the query may be from (or otherwise derived from) the goal embedding and the set of key-value pairs may be derived from the current hidden state 432 (e.g., a linear projection and / or reshaped version of the current hidden state).

[0142] Further, attention neural network 445 is configured to process inputs including, for each of the options, (i) the current hidden state, (ii) a respective option embedding for the option, and (iii) a predicate prediction to generate a respective option attention hidden state for the option. To generate a respective option attention hidden state for each option, the attention mechanism similarly maps a query and a set of key-value pairs to an output (the option attention hidden state), where the query may be an option embedding for the option and the set of key-value pairs may be derived from current hidden state 432.

[0143] Value neural network 450 may be a fully connected neural network, i.e., one that includes one or more fully connected layers, optionally followed by an activation layer (e.g., a ReLU activation layer) that is configured to process the goal attention hidden states to generate goal value estimates for the goals that represent the estimated return that will be achieved, e.g., a cumulative measure of the reward that the agent would receive if it were to attempt to satisfy the goal starting from the current state. Value neural network 450 is also configured, for each of the options, to process, using the value neural network head, a respective option attention hidden state for the option to generate a respective option value estimate for the option that represents the estimated return that will be achieved if the agent were to attempt to satisfy the option starting from the current state.

[0144] The goal attention hidden state and each option attention hidden state are then combined to generate a combined hidden state to be provided to policy neural network 460, which is configured to process the combined hidden state to generate an action selection output. In some implementations, the combined hidden state may be an unweighted combination, while in other implementations, the combined hidden state may be a weighted combination, i.e., combining the goal attention hidden state and each option attention hidden state according to their respective weights.

[0145] In some of these implementations, the respective weights for the goal and for each of the options from the goal value estimate and each option value estimate for the option can be calculated by applying a softmax function to the goal value estimate and each option value estimate for the option to generate respective softmax scores for the goal and for each of the options, which are then used as weights to calculate the weighted combination. Generally, a softmax function is a function that converts numerical values ​​into probabilities. In other of these implementations, the respective weights can be calculated by assigning a weight of 1 to the highest value estimate among the goal value estimate and each option value estimate for the option, and a weight of zero to all other value estimates among the goal value estimate and each option value estimate for the option.

[0146] The policy neural network 460 may include one or more fully connected layers followed by one or more output layers. In some implementations, the one or more output layers may include a single softmax layer configured to process the output of the preceding fully connected layer to generate a probability distribution for the set of actions the agent can perform. In some other implementations, the one or more output layers may include multiple softmax layers, each corresponding to a different subset of the set of actions the agent can perform, each configured to process the output of the preceding fully connected layer to generate a respective probability distribution for the subset. In still other implementations, the one or more output layers may include one or more additional fully connected layers configured to generate a respective Q-value for each action in the set of actions the agent can perform.

[0147] FIG. 4B shows details of one example implementation of an agent neural network 420.

[0148] 5 is a flow diagram of an exemplary process 500 for controlling an agent that interacts with an environment. For convenience, process 500 is described as being performed by a system of one or more computers at one or more locations. For example, a reinforcement learning system, such as the reinforcement learning agent control system 400 of FIG. 4 , appropriately programmed, can perform process 500.

[0149] The system receives observation records characterizing the current state of the environment (step 502). For example, the observation records may include audio data segments, images, or sentences in natural language. Optionally, the observation records may also include information derived from a previous time step, such as a previous action taken, a reward received in the previous time step, or both.

[0150] The system receives goal data (step 504) that represents goals to be satisfied in order to perform a task in the environment. In the received goal data, the goals are represented as a set of options for a respective set of predicates. In the example of Figure 4, the goal is represented as a set of two options, each containing a single predicate "hold the purple ball" or "see the yellow cube," but in other examples, the goal may be represented by more options, each consisting of more predicates, and the same predicate may be shared across multiple options.

[0151] The system processes inputs including the observation records and goal data using an agent neural network to generate an action selection output (step 506), as described further below with reference to FIG. 6. In some implementations, the action selection output may define a probability distribution over a set of actions the agent can perform. In some implementations, the action selection output may include a respective Q-value for each action in the set of actions the agent can perform. In some implementations, the action selection output is an action from a continuous action space, i.e., all of the action values ​​in each action are selected from a continuous range of possible values.

[0152] The system uses the action selection output to select an action to be performed by the agent (step 508). In implementations where the action selection output defines a probability distribution, the system may select an action by sampling the actions according to the probability values ​​for the actions or by selecting the action with the highest probability value. In implementations where the action selection output includes Q-values, the system may process the Q-values ​​(e.g., using a softmax function) to generate a respective probability value for each possible action, which may be used to select the action to be performed by the agent. The system may also select the action with the highest Q-value as the action to be performed by the agent.

[0153] The system causes the agent to perform the selected action (step 510), for example, by commanding the agent or passing a control signal to a control system for the agent to perform the action.

[0154] 6 is a flowchart of an exemplary process 600 for using an agent neural network to generate an action selection output. For convenience, process 600 is described as being performed by a system of one or more computers at one or more locations. For example, a reinforcement learning system, such as the reinforcement learning system 400 of FIG. 4 , appropriately programmed, can perform process 600.

[0155] The system processes the observation records using a state encoder neural network of the agent neural network to generate a current hidden state that represents the current state of the environment (step 602). The state encoder neural network includes one or more recurrent layers and is configured to generate the current hidden state by processing the received current observation records in a manner that modifies the hidden state of the state encoder neural network that was generated by processing previous observation records.

[0156] The system processes the goal data and the current hidden state using the agent neural network's embedded neural network to generate (i) predicate predictions, (ii) goal embeddings for the goals, and (iii) respective option embeddings for each of the options represented by the goal data (step 604).

[0157] The system processes the input, including (i) the current hidden state, (ii) the goal embedding, and (iii) the predicate prediction, using an attention neural network to generate a goal-attention hidden state (step 606). The goal-attention hidden state is generated, at least in part, by applying an attention mechanism to (i) the current hidden state, (ii) the goal embedding, and (iii) the predicate prediction.

[0158] For each of the options, the system processes inputs including (i) the current hidden state, (ii) the respective option embeddings for the options, and (iii) the predicate predictions using an attention neural network to generate a respective option attention hidden state for the option (step 608). The option attention hidden state is generated, at least in part, by applying an attention mechanism to (i) the current hidden state, (ii) the respective option embeddings for the options, and (iii) the predicate predictions.

[0159] The system processes the goal-attention hidden state using the value neural network of the agent neural network to generate a goal value estimate for the goal that represents the estimated return that the agent would achieve if it attempted to satisfy the goal, starting from the current state (step 610).

[0160] For each of the options, the system processes each option attention hidden state for the option using a value neural network to generate a respective option value estimate for the option that represents the estimated return that would be achieved if the agent, starting from the current state, attempted to satisfy the option (step 612).

[0161] The system generates respective weights for the goal and for each of the options from the goal value estimate and the respective option value estimates for the options (step 614). In some implementations, generating the respective weights may include applying a softmax to the goal value estimate and the respective option value estimates for the options. In some other implementations, generating the respective weights may include assigning a weight of one to the highest value estimate of the goal value estimate and the respective option value estimates for the options, and a weight of zero to all other value estimates of the goal value estimate and the respective option value estimates for the options.

[0162] The system combines the goal attention hidden state with each option attention hidden state according to their respective weights to generate a combined hidden state (step 616).

[0163] The system uses the agent neural network's policy neural network to process the combined hidden states and generate an action selection output (step 618).

[0164] This specification uses the term "configured" in the context of systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that, when run, causes the system to perform the operation or action. One or more computer programs configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0165] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, or in tangibly embodied computer software or firmware, or in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to an appropriate receiver apparatus for execution by a data processing apparatus.

[0166] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0167] A computer program may also be called or described as a program, software, software application, app, module, software module, script, or code, and may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to be executed on one computer located at one site or on multiple computers, or it may be distributed across multiple sites and interconnected by a data communications network.

[0168] The term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components and installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine, and in other cases, multiple engines may be installed and running on the same computer or multiple computers.

[0169] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, e.g., FPGAs, or ASICs, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0170] A computer suitable for running a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to the mass storage devices to receive data from, transfer data to, or both. However, a computer need not have such devices. Moreover, a computer may be incorporated in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0171] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0172] To enable user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a reply message in return from the user.

[0173] A data processing apparatus for implementing machine learning models may also include special purpose hardware accelerator units, for example for handling the common and computationally intensive parts of machine learning training or generation, i.e., inference, workload.

[0174] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework.

[0175] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., as data servers, or that include middleware components, e.g., application servers, or that include front-end components, e.g., client computers having graphical user interfaces, web browsers, or apps through which users can interact with implementations of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0176] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device for the purpose of displaying the data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of user interaction, may be received from the device at the server.

[0177] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Also, some features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as functioning in a particular combination, and even initially claimed as such, one or more features from a claimed combination can, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or variations of the subcombination.

[0178] Similarly, although operations may be illustrated in the figures or recited in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order illustrated, or sequentially, or that all of the illustrated operations be performed to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0179] Specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0180] 100 Training System, System 102 agents, reinforcement learning agents 120 Population-Based Training (PBT) Engine 124 Task Generation Engine 130 Training Data Repository 140 Collective Repositories 142 Candidate Agent Neural Network 400 Reinforcement Learning Agent Control System, Reinforcement Learning System, System 408 Agent 420 Agent Neural Network, Neural Network 430 State Encoder Neural Network, Neural Network 440 Embedded Neural Networks, Neural Networks 445 Attention Neural Networks, Neural Networks 450 Value Neural Network, Value Neural Network "V" Head, Neural Network 460 Policy Neural Network, Policy Neural Network "π" Head

Claims

1. 1. A computer-implemented method for training an agent neural network for use in controlling agents to perform multiple tasks, the method comprising: maintaining population data specifying a population of one or more candidate agent neural networks, the population data specifying, for each candidate agent neural network in the population, (i) respective parameter values ​​for parameters of the candidate agent neural network, and (ii) respective hyperparameter values ​​for training the candidate agent neural network; training each candidate agent neural network on a respective set of one or more tasks to update the parameter values ​​of the parameters of the candidate agent neural network in the ensemble data, wherein the training step includes, for each candidate agent neural network: obtaining data identifying candidate tasks for training said candidate agent neural network; obtaining data specifying a control strategy for the candidate task; determining whether to train the candidate agent neural network at the candidate task based on one or more of: (i) performance of the candidate agent neural network at the candidate task; (ii) performance of the control strategy at the candidate task and for the candidate task; and (iii) the hyperparameter values ​​for the candidate agent neural network; training the candidate agent neural network with the candidate task such that, in response to determining to train the candidate agent neural network with the candidate task, the parameter values ​​of the parameters of the agent neural network are updated.

2. The method of claim 1 , wherein training the candidate agent neural network on the candidate task comprises training the candidate agent neural network on the candidate task through reinforcement learning.

3. 3. The method of claim 1, wherein the control strategy for the candidate task is a uniformly random action policy that takes an action uniformly randomly selected from a set of actions when controlling the agent to perform the candidate task.

4. 3. The method of claim 1, wherein the control strategy for the candidate task is a single-task strategy that uses a single-candidate-agent neural network trained only on the candidate task when controlling the agent to perform the candidate task.

5. 3. The method of claim 1 or 2, wherein the control strategy for the candidate task is a strategy that uses an instance of the candidate agent neural network but uses historical parameter values ​​for the parameters of the candidate agent neural network from an earlier point in time during the training of the candidate agent neural network when controlling the agent to perform the candidate task.

6. determining whether to train the candidate agent neural network at the candidate task based on one or more of: (i) performance of the candidate agent neural network at the candidate task; (ii) performance of the control strategy at the candidate task and for the candidate task; and (iii) the hyperparameter values ​​for the candidate agent neural network; 6. The method of claim 1, comprising, for each of a plurality of task episodes of the candidate task, using the candidate agent neural network to determine a respective candidate agent return received from controlling the agent to perform the task episode of the candidate task, and using the control strategy to determine a respective control strategy return received from controlling the agent to perform the task episode of the candidate task.

7. determining whether to train the candidate agent neural network at the candidate task based on one or more of: (i) performance of the candidate agent neural network at the candidate task; (ii) performance of the control strategy at the candidate task and for the candidate task; and (iii) the hyperparameter values ​​for the candidate agent neural network; 7. The method of claim 6, comprising determining to train the candidate agent neural network on the candidate task only when the performance of the candidate agent neural network, as measured by the respective candidate agent returns, does not exceed a threshold level of performance defined by the hyperparameter values ​​for the candidate agent neural network.

8. determining to train the candidate agent neural network on the candidate task only when the performance of the candidate agent neural network, as measured by the respective candidate agent returns, does not exceed a threshold level of performance defined by the hyperparameter values ​​for the candidate agent neural network; determining to train the candidate agent neural network on the candidate task only if the percentage of task episodes in which the respective candidate agent return exceeds a first specified value is less than a second specified value; 8. The method of claim 7, wherein (i) the first designated value, (ii) the second designated value, or (iii) both are hyperparameter values ​​for the candidate agent neural network.

9. determining whether to train the candidate agent neural network at the candidate task based on one or more of: (i) performance of the candidate agent neural network at the candidate task; (ii) performance of the control strategy at the candidate task and for the candidate task; and (iii) the hyperparameter values ​​for the candidate agent neural network; 9. The method of claim 6, comprising deciding to train the candidate agent neural network on the candidate task only when the performance of the candidate agent neural network, as measured by the respective candidate agent returns, exceeds the performance of the control strategy, as measured by the respective control strategy returns, by more than a threshold level of performance defined by the hyperparameter values ​​for the candidate agent neural network.

10. determining to train the candidate agent neural network on the candidate task only if the performance of the candidate agent neural network, as measured by the respective candidate agent returns, exceeds the performance of the control strategy, as measured by the respective control strategy returns, by more than a threshold level of performance defined by the hyperparameter values ​​for the candidate agent neural network; determining to train the candidate agent neural network on the candidate task only if the proportion of task episodes in which the respective candidate agent returns exceed the respective control strategy returns by at least a third specified value is greater than a fourth specified value; 10. The method of claim 9, wherein (i) the third designated value, (ii) the fourth designated value, or (iii) both are hyperparameter values ​​for the candidate agent neural network.

11. determining whether to train the candidate agent neural network at the candidate task based on one or more of: (i) performance of the candidate agent neural network at the candidate task; (ii) performance of the control strategy at the candidate task and for the candidate task; and (iii) the hyperparameter values ​​for the candidate agent neural network; 11. The method of claim 6, comprising deciding to train the candidate agent neural network on the candidate task only when the performance of the control strategies, as measured by the respective control strategy returns, is below a threshold level of performance defined by the hyperparameter values ​​for the candidate agent neural network.

12. determining to train the candidate agent neural network on the candidate task only when the performance of the control strategies as measured by the respective control strategy returns is below a threshold level of performance defined by the hyperparameter values ​​for the candidate agent neural network, comprising: determining to train the candidate agent neural network on the candidate task only if the average of the respective control strategy returns is less than a fifth specified value; The method of claim 11 , wherein the fifth designated value is a hyperparameter value for the candidate agent neural network.

13. After training each candidate agent neural network on its respective set of one or more tasks, 13. The method of claim 1, further comprising adjusting the hyperparameter values ​​for one or more of the candidate agent neural networks in the population.

14. The population of one or more candidate agent neural networks includes a plurality of neural networks, and adjusting the hyperparameter values ​​for one or more of the candidate agent neural networks in the population includes: calculating a respective fitness measure for each of the plurality of candidate agent neural networks measuring the respective performance of each of the plurality of candidate agent neural networks across a plurality of validation tasks; and applying a population-based training technique to the respective adaptive measures to adjust the hyperparameter values ​​for one or more of the candidate agent neural networks.

15. 15. The method of claim 14, wherein the population-based training technique also sets, for each of one or more of the candidate agent neural networks, the respective parameter values ​​for the candidate agent neural network equal to the respective parameter values ​​for another candidate agent neural network in the population data.

16. 16. The method of claim 14 or 15, wherein calculating the respective fitness measures comprises determining a respective normalized percentile metric for the candidate agent neural network in each of the plurality of validation tasks.

17. The step of applying the population-based training technique comprises: determining that each fitness measure for a first candidate agent neural network Pareto dominates each fitness measure for a second candidate agent neural network; and in response, adjusting the hyperparameter values ​​for the second candidate agent neural network to be equal to mutated versions of the hyperparameter values ​​for the first candidate agent neural network.

18. training the candidate agent neural networks on the candidate tasks such that the training of each of the candidate agent neural networks on a respective set of one or more tasks occurs during a current training generation in a sequence of training generations during the training of the agent neural networks, and updating the parameter values ​​of the parameters of the agent neural networks includes:

18. The method of any one of claims 1 to 17, comprising training the candidate agent neural network on the candidate task to optimize expected return while distilled from a best-performing candidate agent neural network in the population at the end of a previous training generation that immediately precedes the current training generation in the sequence.

19. 19. The method of claim 1, wherein during the prior training generation, the candidate agent neural networks in the population are trained for self-reward play purposes.

20. 1. A method implemented by one or more computers for controlling an agent that interacts with an environment to cause the agent to perform a task in the environment, the method comprising: receiving observations characterizing a current state of the environment; receiving goal data representing goals to be satisfied in order to perform the task in the environment, the goals being expressed as a set of options for a respective set of predicates; processing the observation records and the goal data using an agent neural network to generate an action selection output; processing the observation records using a state encoder neural network to generate a current hidden state that represents the current state of the environment; generating a goal embedding for the goal and a respective option embedding for each of the options from the goal data; using an attention neural network to process an input including the current hidden state and the goal embedding to generate a goal-attention hidden state; for each of the options, using the attention neural network to process inputs including the current hidden state and the respective option embeddings for the options to generate a respective option attention hidden state for the options; processing the goal-attention hidden state using a value neural network head to generate a goal value estimate for the goal, the goal representing an estimated return that would be achieved if the agent were to attempt to satisfy the goal starting from the current state; for each of the options, using the value neural network head to process the respective option attention hidden states for the options to generate a respective option value estimate for the options, the respective option value estimate representing an estimated return that would be achieved if the agent, starting from the current state, attempted to satisfy the options; generating respective weights for the goal and for each of the options from the goal value estimates and the respective option value estimates for the options; combining the goal attention hidden state and each of the option attention hidden states according to the respective weights to generate a combined hidden state; processing the combined hidden states using a policy neural network head to generate the action selection output; using the action selection output to select an action to be performed by the agent; causing the agent to perform the selected action.

21. 21. The method of claim 20, wherein the state encoder neural network is a recurrent neural network.

22. processing the observation records and the goal data using an agent neural network to generate an action selection output, 22. The method of claim 20 or 21, further comprising: processing the goal data and the current hidden state using a predicate predictor neural network to generate predicate predictions, wherein the input including the current hidden state and the goal embedding further includes a predicate prediction, and wherein, for each option, the input including the current hidden state and the respective option embedding further includes a predicate prediction.

23. 23. The method of claim 20, wherein generating respective weights for the goal and for each of the options from the goal value estimates and the respective option value estimates for the options comprises applying a softmax to the goal value estimates and the respective option value estimates for the options.

24. 23. The method of any one of claims 20 to 22, wherein generating respective weights for the goal and for each of the options from the goal value estimate and the respective option value estimates for the options comprises assigning a weight of one to a highest value estimate among the goal value estimate and the respective option value estimates for the options, and a weight of zero to all other value estimates among the goal value estimate and the respective option value estimates for the options.

25. 25. The method of any one of claims 20 to 24, wherein the action selection output defines a probability distribution over a set of actions that the agent can perform.

26. 25. The method of any one of claims 20 to 24, wherein the action selection output includes a respective Q-value for each action in a set of actions that the agent can perform.

27. 25. The method of any one of claims 20 to 24, wherein the action selection output is an action from a continuous action space.

28. one or more computers; and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of a respective method of any one of claims 1 to 27.

29. 28. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective methods recited in any one of claims 1 to 27.

30. 30. The method, system, or computer storage medium of claim 1, wherein the agent neural network is used in controlling the agent in a real-world environment and is configured to process observations regarding states of the real-world environment to generate an action selection output regarding an action to be performed by the agent in the real-world environment.

31. 31. The method, system, or computer storage medium of claim 30, wherein the agent is a machine agent, and the agent neural network is used in controlling the machine agent in the real-world environment to perform the task or one of the tasks.

Images