Training agent neural networks through open-ended learning

Abstract

To provide a method of training agent neural networks through open-ended learning.SOLUTION: A method includes the steps of: maintaining population data specifying a population of one or more candidate agent neural networks; and training each candidate agent neural network on a respective set of one or more tasks to update parameter values of the candidate agent neural networks in the population data. The training step comprises, for each candidate agent neural network: obtaining data identifying a candidate task; obtaining data specifying a control policy for the candidate task; determining whether to train the candidate agent neural network on the candidate task; and in response to determining to train the candidate agent neural network on the candidate task, training the candidate agent neural network on the candidate task.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Application No. 63 / 226,124, filed on 27 July 2021. The disclosures of the prior application are deemed to be part of the disclosures of this application and are incorporated by reference.

[0002] This specification relates to data processing using machine learning models. [Background technology]

[0003] A machine learning model receives input and, based on the received input, generates an output, such as a predicted output. Some machine learning models are parameterized models, which receive input and generate an output based on the values ​​of the model's parameters.

[0004] Some machine learning models are deep models that utilize multiple layers of models to generate an output for a given input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each applying a nonlinear transformation to the received input to generate the output. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] PCT Patent Application No. PCT / EP2018 / 082162 [Non-patent literature]

[0006] [Non-Patent Document 1] Shaker, Noor, et al., "Procedural content generation in games. Switzerland: Springer International Publishing," 2016. [Non-Patent Document 2] Song et al., “V-MPO: On-policy maximum a posteriori policy optimization for discrete and continuous control,” International Conference on Learning Representations, 2019. [Overview of the Initiative] [Means for solving the problem]

[0007] This specification generally describes a system implemented as a computer program on one or more computers in one or more locations for training an agent neural network used to select actions to be performed by an agent interacting with an environment. In particular, the system trains the agent neural network so that it can be used to control an agent to perform one of several tasks. Each task may include one or more of the following, for example, navigating to a specified location in the environment, identifying a specific object in the environment, or manipulating a specific object in a specified manner.

[0008] In one embodiment, a computer implementation method for training an agent neural network for use in controlling an agent to perform multiple tasks is described. The method includes the step of maintaining a population data specifying a set of one or more candidate agent neural networks, the population data specifying for each candidate agent neural network in the population (i) the respective parameter values ​​of the candidate agent neural network's parameters, and (ii) the respective hyperparameter values ​​for training the candidate agent neural network. The method then includes the step of training each candidate agent neural network in each set of one or more tasks among several common tasks, so as to update the parameter values ​​of the candidate agent neural network's parameters in the population data. In the implementation, the training steps include, for each candidate agent neural network, acquiring data to identify a candidate task (e.g., one of several tasks) for training the candidate agent neural network; acquiring data specifying a control policy for the candidate task; and deciding whether (or not) to train the candidate agent neural network on the candidate task based on one or more of the following: (i) the performance of the candidate agent neural network on the candidate task; (ii) the performance of the control policy for the candidate task on the candidate task; and (iii) hyperparameter values ​​for the candidate agent neural network. In response to the decision to train the candidate agent neural network on the candidate task, the method includes training the candidate agent neural network on the candidate task, for example using reinforcement learning, to update the parameter values ​​of the agent neural network's parameters. Training may be repeated each time data identifying a candidate task is acquired, i.e., each time one of several tasks is used as a candidate task.

[0009] In such an implementation, the training tasks consumed by the agent are dynamically generated in response to the agent's performance. The effect is that task distribution can be varied throughout training, and the task distribution itself can be optimized to improve the agent's performance. More specifically, the decision to train (or not train) a candidate agent neural network is based on a combination of the three factors listed above. For example, if performance is generally (significantly) better than the control policy, the agent should be trained only on tasks, thereby enabling the agent to perform meaningful actions, i.e., actions that affect returns. The agent's performance can generally be judged, for example, by the returns from tasks. The control policy may be any action selection policy (to control the agent to perform candidate tasks), such as a uniform random action policy or the agent's past policy.

[0010] A trained agent neural network can be used to control an agent to perform multiple tasks in one or more environments. The environment may be a real-world environment, or part or all of the training may take place in a simulation of a real-world environment. In some implementations, particularly when part or all of the training takes place in a simulation, "proposed" tasks for training a candidate agent neural network can be generated procedurally, i.e., automatically based on task parameters, and these tasks can then be used to determine whether or not to train the candidate agent neural network.

[0011] The above method can be used to obtain a population containing one or more candidate agent neural networks. When the population contains two or more candidate agent neural networks, a population of multiple candidate agent neural networks is trained. One or more can then be selected as the trained agent neural network. Within a population of multiple candidate agent neural networks, they may, but do not need to, have the same architecture as one another, and generally, they may have different respective parameter values, such as weights, and different respective hyperparameter values.

[0012] In some implementations, a set of candidate agent neural networks is used for ensemble-based training. That is, a set of candidate agent neural networks is trained, and an adaptive measure is determined for each agent, for example, to compare two (or more) of the agents. This may be done, for example, by evaluating each agent using a fitness function, and there are many suitable adaptive measures, and generally, fitness can be determined from the return from the task.

[0013] As described above, each candidate agent neural network is controlled by its own hyperparameter values. The adaptive measure can be used to modify a population of candidate agent neural networks, in particular by tuning the hyperparameter values ​​for one or more of the candidate agent neural networks, with the aim of improving the performance of the population. Tuning can be carried out in many ways. For example, the hyperparameter values ​​may be perturbed randomly.

[0014] As another example, if an adaptation measure for a first candidate agent neural network (across multiple tasks) is determined to dominate an adaptation measure of a second candidate agent neural network (across multiple tasks), the hyperparameter values for the second candidate agent neural network may be adjusted to be equal to a mutated (modified) version of the hyperparameter values for the first candidate agent neural network. Pareto domination can be considered achieved if the adaptation measure is at least good across multiple tasks and better for at least one of the tasks. The tasks may be candidate tasks or other, e.g., validation tasks.

[0015] In some implementations, a less preferred candidate agent neural network, i.e., one with a smaller adaptation measure (e.g., when comparing two agents), may have its respective parameter values (e.g., weights), and optionally its hyperparameter values as well, set to the respective parameter values, and optionally hyperparameter values, of another candidate agent neural network, e.g., one with a larger adaptation measure (e.g., when comparing two agents).

[0016] As described above, the fitness or performance of an agent can generally be determined from the return from a task. However, when there are many candidate tasks, the return can vary widely. Thus, in some implementations, calculating the adaptation (or performance) measure for a task may include, for example, determining the k-th, e.g., 50th percentile score achieved by the agent in the task, and determining each normalized percentile metric, where the score is normalized (and thus divided) using a normalization constant that is the score achieved by a mixture or set of one or more agents (candidate agent neural networks) that achieve the best score in the task. Here, the score is based on the reward or return achieved by the agent in the task. This approach avoids the need to compare performance, for example, to an optimal policy for a useful task, as it may not be straightforward to determine such an optimal policy.

[0017] The candidate agent neural network may be trained on the candidate task to optimize the predicted return from the task while being distilled from the candidate agent neural network that performs the best in the population, as determined, for example, at the end of the previous training generation that directly precedes the current training generation in the sequence. Such training during distillation may include training using a distillation loss that encourages the action selection output of the candidate agent neural network being trained towards the action selection output of the candidate agent neural network that performs the best, e.g., a loss based on the difference between these outputs.

[0018] Training may involve a self-rewarding play objective, i.e., the objective of sequentially training candidate agent neural networks to achieve goal g and not(g) (where not(g) is the negation of the goal). This can encourage exploration, making it appear as if the agents are competing with themselves. The goal can be expressed as a set of options (disjunctions) for each set of predicates, for example, representing one or more requirements for each option. The goal may or may not be expressed in natural language.

[0019] Generally, an agent neural network (and each of the candidate agent neural networks) is configured for use in controlling an agent to perform multiple tasks in one or more environments. Therefore, generally, an agent neural network (and each of the candidate agent neural networks) is configured to receive and process observational records characterizing the current state of the environment, and in some implementations, it may also be configured to receive and process data that identifies the (specific) tasks to be performed, such as goal data representing goals to be satisfied in order to perform tasks within the environment (alternatively, for example, the tasks may be inferred from the environment).

[0020] Generally, an agent neural network (and each of the candidate agent neural networks) is configured to generate an action selection output that characterizes the actions that the agent should take in response to an observation to perform one or more tasks. In an implementation, the trained agent neural network is used to select the actions that the agent should take in a real-world environment, and the selected actions relate to the actions that the agent should take in a real-world environment (and the observation record relates to the observation record of the real-world environment). For example, the agent may include a robot or a mechanical agent such as an autonomous or semi-autonomous vehicle, and other examples of agents will be given later.

[0021] The subject matter described herein may be implemented in a particular form to achieve one or more of the following advantages:

[0022] The described technique can implement an open-ended reinforcement learning training process, during which training task allocation and training objectives are dynamically adapted so that multiple instances of the agent neural network rarely stop learning, in order to effectively train the network instances sequentially, so that the agent, controlled by one or more of the trained network instances, achieves robust and generally qualified performance across a vast space of different tasks and environments. The described technique for training agent neural networks is universally applicable to any type of complex environment in which the agent may be deployed to perform any type of technically challenging task.

[0023] The techniques described thus yield generally competent agent neural networks that are not only state-of-the-art across a wide range of agent control tasks, but also surpass human-level performance, and are more generalizable, easily adaptable by fine-tuning to new tasks, including novel tasks that are naturally different from any existing tasks on which the network may have been trained. In addition to achieving improved agent performance, by training a collection of agent neural network instances across a very large number of tasks, such as an infinite multitasking continuum, the training process also consumes fewer computational resources, such as memory and processing power, than conventional methods that require training one neural network model from scratch for each single new task or new environment.

[0024] Details of one or more implementations of the subject matter described herein are shown in the accompanying drawings and the following description. Other features, embodiments, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0025] [Figure 1] This is a diagram illustrating an exemplary training system. [Figure 2] This is a flowchart illustrating an exemplary process for training a population of candidate agent neural networks. [Figure 3] This figure illustrates the training of a group of candidate agent neural networks. [Figure 4A] This figure shows an exemplary reinforcement learning agent control system. [Figure 4B] This figure shows an exemplary implementation of an agent-based neural network. [Figure 5] This is a flowchart illustrating an exemplary process for controlling an agent that interacts with its environment. [Figure 6] This is a flowchart illustrating an exemplary process for using an agent neural network to generate an action selection output. [Figure 7] This figure illustrates how to determine the normalized percentile metric. [Modes for carrying out the invention]

[0026] Similar reference numbers and names in various drawings refer to the same elements.

[0027] Figure 1 shows an exemplary training system 100. Training system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below may be implemented.

[0028] The training system 100 is a system that implements an open-ended reinforcement learning training process for training multiple instances of an agent neural network in environment 104, each of which can be used to select actions to be performed by an agent, for example, agent 102A, in order to control the agent to perform a wide range of machine learning tasks. In some cases, the machine learning task is a single-agent task that the agent itself can perform, and in other cases, the machine learning task is a multi-agent task in environment 104 that requires the agent to interact with one or more other reinforcement learning agents, for example, agents 102B-L, for example, to compete or cooperate. That is, the training system 100 takes (i.e., generates or receives) observation records, each observation record characterizing the respective state of environment 104, and in response to each observation, selects an action from a predetermined set of actions to be performed by reinforcement learning agent 102A in response to the observation. In response to some or all of the actions performed by agent 102A, the training system 100 receives a reward. Each reward is a numerical value received by agent 102A from environment 104 as a result of performing an action, and in particular, the reward differs depending on the state to which environment 104 transitions as a result of agent 102A performing an action. In some cases, a given set of actions can define an individual action space; in other cases, a given set of actions can alternatively define a contiguous action space, i.e., all action values ​​in individual actions are selected from a contiguous range of possible values; or, even further alternatively, a hybrid action space can be defined, i.e., one or more action values ​​in individual actions are selected from a contiguous range of possible values.

[0029] In particular, by implementing the open-ended reinforcement learning training process described, it becomes possible to train a single, but generally capable, agent neural network that can be used to control an agent to achieve or exceed not only state-of-the-art but also human-level performance across thousands or millions of tasks in different environments.

[0030] In some implementations, the environment is a real-world environment, the agent is a mechanical agent that interacts with the real-world environment, such as a robot or autonomous or semi-autonomous land, air, or sea vehicle that operates within or navigates the environment, and the action is an action taken by the mechanical agent in the real-world environment to perform a task. For example, the agent may be a robot that interacts with the environment to perform a specific task, such as locating a target object in the environment, moving the target object to a specified location in the environment, or navigating to a specified destination in the environment.

[0031] In these implementations, observational records may include, for example, images, object position data, and sensor data, such as from image, distance, or position sensors, or from actuators, to capture observational records of the agent interacting with the environment. For example, in the case of a robot, observational records may include data characterizing the robot's current state, such as joint position, joint velocity, joint force, torque, or acceleration, such as gravity-compensated torque feedback, and one or more of the overall or relative orientation of the members held by the robot. In the case of a robot or other mechanical agent or vehicle, observational records may similarly include position, line measure or angular velocity, force, torque, or acceleration, and one or more of the overall or relative orientation of one or more parts of the agent. Observational records may be defined in one, two, or three dimensions, and may be absolute and / or relative observational records. Observation records may include, for example, detected electronic signals such as motor current or temperature signals, and / or image or video data from, for example, a camera or LiDAR sensor, and data from, for example, the agent's sensors or sensors placed separately from the agent in the environment.

[0032] In these implementations, actions may be control signals or altitude control commands to control a robot or other mechanical agent, for example, torque on a robot's joints, or torque or altitude control commands to an autonomous or semi-autonomous land, air, or sea vehicle, for example, a vehicle's control surface or other control element, for example, a steering control element. Control signals may include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or a part of another mechanical agent. Control signals may also include, or instead of, electronic control data such as motor control data, or more generally, data for controlling one or more electronic devices in the environment whose control affects the state of the observed environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, control signals may define actions to control navigation, for example, steering, and movement, for example, braking and / or acceleration of the vehicle.

[0033] In some implementations, the environment is a simulation of the real-world environment described above, and the agent is implemented as one or more computers interacting with the simulated environment. For example, the simulated environment may be a simulation of a robot or a vehicle, and a reinforcement learning system is trained in that simulation and then, once trained, can be used in the real world.

[0034] In some implementations, the environment is a real-world manufacturing environment for producing products such as chemical, biological, or mechanical products, or food. As used herein, “manufacturing” a product includes refining starting materials to create a product, or processing starting materials to remove contaminants, for example, to produce a cleaned or recycled product. A manufacturing plant may comprise multiple manufacturing units, such as containers for chemical or biological substances, or machines for processing solid or other materials, such as robots. The manufacturing units are configured such that intermediate versions or components of a product can be moved between manufacturing units during the manufacturing of the product, for example, via pipes or mechanical conveyors. As used herein, manufacturing a product also includes the manufacturing of food by kitchen robots.

[0035] The agent may include an electronic agent configured to control a manufacturing unit, or a machine such as a robot that operates to manufacture a product. In other words, the agent may comprise a control system configured to control the manufacture of a chemical, biological, or mechanical product. For example, the control system may be configured to control one or more of the manufacturing units or machines, or to control the movement of intermediate versions or components of a product between manufacturing units or machines.

[0036] For example, tasks performed by an agent may include tasks to manufacture a product or an intermediate version or component of it. Another example is tasks to control, for example, the use of resources, such as controlling power consumption, water consumption, or the consumption of any materials or consumables used in the manufacturing process, or minimizing such use.

[0037] Actions may include control actions to control the use of a machine or manufacturing unit to process solid or liquid materials to produce a product, or an intermediate or component thereof, or to control the movement of intermediate versions or components of a product within a manufacturing environment, for example, between manufacturing units or machines. Generally, actions may be any action that has an effect on the observed state of the environment, and may, for example, be configured to adjust any of the detected parameters described below. These may include actions to adjust the physical or chemical conditions of a manufacturing unit, or actions to control the movement of mechanical parts of a machine or joints of a robot. Actions may include actions that impose operating conditions on a manufacturing unit or machine, or actions that bring about changes to settings for adjusting, controlling, or turning on / off the operation of a manufacturing unit or machine.

[0038] Rewards or returns may be related to metrics of task performance. For example, in the task of manufacturing a product, metrics may include the quantity of product manufactured, the quality of the product, the speed of production, or, in relation to the physical costs of carrying out the manufacturing task, metrics such as the amount of energy, materials, or other resources used to carry out the task. In the task of controlling resource use, metrics may include any metric of resource use.

[0039] In general, observational records of environmental conditions may include any electronic signals that represent the functionality of the electronic and / or mechanical components of the equipment. For example, a representation of environmental conditions may be derived from observations made by a sensor detecting the condition of the manufacturing environment, for example, a sensor detecting the condition or configuration of a manufacturing unit or machine, or a sensor detecting the movement of material between manufacturing units or machines. As some examples, such sensors may be image or video sensors for capturing image or video observational records of manufacturing units or machines or any other suitable type of sensor, configured to detect mechanical motion or force, electrical conditions such as current, voltage, frequency, impedance, quantity, level, flow rate / movement rate or flow path / transmission path of one or more materials, physical or chemical conditions such as physical condition, shape or configuration or chemical condition such as pH, the configuration of a unit or machine, such as the mechanical configuration of a unit or machine, or valve configuration, and may be image or video sensors for capturing image or video observational records of manufacturing units or machines or motion. In the case of machines such as robots, observational records from sensors may include observational records of the position, linear measure or angular velocity, force, torque or acceleration, or attitude of one or more parts of the machine, for example, data characterizing the current state of the machine or robot, or of a component held or processed by the machine or robot. Observational records may also include detected electronic signals, such as motor current or temperature signals, or image or video data from, for example, a camera or LiDAR sensor. Such sensors may be part of an agent in the environment or located separately from it.

[0040] In some implementations, the environment is the real-world environment of a service facility containing multiple components of electronic equipment, such as a server farm or data center, e.g., a telecommunications data center or a computer data center, for storing or processing data. The service facility may also include auxiliary control equipment that controls the operating environment of the components of the equipment, e.g., environmental control equipment such as temperature control equipment, e.g., cooling equipment, or airflow control equipment or HVAC equipment. Tasks may include tasks that control resource usage, e.g., tasks to minimize it, such as tasks to control power consumption or water consumption. Agents may include electronic agents configured to control the operation of components of the equipment, or auxiliary agents, e.g., the operation of environmental control equipment.

[0041] In general, an action may be any action that has an effect on the observed state of the environment, for example, an action configured to adjust any of the detected parameters described below. These may include actions to control components of equipment or auxiliary control devices, or to impose operating conditions on them, for example, actions that bring about a change in the settings for adjusting, controlling, or turning on / off the operation of components of equipment or auxiliary control devices.

[0042] In general, observational records of environmental conditions may include any electronic signals that represent the functionality of a facility or equipment within a facility. For example, a representation of environmental conditions may be derived from observations made by any sensors that detect the physical environmental conditions of a facility, or from observations made by any sensors that detect the condition of one or more components of equipment or one or more components of auxiliary control equipment. These include sensors configured to detect electrical conditions such as current, voltage, power, or energy; facility temperature; fluid flow, temperature, or pressure within the facility or its cooling system; or physical facility configurations such as whether a vent is open.

[0043] Rewards or returns may be related to metrics of task performance. For example, in the case of a task to control resource usage, such as minimizing it, the metrics could include any metric of resource usage, such as controlling the use of electricity or water.

[0044] In some implementations, the environment is the real-world environment of a power generation facility, such as a renewable power generation facility like a solar or wind power plant. Tasks may include control tasks to control the power generated by the facility to meet demand or reduce the risk of mismatches between elements of the distribution grid, such as controlling the supply of power to the distribution grid, or maximizing the power generated by the facility. Agents may include electronic agents configured to control the coupling of power generated by the facility or the generated power to the distribution grid. Actions may include actions to control the electrical or mechanical configuration of a generator, such as the configuration of a wind turbine, or the configuration of one or more solar panels or mirrors, or the electrical or mechanical configuration of a rotary generator, such as the electrical or mechanical configuration of one or more renewable power generation elements. Mechanical control actions may include actions to control the conversion from energy input to electrical energy output, such as the efficiency of the conversion or the degree of coupling between energy input and electrical energy output. Electrical control actions may include actions to control one or more of the voltage, current, frequency, or phase of the generated power.

[0045] Rewards or returns may relate to metrics of task performance. For example, in a task to control the supply of power to a power grid, metrics may relate to a measure of the power transferred, or a measure of electrical mismatches between the power generation facility and the grid, such as voltage, current, frequency, or phase mismatches, or a measure of power or energy loss at the power generation facility. In a task to maximize the supply of power to a power grid, metrics may relate to a measure of the power or energy transferred to the grid, or a measure of power or energy loss at the power generation facility.

[0046] In general, observational records of environmental conditions may include any electronic signals that represent the electrical or mechanical functionality of power generation equipment within a power generation facility. For example, a representation of environmental conditions may be derived from observations made by any sensors that detect the physical or electrical condition of equipment within a power generation facility that is generating electricity, or the physical environment of such equipment, or the conditions of auxiliary equipment supporting the power generation equipment. Such sensors may include, for example, sensors configured to detect observational records of the electrical conditions of equipment, such as current, voltage, power, or energy, the temperature or cooling of the physical environment, fluid flow, or the physical configuration of equipment, and the electrical conditions of the power grid, from local or remote sensors. Observational records of environmental conditions may also include one or more predictions regarding future conditions for the operation of power generation equipment, such as predictions of future wind levels or solar irradiance, or predictions of future electrical conditions of the power grid.

[0047] In another example, the environment may be a chemosynthesis or protein folding environment, where each state is the respective state of a protein chain, or one or more intermediates or chemical precursors, and the agent is a computer system for determining how to fold the protein chain or synthesize a chemical. In this example, the action is a possible folding action for folding the protein chain or an action for assembling chemical precursors / intermediates, and the outcome to be achieved may include, for example, folding the protein so that it is stable and achieves a specific biological function, or providing an effective synthetic pathway for a chemical. In another example, the agent may be a mechanical agent that automatically performs or controls a protein folding action or chemosynthesis step selected by the system without human interaction. The observational record may include direct or indirect observational records of the state of the protein or chemical / intermediate / precursor, and / or may be derived from simulations.

[0048] Similarly, the environment may be a drug design environment in which each state is a potential state of a pharmaceutical chemical, and the agent is a computer system for determining the elements of the pharmaceutical chemical and / or the synthesis route for the pharmaceutical chemical. The drug / synthesis may be designed based on rewards derived from the drug's target, for example, in a simulation. As another example, the agent may be a mechanical agent that carries out or controls the synthesis of the drug.

[0049] In some further applications, the environment is a real-world environment, and the agent manages the distribution of tasks across computing resources, for example, on mobile devices and / or in a data center. In these implementations, actions may include assigning tasks to specific computing resources.

[0050] As further examples, an action may include presenting an advertisement, observational records may include impressions or click-through counts or rates of an advertisement, and rewards may characterize previous selections of items or content made by one or more users.

[0051] In some cases, observation records may include text or spoken commands provided to the agent by a third party (e.g., the agent's operator). For example, the agent may be an autonomous vehicle, and the user of the autonomous vehicle may provide the agent with text or spoken commands (e.g., to navigate to a specific location).

[0052] As another example, the environment may be an electrical, mechanical, or electromechanical design environment, for example, an environment in which the design of an electrical, mechanical, or electromechanical entity is simulated. The simulated environment may be a simulation of a real-world environment in which the entity is intended to work. The task may be to design the entity. Observations may include observations that characterize the entity, i.e., observations of the entity's mechanical shape, or its electrical, mechanical, or electromechanical configuration, or observations of the entity's parameters or properties. Actions may include actions that modify the entity, for example, actions that modify one or more of the observations. Rewards or returns may include one or more metrics of the performance of the entity's design. For example, rewards or returns may relate to one or more physical properties of the entity, such as weight or strength, or to one or more electrical properties of the entity, such as a measure of efficiency in performing a particular function for which the entity is designed. The design process may include outputting the design for manufacture, for example, in the form of computer executable instructions for manufacturing the entity. The process may include making the entity according to the design. Therefore, the entity design may be optimized, for example, by reinforcement learning, and then the optimized design can be output, for example, as a computer executable instruction, to manufacture the entity, and the entity having the optimized design can then be manufactured.

[0053] As mentioned above, the environment may be a simulated environment. Generally, in the case of a simulated environment, the observation record may include a simulated version of one or more of the aforementioned observation records or observation record types, and the action may include a simulated version of one or more of the aforementioned actions or action types. For example, the simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle that navigates through the motion simulation. In these implementations, the action may be a control input for controlling a simulated user or a simulated vehicle. Generally, the agent may be implemented as one or more computers that interact with the simulated environment.

[0054] The simulated environment may be a simulation of a specific real-world environment and agent. For example, the system may be used to select actions in the simulated environment during the system's training or evaluation, and after training or evaluation or both are complete, it may be deployed to control a real-world agent in the specific real-world environment that was the subject of the simulation. This can avoid unnecessary wear and tear on the real-world environment or the real-world agent and may allow the control neural network to be trained and evaluated based on situations in the real world that rarely occur, are difficult to reproduce, or are unsafe. For example, the system may be partially trained using a simulation of a machine agent in a simulation of a specific real-world environment, and then deployed to control a real machine agent in that specific real-world environment. In such a case, the observational records of the simulated environment are related to the real-world environment, and the selected actions in the simulated environment are related to actions that should be performed by the machine agent in the real-world environment.

[0055] Optionally, in any of the above implementation forms, observation records at any given time step may include data from previous time steps, such as actions performed in the previous time step, rewards received in the previous time step, or both, which may be useful in characterizing the environment.

[0056] In general, an agent neural network may have any suitable neural network architecture that enables it to perform its described function, for example, processing data to identify a task to be performed and observations to generate an action selection output that characterizes the action to be performed by the agent in response to the observation. For example, an agent neural network may include any suitable number of layers (e.g., 5 layers, 10 layers, or 25 layers) connected in any suitable configuration (e.g., as a linear sequence of layers with or without residual connections) and of any suitable type (e.g., fully connected layers, convolutional layers, attention layers, transformer layers, etc.).

[0057] For example, the action selection output may include a numerical probability value for each action in a set of possible actions that the agent can perform. The system can select the action to be performed by the agent, for example, by sampling actions according to their probability values, or by selecting the action with the highest probability value.

[0058] In another example, the action selection output could directly define the action to be performed by the robot agent, for example, by defining the torque value that should be applied to the robot agent's joints.

[0059] In another example, the action selection output might include the respective Q-values ​​for each action in the set of possible actions that the agent can perform. The system can process the Q-values ​​(for example, using a softmax function) to generate the respective probability values ​​for each possible action, which can then be used to select the action that the agent should perform (as previously described). The system could also select the action with the highest Q-value as the action that the agent should perform.

[0060] The Q-value for an action is an estimate of the "return" to the current observation record, resulting from the agent performing an action and then agent 102 selecting a future action to perform according to the current values ​​of the agent neural network parameters.

[0061] Return refers to the cumulative measure of the "reward" an agent receives, for example, the total time-discounted reward. An agent can receive a reward at each time step, which is specified by a scalar number and characterizes, for example, the agent's progress in completing an assigned task.

[0062] In some cases, the system can select the action that the agent should perform according to the search policy. For example, the search policy may be a greedy search policy, and the system selects the action that the agent should perform according to the action selection output with probability 1-, and randomly selects an action with probability ∈. In this example, ∈ is a scalar value between 0 and 1.

[0063] One specific example of an architecture for agent neural networks is further described with reference to Figure 4.

[0064] The training system 100 maintains a population repository 140 that stores population data specifying a group of one or more candidate agent neural networks 142A-M. The population repository 140 is implemented as one or more logical memory devices in one or more physical locations, or as a logical memory space allocated to one or more memory devices in one or more physical locations. At any point during training, the population repository 140 stores data specifying the current group of candidate agent neural networks 142A-M.

[0065] In particular, the population repository 140 stores a set of retention values ​​that define each candidate agent neural network 142A-M in the current population. The set of retention values ​​includes network parameters and hyperparameters, and in implementations, is also convenient for maintaining a performance measure (or simply "performance") for each candidate agent neural network 142A-M on different tasks. For example, for candidate agent neural network A 142A, the set of retention values ​​includes network parameters A 144A, hyperparameters A 146A, and performance A 148A (which may include the respective performance for different tasks). If there are multiple candidate agent neural networks in the population, each candidate agent neural network will generally have the same architecture, but with different parameter values, and possibly different hyperparameter values ​​from other candidate agent neural networks in the population.

[0066] Hyperparameters for a candidate agent neural network are values ​​that influence how network parameter values, such as weights, are updated during training. Hyperparameters may include discount factors, learning rates, objective function values, or weights, which are assigned to various terms of the objective function. Furthermore, hyperparameters may include one or more specified values ​​used to determine whether to train the candidate agent neural network on a given candidate task. Thus, generally speaking, in the systems described, hyperparameters control parameter learning (which can be done, for example, by backpropagating the gradients of one or more objective functions).

[0067] To train the agent neural network, the training system 100 also maintains a training data repository 130 that stores training data for training candidate agent neural networks 142A-M. The training data repository 130 is implemented as one or more logical memory devices in one or more physical locations, or as a logical memory space allocated to one or more memory devices in one or more physical locations. The training data repository 130 stores data defining a set of candidate tasks 132A-N on which the candidate agent neural network can be trained, and corresponding control policies 133A-N for each candidate task, which can be used to control the agent to perform the candidate task, for example, control policy 133A for task A 132A. The training data repository 130 optionally stores data defining a set of validation tasks for evaluating the performance of the candidate agent neural networks 142A-M in validation tasks.

[0068] The training system 100 can receive data in one of several ways. For example, the system 100 can receive data defining a set of candidate tasks and / or validation tasks as an upload from a remote user of the system via a data communication network (for example, using an application programming interface (API) made available by the training system 100). As another example, the training system 100 can receive input from a user specifying which data already maintained by the training system 100 should be used as data defining a set of candidate tasks and / or validation tasks.

[0069] In some implementations, the training data repository 130 remains fixed throughout the training process, while in other implementations, the training data repository 130 expands (for example, to infinity) or shrinks as the training process iterations are performed, allowing for dynamic addition of new candidate tasks to the repository and discarding existing tasks where a given number of candidate agent neural networks have achieved threshold performance. In these other implementations, the training system 100 can use a task generation engine 124 to generate new tasks from the space of candidate tasks, for example, to generate a multitask continuum, in each of the multiple training iterations throughout the training process. This space of candidate tasks can be parameterized by a set of task parameters, each associated with one or more values, such as scores, individual values, or continuous values. For example, the task generation engine 124 can generate tasks randomly, for example, by randomly selecting different task parameter values ​​from a set of task parameters. Next, in order to actually generate the tasks according to these selected task parameter values, the training system 100 may use, in addition to or instead of, other known digital content generation techniques, any of the exemplary techniques described in Shaker, Noor et al., "Procedural content generation in games. Switzerland: Springer International Publishing," 2016.

[0070] Generally, a set of task parameters may include any of a variety of tunable parameters that can collectively define the space of a candidate task. For example, a set of task parameters may include a first set of environment parameters that define different properties or characteristics of the environment in which the agent interacts. More specifically, environment parameters may define or otherwise specify the static topology of the environment (e.g., the layout and structure of the environment's topological building blocks), the lighting of the environment, the type or format of observational records characterizing the state of the environment that can be provided to the agent, the properties of dynamic objects present in the environment (e.g., the location and physical properties of target objects or obstacles), a predetermined set of actions that the agent can take in response to receiving observational records, the number of other agents present in the environment (in the case of competitive or collaborative tasks), or a combination thereof, and possibly more.

[0071] As another example, a set of task parameters may include a second set of goal parameters that define or otherwise specify different goals for each of one or more agents to achieve when interacting with the environment. Goal parameters may include, for example, various terms of a reward function used to calculate the reward (expressed numerically) that should be issued to the agent in response to different actions the agent takes when interacting with the environment. The reward generally reflects the progress the agent is making toward achieving the goals specified for the candidate task it is configured to perform.

[0072] The training system 100 may use a population-based training (PBT) engine 120 to train a population of one or more candidate agent neural networks 142A-M using a population-based training technique described in detail in PCT Patent Application No. PCT / EP2018 / 082162, which is incorporated herein by reference. As part of the training, the PBT engine 120 trains each candidate agent neural network in each set of one or more tasks to update the parameter values ​​of the candidate agent neural network parameters in the population repository 140. However, unlike conventional population-based training, before training a given candidate agent neural network on any given candidate task, the PBT engine 120 obtains data identifying candidate tasks for training the candidate agent neural network and data specifying control policies for the candidate tasks.

[0073] In particular, the PBT engine 120 determines whether to train the candidate agent neural network on a candidate task based on one or more of the following: (i) the performance of the candidate agent neural network on the candidate task (i.e., when used according to the parameter values ​​currently stored in the collective data for the candidate agent neural network), (ii) the performance of the control policy for the candidate task on the candidate task, and (iii) the hyperparameter values ​​for the candidate agent neural network. Generally, given the current learning stage, i.e., given the current values ​​of the candidate's parameters stored in the collective data, if the PBT engine determines that the task is "useful" for the candidate, the PBT engine 120 will simply train the candidate agent neural network on the given task. The PBT engine 120 can make this determination in part based on the performance of the control policy relative to the performance of the candidate.

[0074] In response to a decision to train the candidate agent neural network on the candidate task, the PBT engine 120 trains the candidate agent neural network on the candidate task to update the parameter values ​​of the agent neural network's parameters. In response to a decision not to train the candidate agent neural network on the candidate task, the system refrains from training the candidate agent neural network on the candidate task.

[0075] The use of a control policy thus improves the effectiveness and efficiency of the training process, for example, by allowing the PBT engine 120 to train a candidate agent neural network on a candidate task only when the performance of the candidate agent neural network is significantly better than the performance of the control policy. In some implementations, for each candidate task, for example task A 132A, the corresponding control policy, for example control policy 133A, may be a fixed policy, for example, a random action policy that randomly selects an action, while in other implementations, the corresponding control policy may be a policy controlled by candidates from previous training generations, i.e., according to the historical network parameter values ​​of the candidates from an early point in training. In these other implementations, the use of a control policy allows the PBT engine 120 to further determine whether the performance of the candidate agent neural network has recently improved or deteriorated compared to previous training iterations, and to take appropriate measures accordingly, such as adjusting hyperparameters or simply removing it from the current population.

[0076] In some implementations, the PBT engine 120 keeps the hyperparameters for the candidates fixed during training. In some other implementations, the PBT engine 120 adjusts the hyperparameters for the candidates during training to ensure that the dynamic task selection described above continues to select only tasks that are useful for learning as the candidate parameter values ​​change throughout the training process.

[0077] Some or all of the tasks for training candidates may be multi-agent tasks, i.e., tasks requiring an agent controlled using a candidate to interact with one or more other agents controlled using a different policy. For these tasks, the system may use any of a variety of policies to control one or more other agents. For example, the system may use a fixed policy, such as a random action policy that randomly selects an action, a no-action policy where other agents do not perform any action, an expert policy that represents the behavior of an expert agent, or a policy controlled by high-performance candidates from previous training generations.

[0078] After training, the training system 100 may select one of the candidates from the population to use as the agent neural network, that is, to use in controlling the agent to perform a new task. For example, after training is complete, the training system 100 may select the best-performing candidate. Alternatively, the training system 100 may use an ensemble of multiple candidates from the population as the final agent neural network.

[0079] Figure 2 is a flowchart of an exemplary process 200 for training a population of candidate agent neural networks. For convenience, process 200 is described as being performed by a system consisting of one or more computers in one or more locations. For example, a population-based neural network training system appropriately programmed according to this specification, such as the training system 100 in Figure 1, can perform process 200.

[0080] As described above, the system maintains population data that specifies a group of one or more candidate agent neural networks. For each candidate agent neural network in the group, the population data specifies (i) the parameter values ​​for each parameter of the candidate agent neural network, (ii) the hyperparameter values ​​for training the candidate agent neural network, and (iii) the performance for the candidate agent neural network (which may include performance for each different task).

[0081] The system repeatedly performs process 200 on each candidate neural network in the group. In some implementations, the system repeatedly performs process 200 in parallel for each candidate neural network, and asynchronously with respect to each other candidate neural network in the group.

[0082] The system acquires data that identifies candidate tasks for training a candidate agent neural network (step 202). Generally, the system acquires different candidate tasks in different iterations. Different tasks may involve different environments in which the agent controlled by the candidate agent neural network interacts, different goals that the agent must achieve when interacting with the environment, and different numbers of other agents present in the environment (in the case of competitive or cooperative tasks).

[0083] In some implementations, candidate tasks may be new tasks automatically and dynamically generated by the system (i.e., throughout the training process) according to different sets of task parameters sampled from the space of candidate tasks. In other implementations, candidate tasks may be obtained by sampling a certain number of candidate tasks from a set of candidate tasks contained in the training data repository in each iteration. In some of these implementations, the system may use the same task sampling strategy for all candidate agent neural networks in the population, while in others, the system may use different candidate-specific task sampling strategies for the population. Furthermore, in implementations where the system maintains data specifying a list of historical candidate tasks on which the candidate agent neural networks have previously been trained, the system may update the list to include the obtained candidate tasks.

[0084] The system obtains data specifying a control policy for the candidate task (step 204). Generally, the control policy may be any action selection policy that can be used to control an agent (potentially including other agents in the environment) to interact with the environment. For example, a control policy for a candidate task is a uniform random action policy in which, when controlling an agent to perform the candidate task, the agent performs an action uniformly randomly selected from a given set of actions. Another example is a single-task policy in which, when controlling an agent to perform the candidate task, the agent uses a single candidate agent neural network trained only on the candidate task. Another example is a control policy in which, when controlling an agent to perform the candidate task, the agent uses an instance of the candidate agent neural network, but uses historical parameter values ​​for the network parameters of the candidate agent neural network from an early point in time during the training of the candidate agent neural network.

[0085] The system determines whether to train the candidate agent neural network on the candidate task based on one or more of the following: (i) the performance of the candidate agent neural network on the candidate task, (ii) the performance of the control policy on the candidate task, and (iii) hyperparameter values ​​for the candidate agent neural network (step 206).

[0086] Specifically, to make this decision, the system can use a planning algorithm, such as the Monte Carlo Tree Search (MCTS) algorithm or another lookahead planning algorithm, to predict several future states of the environment after its initial state at a given time step. These are called task episodes. A task episode represents the rollout of the environment at a time after a given time step, assuming that the agent performs a specific action selected using a candidate agent neural network. The system can run a planning algorithm to iteratively generate multiple task episodes for a candidate task, for example, five, ten, twenty, or more task episodes.

[0087] A task episode refers to a sequence of time steps in which an agent interacts with its environment. A task episode may end, for example, when the agent has interacted with its environment for a predetermined number of time steps, or when the agent has completed a task.

[0088] In some implementations, each task episode may include a sequence of multiple trajectories, each trajectory of which may include a current observation characterizing each current state of the environment, a current action performed by the agent in response to the current observation, a next state characterizing each subsequent state of the environment, and a reward received by the agent in response to performing the current action.

[0089] For each of the multiple task episodes of a candidate task, the system uses a candidate agent neural network to determine each candidate agent return received by controlling the agent to perform the task episode of the candidate task. The system also uses a control policy to determine each control policy return received by controlling the agent to perform the task episode of the candidate task. Both candidate agent returns and control policy returns may be task-specific returns calculated according to a reward function defined by selected task parameters, for example, as a cumulative measure of the rewards an agent receives in response to performing one or more actions when interacting with the environment.

[0090] In some implementations, the system decides to proceed with training the candidate agent neural network on the candidate task only when the performance of the candidate agent neural network, as measured by each candidate agent's return, does not exceed a performance threshold level defined by the hyperparameter values ​​for the candidate agent neural network.

[0091] In these implementations, the system may decide to train the candidate agent neural network on a candidate task only if the proportion of task episodes in which each candidate agent return exceeds a first specified value is less than a second specified value. For example, the first specified value may be an integer or floating-point value within the possible range of total rewards the agent can receive, and the second specified value may be a decimal value between zero and one, where the first specified value, the second specified value, or both are hyperparameter values ​​for the candidate agent neural network.

[0092] An example of this criterion in mathematical terms is: Pr[R π (x>m s ] <m solved It is acceptable for R to be such that in the formula, π (x) is the return of each candidate agent, and m s and m solved These are the first and second specified values, respectively.

[0093] In some implementations, the system decides to train a candidate agent neural network on a candidate task only when the performance of the candidate agent neural network, as measured by each candidate agent's return, exceeds the performance of the control policy, as measured by each control policy's return, by a performance threshold level defined by the hyperparameter values ​​for the candidate agent neural network.

[0094] In these implementations, the system may decide to train the candidate agent neural network on a candidate task only if the proportion of task episodes in which each candidate agent return exceeds the respective control policy return by at least a third specified value is greater than the fourth specified value. For example, the third specified value may be an integer or floating-point value within the possible range of total rewards the agent can receive, and the fourth specified value may be a decimal value between zero and one, where the third specified value, the fourth specified value, or both are hyperparameter values ​​for the candidate agent neural network. The third and fourth specified values ​​may be the same as or different from the first and second specified values, respectively.

[0095] An example of this criterion in mathematical terms is:

[0096]

number

[0097] It is acceptable for R to be such that in the formula, π (x) is the return of each candidate agent, and R πcont (x) is the return of each control policy, and m> and m >cont are the third and fourth specified values, respectively.

[0098] In some implementations, the system determines to train the candidate agent neural network on a candidate task only when the performance of the control policy measured by each control policy return is lower than a performance threshold level defined by hyperparameter values for the candidate agent neural network. That is, the system determines to train the candidate agent neural network on a candidate task that requires a control policy that is, for example, more sophisticated or advanced than a uniform random action policy to achieve the specified goal of the candidate task.

[0099] In these implementations, the system may determine to train the candidate agent neural network on a candidate task only when the average (or median, or other representative value) of each control policy return is less than a fifth specified value. For example, the fifth specified value may be an integer or floating point value within the range of possible total rewards that an agent can receive, where the fifth specified value is a hyperparameter value for the candidate agent neural network. The fifth specified value may be the same as or different from the first or third specified values.

[0100] An example of this criterion in mathematical notation is

[0101]

Number

[0102] which may be, where V πcont (x) is the expected return of each control policy, and m cont is the fifth specified value.

[0103] In any of the above implementations, in response to the decision to train the candidate agent neural network on the candidate task, the system trains the candidate agent neural network on the candidate task by using any appropriate reinforcement learning technique to update the maintenance values ​​of the agent neural network's network parameters, i.e., by optimizing the objective function determined by the rewards received from the environment by the agent controlled using the candidate agent neural network (step 208). For example, the reinforcement learning technique may be an on-policy RL training technique, such as one of the RL algorithms described in more detail in Song et al., "V-MPO: On-policy maximum a posteriori policy optimization for discrete and continuous control," International Conference on Learning Representations, 2019, and the system can train the candidate agent neural network on the candidate task for a certain number of iterations or a set time period.

[0104] Applied in parallel with reinforcement learning techniques is the population-based training technique. Specifically, the system uses retention values ​​for the candidate agent neural network's hyperparameters and network parameters to train the candidate neural network on each of one or more sets of candidate tasks, iteratively generating updated network parameters for the candidate agent neural network until the PBT termination criteria are met. The PBT termination criteria are one or more set conditions, which, when met by the candidate agent neural network, cause the system to update the repository for the candidate agent neural network with new network parameters, new hyperparameters, and new performance measures. An example of when the PBT termination criteria are met is when the candidate agent neural network has been trained on each of one or more sets of candidate tasks for a certain number of iterations of the iterative training process (e.g., 1e4, 1e6, 1e8, etc.) or a set time period (e.g., 1 hour, 2 hours, 10 hours, etc.). Another example of when the PBT termination criteria are met is when the candidate agent neural network falls below a certain performance threshold.

[0105] Upon meeting the PBT termination criteria, the system performs a collective repository update process, which includes determining the updated performance for the candidate agent neural network according to the updated values ​​of the network parameters for the candidate agent neural network. The updated performance reflects the potential performance increase of the candidate agent neural network as a result of the updated network parameters.

[0106] The system also determines new values ​​for hyperparameters and network parameters for the candidate neural network. In some implementations, the system determines new values ​​for hyperparameters (i.e., adjusting the maintained hyperparameter values ​​for the candidate agent neural network) and network parameters for the candidate agent neural network based at least on the maintained performance of the population of candidate agent neural networks in the population repository and the updated performance of the candidate agent neural network.

[0107] After determining new network parameters, hyperparameters, and performance for the candidate agent neural network, the iterative training process continues. That is, the system uses the new hyperparameters and network parameters of the candidate agent neural network to train it through reinforcement learning on one or more newly acquired set of candidate tasks, iteratively generating updated network parameters for the candidate agent neural network. The system continues the iterative training process for the candidate agent neural network until the next PBT termination criterion is met (and the system also repeats the collective repository update process for the candidate agent neural network). In some implementations, the system may terminate the iterative training process for the candidate agent neural network, for example, when the PBT termination criterion has been met a predetermined number of times, or when the performance criterion is met to instruct the system to stop training; in other implementations, the system may continue the iterative training process indefinitely.

[0108] Figure 3 illustrates the training of a population of candidate agent neural networks. A training system, for example, training system 100 in Figure 1, performs an iterative training process for candidate agent neural networks A-G in a population ("Population 1") in parallel and until the PBT termination criteria are satisfied. As the training system performs the iterative training process for candidate agent neural networks A-G, the network parameters A-G for the candidate agent neural networks are updated accordingly.

[0109] The system updates the performance of each candidate agent neural network based on an evaluation of its performance on each of several validation tasks. Multiple validation tasks can be obtained by sampling from a set of validation tasks. The number of tasks in a validation set is generally much smaller than the total number of tasks in a set of candidate tasks. For example, a training data repository may store a set of candidate tasks numbering in the millions, 5 million, or more, while the number of validation tasks is only in the thousands. In some implementations, sampling may be random sampling, while in others, sampling may be sloped sampling to ensure uniform coverage of the set of validation tasks. The updated performance for each validation task may be determined, for example, by task-specific returns calculated according to a reward function defined by selected task parameters, as a cumulative measure of the reward an agent receives in response to performing one or more selected actions using the candidate agent neural network when interacting with the environment.

[0110] Unlike traditional population-based training, where performance is directly used to update hyperparameters and network parameters for each candidate agent neural network, the evaluation tasks can vary by task complexity, return scale, or both. Therefore, the system further computes a unique PBT adaptive measure (or, simply, "adaptive measure") 310 for each candidate agent neural network, which is then used to update the hyperparameters and network parameters for that network. The adaptive measure 310 allows for better characterization of network performance and robustness across a set of evaluation tasks.

[0111] In particular, each adaptive measure, which can be considered a multidimensional measure of the performance of each candidate agent neural network across multiple validation tasks, is calculated by determining the respective normalized percentile metric for the candidate agent neural network in each of the multiple validation tasks.

[0112] Figure 7 illustrates how to determine the normalized percentile metric. As shown, for each candidate agent neural network, in order to determine the normalized percentile metric for the group of candidate agent neural networks for each of the validation tasks 711, 712, 713, 714, and 715, the system can use the best performance of the candidate in the validation task (e.g., best performance 701B for task 711) as a normalization constant, and then normalize the performance of the candidate agent neural network (e.g., candidate agent neural network performance 701A for task 711) using the normalization constant. For each candidate agent neural network, the normalized performance of the candidate across multiple validation tasks (e.g., normalized candidate agent neural network performance 724C for task 714) can then be sorted, for example, in monotonically ascending order (as shown by the normalized percentile curve in Figure 7), from which the respective normalized percentile metric for the candidate in each validation task can be determined.

[0113] In the example in Figure 7, the best performance of a candidate on any given validation task may be the best performance achievable by a candidate agent neural network, measured by the candidate agent returns received by each candidate on the validation task (in the case of a single-agent task), or alternatively, the best performance achievable by one of a group of candidate agent neural networks (a Nash equilibrium), measured by the candidate agent returns received by each candidate on the validation task (in the case of a multi-agent task).

[0114] The system then applies population-based training techniques to each adaptive measure to adjust the respective hyperparameter values ​​for each of the one or more candidate agent neural networks 142A-G in the population. For example, by comparing the adaptive measures of each candidate agent neural network with one another, and in response to a judgment that the adaptive measure for candidate agent neural network 142D is better than the respective adaptive measures for another candidate agent neural network in the population, for example, candidate agent neural network 142E, the system identifies candidate agent neural network 142D as the candidate that performs "better" in the current population. For example, the comparison result may be based on Pareto dominance. As an example here, if the adaptive measures of a candidate for multiple tasks, for example, multiple validation tasks, such as the normalized percentile metric, are all as good as those for at least multiple tasks, for example, multiple validation tasks, but more precisely, the adaptive measure, for example, the normalized percentile metric, is better for at least one of the tasks, for example, the validation task, then the performance of a candidate may be considered to Pareto dominance over the performance of other candidates.

[0115] In this example, the system can then determine new values ​​for the network hyperparameters of other candidate agent neural networks by "utilizing" the network hyperparameters for candidate agent neural network 142D, that is, by adjusting the hyperparameter values ​​for other candidate agent neural networks to be equal to the hyperparameter values ​​for candidate agent neural network 142D. Alternatively, in this example, the system can determine new values ​​for the network hyperparameters of other candidate agent neural networks by "exploring" the network hyperparameters for candidate agent neural network 142D, that is, by adjusting the hyperparameter values ​​for other candidate agent neural networks to be equal to variant (e.g., randomly perturbed) versions of the hyperparameter values ​​for candidate agent neural network 142D.

[0116] The system similarly applies population-based training techniques to each adaptive measure to adjust the respective parameter values ​​for each candidate agent neural network 142A-G in the population. Continuing the above example where candidate agent neural network 142D is identified as the candidate that performs "better" in the current population, the system can set the respective parameter values ​​for another candidate agent neural network, for example, candidate agent neural network 142E, to be equal to the respective parameter values ​​(or variant versions of the respective parameter values) for candidate agent neural network 142D.

[0117] As shown in Figure 3, in some implementations, the system further incorporates generational training techniques into population-based training techniques to further improve the effectiveness and speed of RL training. In these implementations, training of candidate agent neural networks takes place over a sequence of training generations, where each candidate agent neural network trained in each training generation can be considered collectively as a specific population. For example, each training generation may consist of a fixed number of iterations of the iterative training process (e.g., 1e8, 5e8, 10e8, etc.) or a set time period (e.g., 12 hours, 24 hours, 48 ​​hours, etc.).

[0118] Figure 3 shows an exemplary implementation in which each group of seven candidate agent neural networks is trained in each training generation in a sequence of four training generations. However, there may be many more, and sometimes several times more, candidate agent neural networks trained for more or fewer training generations.

[0119] Generational population-based training allows the use of policy distillation techniques during training, which generally allows a candidate agent neural network in the current training generation to bootstrap its behavior from another candidate in the preceding training generation. Implementing policy distillation techniques involves distilling a candidate agent neural network from the best-performing candidate agent neural network during the current training generation of a sequence of training generations of a population of candidate agent neural networks, while training the candidate agent neural network to optimize expected returns on the same or different candidate task where a candidate agent neural network that performs "better" than a preceding training generation (e.g., in terms of task-specific performance or adaptive measure) has already been trained, for example, distilling from the best-performing candidate at the end of a preceding training generation that directly precedes the current training generation in the sequence.

[0120] Specifically, the system adds a backup policy distillation loss term to the objective function used in RL training of the candidate agent neural network.

[0121]

number

[0122] It can be incorporated here

[0123]

number

[0124] And in the formula, π teacherAnd π are the action selection outputs of the “better” performance candidate agent neural network and the “trained” candidate agent neural network, respectively. In the illustrative equation above, the policy loss term is masked over the environmental state, and the reserve loss (in Kullback-Leibler divergence) only affects RL training at specific time steps in various training task episodes where no reward is received.

[0125] Furthermore, in these implementations, candidate agent neural networks that perform "better" than previous training generations (for example, in terms of task-specific performance or adaptive measures) can be used to control other agents in a multi-agent candidate task to increase the diversity of RL training.

[0126] In some implementations, one or more candidate agent neural networks in one of the training generations in a sequence, each corresponding to a certain population, can be trained for the Self-Reward Play objective. This can be used to aid RL training by preventing the candidate agent neural network from achieving only the corresponding goal in fewer tasks. Specifically, the Self-Reward Play objective rewards the candidate agent neural network for satisfying goal g, and after success, the candidate is rewarded for satisfying not(g) in the same environment, and this flip in the goal is repeated after each satisfaction. For example, suppose goal g is to move an object toward a target location, and then not(g) is to move the object toward the target location. This can be seen as two agents playing against themselves, one agent having to satisfy g and the other agent having to satisfy not(g), but the agents acting sequentially and controlled by the same candidate agent neural network.

[0127] Figure 4A shows an exemplary reinforcement learning agent control system 400. The reinforcement learning agent control system 400 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below may be implemented.

[0128] The reinforcement learning system 400 includes an agent neural network 420 obtained as a result of an open-ended reinforcement learning training process described herein.

[0129] The reinforcement learning agent control system 400 uses an agent neural network 420 to select an action 410 that should be performed by agent 408, and then controls agent 408 as it interacts with the environment 402 by having agent 408 perform the selected action 410.

[0130] The execution of the selected action 410 by agent 408 generally transitions the environment 402 to a new state. By repeatedly having agent 408 act in the environment 402, system 400 can control agent 408 to complete each of several specified tasks.

[0131] In each of the multiple time steps, the agent neural network 420 is configured to process an input including (i) a current observation record 404 characterizing the current state of the environment 402 and (ii) goal data 406 representing a goal to be satisfied in order to perform a task in the environment, according to trained values ​​of the network parameters, in order to produce an action selection output. The goal may be represented as a set of options for each set of predicates, each predicate may map the current state of the environment to a corresponding reward value, and each option is a concatenation of one or more predicates such that a reward is received only if all predicates return a non-zero reward. In some implementations, the goal data 406 may be represented as one or more tensors of numbers, for example, where each predicate is represented as a multi-hot coded vector.

[0132] Specifically, in some implementations, the current observation record 404 characterizing the current state of the environment 402 may include information defining the position, orientation, velocity, etc., of different entities present in the environment (e.g., target objects, obstacles, and other agents). In these implementations, a set of atomic predicates φ in the form of physical relationships relating to some or all of the entities may be included. j It is good if it is defined, and the goal is a set of atomic predicates φj It can be defined as a Boolean representation of φ. These physical relations may include, for example, being near, above, seeing, and holding, as well as their negations, and entities are static topological building blocks such as target objects, obstacles, agents, or the ground of the environment. The exemplary predicate may then be “near (purple sphere, opponent),” which represents the goal of bringing one of the opponent agents close to the purple sphere in the environment. With a fixed set of possible predicates, an agent’s goal can be represented by a set of options (disjunctions) to a set of related (or required) predicates for this option (conjunction). In other words, the goal is a mapping φ j :S→{0,1} d This may be the case, where each current state is assigned a starting point to a binary vector of d predicate true values, where 0 indicates a predicate unrelated to the goal when the environment is in the state characterized by the observation record S, and 1 indicates a predicate related to the goal when the environment is in the state characterized by the observation record S. For example, the goal represented by goal data 406 is,

[0133]

number

[0134] It can be like this, and for some exemplary predicates, "yellow ball (φ j2 While near ) the purple sphere (φ j1 ) hold or yellow ball (φ j4 Counter agents (φ) that do not possess j3 While watching ), the yellow ball (φ j2 It can sometimes mean "being near (someone)".

[0135] The system 400 then uses the action selection output to control the agent, that is, to select an action 410 to be performed by the agent in the current time step according to the action selection output, and then causes the agent to perform the action 410, for example, by directly sending a control signal to the agent or by sending data identifying the action 410 to the control system for the agent. Exemplary action selection outputs and how they may be used to select an action are described above with reference to Figure 1.

[0136] The agent neural network 420 is implemented with a neural network architecture that enables the network to perform its described function. As shown in Figure 4A, the agent neural network 420 includes a state encoder neural network 430, an embedded neural network 440, an attention neural network 445, a value neural network (value neural network "V" head) 450, and a policy neural network (policy neural network "π" head) 460. That is, the value neural network may also be called the value neural network head, and similarly, the policy neural network may be called the policy neural network head, the term "head" may indicate, but is not required, that both the value neural network and the policy neural network receive input shared data generated by one or more layers of the attention neural network 445. Each of the neural networks 420, 430, 440, 445, and 450 contains a different subset of multiple neural network layers within the agent neural network 420.

[0137] The state encoder neural network 430 may include a stack of multiple convolutional layers, followed by one or more pooling layers (e.g., a maximum pooling layer), and followed by one or more recurrent layers (e.g., a long-short-term memory (LSTM) layer). The state encoder neural network 430 is configured, i.e., throughout training, to update its hidden state by receiving and processing observation records 404, i.e., generating a current hidden state 432 that represents the current state of the environment. Typically, an embedding is an ordered set of numerical or other values ​​with a fixed number of dimensions. The fixed number of dimensions may depend on the actual number of LSTM neurons contained within each LSTM layer of the state encoder neural network 430, e.g., 128, 256, etc.

[0138] The embedded neural network 440 may be a fully connected neural network, i.e., comprising multiple fully connected layers, followed optionally by an activation layer (e.g., a ReLU activation layer), the activation layer configured to process the goal data 406 and the current hidden state 432 to generate (i) predicate predictions, (ii) goal embeddings for the goals, and (iii) option embeddings for each of the options represented by the goal data. The embedded neural network 440 may then comprise a predicate predictor neural network, a goal embedding neural network, and an option embedding neural network, each comprising a subset of the multiple fully connected layers of the embedded neural network 440.

[0139] The predicate predictor neural network is configured to process goal data 406 and the current hidden state 432 to generate predicate predictions. In some implementations, the predicate prediction is a set of atomic predicates φ, assuming the environment is in a state characterized by the current observation. jThis may be a multi-label binary classification prediction that specifies which predicates are related to (for example, included in) the goals represented in goal data 406.

[0140] The option-embedded neural network is configured to process goal data 406 and current hidden state 432, data derived from goal data 406 and current hidden state 432, or both, to generate option embeddings for each of the options. Similarly, the goal-embedded neural network is configured to process goal data 406 and current hidden state 432, data derived from goal data 406 and current hidden state 432, or both, to generate goal embeddings for the goals. Typically, an embedding is an ordered set of numerical or other values ​​with a fixed number of dimensions. The fixed number of dimensions may depend on the actual number of hidden units contained within each fully connected layer of the embedding neural network 440, for example, 128, 256, etc.

[0141] The attention neural network 445 is configured to process inputs including (i) the current hidden state, (ii) a goal embedding, and (iii) a predicate prediction for generating a goal attention hidden state. As used herein, an attention neural network is a neural network comprising one or more attention layers, each attention layer being a neural network layer comprising an attention mechanism, e.g., a scaled dot product attention mechanism. To generate a goal attention hidden state, the attention mechanism maps a query and a set of key-value pairs to an output (goal attention hidden state), the query may be from (or otherwise derived from) the goal embedding, and the set of key-value pairs may be derived from the current hidden state 432 (e.g., a linear projection and / or reshaped version of the current hidden state).

[0142] Furthermore, the attention neural network 445 is configured to process an input for each option, including (i) the current hidden state, (ii) the respective option embedding for the option, and (iii) a predicate prediction, to generate a corresponding option attention hidden state for each option. To generate a corresponding option attention hidden state for each option, the attention mechanism maps a query and a set of key-value pairs to the output (option attention hidden state), where the query can be the option embedding for the option, and the set of key-value pairs can be derived from the current hidden state 432.

[0143] The value neural network 450 is a fully connected neural network, i.e., it may include one or more fully connected layers, optionally followed by an activation layer (e.g., a ReLU activation layer), which is configured to process the goal attention hidden state and generate a goal value estimate for the goal, representing the estimated return achieved, for example, a cumulative measure of the reward the agent would receive if it attempted to satisfy the goal starting from the current state. The value neural network 450 is also configured, for each of the options, to use the value neural network head to process the respective option attention hidden state for the option and generate a respective option value estimate for the option, representing the estimated return achieved if the agent attempted to satisfy the option starting from the current state.

[0144] The goal attention hidden state and each option attention hidden state are then combined to generate a combination hidden state to be provided to the policy neural network 460, which is configured to process the combination hidden state and generate an action selection output. In some implementations, the combination hidden state may be an unweighted combination, while in other implementations, the combination hidden state may be a weighted combination, i.e., a combination of the goal attention hidden state and each option attention hidden state according to their respective weights.

[0145] In some of these implementations, the respective weights for the goal, and for each option from the respective option value estimates for the goal and the options, can be calculated by applying a softmax function to the respective option value estimates for the goal and the options to generate softmax scores for the goal and for each option, which are then used as weights for calculating weighted combinations. Generally speaking, a softmax function is a function that converts numbers into probabilities. In other implementations, the respective weights can be calculated by assigning a weight of 1 to the highest value estimate among the respective option value estimates for the goal and the options, and a weight of 0 to all other value estimates for the respective option value estimates for the goal and the options.

[0146] The policy neural network 460 may include one or more fully connected layers and one or more subsequent output layers. In some implementations, the one or more output layers may include a single softmax layer configured to process the output of the preceding fully connected layer to generate a probability distribution for a set of actions that the agent can perform. In some other implementations, the one or more output layers may include multiple softmax layers, each corresponding to a different subset of the set of actions that the agent can perform, each configured to process the output of the preceding fully connected layer to generate its respective probability distribution for the subset. In yet another implementation, the one or more output layers may include one or more additional fully connected layers configured to generate their respective Q-values ​​for each action in the set of actions that the agent can perform.

[0147] Figure 4B shows details of one exemplary implementation of the agent neural network 420.

[0148] Figure 5 is a flowchart of an exemplary process 500 for controlling an agent that interacts with its environment. For convenience, the process 500 is described as being carried out by a system consisting of one or more computers in one or more locations. For example, a reinforcement learning system, such as the appropriately programmed reinforcement learning agent control system 400 in Figure 4, can carry out the process 500.

[0149] The system receives an observational record characterizing the current state of the environment (step 502). For example, the observational record may include an audio data segment, an image, or a sentence in natural language. Optionally, the observational record may also include information derived from the previous time step, such as the most recent action taken, the reward received in the most recent time step, or both.

[0150] The system receives goal data representing the goals to be satisfied in order to perform a task within the environment (step 504). In the received goal data, a goal is represented as a set of options for each set of predicates. In the example in Figure 4, the goal is represented as two sets of options, each containing a single predicate, "hold the purple sphere" or "see the yellow cube," but in other examples, a goal may be represented by many options, each consisting of more predicates, and the same predicate may be shared across multiple options.

[0151] The system processes the input, including observation records and goal data, using an agent neural network to generate an action selection output, as further described below with reference to Figure 6 (step 506). In some implementations, the action selection output can define a probability distribution over a set of actions that the agent can perform. In some implementations, the action selection output may include the respective Q-values ​​for each action in the set of actions that the agent can perform. In some implementations, the action selection output is an action from a continuous action space, i.e., all action values ​​within each individual action are selected from possible values ​​in a continuous range.

[0152] The system uses the action selection output to select the action to be performed by the agent (step 508). In implementations where the action selection output defines a probability distribution, the system can select an action by sampling the actions according to their probability values, or by selecting the action with the highest probability value. In implementations where the action selection output includes Q values, the system can process the Q values ​​(for example, using a softmax function) to generate a probability value for each possible action, which can then be used to select the action to be performed by the agent. The system can also select the action with the highest Q value as the action to be performed by the agent.

[0153] The system causes the agent to perform the selected action, for example, by instructing the agent to perform the action or by passing a control signal to the agent's control system (step 510).

[0154] Figure 6 is a flowchart of an exemplary process 600 that uses an agent neural network to generate an action selection output. For convenience, process 600 is described as being carried out by a system consisting of one or more computers in one or more locations. For example, a reinforcement learning system, such as the appropriately programmed reinforcement learning system 400 in Figure 4, can carry out process 600.

[0155] The system processes observation records using an agent neural network state encoder neural network to generate a current hidden state that represents the current state of the environment (step 602). The state encoder neural network includes one or more recurrent layers and is configured to generate the current hidden state by processing the received current observation records in such a way that it modifies the hidden state of the state encoder neural network, which was generated by processing previous observation records.

[0156] The system uses an embedded neural network of the agent neural network to process the goal data and the current hidden state to generate (i) predicate predictions, (ii) goal embeddings for the goal, and (iii) option embeddings for each of the options represented by the goal data (step 604).

[0157] The system processes an input including (i) the current hidden state, (ii) the goal embedding, and (iii) the predicate prediction using an attention neural network to generate a goal attention hidden state (step 606). The goal attention hidden state is generated at least in part by applying an attention mechanism to (i) the current hidden state, (ii) the goal embedding, and (iii) the predicate prediction.

[0158] For each option, the system processes the input, which includes (i) the current hidden state, (ii) the respective option embedding for the option, and (iii) the predicate prediction, using an attentional neural network to generate the respective option attentional hidden state for the option (step 608). The option attentional hidden state is generated at least in part by applying an attentional mechanism to (i) the current hidden state, (ii) the respective option embedding for the option, and (iii) the predicate prediction.

[0159] The system uses the value neural network of the agent neural network to process the goal attention hidden state and generate a goal value estimate for the goal, which represents the estimated return that would be achieved if the agent attempted to satisfy the goal starting from the current state (step 610).

[0160] For each option, the system uses a value neural network to process the option attention hidden state for each option and generate an option value estimate for each option that represents the estimated return that would be achieved if the agent, starting from the current state, attempted to satisfy the option (step 612).

[0161] The system generates weights for the goal and for each of the options from the goal value estimate and the respective option value estimates for the options (step 614). In some implementations, generating the respective weights may involve applying softmax to the goal value estimate and the respective option value estimates for the options. In some other implementations, generating the respective weights may involve assigning a weight of 1 to the highest value estimate among the goal value estimate and the respective option value estimates for the options, and a weight of zero to all other value estimates of the goal value estimate and the respective option value estimates for the options.

[0162] The system generates a combined hidden state by combining the goal attention hidden state and each option attention hidden state according to their respective weights (step 616).

[0163] The system uses a policy neural network of an agent neural network to process the combinational hidden state and generate an action selection output (step 618).

[0164] This specification uses the term “configured” in relation to system and computer program components. A system consisting of one or more computers being configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action when it is running. A computer program being configured to perform a particular operation or action means that the program contains instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0165] The subject matter and functional operating embodiments described herein can be implemented in digital electronic circuit mechanisms, or in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. The subject matter embodiments described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as mechanically generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0166] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may be a special-purpose logic circuit mechanism, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), or may further include these. Optionally, in addition to hardware, a device may include code that creates an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or code comprising one or more of these.

[0167] Computer programs may be called, or described as, programs, software, software applications, apps, modules, software modules, scripts, or code, and may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but may not, correspond to a file in a file system. A program may be stored in a single file dedicated to the program, in a part of a file that holds one or more scripts stored in other programs or data, such as a markup language document, or in multiple collaborative files, such as a file that stores one or more modules, subprograms, or parts of code. Computer programs may be deployed to run on one computer or on multiple computers located in one place, or they may be distributed across multiple locations and interconnected by data communication networks.

[0168] In this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same computer or on multiple computers.

[0169] The processes and logic flows described herein may be implemented by one or more programmable computers running one or more computer programs to perform their functions by acting on input data and producing outputs. The processes and logic flows may also be implemented by special-purpose logic circuit mechanisms, such as FPGAs or ASICs, or by a combination of special-purpose logic circuit mechanisms and one or more programmed computers.

[0170] A computer suitable for running computer programs may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from read-only memory or random-access memory, or both. Essential elements of a computer are a central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory can be complemented by or incorporated into special-purpose logic circuit mechanisms. Generally, a computer may include one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or be operably coupled to mass storage devices to receive data from or transfer data, or both. However, a computer is not required to have such devices. Moreover, a computer can be incorporated into another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive, to name just a few.

[0171] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks, and all forms of non-volatile memory, media, and memory devices.

[0172] To enable interaction with the user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, on which the user can provide input to the computer. Interaction with the user can also be provided using other types of devices, for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user can be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0173] Data processing devices for implementing machine learning models may include, for example, the common and computationally intensive parts of machine learning training or generation, i.e., inference, as well as special-purpose hardware accelerator units for handling the workload.

[0174] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0175] Embodiments of the subject matter described herein can be implemented in a computing system that includes a backend component, for example, a data server, or a middleware component, for example, an application server, or a frontend component, for example, a client computer having a graphical user interface, a web browser, or an application through which the user can interact with the implementation of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), for example, the Internet.

[0176] A computing system may include a client and a server. The client and server are generally separate from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer and having a client-server relationship with each other. In some embodiments, the server transmits data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from the user. Data generated on the user device, such as the results of user interaction, may be received from the device to the server.

[0177] This specification includes many specific implementation details, but these should not be construed as limiting the scope of any invention or claim, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Furthermore, some features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable partial combination in multiple embodiments. Moreover, features are described above as functioning in a particular combination, and may even be initially claimed as such, but one or more features from a claimed combination may, in some cases, be removed from that combination, and the claimed combination may be a partial combination or a variation of a partial combination.

[0178] Similarly, although the operations are shown in the drawings and enumerated in the claims in a specific order, this should not be understood as requiring such operations to be performed in a specific illustrated order or sequentially, or that all shown operations must be performed to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0179] This document describes specific embodiments of the subject matter. Other embodiments are within the scope of the following claims. For example, the actions enumerated in the claims can still achieve the desired results even if they are performed in a different order. As an example, the process shown in the accompanying diagrams does not necessarily require the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of Symbols]

[0180] 100 training systems, systems 102 Agents, Reinforcement Learning Agents 120 Group Base Training (PBT) Engine 124 Task Generation Engine 130 Training Data Repositories 140 collective repositories 142 Candidate Agent Neural Network 400 Reinforcement Learning Agent Control System, Reinforcement Learning System, System 408 Agent 420 Agent Neural Network, Neural Network 430 State Encoder Neural Network, Neural Network 440 Embedded Neural Networks, Neural Networks 445 Attention Neural Network, Neural Network 450 Value Neural Network, Value Neural Network "V" Head, Neural Network 460 Policy Neural Network, Policy Neural Network "π" Head

Claims

1. A method, performed by one or more computers, for controlling agents that interact with an environment to cause the agents to perform tasks in the environment, The steps include receiving observational records that characterize the current state of the environment, A step of receiving goal data representing a goal to be satisfied in order to perform the task in the environment, wherein the goal is represented as a set of options for each set of predicates, A step of using an agent neural network to process the observation records and goal data to generate an action selection output, The steps include: using a state encoder neural network to process the observation record and generate a current hidden state that represents the current state of the environment; A step of generating goal embeddings for the aforementioned goal and option embeddings for each of the aforementioned options from the goal data, The steps include using an attention neural network to process the input, including the current hidden state and the goal embedding, to generate a goal attention hidden state, For each of the options, the attention neural network is used to process an input including the current hidden state and the respective option embedding for the option to generate a respective option attention hidden state for the option. The steps include: using a value neural network head to process the goal attention hidden state and generate a goal value estimate for the goal, which represents the estimated return that would be achieved if the agent attempted to satisfy the goal starting from the current state; For each of the options, the value neural network head processes the respective option attention hidden state for the option to generate an option value estimate for the option that represents the estimated return that would be achieved if the agent attempted to satisfy the option, starting from the current state. A step of generating weights for the goal and for each of the options from the goal value estimation and the respective option value estimation for the options, A step of generating a combined hidden state by combining the goal attention hidden state and the respective option attention hidden states according to the respective weights, The steps include processing the combinational hidden state using a policy neural network head to generate the action selection output, The steps include selecting an action to be performed by the agent using the action selection output, A method comprising the step of causing the agent to perform the selected action.

2. The method according to claim 1, wherein the state encoder neural network is a recurrent neural network.

3. The step of using an agent neural network to process the observation records and goal data to generate an action selection output is: The method according to claim 1, further comprising the step of using a predicate predictor neural network to process the goal data and the current hidden state to generate a predicate prediction, wherein the input, including the current hidden state and the goal embedding, further includes a previous describer word prediction, and for each option, the input, including the current hidden state and the respective option embedding, further includes a previous describer word prediction.

4. The method according to claim 1, wherein the step of generating weights for the goal and for each of the options from the goal value estimate and the respective option value estimate for the options includes the step of applying softmax to the goal value estimate and the respective option value estimate for the options.

5. The method according to claim 1, wherein the step of generating weights for the goal and for each of the options from the goal value estimate and the respective option value estimates for the options includes assigning a weight of 1 to the highest value estimate among the goal value estimate and the respective option value estimates for the options, and a weight of zero to all other value estimates of the goal value estimate and the respective option value estimates for the options.

6. The method according to claim 1, wherein the action selection output defines a probability distribution over a set of actions that the agent can perform.

7. The method according to claim 1, wherein the action selection output includes a Q value for each action in the set of actions that the agent can perform.

8. The method according to claim 1, wherein the action selection output is an action from a continuous action space.

9. One or more computers, A system comprising one or more storage devices communicably coupled to one or more computers, wherein the one or more storage devices, when executed by the one or more computers, store instructions causing the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 8.

10. One or more non-temporary computer storage media that, when executed by one or more computers, store instructions causing the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 8.

Images