Intelligent agent control by reinforcement learning in context

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling agents. In particular, when controlling an agent on a new task, the agent may be controlled using an action selection neural network that performs reinforcement learning in the context.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 411,089, filed on September 28, 2022. The disclosure of the prior application is considered part of the disclosure of the present application and is incorporated by reference into the disclosure of the present application. Background Art

[0002] This specification relates to using machine learning models to process data.

[0003] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate output based on the received input and the values ​​of the model parameters.

[0004] Some machine learning models are deep models that use multiple layers of models to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to a received input to generate an output. Summary of the invention

[0005] This specification generally describes a system implemented as a computer program on one or more computers at one or more locations that controls an agent that interacts with an environment to perform tasks in the environment using an action-selecting neural network.

[0006] More specifically, the agent selection neural network represents an "in-context" reinforcement learning algorithm. That is, because contextual data from previous interactions with the environment are conditioned while using the action selection neural network to control the agent, as the amount of contextual data increases, the action selection neural network can select actions that result in improved performance on the task (relative to earlier points during control of the agent) without requiring updates to the parameters of the action selection neural network. In other words, as the amount of available data increases, the action selection neural network can "mimic" the performance of a reinforcement learning algorithm without requiring updates to the parameters of the neural network, i.e., without requiring further training of the neural network.

[0007] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0008] This specification describes an action selection neural network that can be used to perform reinforcement learning in the context of a new task after being trained on a training dataset. When performing a new task, the action selection neural network can be much more data efficient than the reinforcement learning algorithm used to generate the training dataset, for example, because the multi-actor algorithm is "distilled" to a single-actor algorithm, because the system subsamples training episodes when generating the training dataset, or both.

[0009] Moreover, "learning" a new task using an action selection neural network also consumes much less computational resources than learning a policy for a new task using an RL algorithm, since no computationally expensive back-propagation is required since the action selection neural network no longer needs to be updated during "learning." That is, since the action selection neural network improves on the new task simply by observing more context and without updating the weights of the neural network, no gradient calculations are required, and thus no back-propagation is required, while still improving performance on the new task.

[0010] Additionally, distributed reinforcement learning algorithms require a significant amount of network communication between distributed actors and learners during the learning process. By leveraging an already trained action selection neural network, this network communication is greatly reduced or even completely eliminated. That is, by utilizing the described "in-context" reinforcement learning scheme, the system can utilize a single actor implemented on a single set of one or more hardware devices to achieve comparable or better performance on a given new task than when learning the task using a distributed reinforcement learning algorithm that requires multiple actors and one or more learners, each implemented on a different set of one or more hardware devices. Thus, network communication is greatly reduced because weight updates do not need to be transmitted between actors and learners, and transitions do not need to be sampled from a replay buffer.

[0011] Additionally, after training and when "learning in context" a new task, the storage requirements are at least comparable to, and in many cases significantly reduced relative to, distributional reinforcement learning algorithms, since only the tokenized observations, actions, and rewards needed by the current context for the action selection neural network at the current time step need to be stored, and any tokenized observations and rewards no longer needed for the context can be discarded. Distributed reinforcement learning algorithms, on the other hand, need to maintain a replay buffer that includes a large number of transitions that the learner can sample from to train the neural network. By removing the need to maintain this large replay buffer, the described techniques significantly reduce the storage requirements for "learning" a new task.

[0012] Furthermore, the action selection neural network can be trained completely “offline”, i.e., without being used to control the agent. This avoids damage or wear and tear on the real-world agent, and also provides the ability to amortize and parallelize the training workload.

[0013] Moreover, once trained, the same action-selection neural network can be used to "learn" many new tasks without requiring any further training, thereby greatly reducing the amount of computational resources required within a multi-tasking system.

[0014] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 An example action selection system is shown.

[0016] Figure 2 is a flow chart of an example process for training an action selection neural network.

[0017] Figure 3 is a flowchart of an example process for generating a training history sequence for a given task.

[0018] Figure 4 is an illustration of an example of training an action selection neural network.

[0019] Figure 5 is a flow chart of an example process for controlling an agent using an action selection neural network.

[0020] Figure 6 An example of an attention map generated by an action selection neural network is shown.

[0021] Figure 7 The performance of the described technique relative to a baseline technique is shown.

[0022] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0023] Figure 1 An example action selection system 100 is shown. Action selection system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0024] The action selection system 100 controls an agent 104 that interacts with an environment 106 to complete a task by selecting an action 108 to be performed by the agent 104 at each of a plurality of time steps during performance of an episode of the task.

[0025] As a general example, a task may include, for example, one or more of the following: navigating to a specified location in an environment, identifying a specific object in an environment, manipulating a specific object in a specified manner, controlling an item of equipment to meet a criterion, allocating resources across devices, etc. More generally, the task is specified by the reward received, i.e., such that when the task is successfully completed, the round reward is maximized. Rewards and rewards are described in more detail below. Examples of agents, tasks, and environments are also provided below.

[0026] An "episode" of a task is a sequence of interactions during which the agent attempts to perform a single instance of the task starting from some starting state of the environment. In other words, each task episode begins with the environment in an initial state, e.g., a fixed initial state or a randomly selected initial state, and ends when the agent has successfully completed the task or when some termination criterion is met, e.g., the environment enters a state that has been designated as a terminal state or the agent performs a threshold number of actions without successfully completing the task.

[0027] At each time step during any given task episode, the system 100 receives an observation 110 characterizing the current state of the environment 106 at the time step, and in response, selects an action 108 to be performed by the agent 104 at that time step. After the agent performs the action 108, the environment 106 transitions to a new state, and the system 100 receives a reward 130 from the environment 106.

[0028] Generally speaking, reward 130 is a scalar value and represents the progress of agent 104 toward completing a task.

[0029] As a specific example, reward 130 may be a sparse binary reward that is zero unless the task is successfully completed as a result of the action being performed, i.e., the sparse binary reward is only non-zero, such as equal to one, if the task is successfully completed as a result of the action being performed.

[0030] As another specific example, reward 130 may be a dense reward that measures the agent's progress toward completing the task as each observation is received during an episode of attempting to perform the task, i.e., such that a non-zero reward can be and often is received before the task is successfully completed.

[0031] While executing any given mission round, the system 100 selects actions in an attempt to maximize the rewards received during the mission round.

[0032] That is, at each time step during an episode, the system 100 selects actions that attempt to maximize the reward that will be received for the remainder of the task episode starting at that time step.

[0033] In general, at any given time step, the reward that will be received is a combination of the rewards that will be received at the time step after the given time step in the episode.

[0034] For example, at time step t, the reward can satisfy: , where i ranges from all time steps after t in the episode or a fixed number of time steps after t in the episode, is a discount factor greater than zero and less than or equal to one, and is the reward at time step i.

[0035] To control the agent, at each time step in an episode, the action selection subsystem 102 of the system 100 uses the action selection neural network 102 to select an action 108 to be performed by the agent 104 at the time step.

[0036] At a high level, the system 100 generates an input sequence 112 from observations 110 and then processes the input sequence 112 using the action selection neural network 102 to generate a policy output for a time step.

[0037] As will be described in more detail below, input sequence 112 generally represents observations 110 and additionally includes contextual data from previous time steps in the current episode and in previous episodes of performing the task.

[0038] The system then uses the policy output to select an action 108 and causes the agent 104 to perform the selected action 108 .

[0039] More specifically, the agent selection neural network 102 represents an “in-context” reinforcement learning algorithm.

[0040] That is, as a result of being conditioned on contextual data from previous interactions with the environment while using the action selection neural network 102 to control the agent 104 and as the amount of available contextual data increases, the action selection neural network 102 can select actions that result in improved performance on the task (relative to earlier points during control of the agent) without having to update (learnable) parameters (e.g., weights) of the action selection neural network 102.

[0041] In other words, as the amount of available data increases, the action selection neural network 102 can "mimic" the performance of the reinforcement learning algorithm without the need to update the parameters of the neural network 102, i.e., without the need to further train the neural network.

[0042] In particular, system 100 trains action-selection neural network 102 and then, after training, uses action-selection neural network 120 as an “in-context” reinforcement learning algorithm while performing new tasks, i.e., performing new tasks without further training action-selection neural network 102.

[0043] For each new task, the performance of the action selection neural network 102 in controlling the agent on the task, e.g., in terms of the average reward obtained during any given task episode, improves as the amount of available context data increases, and no training of the action selection neural network 102 is required.

[0044] To train the action selection neural network 102, the system obtains a training data set 150 that includes a corresponding training history sequence 152 for each of a plurality of tasks.

[0045] In general, each training history sequence 152 includes a sequence of tokens representing transitions from multiple task rounds executed while training a policy for the task (particularly an action selection policy) through reinforcement learning. The policy for the task can be any suitable policy, for example, any suitable policy that can be adjusted through training to improve the performance of the policy on the task, and can be represented by a neural network or other machine learning model having the same architecture as the neural network 102 or an architecture different from the neural network 102. That is, any suitable training history sequence 152 for training any suitable policy can be incorporated into the training data set 150. In general, an action selection policy can be a mechanism for selecting an action in response to an observation (of the environment).

[0046] As described above, an "episode" of a task (or "task episode") is a sequence of interactions during which the agent attempts to perform an instance of the task starting from some starting state of the environment.

[0047] In other words, each task episode begins with the environment in an initial state, e.g., a fixed initial state or a randomly selected initial state, and ends when the agent has successfully completed the task or when some termination criterion is met, e.g., the environment enters a state that has been designated as a terminal state or the agent performs a threshold number of actions without successfully completing the task. At each time step in a task episode, the agent receives an observation representing the state of the environment up to the end time step, performs an action in response to the observation, and receives a reward.

[0048] In other words, a task episode results in a series of "transitions" that each correspond to a different time step and each include a corresponding observation, a corresponding action, and a corresponding reward received in response to performing the action.

[0049] The labeled sequence in training history sequence 152 includes a corresponding episode subsequence for each of a plurality of task episodes that occurred during training of the policy for the task.

[0050] For each transition from a task round, a corresponding round subsequence of each task round in the task round includes a corresponding transition subsequence, which includes (i) one or more markers representing observations in the transition, (ii) one or more markers representing actions in the transition, and (iii) one or more markers representing rewards in the transition.

[0051] As used in this specification, a "tag" is a vector of numerical values ​​of specified dimensions.

[0052] In other words, the tokenization system 120 within the system 100 or a different tokenization system preprocesses the observations, actions, and rewards to "tokenize" them so that each observation, action, and reward is represented as one or more tokens, each having predetermined dimensions.

[0053] System 120 may represent observations, actions, and rewards as tokens in any suitable manner.

[0054] An example technique for doing this is described in Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch, “Decision transformer: Reinforcement learning via sequence modeling,” Advances in neural information processing systems, 34:15084–15097, 2021.

[0055] Another example technique for doing this is described in Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas, “Generalist Agent,” 2022, arXiv:2205.06175.

[0056] That is, the system 120 may use one or another of the techniques described above to represent any given received action, observation, and reward as a corresponding set of one or more tags having predetermined dimensions.

[0057] The system 120 may generally use the same tokenization technique when generating data in a training sequence and when generating input sequences 112 processed by the action selection neural network 102 after training.

[0058] The system 100 then trains the action selection neural network 102 on the training data set. As a result of the training, the action selection neural network 102 is able to perform the "in-context" reinforcement learning described above and below.

[0059] References below Figure 2 and Figure 3 Describe this training in more detail.

[0060] After training, the system 100 can control the agent 104 to perform a task by controlling the agent 104 at each of a plurality of time steps in a sequence of time steps in a “current” task round using the action selection neural network 102 .

[0061] At each time step, the system 100 receives a current observation 110 that characterizes the state of the environment 106 at the time step.

[0062] The tokenization system 120 then generates the tokenized input sequence 112 .

[0063] The input sequence 112 typically includes: (i) one or more labels representing the current observation 110, (ii) a corresponding current transition subsequence for each current round transition in one or more current round transitions, and (iii) a corresponding previous transition subsequence for each previous round transition in one or more previous round transitions.

[0064] In some cases, tokens in the input sequence 112 other than the one or more tokens representing the current observation 110 have already been generated, and the tokenization system 120 only needs to tokenize the current observation 110 to generate the one or more tokens representing the current observation 110 .

[0065] Each current-round transition corresponds to a corresponding earlier time step in the current task round, and a corresponding transition subsequence of the current-round transitions includes: (a) one or more tokens representing observations received at the earlier time step, (b) one or more tokens representing actions performed by the agent in response to receiving the observations at the earlier time step, and (c) one or more tokens representing rewards received in response to the agent performing the actions.

[0066] Each previous-round transition corresponds to a corresponding earlier time step in a corresponding previous task round (of the current task) performed by the agent prior to the current task round, and the corresponding previous-round transition subsequence includes: (a) one or more tokens representing observations received at the earlier time step, (b) one or more tokens representing actions performed by the agent in response to receiving the observations at the earlier time step, and (c) one or more tokens representing rewards received in response to the agent performing the actions.

[0067] That is, the input sequence 112 provides context from earlier in the current round and also from previous task rounds.

[0068] As described above, the system 100 then processes the labeled input sequence 112 using an action selection neural network to generate a policy output for a time step, uses the policy output to select an action; and causes the agent to perform the selected action.

[0069] The policy output may be any suitable output defining a probability distribution of an action set. For example, the policy output may include a corresponding probability of each action in the action set. As another example, the policy output may include parameters of a probability distribution of an action set.

[0070] The system 100 may select an action 108 by, for example, selecting the action with the highest probability based on the probability output or sampling actions from a probability distribution defined by the policy output.

[0071] The action selection neural network 102 may be any suitable sequence model, for example, may have any suitable architecture that allows the neural network 102 to map a labeled input sequence to a probability distribution.

[0072] As an example, the action selection neural network 102 can be a causal Transformer neural network, i.e., a neural network that includes one or more causally masked self-attention layers, e.g., such that at each time step, the self-attention neural network layer only sees past inputs in the sequence of processed inputs. A self-attention layer can be a layer that maps a query and a set of key-value pairs to an output from which the output of the self-attention layer is derived, each key-value pair being derived from an input to the self-attention layer (e.g., the entire vector). The output can be computed as a weighted sum of values ​​weighted by a similarity function of the query to each corresponding key.

[0073] When the action selection neural network 102 is a causal Transformer, "processing" the input sequence 112 can mean processing the entire sequence to recompute hidden states for earlier tokens in the sequence, accessing cached hidden states from memory and computing only the hidden state for the last token in the sequence, or utilizing any other technique to extend the context window of attention to reduce inference latency, or both.

[0074] As another example, the action selection neural network 102 may be a recurrent neural network (RNN), i.e., a neural network including one or more recurrent neural network layers. For example, the neural network 102 may be a long short-term memory (LSTM) neural network or a gated recurrent unit (GRU) neural network.

[0075] When action selection neural network 102 is a recurrent neural network, "processing" input sequence 112 may mean processing the entire sequence to recalculate the hidden state for earlier tokens in the sequence or accessing the most recently updated hidden state from memory and updating the hidden state by processing only the last token in the sequence.

[0076] In some implementations, the environment is a real-world environment, the agent is a mechanical agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle that operates in or navigates through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform a task. For example, the agent can be a robot that interacts with the environment to complete a specific task, such as locating an object of interest in the environment or moving an object of interest to a specified location in the environment or navigating to a specified destination in the environment.

[0077] In these implementations, observations may include, for example, one or more of the following: images, object position data, and sensor data to capture observations when the agent interacts with the environment, such as sensor data from images, distance or position sensors, or from actuators. For example, in the case of a robot, observations may include data characterizing the current state of the robot, such as one or more of the following: joint positions, joint velocities, joint forces, torques, or accelerations (e.g., gravity-compensated torque feedback), and the global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle, observations may similarly include one or more of the following: position, linear or angular velocity, force, torque, or acceleration of one or more parts of the agent, and global or relative pose. Observations may be defined in 1, 2, or 3 dimensions, and may be absolute and / or relative observations. Observations may also include, for example, sensed electronic signals, such as motor currents or temperature signals; and / or, for example, image or video data from a camera or lidar sensor, such as data from a sensor of the agent or data from a sensor located separately from the agent in the environment.

[0078] In these implementations, the action may be a control signal for controlling a robot or other mechanical agent, such as a torque or higher-level control command for a joint of the robot; or may be a control signal for controlling an autonomous or semi-autonomous land, air, or sea vehicle, such as a torque to a control surface or other control element (e.g., a steering control element of a vehicle), or a higher-level control command. The control signal may include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or parts of another mechanical agent. The control signal may also or alternatively include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within an environment, the control of which has an effect on the state of the observed environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, the control signal may define actions for controlling navigation (e.g., steering) and movement (e.g., braking and / or acceleration of the vehicle).

[0079] In some implementations, the environment is a simulation of the real-world environment described above, and the agent is implemented as one or more computers that interact with the simulated environment. For example, the simulated environment can be a simulation of a robot or a vehicle, and the reinforcement learning system can be trained on the simulation and then, once trained, used in the real world.

[0080] In some implementations, the environment is a real-world manufacturing environment for manufacturing products such as chemical, biological, or mechanical products or food. As used herein, "manufacturing" a product also includes refining starting materials to create a product, or processing starting materials, for example, to remove contaminants, to generate a cleaned or recycled product. The manufacturing environment may include multiple manufacturing units, such as containers for chemical or biological substances, or machines for processing solids or other materials, such as robots. The manufacturing units are configured so that intermediate versions or components of the product can be moved between manufacturing units during product manufacturing, such as via pipelines or mechanical transport. As used herein, the manufacture of a product also includes the manufacture of food by a kitchen robot.

[0081] The agent may include an electronic agent configured to control a manufacturing cell or a machine, such as a robot, that operates to manufacture a product. That is, the agent may include a control system configured to control the manufacture of a chemical, biological, or mechanical product. For example, the control system may be configured to control one or more of the manufacturing cells or machines, or to control the movement of intermediate versions or components of a product between manufacturing cells or machines.

[0082] As one example, the tasks performed by the agent may include the task of manufacturing a product or an intermediate version or component thereof. As another example, the tasks performed by the agent may include the task of controlling (e.g., minimizing) the use of resources, such as the task of controlling the consumption of electricity or water or the consumption of any materials or consumables used in the manufacturing process.

[0083] Actions may include control actions to control the use of a machine or manufacturing unit for processing solid or liquid materials to manufacture a product or an intermediate product or component thereof, or to control the movement of an intermediate version or component of a product within a manufacturing environment, such as between manufacturing units or machines. In general, an action may be any action that has an effect on the state of the observed environment, such as an action configured to adjust any of the sensed parameters described below. These actions may include actions to adjust the physical or chemical conditions of a manufacturing unit, or actions to control the movement of mechanical parts of a machine or joints of a robot. Actions may include actions that impose operating conditions on a manufacturing unit or machine, or actions that cause changes in settings to adjust, control, or turn on or off the operation of a manufacturing unit or machine.

[0084] The reward or reward may be associated with a measure of the performance of the task. For example, in the case of a task to manufacture products, the measure may include a measure of the number of products manufactured, the quality of the products, the speed at which the products are produced, or a measure of the physical cost of performing the manufacturing task, such as a measure of the amount of energy, material, or other resource used to perform the task. In the case of a task to control resource usage, the measure may include any measure of the use of the resource.

[0085] In general, observations of the state of the environment may include any electronic signals representing the function of electronic and / or mechanical items of equipment. For example, representations of the state of the environment may be derived from observations made by sensors that sense the state of the manufacturing environment (e.g., sensors that sense the state or configuration of a manufacturing unit or machine, or sensors that sense the movement of materials between manufacturing units or machines). As some examples, such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow / movement rate, or flow / movement path of one or more materials; physical or chemical conditions, such as physical state, shape, or configuration, or chemical state such as pH; configuration of a unit or machine such as the mechanical configuration of a unit or machine, or valve configuration; image or video sensors for capturing image or video observations of a manufacturing unit or machine or movement; or any other suitable type of sensor. In the case of a machine such as a robot, observations from sensors may include observations of the position, linear or angular velocity, force, torque, acceleration, or posture of one or more parts of the machine, such as data characterizing the current state of the machine or robot or an item held or handled by the machine or robot. Observations may also include, for example, sensed electronic signals, such as motor current or temperature signals, or image or video data, such as from a camera or lidar sensor. Sensors such as these may be part of an agent in the environment or located separately from the agent.

[0086] In some implementations, the environment is a real-world environment of a service facility including multiple items of electronic equipment, such as a server farm or data center, such as a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include auxiliary control devices that control the operating environment of the items of equipment, such as environmental control devices, such as temperature control (e.g., cooling) devices, or air flow control or air conditioning devices. Tasks may include tasks that control (e.g., minimize) the use of resources, such as tasks that control power consumption or water consumption. Agents may include electronic agents configured to control the operation of items of equipment or to control the operation of auxiliary (e.g., environmental) control devices.

[0087] In general, an action may be any action that has an effect on the state of the observed environment, such as an action configured to adjust any of the sensed parameters described below. These actions may include actions that control a device or an item of auxiliary control equipment or impose operating conditions on a device or an item of auxiliary control equipment, such as actions that cause a change in a setting to adjust, control, or turn on or off the operation of an item of device or an item of auxiliary control equipment.

[0088] In general, observations of the state of the environment may include any electronic signal that represents the function of the facility or equipment in the facility. For example, representations of the state of the environment may be derived from observations made by any sensor that senses the state of the physical environment of the facility or by any sensor that senses the state of one or more items of equipment or one or more items of auxiliary control equipment. These include sensors configured to sense: electrical conditions such as current, voltage, power, or energy; temperature of the facility; fluid flow, temperature, or pressure within the facility or within the cooling system of the facility; or physical facility configurations, such as whether a vent is open.

[0089] The reward or reward may be related to a measure of the performance of the task. For example, in the case of a task to control (eg, minimize) the use of a resource, such as in the case of a task to control the use of electricity or water, the measure may include any measure of the use of the resource.

[0090] In some implementations, the environment is a real-world environment of a power generation facility (e.g., a renewable power generation facility, such as a solar farm or a wind farm). Tasks may include control tasks that control the power generated by the facility (e.g., control the delivery of power to a distribution network, such as to meet demand or reduce the risk of mismatch between elements of the power grid, or maximize the power generated by the facility). The agent may include an electronic agent configured to control the generation of power by the facility or the coupling of the generated power to the power grid. The action may include an action that controls the electrical or mechanical configuration of a generator (such as the electrical or mechanical configuration of one or more renewable power generation elements) (e.g., controlling the configuration of a wind turbine or solar panel or mirror, or the electrical or mechanical configuration of a rotating power generation machine). Mechanical control actions may, for example, include actions that control the conversion of energy input to electrical energy output (e.g., conversion efficiency or the degree of coupling of energy input to electrical energy output). Electrical control actions may, for example, include actions that control one or more of the voltage, current, frequency, or phase of the generated power.

[0091] The reward or reward may be related to a measure of the performance of the task. For example, in the case of a task to control the delivery of power to a distribution network, the metric may be related to a measure of the power transmitted, or to a measure of an electrical mismatch (such as a voltage, current, frequency, or phase mismatch) between the generating facility and the grid, or to a measure of power or energy lost in the generating facility. In the case of a task to maximize the delivery of power to a distribution network, the metric may be related to a measure of power or energy transmitted to the grid, or to a measure of power or energy lost in the generating facility.

[0092] In general, observations of the state of the environment may include any electronic signal representing the electrical or mechanical function of a power generation device in a power generation facility. For example, a representation of the state of the environment may be derived from observations made by any sensor sensing the physical or electrical state of a device in a power generation facility that is generating electricity, or the physical environment of such a device, or the condition of an auxiliary device supporting the power generation device. Such sensors may include sensors configured to sense: electrical conditions of the device, such as current, voltage, power, or energy; temperature or cooling of the physical environment; fluid flow; or the physical configuration of the device; and observations of electrical conditions of the power grid, such as from local or remote sensors. Observations of the state of the environment may also include one or more predictions about future operating conditions of the power generation device, such as predictions of future wind levels or solar irradiance or predictions of future electrical conditions of the power grid.

[0093] As another example, the environment can be a chemical synthesis or protein folding environment, so that each state is a corresponding state of a protein chain or one or more intermediate or precursor chemicals, and the agent is a computer system for determining how to fold a protein chain or synthesize a chemical. In this example, the action is a possible folding action for folding a protein chain or an action for assembling a precursor chemical / intermediate, and the results to be achieved can include, for example, folding a protein so that the protein is stable, and so that it achieves a specific biological function, or providing an effective synthesis route for the chemical. As another example, the agent can be a mechanical agent that indirectly performs or controls a protein folding action or a chemical synthesis step, such as by controlling a synthesis step automatically selected by the system without human intervention. Observations can include direct or indirect observations of the state of a protein or chemical / intermediate / precursor, and / or can be derived from a simulation. Therefore, the system can be used to automatically synthesize proteins with specific functions, such as proteins with a binding site shape, such as a ligand that binds with sufficient affinity to obtain its biological effect that can be used as a drug. For example, it can be an agonist or antagonist of a receptor or enzyme; or it can be an antibody that is configured to bind to an antibody target (such as a viral coat protein, or a protein expressed on a cancer cell), for example to act as an agonist of a specific receptor or to prevent binding of another ligand and thereby prevent activation of a related biological pathway.

[0094] In a similar manner, the environment can be a drug design environment, such that each state is a corresponding state of a potential pharmaceutical drug, and the agent is a computer system for determining elements of the pharmaceutical drug and / or a synthetic pathway for the pharmaceutical drug. For example, in a simulation, the drug / synthesis can be designed based on a reward derived from the drug's goal. As another example, the agent can be a mechanical agent that performs or controls the synthesis of the drug.

[0095] In some further applications, the environment is a real-world environment, and the agent manages the allocation of tasks across computing resources, for example on a mobile device and / or in a data center. In these implementations, an action may include assigning a task to a specific computing resource.

[0096] As further examples, an action may include presenting an advertisement, an observation may include an advertisement impression or a click count or click rate, and a reward may characterize a previous selection of an item or content by one or more users.

[0097] In some cases, observations may include textual or verbal instructions provided to the agent by a third party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and the user of the autonomous vehicle may provide textual or verbal instructions to the agent (e.g., to navigate to a particular location).

[0098] As another example, the environment may be an electrical, mechanical or electromechanical design environment, such as an environment in which the design of an electrical, mechanical or electromechanical entity is simulated. The simulated environment may be a simulation of a real-world environment in which the entity is intended to work. The task may be to design an entity. The observations may include observations that characterize the entity, i.e., observations of the mechanical shape or electrical, mechanical or electromechanical configuration of the entity, or observations of parameters or properties of the entity. The actions may include actions that modify the entity, such as actions that modify one or more of the observations. The reward or reward may include one or more metrics of the performance of the design of the entity. For example, the reward or reward may be related to one or more physical properties of the entity, such as weight or strength; or to one or more electrical properties of the entity, such as a measure of the efficiency of performing a specific function for which the entity is designed. The design process may include outputting a design for manufacture, such as in the form of computer executable instructions for manufacturing the entity. The process may include making the entity according to the design. Thus, the design of the entity may be optimized, such as by reinforcement learning, and then the optimized design is output for manufacturing the entity, such as output as computer executable instructions; then the entity with the optimized design may be manufactured.

[0099] As previously mentioned, the environment can be a simulated environment. Generally speaking, in the case of a simulated environment, observations can include simulated versions of one or more observations or observation types described previously, and actions can include simulated versions of one or more actions or action types described previously. For example, the simulated environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent can be a simulated vehicle that navigates through the motion simulation. In these implementations, the action can be a control input that controls a simulated user or a simulated vehicle. Generally speaking, the agent can be implemented as one or more computers that interact with the simulated environment.

[0100] The simulated environment can be a simulation of a specific real-world environment and agent. For example, the system can be used to select actions in the simulated environment during training or evaluation of the system, and after training or evaluation or both are completed, it can be deployed to control a real-world agent in a specific real-world environment that is the object of simulation. This can avoid unnecessary wear and tear on the real-world environment or the real-world agent and damage to it, and can allow the control neural network to be trained and evaluated in situations that rarely occur in the real-world environment or are difficult or unsafe to recreate. For example, the system can be partially trained using a simulation of a mechanical agent in a simulation of a specific real-world environment, and then deployed to control a real mechanical agent in a specific real-world environment. Therefore, in such cases, observations of the simulated environment are related to the real-world environment, and the selected actions in the simulated environment are related to the actions to be performed by the mechanical agent in the real-world environment.

[0101] Optionally, in any of the above implementations, observations at any given time step may include data from a previous time step that may be helpful in characterizing the environment, such as actions performed at the previous time step, rewards received at the previous time step, or both.

[0102] Figure 2 is a flow chart of an example process 200 for training an action selection neural network. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to the present specification, such as Figure 1 The action selection system 100 can perform process 200.

[0103] The system obtains a training data set including a corresponding training history sequence for each of a plurality of tasks (step 202 ).

[0104] The training history sequence for a given task includes a sequence of labels representing transitions from multiple task episodes executed while training a policy for the task through reinforcement learning.

[0105] In other words, each training history sequence includes transitions generated while the agent for the task was being controlled by the policy at multiple different time points during training of the policy for the task.

[0106] Generally speaking, a labeled sequence in a training history sequence includes a corresponding round subsequence for each of a plurality of task rounds, and the corresponding round subsequence for each of the task rounds includes a corresponding transition subsequence for each transition from the task round.

[0107] Each transition subsequence further includes: (i) one or more tags representing observations in the transition, (ii) one or more tags representing actions in the transition, and (iii) one or more tags representing rewards in the transition.

[0108] Therefore, when there are a total of N tasks , the dataset D can be represented as: , in is the labeled sequence in the training history sequence of the task It means that when the task one or more labels for the observations received at time step t when training, It means that when the task One or more labels for the action to be performed at time step t during training, It means that in the task One or more labels of the reward received in response to performing an action at time step t during training, where t ranges from 0 to T, where T is the total number of time steps in the training history of the task ,and is a reinforcement learning algorithm used to train policies to perform tasks .

[0109] In general, within each training history sequence, the episode subsequences are ordered according to the order in which the corresponding task episodes were executed during training. occurs during the last task episode executed during training and transforms the subsequence It first happens during training.

[0110] Additionally, for any given task, the policy for the task is represented by a machine learning model (e.g., a neural network or other type of machine learning model with multiple weights). Therefore, because the transition history sequence includes transitions for different points in time during the training of this machine learning model, and because the values ​​of the weights are repeatedly updated during training, the training history sequence for any given task will include transitions generated while the agent for the task is being controlled according to multiple different sets of weight values ​​for the multiple weights.

[0111] Thus, any given training history sequence will reflect the use of policies to control the agent for a task at different stages of training, and may, for example, begin with transitions generated by controlling the agent using a random or near-random policy and then include subsequent transitions that reflect improvements in the quality of actions chosen as training progresses.

[0112] In some implementations, the same reinforcement learning algorithm is used to train the policies for all tasks. For example, each policy may be trained using a UCB (confidence upper bound) exploration technique, an online policy reinforcement learning technique (e.g., an on-policy actor-critic), an offline policy reinforcement learning technique (e.g., an offline policy DQN), or any other suitable reinforcement learning technique. Thus, the resulting training causes the action selection neural network to "distill" or "approximate" the reinforcement learning algorithm. In particular, the training may cause the action selection neural network to implement a more computationally efficient version of the algorithm after training, i.e., a version that requires fewer rounds of tasks to be executed to reach or exceed the final performance of the trained policy trained using the algorithm.

[0113] In some other implementations, different reinforcement learning algorithms may have been used to train respective policies for different ones of the tasks.

[0114] That is, the training history sequence will include both: (i) a first training history sequence including a labeled sequence representing transitions from a plurality of task episodes executed while training a policy for a first task by reinforcement learning using a first reinforcement learning algorithm, and (ii) a second training history sequence from the plurality of training history sequences including a labeled sequence representing transitions from a plurality of task episodes executed while training a policy for a second task by reinforcement learning using a second, different reinforcement learning algorithm. Thus, the resulting training of the action selection neural network causes the action selection neural network to generalize across a plurality of different reinforcement algorithms.

[0115] Reference below Figure 3 An example technique for generating a training history sequence is described.

[0116] The system then trains the action selection neural network on the training dataset by repeatedly performing steps 204 and 206.

[0117] The system selects a subsequence of a corresponding training history sequence from the training data set (step 204).

[0118] In general, a subsequence represents transitions from multiple rounds of tasks represented in the training history sequence. That is, a subsequence includes transition subsequences corresponding to transitions from multiple different rounds of tasks represented in the training history sequence.

[0119] For example, the system may randomly sample subsequences of a fixed length, i.e., a subsequence that includes a fixed number of transition subsequences, with the only constraint being that the subsequence includes transition subsequences from multiple episodes. When the fixed length exceeds the maximum number of time steps in an episode, this constraint is automatically satisfied, and the system may randomly sample transition subsequences having a fixed length. The fixed length may be determined, for example, based on the context size of the action selection neural network, where the context size is the maximum number of tokens in any given input sequence of the action selection neural network.

[0120] The system trains an action selection neural network to predict, for each transition in the subsequence, one or more tokens representing actions in the transition conditioned on tokens preceding the one or more tokens representing actions in the subsequence (step 206).

[0121] Any suitable objective function may be used. As just one example, the system may train an action selection neural network on a negative log-likelihood loss function L that satisfies:

[0122] Therefore, as the reinforcement learning policy improves with each history sequence, by accurately predicting actions, the action selection neural network learns to output an improved policy relative to the policy seen in context.

[0123] In some implementations, the system can utilize label smoothing during training. Label smoothing can involve smoothing a target for prediction, such as an action in a transition, relative to other possible targets (e.g., other possible actions).

[0124] In particular, as part of training on the negative log-likelihood loss function above, the system can use label smoothing regularization, where the system uses a smoothed distribution as the target probability distribution for time steps, which assigns a probability of 1–α to the action in the transition and a probability of α / (k–1) to each other action in the action set, where k is the total number of actions in the set and α is a positive hyperparameter between zero and one.

[0125] Figure 3 is a flow chart of an example process 300 for generating a training history for a given task. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to the present specification, such as Figure 1 The action selection system 100 can perform process 300.

[0126] The system trains a policy, such as a machine learning model, for a task through reinforcement learning (step 302).

[0127] As part of a training strategy, the system repeatedly performs steps 304 to 310 .

[0128] In particular, the system performs episodes of the task by controlling an agent for the task using a policy for the task to generate transitions (step 304), and stores the transitions in a replay memory (step 306). The system also samples one or more transitions from the replay memory (step 308), and trains the policy on the one or more sampled transitions using a reinforcement learning algorithm (step 310).

[0129] In some implementations, the algorithm is a single-stream algorithm, and steps 304 to 310 are all performed by the same set of one or more hardware devices.

[0130] In some other implementations, the algorithm is a distributed algorithm, and steps 304 and 306 are performed by multiple actors ("actor computing units"), each of which is implemented on a different set of hardware devices, while steps 308 and 310 are performed by one or more learners ("learner computing units") implemented on different sets of hardware devices. In general, multiple actors can perform rounds of the task in parallel with each other during training, i.e., each actor can perform iterations of steps 304 and 306 in parallel with each other actor. Optionally, one or more learners perform steps 308 and 310 asynchronously with the actors performing steps 304 and 306.

[0131] The system then generates a training history sequence for the task based on transitions generated as a result of performing the task (step 312).

[0132] When multiple actors perform episodes of a task in generating training data, a corresponding training history sequence for the task includes an episode subsequence of one or more episodes performed by each of the multiple actors.

[0133] In some implementations, the system includes in the training history sequence for the task a corresponding subsequence of each of the episodes generated during training.

[0134] However, in some other implementations, the system subsamples the episodes when generating the training history sequence. In particular, in the corresponding training history sequence of the task, the system only includes a subsequence of episodes for every kth episode of the task performed during training, where k is an integer greater than 1.

[0135] In some cases, instead of subsampling the rounds when generating the training history sequence, the system may instead obtain an original training history sequence for the task, the original training history sequence including a corresponding round subsequence for each of a plurality of original task rounds executed during training; and then subsample the original training history sequence by generating a corresponding historical training sequence including only a round subsequence for every k-th original task round in the original training history sequence, where k is an integer greater than 1. For example, the original training history may have been generated by another training system.

[0136] In both of these cases, when performing “in-context” reinforcement learning, this subsampling can help the action-selection neural network become more data-efficient after training, for example, because the action-selection neural network can learn to “distill” a more aggressive reinforcement learning algorithm through subsampling.

[0137] Figure 4 is a diagram illustrating an example 400 of training the action selection neural network 102 .

[0138] In particular, in example 400 , the training includes a data generation phase 410 and a model training phase 420 .

[0139] In the data generation phase 410, the system generates a training history for each of the N tasks by training a corresponding policy for each task using reinforcement learning. .

[0140] like Figure 4 As shown and described above, the training history for a given task includes data from transitions generated after different amounts of learning progress have occurred during training of the policy for the task.

[0141] In the model training phase 420, the system trains the action selection neural network 120 (in Figure 4 In the example of , a causal Transformer neural network is used to predict actions given the cross-turn context described above. That is, for a transition at time step t, the system predicts actions based on the following probabilities to train the action selection neural network.

[0142] Figure 5 is a flow chart of an example process 500 for controlling an agent at time step 1 using an action selection neural network. For convenience, process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to the present specification, such as Figure 1 The action selection system 100 can perform process 500.

[0143] The system may perform process 200 at each time step during a sequence of time steps (eg, at each time step during a mission round).

[0144] The system can also continue the process across different task rounds of the same task in order to perform "in-context" reinforcement learning of the task as described above.

[0145] That is, the system can repeatedly perform process 500 to perform a sequence of task rounds of the task and reinforce learning “in context” how to perform the task during the task rounds.

[0146] The system receives a current observation characterizing a state of the environment at a time step (step 502).

[0147] The system generates a labeled input sequence (step 504). The labeled input sequence includes: (i) one or more labels representing a current observation, (ii) a corresponding current transition subsequence for each current round transition in one or more current round transitions, and (iii) a corresponding previous transition subsequence for each previous round transition in one or more previous round transitions.

[0148] Each current-round transition corresponds to a corresponding earlier time step in the current task round, and a corresponding transition subsequence of the current-round transitions includes: (a) one or more tokens representing observations received at the earlier time step, (b) one or more tokens representing actions performed by the agent in response to observations received at the earlier time step, and (c) one or more tokens representing rewards received in response to the agent performing the actions.

[0149] Each previous-round transition corresponds to a corresponding earlier time step in a corresponding previous task round performed by the agent prior to the current task round, i.e., corresponds to a task round different from the current task round being performed, and the corresponding previous-round transition subsequence includes: (a) one or more tags representing observations received at the earlier time step, (b) one or more tags representing actions performed by the agent in response to observations received at the earlier time step, and (c) one or more tags representing rewards received in response to the agent performing the actions.

[0150] Thus, the input sequence includes subsequences of transitions from multiple different rounds (ie, the current round and at least one earlier round).

[0151] For example, the system can include in the input sequence corresponding subsequences of up to a maximum number of most recently generated transitions, e.g., where the maximum number is determined based on a context size of the action neural network. If more than the maximum number of transitions have been generated for the current task, the system selects the most recently generated transition to represent in the input sequence.

[0152] In some implementations, if fewer than a maximum number of transitions have been generated, and the system has access to external data (e.g., from an expert policy for the task), the system can use the external data to "hint" the action-selection neural network by including in the input sequence a subsequence representing transitions from the expert data in addition to a subsequence from historical data generated as a result of agent control using the action-selection neural network.

[0153] The system processes the labeled input sequence using an action selection neural network to generate a policy output for a time step (step 506), and uses the policy output to select an action (step 508).

[0154] The system then causes the agent to perform the selected action (step 510).

[0155] In general, the agent is controlled while keeping the values ​​of the parameters of the action-selection neural network fixed to their trained values, determined, for example, by training the action-selection neural network as described above.

[0156] Figure 6 An example 600 of an attention map generated by an action selection neural network across time steps when controlling an agent for a new task is shown. Figure 6 In the example of , each episode is 50 steps, the maximum size is the transition from 200 time steps, the left column 610 shows the attention map from step 0 to 200, and the right column 620 shows the attention map from step 1700 to 1900. White and gray correspond to low attention and high attention, respectively. From these patterns, it is clear that the action selection neural network pays attention to the tokens across several episodes to predict its next action.

[0157] Figure 7 Examples of results achieved by the described technique ("AD") on four different tasks relative to two benchmarks ("ED" and "source") are shown. Figure 7 As can be seen, AD is consistently effective in the context of reinforcement learning for all four tasks while being more data efficient than any of the baselines. Figure 7 Also shown is an online RL algorithm (“RL 2 ”) as an upper bound on the performance of AD.

[0158] In particular, Figure 7 In the example of , one of the two benchmarks is a source RL (“source”) algorithm for generating training data for a neural network used in AD technology. Figure 7 As can be seen in Figure 2, the described techniques produce policies that learn the task more effectively while being much more data efficient than the source RL algorithm.

[0159] One of the other two benchmarks ("ED") is the expert distillation technique, which uses the same action selection neural network as AD, but is trained only on "expert" trajectories instead of learning histories. As can be seen from the figure, AD significantly outperforms ED, for example, because AD uses learning histories during training instead of simply using high-performance expert trajectories.

[0160] This specification uses the term "configured" in connection with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0161] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal), generating an artificially generated propagation signal to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0162] The term "data processing equipment" refers to data processing hardware, and includes all kinds of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer or a plurality of processors or computers. The equipment may also be or further include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may also optionally include code for creating an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0163] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script or code, may be written in any form of programming language (including compiled or interpreted languages ​​or declarative or procedural languages); and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0164] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0165] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmed computers.

[0166] A computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for fulfilling or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit system or incorporated into the special logic circuit system. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or operatively coupled to receive data from it or transfer data to it or both. However, the computer does not have to have such a device. In addition, the computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0167] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0168] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device (e.g., a smart phone running a messaging application), and receiving a response message from the user in return.

[0169] A data processing device used to implement a machine learning model may also include, for example, dedicated hardware accelerator units for processing general-purpose and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0170] Machine learning models can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework or the Jax framework).

[0171] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0172] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from it. Data generated at the user device (e.g., the result of the user interaction) can be received from the device at the server.

[0173] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve a sub-combination or a variation of the sub-combination.

[0174] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in an ordered order, or that all of the operations shown be performed, to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0175] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the ordered order to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous.

[0176] Various aspects of the present disclosure may be set out in the following terms:

[0177] Clause 1. A method performed by one or more computers, the method comprising: A training data set is obtained, wherein the training data set includes a corresponding training history sequence of each task in a plurality of tasks, wherein: Each training history sequence comprises a sequence of labels representing transitions from a plurality of task episodes executed while training a policy for said task via reinforcement learning, The tag sequence includes a corresponding round subsequence of each task round in the plurality of task rounds, and The respective round subsequence of each of the task rounds comprises, for each transition from the task round, a respective transition subsequence comprising: (i) one or more labels representing observations in said transition, (ii) one or more tokens indicating the action in the transition, and (iii) one or more indicia representing the reward being transformed; and Training an action selection neural network on the training data set, wherein each training step in a plurality of training steps comprises: Selecting from the training data set a subsequence from a corresponding training history sequence of the training data set, the subsequence representing transitions from a plurality of the task rounds represented in the training history sequence; and The action selection neural network is trained to predict, for each transition in the subsequence, the one or more tokens representing the action in the transition conditioned on the tokens preceding the one or more tokens representing the action in the subsequence.

[0178] Clause 2. The method of clause 1, wherein, within each training history sequence, the round subsequences are ordered according to the order in which corresponding task rounds were executed during the training.

[0179] Clause 3. A method as described in any preceding clause, wherein each training history sequence includes transitions generated while an agent for the task is being controlled by the policy at multiple different time points during the training of the policy for the task.

[0180] Clause 4. A method as described in Clause 3, wherein the strategy for the task is represented by a machine learning model having multiple weights, and wherein the training history sequence includes transitions generated while the agent for the task is being controlled according to multiple different sets of weight values ​​for the multiple weights.

[0181] Clause 5. A method as described in any preceding clause, wherein: A first training history sequence of the plurality of training history sequences includes a sequence of labels representing transitions from a plurality of task episodes executed while training a policy for a first task by reinforcement learning using a first reinforcement learning algorithm; and A second training history sequence in the plurality of training history sequences includes a sequence of labels representing transitions from a plurality of task episodes executed while training a policy for a second task by reinforcement learning using a second, different reinforcement learning algorithm.

[0182] Clause 6. A method as described in any preceding clause, wherein obtaining a training data set including a corresponding training history sequence for each task in a plurality of tasks comprises, for each of one or more of the tasks: The policy for the task is trained by reinforcement learning, the training comprising repeatedly performing the following operations: executing episodes of the task by controlling an agent for the task using the policy for the task to generate transitions; storing the transition in a replay memory; sampling one or more transitions from the replay memory; and The policy is trained on the one or more sampled transitions.

[0183] Clause 7. The method of clause 6, wherein obtaining a training data set including a corresponding training history sequence for each of the plurality of tasks further comprises, for each of the one or more tasks: The corresponding training history sequence of the task only includes a round subsequence of every k-th round of the task performed during the training, where k is an integer greater than 1.

[0184] Clause 8. The method of clause 6 or clause 7, wherein repeatedly performing the following operations comprises: Multiple rounds of the task are performed in parallel using multiple actor computing units, and wherein, The corresponding training history sequence for the task includes a turn subsequence of one or more turns performed by each of the plurality of actors.

[0185] Clause 9. The method of any one of clauses 1 to 8, wherein obtaining a training data set including a corresponding training history sequence for each of a plurality of tasks comprises, for each of one or more of the tasks: obtaining an original training history sequence of the task, the original training history sequence comprising a corresponding round subsequence of each original task round in a plurality of original task rounds executed during the training; and Generate a corresponding historical training sequence, wherein the corresponding historical training sequence only includes a round subsequence of every k-th original task round in the original training historical sequence, where k is an integer greater than 1.

[0186] Clause 10. A method as described in any preceding clause, wherein one or more training steps in the plurality of training steps: Training the action selection neural network to predict, for each transition in the subsequence, the one or more tokens representing the action in the transition conditioned on the token preceding the one or more tokens representing the action in the subsequence comprises: The action selection neural network is trained using label smoothing.

[0187] Clause 11. A method for controlling a sequence of task rounds of an agent to perform a task, executed by one or more computers, the method comprising, for each time step in a sequence of time steps in a current task round: receiving a current observation representing a state of the environment at the time step; Generate a labeled input sequence, the input sequence comprising: (i) one or more tags representing the current observation, (ii) a corresponding current-round transition subsequence for each of one or more current-round transitions, each current-round transition corresponding to a corresponding earlier time step in the current task round, and the corresponding transition subsequence of the current-round transitions comprising: (a) one or more indicia representing observations received at the earlier time step, (b) one or more indicia representing actions performed by the agent in response to receiving the observations at the earlier time step, and (c) one or more indicia representing rewards received by the agent in response to performing the actions, and (iii) a corresponding previous-round transition subsequence for each of one or more previous-round transitions, each previous-round transition corresponding to a corresponding earlier time step in a corresponding previous task round performed by the agent prior to the current task round, and the corresponding previous-round transition subsequence of the previous-round transitions comprising: (a) one or more indicia representing observations received at the earlier time step, (b) one or more indicia representing actions performed by the agent in response to receiving the observations at the earlier time step, and (c) one or more indicia representing rewards received by the agent in response to performing the actions; processing the labeled input sequence using the action selection neural network to generate a policy output for the time step; Using the policy output to select an action; and Causes the agent to perform the selected action.

[0188] Clause 12. A method as described in Clause 11, wherein controlling the sequence of task turns of an agent to perform a task includes controlling the agent while keeping the values ​​of the parameters of the action selection neural network fixed to the trained values ​​determined by training the action selection neural network by performing the corresponding operations as described in any of Clauses 1 to 10.

[0189] Clause 13. A method as described in Clause 11 or Clause 12, wherein the action selection neural network is a causal Transformer neural network.

[0190] Clause 14. The method of clause 11 or clause 12, wherein the action selection neural network is a recurrent neural network.

[0191] Clause 15. A method as described in any of Clauses 11 to 14, wherein the input sequence includes the previous transition subsequence, followed by the current transition subsequence, and followed by the one or more markers representing the current observation, and wherein the previous transition subsequence and the current transition subsequence are ordered within the input sequence according to the order in which the corresponding transitions are generated.

[0192] Clause 16. A method as described in any preceding clause, wherein each task requires the strategy to control a corresponding agent that interacts with a corresponding environment, wherein, for one or more of the tasks, the corresponding agent is a mechanical agent that interacts with a real-world environment.

[0193] Clause 17. A method as described in Clause 16, wherein the mechanical agent is a robot.

[0194] Clause 18. A method as described in any of the preceding clauses, wherein each task requires the strategy to control a corresponding agent interacting with a corresponding environment, wherein, for one or more of the tasks, the environment is a real-world environment of a service facility including multiple items of electronic devices, and the agent is an electronic agent configured to control the operation of the service facility.

[0195] Clause 19. A method as described in any of the preceding clauses, wherein each task requires the strategy to control a corresponding agent interacting with a corresponding environment, wherein, for one or more of the tasks, the environment is a real-world manufacturing environment for manufacturing products, and the agents include electronic agents configured to control manufacturing units or machines that operate to manufacture the products.

[0196] Clause 20. A system comprising: one or more computers; and One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of a corresponding method as described in any one of clauses 1 to 19.

[0197] Clause 21. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective methods of any of Clauses 1-19.

Claims

1. A method executed by one or more computers, the method comprising: A training data set is obtained, wherein the training data set includes a corresponding training history sequence of each task in a plurality of tasks, wherein: Each training history sequence comprises a sequence of labels representing transitions from a plurality of task episodes executed while training a policy for said task via reinforcement learning, The tag sequence includes a corresponding round subsequence of each task round in the plurality of task rounds, and The respective round subsequence of each of the task rounds comprises, for each transition from the task round, a respective transition subsequence comprising: (i) one or more labels representing observations in said transition, (ii) one or more tokens indicating the action in the transition, and (iii) one or more indicia representing the reward being transformed; and Training an action selection neural network on the training data set, wherein each training step in a plurality of training steps comprises: Selecting from the training data set a subsequence from a corresponding training history sequence of the training data set, the subsequence representing transitions from a plurality of the task rounds represented in the training history sequence; and The action selection neural network is trained to predict, for each transition in the subsequence, the one or more tokens representing the action in the transition conditioned on the tokens preceding the one or more tokens representing the action in the subsequence.

2. The method of claim 1, wherein: Within each training history sequence, the round subsequences are ordered according to the order in which the corresponding task rounds were executed during the training.

3. A method as claimed in any preceding claim, wherein: Each training history sequence includes transitions generated while the agent for the task was being controlled by the policy at multiple different points in time during the training of the policy for the task.

4. The method of claim 3, wherein: The policy for the task is represented by a machine learning model having a plurality of weights, and wherein the training history sequence includes transitions generated while the agent for the task is being controlled according to a plurality of different sets of weight values ​​for the plurality of weights.

5. A method as claimed in any preceding claim, wherein: A first training history sequence of the plurality of training history sequences includes a sequence of labels representing transitions from a plurality of task episodes executed while training a policy for a first task by reinforcement learning using a first reinforcement learning algorithm; and A second training history sequence in the plurality of training history sequences includes a sequence of labels representing transitions from a plurality of task episodes executed while training a policy for a second task by reinforcement learning using a second, different reinforcement learning algorithm.

6. A method as claimed in any preceding claim, wherein: Obtaining a training data set including a corresponding training history sequence for each of a plurality of tasks comprises, for each of one or more of the tasks: The policy for the task is trained by reinforcement learning, the training comprising repeatedly performing the following operations: executing episodes of the task by controlling an agent for the task using the policy for the task to generate transitions; storing the transition in a replay memory; sampling one or more transitions from the replay memory; and The policy is trained on the one or more sampled transitions.

7. The method of claim 6, wherein: Obtaining a training data set including a corresponding training history sequence for each of the plurality of tasks further includes, for each of the one or more tasks: The corresponding training history sequence of the task only includes a round subsequence of every k-th round of the task performed during the training, where k is an integer greater than 1.

8. The method of claim 6 or claim 7, wherein: Repeat the following actions: Multiple rounds of the task are performed in parallel using multiple actor computing units, and wherein, The corresponding training history sequence for the task includes a turn subsequence of one or more turns performed by each of the plurality of actors.

9. The method according to any one of claims 1 to 8, wherein: Obtaining a training data set including a corresponding training history sequence for each of a plurality of tasks comprises, for each of one or more of the tasks: obtaining an original training history sequence of the task, the original training history sequence comprising a corresponding round subsequence of each original task round in a plurality of original task rounds executed during the training; and Generate a corresponding historical training sequence, wherein the corresponding historical training sequence only includes a round subsequence of every k-th original task round in the original training historical sequence, where k is an integer greater than 1.

10. A method as claimed in any preceding claim, wherein: One or more training steps in the plurality of training steps: Training the action selection neural network to predict, for each transition in the subsequence, the one or more tokens representing the action in the transition conditioned on the token preceding the one or more tokens representing the action in the subsequence comprises: The action selection neural network is trained using label smoothing.

11. A method for controlling a task round sequence of an agent to perform a task, executed by one or more computers, the method comprising, for each time step in a plurality of time steps in a time step sequence in a current task round: receiving a current observation representing a state of the environment at the time step; Generate a labeled input sequence, the input sequence comprising: (i) one or more tags representing the current observation, (ii) a corresponding current-round transition subsequence for each of one or more current-round transitions, each current-round transition corresponding to a corresponding earlier time step in the current task round, and the corresponding transition subsequence of the current-round transitions comprising: (a) one or more indicia representing observations received at the earlier time step, (b) one or more indicia representing actions performed by the agent in response to receiving the observations at the earlier time step, and (c) one or more indicia representing rewards received by the agent in response to performing the actions, and (iii) a corresponding previous-round transition subsequence for each of one or more previous-round transitions, each previous-round transition corresponding to a corresponding earlier time step in a corresponding previous task round performed by the agent prior to the current task round, and the corresponding previous-round transition subsequence of the previous-round transitions comprising: (a) one or more indicia representing observations received at the earlier time step, (b) one or more indicia representing actions performed by the agent in response to receiving the observations at the earlier time step, and (c) one or more indicia representing rewards received by the agent in response to performing the actions; processing the labeled input sequence using the action selection neural network to generate a policy output for the time step; Using the policy output to select an action; and Causes the agent to perform the selected action.

12. The method of claim 11, wherein: Controlling an agent to perform a task turn sequence includes controlling the agent while keeping the values ​​of the parameters of the action selection neural network fixed to the trained values ​​determined by training the action selection neural network by performing the corresponding operations described in any one of claims 1 to 10.

13. The method of claim 11 or claim 12, wherein: The action selection neural network is a causal Transformer neural network.

14. The method of claim 11 or claim 12, wherein: The action selection neural network is a recurrent neural network.

15. The method according to any one of claims 11 to 14, wherein: The input sequence comprises the previous transition subsequence, followed by the current transition subsequence, and followed by the one or more markers representing the current observation, and wherein the previous transition subsequence and the current transition subsequence are ordered within the input sequence according to an order in which corresponding transitions are generated.

16. A method as claimed in any preceding claim, wherein: Each task requires the policy to control a corresponding agent that interacts with a corresponding environment, wherein, for one or more of the tasks, the corresponding agent is a mechanical agent that interacts with a real-world environment.

17. The method of claim 16, wherein: The mechanical agent is a robot.

18. A method as claimed in any preceding claim, wherein: Each task requires the strategy to control a corresponding agent interacting with a corresponding environment, wherein, for one or more of the tasks, the environment is a real-world environment of a service facility including multiple items of electronic devices, and the agent is an electronic agent configured to control the operation of the service facility.

19. A method as claimed in any preceding claim, wherein: Each task requires the strategy to control a corresponding agent interacting with a corresponding environment, wherein, for one or more of the tasks, the environment is a real-world manufacturing environment for manufacturing products, and the agents include electronic agents configured to control manufacturing units or machines that operate to manufacture the products.

20. A system comprising: one or more computers; as well as One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 19.

21. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective methods of any one of claims 1 to 19.

Images