Reinforcement learning with centralized reasoning and decentralized training

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing reinforcement learning with centralized reasoning and training. One of the methods includes receiving, at a current time step of a plurality of time steps, a respective observation for each of a plurality of environments by an actor; obtaining, for each environment, a respective reward for the actor as a result of the actor performing a respective action at a previous time step preceding the current time step; processing, for each environment, the respective observation and the respective reward by a policy model; providing, to the actor, a respective policy output for each of the plurality of environments; maintaining, at a repository, for each environment, a respective sequence of tuples corresponding to the actor; determining that the maintained sequence satisfies a threshold condition; and in response, training the policy model on the maintained sequence.

Description

Technical Field

[0001] This manual relates to reinforcement learning. Background Technology

[0002] In reinforcement learning systems, agents interact with the environment by performing actions selected by the reinforcement learning system in response to observations that represent the current state of the environment.

[0003] Some reinforcement learning systems respond to receiving a given observation based on the output of a neural network and select an action to be performed by an agent.

[0004] A neural network is a machine learning model that uses one or more non-linear units to predict the output of a received input. Some neural networks are deep neural networks, which include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates its output from the received input based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0005] This specification describes techniques for performing reinforcement learning using a centralized policy model.

[0006] In one aspect, this specification relates to a method comprising: receiving corresponding observations generated by a corresponding actor for each environment in a plurality of environments; for each environment, processing a corresponding policy input, including the corresponding observations of the environment, through a policy model to obtain a corresponding policy output for the actor; providing a corresponding action to the corresponding actor for each environment; for each environment, obtaining a corresponding reward for the corresponding actor in the environment as a result of performing the provided action in the environment; maintaining a corresponding sequence of tuples for each environment; determining that the maintained sequence satisfies a threshold condition; and, in response, training a policy model on the maintained sequence.

[0007] The implementation may include one or more of the following features: A policy model has multiple model parameter values. A corresponding policy output defines a control policy for performing a task in the environment. A corresponding action is determined based on the control policy defined by the corresponding policy output. At least one tuple in the corresponding tuple sequence includes the corresponding observation, action, and reward obtained by the actor in response to performing the action in the environment. The corresponding tuple sequence is stored in a priority replay buffer and sampled from the priority replay buffer to train the policy model. Policy inputs may include batches of corresponding policy model inputs, and policy outputs may include batches of corresponding policy outputs for each batch of corresponding policy model inputs. The actor does not include the policy model.

[0008] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0009] By using a centralized policy model, the system implementing the subject matter of this specification can be easily scaled to handle observations from any number of actors in any number of different environments. Because the policy model is centralized at the learner engine, the learner engine does not need to synchronize model parameter values ​​and other values ​​used for the policy model between each actor interconnected with it. Instead, the network communication between the actors and the learner engine, i.e., data transmission, is reduced to only inference calls from actors to the learner engine and actions generated by the learner engine in response to those inference calls.

[0010] Because inference and training are centralized, computationally more expensive and scarce computing resources can be concentrated on executing and training the policy model, rather than on hardware that implements a less computationally efficient actor engine. For example, the learner engine can be implemented on multiple hardware accelerators (e.g., neural network accelerators such as Tensor Processing Units (“TPUs”), where separate processing threads are dedicated to handling inference calls, training, and data prefetching operations, such as batching training data, queuing data, or sending data to a priority replay buffer and / or a device buffer for one or more hardware accelerators. The actor does not need to alternate between operations for executing actions in the environment and operations for generating new policy outputs that define future actions more suitable for execution on the hardware accelerator.

[0011] The learning engine can automatically or in response to user input adjust the ratio between accelerators configured to perform inference operations and accelerators configured to perform training operations. In some implementations, a specific ratio of inference to training allocation increases the overall throughput of the system implementing the learner engine.

[0012] Furthermore, the learner engine is configured to receive and respond to inference calls from actors while maintaining training data for later updating the parameter values ​​of the policy model. The learner engine is configured to train the policy model on the maintained data, and once the policy model is trained, the learner engine is configured to respond to subsequent inference calls by processing the received observations via the newly updated policy model and providing actions sampled from the newly updated policy model. This eliminates the need to update each actor individually using the updated policy model, thus improving system efficiency and accuracy.

[0013] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0014] Figure 1 An exemplary centralized reasoning reinforcement learning system is shown.

[0015] Figure 2 The exemplary learner engine of the exemplary centralized reasoning reinforcement learning system is illustrated in detail.

[0016] Figure 3A The illustration shows an exemplary off-policy reinforcement learning process utilized by the system.

[0017] Figure 3B The illustration shows another exemplary off-policy reinforcement learning process utilized by the system.

[0018] Figure 4 An exemplary process for centralized reinforcement learning is illustrated. Detailed Implementation

[0019] This specification generally describes a reinforcement learning system that trains a policy model in a centralized manner. A policy model is a machine learning model used to control an agent interacting with the environment in response to observations representing the state of the environment, for example, to perform a specific task within the environment.

[0020] In some implementations, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent may be a robot that interacts with the environment, such as to locate an object of interest in the environment, move the object of interest to a designated location in the environment, physically manipulate the object of interest in the environment, and / or navigate to a designated destination in the environment; or the agent may be an autonomous or semi-autonomous land, air, or sea vehicle that navigates to a designated destination in the environment.

[0021] In these implementations, observation may include one or more of, for example, images, object location data, and sensor data, to capture observations as the agent interacts with the environment, such as sensor data from images, distance or position sensors, or from actuators.

[0022] For example, in the case of a robot, observations may include data characterizing the robot’s current state, such as one or more of the following: joint positions, joint velocities, joint forces, global orientation, torque and / or acceleration, such as gravity-compensated torque feedback, and the global or relative pose of an object held by the robot.

[0023] In the case of robots or other mechanical agents or vehicles, observation can similarly include one or more of the following: position, linear or angular velocity, force, torque and / or acceleration, and global or relative posture of one or more parts of the agent. Observation can be defined in 1D, 2D, or 3D and can be absolute and / or relative.

[0024] Observations may also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or, for example, image or video data from a camera or lidar (LIDAR) sensor, such as data from a sensor located separately from the agent in the environment.

[0025] In the case of electronic agents, observation may include data from one or more sensors monitoring a part of the plant or service facility, such as current, voltage, power, temperature, and other sensor and / or electronic signals representing the function of the electronic and / or mechanical parts of the instrument.

[0026] Actions can be control inputs for controlling robots or autonomous or semi-autonomous land, air, or sea vehicles, such as the torque of a robot's joints or higher-level control commands, such as the torque of a vehicle's control surfaces or other control elements.

[0027] In other words, actions can include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a part of another mechanical agent. Actions may additionally or alternatively include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within an environment, the control of which affects the observed environmental state. For example, in the case of autonomous or semi-autonomous land, air, or sea vehicles, actions can include actions controlling navigation, such as steering and movement, such as braking and / or acceleration of the vehicle.

[0028] In some implementations, the environment is a simulated environment, and the agent is implemented as one or more computers that interact with the simulated environment.

[0029] Training an agent in a simulated environment allows the agent to learn from a large amount of simulated training data, while avoiding the risks associated with training an agent in a real-world environment, such as damage caused by performing poorly chosen actions. An agent trained in a simulated environment can then be deployed in a real-world environment.

[0030] For example, the simulation environment can be a motion simulation of a robot or vehicle, such as a driving simulation or a flight simulation. In these implementations, the action can be a control input that controls the simulated user or the simulated vehicle.

[0031] In another example, the simulated environment could be a video game, and the agent could be a simulated user playing the video game.

[0032] In another example, the environment can be a protein folding environment, such that each state is a corresponding state of the protein chain, and the agent is a computer system for determining how to fold the protein chain. In this example, actions are possible folding actions for folding the protein chain, and the desired outcome can include, for example, folding the protein so that it is stable and thus performs a specific biological function. As another example, the agent can be a mechanical agent that automatically executes or controls protein folding actions selected by the system without human interaction. Observations can include direct or indirect observations of protein states and / or can be derived from simulation.

[0033] Typically, in a simulated environment, observations may include simulated versions of one or more of the previously described observations or observation types, and actions may include simulated versions of one or more of the previously described actions or action types.

[0034] In some other applications, the agent can control actions in a real-world environment, including those involving equipment components, such as actions in a data center or power or water distribution system on a grid trunk line, or actions in a manufacturing plant or service facility. Observations can then be related to the operation of the plant or facility. For example, observations could include observations of the power or water usage of equipment, or observations of power generation or distribution control, or observations of resource utilization or waste generation. The agent can control actions in the environment to improve efficiency, for example, by reducing resource utilization, and / or reducing the environmental impact of operations in the environment, such as by reducing waste. Actions can include actions that control or impose operating conditions on components of equipment in the plant / facility, and / or actions that cause changes in setpoints in the operation of the plant / facility, such as adjusting or turning on / off components of the plant / facility.

[0035] In some applications, the environment is a content recommendation environment, and actions correspond to different content items that can be recommended to the user. That is, each action recommends a specific content item to the user. In these applications, observations are data representing the context of content recommendations, such as data characterizing the user, data characterizing content items previously presented to the user, currently presented to the user, or both.

[0036] Optionally, in any of the above embodiments, observations at any given time step may include data from a previous time step that may be advantageous in characterizing the environment, such as actions performed at the previous time step, rewards received at the previous time step, or both.

[0037] Reinforcement learning systems, facilitated by accelerators such as Tensor Processing Units (TPUs) and Graphics Processing Units (GPUs), have demonstrated the ability to perform tasks related to distributed training at scale by processing large amounts of data collected from multiple environments (i.e., multiple versions of the target environment that will control the agent after distributed training).

[0038] Typically, the various tasks performed by a reinforcement learning system during distributed training are inherently heterogeneous; that is, even in the same environment, the tasks differ from one another. For example, different tasks may include observing the environment and collecting data representing the observations from the environment, using the collected data to make inference calls to a policy model, generating policy outputs using the policy model in response to the inference calls, defining policy outputs that allow actors to take appropriate actions to function in the corresponding environment, and training the policy model based on the collected data.

[0039] The aforementioned heterogeneous tasks require corresponding computational resources, such as computing power, storage, and data transfer bandwidth, and the total computational cost increases dramatically with the increase in the size of the data observed from the environment, the average complexity of the task to which the policy model is trained, and the number of agents and environments in the reinforcement learning environment.

[0040] The described technique addresses the aforementioned problems by efficiently allocating distributed training tasks to minimize the computational cost of performing distributed training. In short, a system implemented using the described technique centralizes tasks in a learner engine that trains a policy model and responds to inference calls from multiple distributed actors. Simultaneously, the system distributes tasks to the corresponding actors, such as observing data representing observations in the corresponding environment (i.e., current state and reward), and invoking the learner engine to make inference calls on actions generated by the policy model based on the observed data. Therefore, the only data transfer occurring between the learner engine and the distributed actors is the data representing observations and actions. Each distributed actor does not need to communicate with the learner engine to obtain model parameters defining the policy model, nor does it need to train the policy model itself.

[0041] Figure 1 An exemplary centralized inference reinforcement learning system 100 is shown. System 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, wherein the systems, components and techniques described below can be implemented.

[0042] System 100 trains a policy model 108 used to control the agent, that is, to select actions to be performed by the agent when the agent interacts with the environment so that the agent can perform one or more tasks.

[0043] The policy model 108 is configured to receive policy inputs including input observations characterizing the state of the environment, and to process the policy inputs to generate a policy output that defines a control policy for controlling the agent.

[0044] In some implementations, the policy output may be, or can define, a probability distribution of a set of actions that can be performed by the agent. The system 100 can then sample from the probability distribution to obtain actions from the action set. Alternatively, the policy output can directly identify actions from the action set. As another example, the policy input may also include actions from the action set, and the policy output may include the Q-values ​​of the input actions. The system can then generate a corresponding Q-value for each action in the action set and select actions based on the corresponding Q-values, for example, by selecting the action with the highest Q-value or by transforming the Q-values ​​into a probability distribution and then sampling from the probability distribution.

[0045] In some implementations, the control strategy used by the system allows the agent to explore the environment. For example, the system can apply an exploration strategy to the policy output, such as an ε-greedy exploration strategy.

[0046] Typically, the policy model 108 can be implemented as a machine learning model. In some implementations, the policy model is a neural network with multiple layers, each layer having a corresponding set of parameters. In this specification, the parameters of the neural network are collectively referred to as "model parameters".

[0047] Typically, policy model 108 can have any suitable type of neural network architecture. For example, policy model 108 can be one or more convolutional neural networks, one or more recurrent neural networks, or any combination of both convolutional neural networks and recurrent neural networks. In embodiments where the neural network includes a recurrent neural network, policy model 108 can take the form of a long short-term memory (“LSTM”) network, a gated recurrent unit network, a multiplicative LSTM network, or an LSTM network incorporating attention.

[0048] System 100 trains policy model 108 in a centralized manner. To allow system 100 to perform centralized training, system 100 includes multiple actors 102 and a learning engine 110, which can communicate with the multiple actors 102, for example, by sending data over a data communication network, a physical connection, or both. Actors 102 can send input data 112 to the learning engine 110 and receive output data 122 from the learning engine 110 via the communication network and / or physical connection.

[0049] Each actor 102a-102z is implemented as one or more computer programs on one or more computers and is configured to observe one or more environments 104. In some cases, actors 102a-102z may be implemented on the same computer, while in other cases, actors 102a-102z may be implemented on different computers from each other.

[0050] In other words, for each of one or more environments 104, each actor 102a-102z is configured to control one or more corresponding copies of the agent when the agent copy interacts with the environment 104, i.e., to select an action to be performed by the corresponding agent copy in response to an observation that at least characterizes the state of the agent copy in the environment 104. Each agent copy is a corresponding version of the target agent that the policy model will be used to control after training. For example, when the target agent is a mechanical agent, each agent copy may be a different instance of the same mechanical agent or a corresponding computer simulation of the mechanical agent. When the target agent is a simulated agent or other computerized agent, each agent copy is also a computerized agent. In this specification, the agent copy that performs the action may also be referred to as the corresponding actor that performs the action.

[0051] Each environment 104 is a version of the target environment to which the target agent will be deployed after the policy model has been trained. In particular, each environment 104a-104z can be a real-world environment or a simulated environment, and in some cases, one or more of environments 104a-104z are real-world environments, while one or more other environments are simulated environments.

[0052] In order to control the agent copy in a given environment at a given time step, actor 102 receives an observation 114 at that time step, which characterizes the environmental state at that time step, and provides input data 112 representing the observation and optional other data to learner engine 110 to request output data 122 representing the action that the agent copy should perform in response to the observation. Actor 102, which requests actions from learner engine 110 using input data 112, can also be referred to as making an inference call to learner engine 110.

[0053] In other words, instead of each actor 102 maintaining a separate backup of the trained policy model 108 and the learner engine 108 sending data representing the parameters used for the trained policy model to each actor, actor 102 submits inference calls to the learner engine 100.

[0054] In response to each inference call, the learner engine 110 uses the policy model 108 to obtain the corresponding action and sends output data 122 to the actor that transmitted the inference call, which identifies the action obtained using the policy model 108. The actor then interacts with the corresponding agent copy to execute the received action in the appropriate environment.

[0055] Each actor 102 is associated with one or more environments 104. For example, actor 102z is assigned from environments 104a-104z. When each agent replica interacts with environment 104, the corresponding actor 102a-102z can receive observation 114 from the corresponding environment 104 at each of multiple time steps. For example, actor 102a can receive observation 114a from environment 104a at one of multiple time steps. As another example, actor 102z can receive observation 114c from environment 104z at one of multiple time steps. At each of the multiple time steps, each agent replica executes an action generated using policy model 108 while being controlled by the corresponding actor 102, wherein the received action is generated by policy model 108.

[0056] At each time step, each agent replica can also receive a reward in response to the action taken by the agent replica in the corresponding environment at that time step.

[0057] Typically, the reward is a numerical value (e.g., a scalar) and represents the agent's progress toward completing the task at the time step in which the reward is received.

[0058] As a specific example, the reward can be a sparse binary reward, which is zero unless the task is successfully completed in the plot, and 1 if the task is successfully completed because the action is performed in the plot.

[0059] As another specific example, the reward could be a dense reward, which measures the robot's progress toward completing the task during the episode of attempting to perform the task, up to individual observations received at multiple time steps.

[0060] The learner engine 110 includes a policy model 108 and a queue 148.

[0061] Typically, the learner engine 110 trains the policy model 108 in a centralized manner by repeatedly receiving input data 112 representing observations from multiple actors 102 and sending output data 122 representing actions of agent replicas to the multiple actors 102. While the policy model is being trained by the learner engine 110, the output data 122 is generated by the policy model 108. The input data 112 can further identify the reward received by the corresponding agent in response to an action taken in the corresponding environment at a previous time step.

[0062] As a specific example, the learner engine 110 can batch input data 112 representing observations 114 from actor 102 into multiple batches of input data 132, that is, batch them into a batch of inputs that can be processed in parallel by the policy model 108. In this specification, the batched input data 132 may also be referred to as batch inference.

[0063] The learner engine 110 can then use the policy model 108 to process each batch inference 132 to generate a batch of output data 142 that specifies the corresponding action to be performed in response to each observation represented in the batch inference 132. Specifically, the batch of output data 142 includes the corresponding policy output for each observation in the batch inference 132. In this specification, the batch of output data 142 may also be referred to as batch-processed actions.

[0064] Then, the learner engine 110 can use a control policy suitable for the type of policy output generated by the policy model 108 to select the corresponding action using each policy output (e.g., by sampling from a probability distribution or selecting the action with the highest Q value). Optionally, the learner engine 110 can apply an exploration policy when selecting each action.

[0065] Then, the learner engine 110 can send output data 122 to each actor 102, which specifies the action to be performed by the agent copy controlled by the actor 102.

[0066] The learner engine 110 also stores training data 118 derived from the input data 112 and output data 122 and associated with the training policy model 108 in a queue 148. Specifically, the learner engine 110 generates a trajectory for each environment and stores the generated trajectory for each environment in the queue 148. The trajectory comprises a sequence of tuples, each tuple including a corresponding observation, action, and reward obtained in response to performing the action in the environment. In some cases, when the trajectory ends at the termination of an episode in which the task is performed, the last tuple in the trajectory may not include the action and reward; that is, because the last observation represents the termination state of the episode and no further action is performed.

[0067] Then, the learner engine 110 can repeatedly update the parameters of the policy model 108 using the training data 118 stored in the queue 148.

[0068] Figure 2 An exemplary learner engine 110 of an exemplary centralized reasoning reinforcement learning system 100 is illustrated in detail.

[0069] The learner engine 100 includes multiple accelerators 130, which are used by the engine 100 to perform tasks related to performing inference, namely, tasks related to providing actions to the agent to perform in the environment, and tasks related to training the policy model 108.

[0070] Each accelerator includes one or more cores that can be assigned to perform tasks. Typically, accelerator 130 can be any processor with suitable bandwidth and computational power for training neural networks, i.e., a TPU, GPU, or some type of central processing unit (CPU).

[0071] The multiple cores of accelerator 130 are assigned to two groups: inference cores 130a and training cores 130b. Inference cores 130a are configured to process the corresponding policy inputs, i.e., batch inference 132, using policy model 108. Training cores 130b are different from inference cores 130a and are configured to train policy model 108 based on the maintained trajectory sequence.

[0072] At each time step in the plot where the actor performs the task, the learner engine 110 generates one or more batch inferences 132 based on the input data 112. The learner engine 110 then uses the policy model 108 to process the batch inferences 132 according to the current values ​​of the model parameters of the policy model 108. If the policy model 108 is based on one or more recurrent neural networks, the learner engine 110 also loads previously stored recurrent states 202 (if any) into the policy model 108. The learner engine 110 uses the policy model 108 to compute the policy output and, using the inference kernel 130a, samples the batched actions 142 of the corresponding agent copy in the corresponding environment 104 based on the policy output. If the policy model 108 is based on one or more recurrent neural networks, the learner engine 110 also obtains the new recurrent states 202 of that time step as part of the output of the policy model and stores them for use at the next time step.

[0073] Next, the learner engine 110 integrates the batch-processed actions 142 into output data 122, which represents the actions of each agent copy in the corresponding environment at that time step, and sends the output data to the corresponding actor. The actor receives the output data 112 from the learner engine 110 and instructs the corresponding agent copy to perform the corresponding action in the corresponding environment.

[0074] The learner engine 110 stores data tuples 118 (i.e., a set of data tuples, each representing observations, actions, and rewards based on the corresponding agent copy of batch inference 132) generated using the inference kernel 130a into, for example, a dedicated storage device 128. For the appropriate environment, the data tuples 118 stored in storage device 128 are considered incomplete trajectories until a predetermined length of trajectory or other predetermined condition is met, i.e., when the length of each trajectory used for all agent copies exceeds a threshold.

[0075] The total number of time steps spanned by the trajectory can be determined by the system 100 or the user. As mentioned earlier, the total length of the trajectory can be arbitrary, ranging from one time step to all time steps of the plot.

[0076] When a trajectory meets a predetermined length or other predetermined condition (e.g., the episode ends), the trajectory becomes a complete trajectory. The learner engine 110 transmits data 212 representing the complete trajectory to queue 148. For example, queue 148 can be designed in a first-in, first-out (FIFO) manner, such that the first trajectory of the corresponding agent copy stored in the queue is the first trajectory to be read by the learner engine 110. Queue 148 is also referred to as the complete trajectory queue.

[0077] Then, the learner engine 110 obtains data 214 representing a subset of the complete trajectory of the corresponding agent copy into the priority replay buffer 204. The priority replay buffer 204 is a data structure configured to store data and implemented by a computer program based on the accelerator 130. The priority replay buffer can also be implemented in a FIFO manner. The learner engine 110 can use either the inference core 130a or the training core 130b to transfer the complete trajectory 214 from the queue to the priority replay buffer. In some implementations, the learner engine 110 uses multiple threads of the accelerator 130 to transfer the trajectory from the storage device 128 to the queue 148 and then to the replay buffer 204.

[0078] The learner engine 110 samples the trajectory from the replay buffer 204, for example, randomly or using a priority replay technique, and sends the sampled trajectory 216 to the device buffer 206 maintained by the training core 130b. The device buffer is also a data structure configured to store data and implemented by a computer program based on the accelerator 130.

[0079] The learner engine 110 trains the policy model 108 using the training kernel 130b based on the sampled trajectory 218 by performing optimization 120 on an off-policy reinforcement learning algorithm (e.g., an actor-judge algorithm or other suitable algorithm).

[0080] The learner engine 110 updates the trained policy model 108 synchronously on both the inference kernel 130a and the training kernel 130b. That is, in future time steps, the inference kernel 130a of the learning engine 110 answers the batch inference 132, and the training kernel 130b of the learning engine 110 trains the policy model 108 based on the updated policy model (i.e., based on the updated values ​​of the model parameters generated as a result of training).

[0081] The learning engine 110 can be configured to periodically train the policy model based on some criteria (e.g., after a certain number of time steps, or at the end of an episode). The criteria used to determine trajectory integrity can be the same as the criteria used to periodically update the policy model by the learner engine 110.

[0082] To optimize the use of computing resources and minimize the computational cost of system 100, the system can adjust the ratio of the number of inference cores 130a to the number of training cores 130b. For example, for an accelerator 130 with a TPU having 8 cores, the learner engine can allocate 6 cores to inference cores 130a and 2 cores to training cores 130b, i.e., a ratio of 3. As another example, the accelerator 130 can have 32 cores, with 20 cores allocated to inference cores 130a and 12 cores allocated to training cores 130b, i.e., a ratio of 1.67. Typically, system 100 can determine the optimal computational efficiency ratio by training a policy model 108 with the same settings, in addition to allocating cores separately based on different ratios. The system can choose one of the different ratios that produce optimal computational performance. In some implementations, the user can override the ratio selected by system 100.

[0083] Figure 3A An exemplary off-policy reinforcement learning process 300a utilized by system 100 is illustrated.

[0084] In traditional off-policy reinforcement learning schemes, the same policy model is used to obtain each data tuple that forms the trajectory. In other words, a given actor will use the same parameter values ​​of the policy model to generate the entire trajectory, even though the policy model may have been updated by the learner engine when generating the trajectory.

[0085] On the other hand, process 300a allows each tuple in the trajectory to be obtained through the most recently updated policy model until an action in the tuple is selected. For example... Figure 3AAs shown, for the first time step along time step axis 310, each of the multiple actors 102 obtains data including observations at environment step 305a and makes an inference call to learner engine 110. Learner engine 110 responds to the inference call by providing actions to actor 102 via inference step 303a, based on the policy model updated prior to the first time step (i.e., at optimization step 301a). Actor 102 then instructs the corresponding agent copy to perform the appropriate action in the corresponding environment in the first time step, causing each environment to transition to the state at the next environment time step 305b. At this time step, actor 102 collects data including observations at environment step 305b and makes an inference call to learner engine 100. Learner engine 100 then responds to the inference call at inference step 303b, based on the policy model updated via optimization step 301b after the first step. Therefore, based on the most recently updated policy model, each tuple of the corresponding agent's action, reward, and observation is added to the trajectory until the end of the time step, i.e., the end of the episode, and the completed trajectory is sent to queue 148. Therefore, from Figure 3A As can be seen, the same trajectory includes actions selected using at least two different sets of parameter values, namely those actions generated at optimization step 301a and those actions generated at optimization step 301b.

[0086] Figure 3B The illustration shows another exemplary policy reinforcement learning process 300b utilized by system 100.

[0087] and Figure 3A Unlike the exemplary policy scheme shown, the frequency of updating the policy model (i.e., the frequency of optimization steps) can be less frequent than every time step. For example, the frequency could be every other time step. Figure 3B As shown, the learner engine 110 responds to inference calls from actor 102 at the first and second time steps by using a policy model trained before the first time step in optimization step 301a. However, at the third time step, the learner engine 110 responds to the inference calls by using an updated policy model trained after the second time step but before the third time step in optimization step 301b. Although for ease of illustration, Figure 3B The update frequency shown is every other step, but it can be every three time steps, every ten time steps, or more. In some implementations, the frequency can be based on time intervals of computation time and therefore can be non-uniform. That is, the frequency can be different for different time steps along time step axis 310. For example, the first update can be computed after the first time step, and the second update can be computed after three time steps. However, even with such a non-uniform frequency, the policy model can still be updated within the trajectory.

[0088] Figure 4 An exemplary process for centralized reinforcement learning is illustrated. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed centralized inference reinforcement learning system (e.g., Figure 1 The system 100) can execute process 400.

[0089] Specifically, the system can repeatedly execute steps 402-406 of the process to generate training data for training the policy model.

[0090] The system receives the corresponding observations generated by the respective actors for each environment in the multiple environments (step 402).

[0091] The system processes the corresponding policy inputs for each environment through a policy model to obtain the corresponding policy outputs for actors that define control policies for performing tasks within the environment (step 404). The corresponding policy inputs for each environment include observations characterizing the state of the environment. In some implementations, if the policy model is a recurrent neural network, the corresponding policy inputs further include stored recurrent states. The system can maintain and store separate recurrent states for each environment to ensure that the policy model is conditioned on appropriate data.

[0092] The system provides each actor in each environment with a corresponding action determined by a control policy defined by the corresponding policy output of the environment (step 406). The corresponding actor then instructs its corresponding agent copy to perform the corresponding action in the corresponding environment to transition the environment to a new state, and then receives a reward from the environment.

[0093] The system obtains the corresponding reward for each actor based on the result of the provided actions performed in the environment (step 408).

[0094] The system maintains a trajectory for each environment, which is a sequence of corresponding tuples with corresponding observations, actions, and rewards (step 410). That is, each time the system receives an observation for a given environment, provides an action to the agent for the given environment, and then receives a reward in response to the provided action being executed, the system generates a tuple and adds it to the trajectory for the given environment.

[0095] The system determines that the maintained sequence meets the threshold condition (step 412). As mentioned earlier, the threshold condition may be that the length of the tuple sequence reaches a threshold, or that the plot of the task terminates.

[0096] In response, the system trains a policy model 108 on the maintained sequence (step 414). For example, the system can use off-policy reinforcement learning techniques to train the policy model 108 on the maintained sequence.

[0097] By repeatedly training the policy model 108 on the maintenance sequence generated using the above-described technique, the system trains the policy model 108 to allow the policy model to be effectively used to control the agent to perform tasks.

[0098] This specification uses the term "configured" together with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that the system has software, firmware, hardware, or a combination of software, firmware, and hardware installed thereon that causes the system to perform those operations or actions in operation. For one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0099] Embodiments of the subject matter and functional operation described in this specification may be implemented using digital electronic circuits, computer software or firmware tangibly implemented, computer hardware including the structures disclosed in this specification and their equivalents, or a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, which are one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0100] The term "data processing device" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0101] A computer program, also referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data, such as one or more scripts stored in a markup language document; in a single file dedicated to said program; or in multiple coordinating files, such as a file storing portions of one or more modules, subroutines, or code. A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a data communication network.

[0102] In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or not at all, and it can be stored on storage devices in one or more locations. Therefore, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0103] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines may be installed and run on the same computer or on multiple computers.

[0104] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry, such as FPGAs or ASICs, or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0105] A computer suitable for executing computer programs can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory, random access memory, or both. Essential components of a computer are the central processing unit for executing or carrying out instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to said one or more mass storage devices, or both, for storing data. However, a computer does not necessarily have to have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game controller, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0106] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0107] To provide interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user and a keyboard and pointing device that the user can use to provide input to the computer. The display device is, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and the pointing device is, for example, a mouse or trackball. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents to and from a device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application.

[0108] The data processing apparatus for implementing machine learning models may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0109] Machine learning frameworks can be used to implement and deploy machine learning models. These frameworks include, for example, TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0110] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server; or middleware components, such as an application server; or frontend components, such as a client computer having a graphical user interface, web browser, or app that a user can use to interact with an implementation of the subject matter described in this specification; or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication through any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0111] A computing system may include clients and servers. Clients and servers are generally geographically distant and typically interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data, such as HTML pages, to a user device, for purposes such as displaying data to a user interacting with the device as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0112] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or potentially claimed claims, but rather as descriptions of features that may be specific to particular embodiments of the invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0113] Similarly, although operations are depicted in the drawings and described in the claims in a specific order, this should not be construed as requiring the operations to be performed in the specific order shown or in a sequential manner, or requiring the execution of all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0114] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method for performing reinforcement learning, comprising: For each of the multiple environments, a corresponding observation generated by the corresponding distributed actor is received at the centralized learner engine of the centralized learner system, wherein the corresponding observation includes one or more of image data, object location data and sensor data, or a simulated version of one or more of image data, object location data and sensor data; For each of the plurality of environments, the centralized learner engine processes the corresponding policy input, which includes the corresponding observations for that environment, through a policy model with multiple model parameter values, to obtain a corresponding policy output that defines a control policy for performing a task in that environment for the corresponding distributed actor. The centralized learner engine provides the corresponding actions determined by the control policy defined by the corresponding policy output of the corresponding distributed actor in each of the multiple environments; For each of the plurality of environments, obtain the corresponding reward for the corresponding distributed actor in that environment as a result of the provided action being performed in that environment; For each of the plurality of environments, the centralized learner engine maintains a corresponding sequence of tuples, in which at least one tuple includes an observation, an action, and a reward obtained by the distributed actor in performing the action in that environment. The centralized learner engine determines that the maintained tuple sequence satisfies a threshold condition; and In response to the determination, the policy model is trained by the centralized learner engine using the maintained sequence of tuples. The receiving, providing, and obtaining processes each include communication via a data communication network.

2. The method according to claim 1, further comprising: The distributed actor is made to perform the corresponding action defined by the corresponding policy output provided to the distributed actor in the environment.

3. The method according to claim 2, wherein, The environment is a real-world environment, and wherein, enabling the distributed actor to perform the corresponding action defined by the corresponding policy output provided to the distributed actor in the environment includes: The distributed actor sends one or more inputs corresponding to the corresponding action to a real-world agent in the real-world environment, wherein the real-world agent is configured to receive the one or more inputs from the distributed actor and perform the corresponding action in the real-world environment.

4. The method according to claim 2, wherein, The environment is a simulated environment, and wherein, causing the distributed actor to perform a corresponding action defined by the corresponding policy output provided to the distributed actor in the environment includes: The distributed actors are then instructed to perform the corresponding actions within the simulated environment.

5. The method according to claim 1, wherein, For each environment, the distributed actor receives its corresponding reward as a result of performing the corresponding action at a previous time step before the current time step: For each environment, the corresponding reward is generated based on the corresponding observations received for that environment.

6. The method according to claim 1, wherein, Maintaining the corresponding tuple sequence for each environment includes: For the environment-generated tuples in the multiple environments, the tuples include: (i) Relevant observations of the environment received by distributed actors, (ii) The corresponding actions provided to the distributed actors in response to the environment, and (iii) A corresponding reward for the corresponding distributed actor in the environment, generated as a result of the distributed actor performing the corresponding action in the environment; and The tuple is added to the corresponding tuple sequence for the environment and the distributed actor.

7. The method according to claim 1, wherein, The policy model is a Long Short-Term Memory (LSTM) neural network, and the processing of the corresponding observation and corresponding reward for each environment through the policy model includes maintaining the cyclic state of the LSTM neural network.

8. The method according to claim 1, wherein, Training the policy model includes using off-policy reinforcement learning techniques.

9. The method according to claim 1, wherein, Training the policy model further includes: Add the tuples of the maintained tuple sequence to the priority replay buffer; and The policy model is trained using tuples sampled from the priority replay buffer.

10. The method according to claim 1, wherein, For each environment, the corresponding policy input includes: Batch processing of the corresponding strategy inputs; and The policy model is used to process the policy inputs in batch processing to obtain the policy outputs in batch processing, wherein the policy outputs in batch processing include the corresponding policy outputs for each policy input in the policy inputs in batch processing.

11. The method according to claim 1, wherein, The distributed actor does not include the policy model.

12. The method according to claim 1, wherein, Receiving the corresponding observations generated by the corresponding distributed actors for each environment includes receiving the corresponding observations as part of one or more remote procedure calls.

13. The method according to any one of claims 1-12, in, For each environment, processing the corresponding policy input includes: for each environment, processing the corresponding policy input on one or more first hardware accelerators among a plurality of hardware accelerators, and Training the policy model on the maintained sequence includes: training the policy model on one or more second hardware accelerators, which are different from the one or more first hardware accelerators; and The one or more first hardware accelerators and the one or more second hardware accelerators define a predetermined ratio of hardware accelerators.

14. A computer-readable storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform a corresponding operation of the method according to any one of claims 1-13.

15. A system comprising one or more computers and one or more storage devices and storage instructions, wherein the instructions, when executed by the one or more computers, cause the one or more computers to perform a corresponding operation of the method according to any one of claims 1-13.

Images