Asynchronous deep reinforcement learning

Abstract

The present disclosure relates to asynchronous deep reinforcement learning. Methods, systems, and apparatus, including computer programs encoded on computer storage media, for asynchronous deep reinforcement learning. One of the systems includes one or more computers configured to implement a plurality of workers, where each worker is configured to operate independently of each other worker, and where each worker is associated with an actor that interacts with a respective replica of an environment during training of a deep neural network. Aspects of the present specification have the technical effect of training a neural network more quickly and / or reducing the memory requirements of the training.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201680065534.X, filed on November 11, 2016. Technical Field

[0003] This manual relates to asynchronous deep reinforcement learning. Background Technology

[0004] In a reinforcement learning system, an agent interacts with the environment by performing actions selected by the reinforcement learning system in response to receiving observations that represent the current state of the environment.

[0005] Some reinforcement learning systems respond to receiving a given observation based on the output of a neural network and select an action to be performed by the agent.

[0006] A neural network is a machine learning model that uses one or more layers of non-linear units to predict the output from a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, which is either the next hidden layer or the output layer. Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0007] This specification describes techniques related to reinforcement learning. Specifically, it describes techniques for asynchronously training deep neural networks used in reinforcement learning systems to select actions an agent will perform when interacting with its environment. The aspects of this specification offer the technical advantages of faster neural network training and / or reduced training memory requirements.

[0008] For a system of one or more computers to be configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon that causes the system to perform the operation or action in operation. For one or more computer programs to be configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0009] The subject matter described in this specification can be implemented in specific embodiments, thereby achieving one or more of the following advantages. By parallelizing training, the neural networks used by the reinforcement learning system can be trained faster. More specifically, by parallelizing training using multiple workers operating independently on a single machine, the communication costs resulting from parallelism can be reduced. Furthermore, by eliminating the need to store experience tuples in relay memory or other storage, as is typically required in parallelization techniques involving multiple workers on multiple machines, the memory requirements for training can be reduced, and on-policy reinforcement learning methods can be employed.

[0010] Details of one or more embodiments of the subject matter of this specification are set forth in the following drawings and description. Other methods, features, and advantages of the subject matter will become apparent from the description, drawings, and claims. For example, the specification provides methods, systems, and apparatus for implementing the techniques disclosed herein, including computer programs encoded on a computer storage medium. In some embodiments, a computer program includes instructions that, when executed by a computing device, cause it to perform any of the techniques disclosed herein. Attached Figure Description

[0011] Figure 1 An example neural network training system is shown.

[0012] Figure 2 This is a flowchart of an example process for training a deep neural network.

[0013] Figure 3 This is a flowchart of an example process for performing an iteration of the Q-learning technique.

[0014] Figure 4 This is a flowchart of an example process for performing iterations of the SARSA technique.

[0015] Figure 5 This is a flowchart of an example process for training a policy neural network.

[0016] Similar reference numerals and names in the figures indicate similar elements. Detailed Implementation

[0017] This specification provides a general description of techniques used for asynchronous deep reinforcement learning. Specifically, it describes how multiple workers can asynchronously train the deep neural networks used by a reinforcement learning system to select actions to be performed by an agent interacting with the environment.

[0018] In some implementations, the environment is a simulation environment and the agent is implemented as one or more computer programs that interact with the simulation environment. For example, the simulation environment may be a video game, and the agent may be a simulated user playing the video game. As another example, the simulation environment may be a motion simulation environment, such as a driving simulation or flight simulation, and the agent may be a simulated vehicle navigating through motion simulation. In these implementations, actions may be control inputs used to control the simulated user or the simulated vehicle.

[0019] In some other implementations, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent could be a robot that interacts with the environment to perform a specific task. As another example, the agent could be an autonomous or semi-autonomous vehicle that navigates through the environment. In these implementations, actions can be control inputs used to control the robot or autonomous vehicle.

[0020] Typically, a reinforcement learning system receives observations representing the current state of the environment and uses these observations to select an action to be performed by the agent. In response to the agent performing the selected action, the environment changes to a new state, and the reinforcement learning system receives feedback. This feedback is a numerical value as a function of the environment's state. While interacting with the environment, the reinforcement learning system attempts to maximize the long-term feedback received by the agent in response to actions performed by the agent.

[0021] Specifically, in response to a given observation, a reinforcement learning system selects the action to be performed by the agent based on the output generated by the deep neural network.

[0022] In order to allow reinforcement learning systems to effectively select actions to be performed by an agent, neural network training systems train deep neural networks to adjust the parameter values ​​of the deep neural network from their initial values.

[0023] Specifically, the neural network training system includes multiple actors that asynchronously train the deep neural network by selecting actions to be performed by actors that interact with a copy of the environment.

[0024] Each actor is an instance of an agent that operates independently of every other actor. For example, when the agent is a mechanical agent, each actor is also a separate mechanical agent that is the same as or nearly the same as the agent and operates independently of every other actor, such as having the same or nearly the same properties as the agent and being able to perform the same actions.

[0025] Figure 1An example neural network training system 100 is shown. The neural network training system 100 is an example of a system of computer programs implemented in one or more locations on one or more computers, as described below.

[0026] The neural network training system 100 includes multiple workers 102A-N and a shared memory 110. Each worker 102A-N is associated with a corresponding action 104A-N, which interacts with a corresponding environment copy 106A-N. For example, worker 102A is associated with action 104A, which interacts with environment copy 106A.

[0027] In some implementations, each of the workers 102A-N is implemented on the same computer, but in a separate thread, process, or other hardware or software capable of performing computations independently for the worker. In these implementations, shared memory 110 is memory on the computer accessible by each of the workers 102A-N. In implementations where the environment is a virtual environment, the action and environment copy associated with a given worker execute in the same thread or other hardware or software as the given worker.

[0028] Each of the workers 102A-N maintains an instance of the trained deep neural network and accesses the shared memory 110 to obtain the current parameter values ​​of the deep neural network already stored in the shared memory 110. Then, each worker uses its instance of the deep neural network to select an action to be performed by the actor associated with that worker, based on the current parameter values ​​of the neural network obtained from the shared memory 110.

[0029] Specifically, during training, each worker receives observations representing the state of a replica of the environment associated with that worker, and in response to each observation, uses a deep neural network to select an action to be performed by an actor associated with that replica of the environment. The worker then receives a next observation representing the next state of the replica of the environment, along with feedback resulting from the actor performing the selected action—the state the replica of the environment transitions to after the actor performs the selected action. The worker uses this feedback and the next observation to determine gradients, and periodically uses these gradients to update the parameter values ​​stored in shared memory 110. The determination of gradients and updating the parameter values ​​stored in shared memory are described below. Figure 2 To describe in more detail.

[0030] By repeatedly and asynchronously updating the parameter values ​​stored in the shared memory 110, each of the workers 102A-102N trains the deep neural network together to determine the training parameter values ​​of the deep neural network.

[0031] Figure 2 This is a flowchart of an example process 200 for training a deep neural network. For convenience, process 200 will be described as being performed by, for example... Figure 1 The work body 120A-N is executed by one of the work bodies.

[0032] This task determines the current parameter values ​​of the deep neural network (step 202).

[0033] In some implementations, the worker accesses memory shared among all workers—for example, Figure 1 Shared memory 110 — obtain the current value.

[0034] In some other implementations, the worker only retrieves the current value from the memory and writes it to the local memory after updating the shared memory as described below. In these implementations, the worker determines the current value as the value stored in the local memory.

[0035] The worker receives a current observation representing the current state of a copy of the environment that the actors associated with it interact with (step 204).

[0036] The worker uses a deep neural network to select the current action to be performed by the actor when the environment is in its current state (step 206).

[0037] Specifically, in some embodiments, the deep neural network is a Q-network, which is configured to receive observations and actions as input and process the input according to the current parameter values ​​of the Q-network to generate a neural network output. The neural network output, referred to herein as the Q-value, is an estimate of the long-term feedback generated by an agent or actor performing an input action when the environment is in the state represented by the input observation.

[0038] In these implementations, for each action in a predetermined set of actions, the worker uses a Q-network to process the action and the current observation to generate a corresponding current Q-value for each action.

[0039] Then, the worker uses the current Q and, according to the worker's action selection strategy, selects an action from the predetermined action set as the current action to be performed by the actor.

[0040] Typically, the action selection strategy of a given job differs from that of every other job. For example, in some implementations, each job's action selection strategy is an ε-greedy strategy, where the job randomly selects an action from a predetermined set of actions with probability ε, and selects the action with the highest Q value with probability 1-ε, but the value of ε is different for each job. For example, each job can periodically sample the value of its ε from the distribution of possible values ​​of ε to determine the ε to use in its action selection strategy. As another example, a job can select actions by applying a softmax nonlinearity to the Q value and sampling actions from the resulting distribution. Then, each job can use different temperature parameters in the softmax to ensure that each job has a different action selection strategy.

[0041] The worker receives the next observation and actual feedback (step 208). The next observation represents the next state of the environment copy, that is, the state that the environment copy changes to because the actor associated with the worker performs the current action. The actual feedback is the value received from the environment copy due to the change from the current state to the next state.

[0042] The worker uses the current feedback and the next observation to perform iterations of the reinforcement learning process to determine the current gradient (step 210). This gradient is the gradient of the loss function that is being optimized to train the deep neural network.

[0043] The loss function and the way the working body determines the gradient depend on the reinforcement learning process used to train the deep neural network, which in turn depends on how the deep neural network is configured.

[0044] For example, when the deep neural network is a Q-network, the worker can use Q-learning techniques or SARSA techniques. See below for reference. Figure 3 Describe the iterations of performing Q-learning techniques. See below for reference. Figure 4 Describe the iterations of implementing SARSA technology.

[0045] The worker uses the current gradient to update the accumulated gradient to determine the updated accumulated gradient (step 212). That is, the worker maintains the gradient in local memory and updates the maintained gradient with the new gradient each time a new gradient is computed, for example by adding the new gradient to the accumulated gradient.

[0046] The worker determines whether the criteria for updating the current parameter values ​​of the neural network are met (step 214). For example, the criterion for updating the current value could be specified as follows: the worker updates the current value after performing a specified number of reinforcement learning iterations since the last time the parameter values ​​were updated. As another example, the criterion could be specified as follows: the worker updates the current value when the total number of reinforcement learning iterations performed by the worker since the last time all multiple workers updated their parameter values ​​exceeds a specified threshold.

[0047] If the criterion is met, the worker uses the accumulated gradient to determine the updated parameter values ​​(step 216) and writes the updated parameter values ​​to shared memory (step 218). Specifically, the worker determines the parameter value updates from the accumulated gradient, accesses the current values ​​in shared memory, and uses the parameter value updates to update those values. In some examples, the worker multiplies the accumulated gradient by the learning rate parameter and then sums the current parameter values ​​and the parameter value updates to determine the updated parameter values. In some other examples, the worker uses an asynchronous variation of the RMSProp optimizer to update the parameter values.

[0048] After the working body writes the updated parameter values ​​to the shared memory, it clears the accumulated gradients, so that no gradients are maintained by the working body when the next iteration of process 200 is initiated.

[0049] If the criterion is not met, the work body avoids writing anything to the shared memory (step 220), that is, the work body continues to execute another iteration of process 200 without updating the current parameter value.

[0050] Each of the multiple workers can asynchronously and repeatedly execute process 200 to collectively determine training values ​​for the parameters of the deep neural network. Once the training values ​​have been determined, the reinforcement learning system can use the deep neural network to select values ​​to be performed by the agent interacting with the environment, i.e., by processing the neural network input using the deep neural network based on the training values ​​of the neural network parameters. In some other cases, the reinforcement learning system uses the deep neural network to select values ​​to be performed by the agent while the workers continue executing process 200, i.e., by periodically obtaining values ​​stored in shared memory and using those values ​​to select the action to be performed by the agent.

[0051] In some cases, once the training values ​​have been determined, they are stored and then transmitted over the network to another system for use when instantiating a trained version of a deep neural network, thus allowing the agent to effectively select actions in its interaction with the environment.

[0052] Figure 3This is a flowchart of an example process 300 for performing iterations of the Q-learning technique. For convenience, process 300 will be described as being performed by, for example... Figure 1 The work body 120A-N is executed by one of the work bodies.

[0053] The working entity obtains the current observation, current action, actual feedback, and the next observation (step 302), as described above. Figure 2 As described.

[0054] The worker determines the maximum output of the target network when the environment copy is in the state represented by the next observation (step 304). The target network is the same neural network as the Q network, i.e., it has the same neural network architecture / form, but with potentially different parameter values. Specifically, each worker periodically synchronizes the parameter values ​​of the target network with the current parameter values ​​of the Q network stored in shared memory. However, compared to updating the parameter values ​​stored in shared memory, the worker synchronizes the parameter values ​​of the target network less frequently, resulting in the target network and the Q network often having different parameter values.

[0055] The system determines the maximum output of the target network by: for each action in a predetermined set of actions, processing the action using the target network in conjunction with the next observation to determine the target network output for that action. Then, the system selects the maximum value of the target network output for each action in the predetermined set of actions as the maximum output of the target network.

[0056] The worker determines the error from the target network's maximum output, actual feedback, and the current Q-value previously determined for the current action when the environment is in its current state. This current Q-value is used to select the current action as the action to be performed when the environment is in its current state (step 306). In some implementations, the error E satisfies:

[0057] E = r + γmax(Q) target )-Q current ,

[0058] Where r is the actual feedback, γ is the pre-order discount factor, and max(Q) target ) is the maximum output of the target network, and Q current It is the current Q value determined for the current action when the environment was in its current state.

[0059] The worker uses the determined error to determine the current gradient (step 308). That is, the worker uses backpropagation to update the value, where the backpropagation error is the determined error for the selected action and zero for all other actions.

[0060] Figure 4This is a flowchart of an example process 400 for performing iterations of the SARSA technique. For convenience, process 400 will be described as being executed by a worker in a neural network training system, for example... Figure 1 One of the working bodies 120A-N in the neural network training system 100.

[0061] The working unit receives the current observation, current action, actual feedback, and the next observation (step 402).

[0062] The worker selects the next action to be performed by the actor when the environment copy is in the next state (step 404). The worker selects the next action as described above with reference to step 206. That is, for each action in the predetermined set of actions, the worker uses a Q-network to process the action and the next observation to generate a corresponding next Q-value for each action. Then, the worker uses the next Q-value and, according to the worker's action selection strategy, selects an action from the predetermined set of actions as the next action to be performed by the actor.

[0063] The working unit determines the target network output for the next action (step 406). That is, the working unit uses the target network to process the next action and the next observation to determine the target network output for the next action.

[0064] The worker determines the error from the target network output for the next action, the actual feedback, and the current Q value previously determined for the current action (step 406). In some implementations, the error E satisfies:

[0065] E=r+γQ target )-Q current

[0066] Where r is the actual feedback, γ is the pre-order discount factor, and Q... target It is the maximum output of the target network, and Q current It is the current Q value determined for the current action when the environment was in its current state.

[0067] The working body uses the determined error to determine the current gradient (step 408).

[0068] When executing process 300 or 400, the worker also determines whether a criterion for synchronizing the target network parameters with the parameters in shared memory is met. For example, this criterion can be the same as the criterion described above with reference to step 216, but with a specified threshold or number greater than the value or number used to determine whether to update the parameters stored in shared memory. Therefore, the worker synchronizes the target network's parameter values ​​less frequently than it updates the parameter values ​​stored in shared memory, resulting in the target network and the Q network often having different parameter values.

[0069] The above description has described a deep neural network as an implementation of a Q-network. However, in some other implementations, the deep neural network is a policy neural network that receives input observations and outputs a score for each action in a predetermined set of actions, where each score represents the probability that the corresponding action is one that should be performed to maximize long-term feedback, which will come from the environment in the state represented by the input observation.

[0070] In these implementations, in order to select an action in response to a given observation, the reinforcement learning system uses a policy neural network to process a given observation and selects the action to be performed by the agent by sampling from actions based on their scores.

[0071] To ensure that each worker has a different action selection strategy, each worker can use a different random seed when sampling from actions, or alternatively, each worker can generate a different temperature for the output layer of the neural network that generates scores for actions.

[0072] Figure 5 This is a flowchart of an example process for training a policy neural network. For convenience, process 500 will be described as being executed by a worker in the neural network training system, for example... Figure 1 One of the working bodies 120A-N in the neural network training system 100.

[0073] The worker determines the current parameter values ​​of the policy neural network (step 502). Specifically, the worker retrieves the values ​​from memory shared among all workers—for example, Figure 1 Shared memory 110 — Get the current value.

[0074] The agent also obtains the current parameter values ​​of the baseline neural network from shared memory. This baseline neural network is configured to receive input observations and generate baseline scores, which represent estimated long-term feedback received by the agent, starting from the state represented by the input observations. In some implementations, the baseline and policy networks share some of their parameters.

[0075] The agent receives observations representing the state of a replica of the environment and selects actions to be performed by the agent based on the current parameter values ​​of the policy neural network until the replica of the environment transforms into a state that meets a specific criterion (step 504). That is, for each observation, the policy neural network processes the observation based on the current value, and the agent selects actions to be performed by sampling actions based on their scores. For example, the specific criterion could be established after a predetermined number t of observations have been received. max It is satisfied after observation or until the predetermined termination state has been reached.

[0076] Specifically, for each received observation of the state of a replica of the environment, the worker uses a policy neural network to process the observation to generate a corresponding score for each action in a predetermined action set based on the current parameter value of the policy neural network, and selects the action to be performed by the worker by sampling from the distribution.

[0077] For each received observation, the working entity also uses the baseline neural network to process actions and observations based on the current parameter values ​​of the baseline neural network in order to generate a baseline score for the observation.

[0078] In some implementations, one or both of the baseline neural network and the policy neural network may be a recurrent neural network.

[0079] The working group determines the actual long-term feedback for each received observation (step 506).

[0080] That is, for the latest observation received, the system sets the actual long-term feedback to, for example, the baseline score predicted by the baseline neural network.

[0081] For a given observation other than the latest observation, the system sets the actual long-term feedback of that given observation as the sum of the feedback received when the environment copy is transformed into the state represented by the latest observation and the discounted actual long-term feedback of the observation received after the actor performs the action selected in response to receiving the given observation.

[0082] The working body determines the corresponding gradient update of the policy neural network and the corresponding gradient update of the baseline neural network for each observation (step 510).

[0083] Specifically, regarding observation S t The gradient update of the strategy upgrade network satisfies:

[0084]

[0085] Where π(a) t |s t ;θ′) is for the response to observation S t The score R generated by the action performed. t It is for observing S t The determined actual long-term feedback, and b t It is to observe S t The baseline score.

[0086] For observation S t The gradient update of the baseline neural network satisfies:

[0087]

[0088] Where b(s) t ;θ′ b ) is to observe S t The baseline score.

[0089] For example, by adding the new gradient of each neural network to the accumulated gradient of that neural network, the worker uses the gradient update of the policy neural network to update the accumulated gradient of the policy neural network to determine the updated accumulated gradient of the policy neural network, and uses the gradient update of the baseline neural network to update the accumulated gradient of the baseline neural network to determine the updated accumulated gradient of the baseline neural network (step 512).

[0090] The working body determines whether the criteria for updating the current values ​​of the parameters of the baseline neural network and the policy neural network are met (step 514).

[0091] For example, the criteria for updating the current value can be specified as follows: the current value is updated after the working body has performed the specified number of iterations of step 504 since the last time the parameter value was updated.

[0092] As another example, the standard can specify that a work body updates its current value when the total number of iterations of step 504, which has been performed since all work bodies last updated their parameter values, exceeds a specified threshold.

[0093] If the criterion is met, the working body uses the corresponding accumulated gradient to determine updated parameter values ​​for the baseline neural network and the policy neural network (step 516), and writes the updated parameter values ​​to the shared memory (step 518), for example, as described above with reference to steps 216 and 218.

[0094] After the working body writes the updated parameter values ​​to the shared memory, it clears the accumulated gradients, so that no gradients are maintained by the working body when the next iteration of process 500 is initiated.

[0095] If the criterion is not met, the work body avoids writing anything to the shared memory (step 520), that is, the work body continues to execute another iteration of process 500 without updating the current parameter value.

[0096] Embodiments of the subject matter and functional operation described in this specification can be implemented as digital electronic circuits, or as tangibly embodied computer software or firmware, as computer hardware including the structures disclosed in this specification and their equivalents, or as a combination of one or more of these. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., modules of one or more computer program instructions encoded on a tangible, non-transient program carrier for execution by or control of the operation of a data processing apparatus. Alternatively or otherwise, the program instructions can be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0097] The term "data processing apparatus" refers to data processing hardware and includes all types of devices, apparatuses, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0098] Computer programs (also referred to as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be adopted in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program may correspond to a file in a file system, but this is not mandatory. A program may be stored as a part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). Computer programs can be deployed to execute on a single computer or on multiple computers located in one location or distributed across multiple locations and interconnected via a communication network.

[0099] The processes and logic flows described in this specification can be implemented by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processing and logic flows can also be implemented by dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and devices can also be implemented as said dedicated logic circuits.

[0100] A computer suitable for executing computer programs may be based on a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. Essential components of a computer are the central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large storage devices for storing data, or operatively coupled to them for receiving and transferring data, such as disks, magneto-optical disks, or optical disks. However, a computer is not required to have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name just a few.

[0101] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices like EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or portable hard disks; magneto-optical disks; and CD-ROM and DVD-ROM discs. Processors and memory can be supplemented with dedicated logic circuitry or integrated therein.

[0102] To provide interaction with the user, the embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing devices, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's client device.

[0103] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes, for example, a back-end component as a data server, or a middleware component as an application server, or a front-end component as a client computer, or any combination of one or more such back-end, middleware, or front-end components, the client computer having a graphical user interface or web browser through which a user can interact with embodiments of the subject matter described in this specification. The components of the system can be interconnected via digital data communication in any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs) such as the Internet.

[0104] A computing system can include clients and servers. Clients and servers are typically geographically separated and interact via communication networks. The client-server relationship originates from computer programs running on the respective computers and having a client-server relationship with each other.

[0105] While this specification contains numerous specific details of implementation, these should not be construed as limiting any invention or the scope of any claimable protection, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in this specification in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in a certain combination and even initially claimed in this way, one or more features from the claimed combination may in some cases deviate from that combination, and the claimed combination may be for sub-combinations or variations thereof.

[0106] Similarly, although the operations are depicted in a specific order in the diagram, this should not be construed as requiring such operations to be performed in the shown specific order or in a sequential order, or to perform all the illustrated operations, in order to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components described above should not be construed as requiring such separation, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0107] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims may be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the desired order or sequential sequence to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A system for training a deep neural network, the deep neural network being used to select actions to be performed by an agent, the agent interacting with an environment by performing actions selected from a predetermined set of actions, the system comprising: A shared memory accessible to all workers in a plurality of workers, wherein the shared memory stores the parameter values ​​of the deep neural network; Multiple processors; Multiple worker entities, wherein each worker entity is configured to operate independently of and in parallel with other worker entities on one of the multiple processors, wherein each worker entity is associated with a corresponding actor that interacts with a corresponding copy of the environment during the training of the deep neural network, and wherein each worker entity is further configured to repeatedly perform operations including: Determine the current parameter values ​​of the deep neural network; Receive the current observation representing the current state of a copy of the environment interacted with by the actors associated with the work unit; The action selection strategy based on the corresponding action of the worker uses one or more outputs generated by the deep neural network based on the current parameter value to select the current action to be performed by the actor associated with the worker in response to the current observation; When the environment copy is in the current state, identify the actual feedback generated by the actor performing the current action; Receive a next observation representing the next state of a copy of the environment interacting with the actor, wherein the copy of the environment changes from the current state to the next state in response to the actor performing the current action; Perform iterations of reinforcement learning techniques to determine the current gradient using the actual feedback and the next observation; The current gradient is used to update the accumulated gradient to determine the updated accumulated gradient; Whether the criteria for determining whether the parameter values ​​used to update the deep neural network stored in the shared memory have been met; and When the criterion used to update the current parameter values ​​of the deep neural network stored in the shared memory has been met: The updated accumulated gradient and the parameter values ​​of the deep neural network stored in the shared memory are used to determine the updated parameter values ​​of the deep neural network; The updated parameter values ​​of the deep neural network are stored in a shared memory that can be accessed by all of the plurality of workers.

2. The system according to claim 1, wherein, Each worker's action selection strategy differs from that of each other worker.

3. The system according to claim 1, wherein the operation further comprises: When the criterion for updating the parameter values ​​of the deep neural network stored in the shared memory has been met, the accumulated gradient of the update is cleared.

4. The system according to claim 1, wherein, Determining whether the criteria for updating the parameter values ​​of the deep neural network stored in the shared memory have been met includes: determining that a specified number of iterations of the reinforcement learning technique have been performed since the parameter values ​​of the deep neural network stored in the shared memory were previously updated.

5. The system according to claim 1, wherein, The deep neural network is a Q-network, configured to receive input observations and input actions characterizing the input state of the environment, and to generate a Q-value, which is an estimate of the long-term feedback generated by performing the input action when the environment is in the input state, wherein selecting the current action to be performed by the actor in response to the current observation includes: For each action in the predetermined set of actions, the current observation and the action are processed using the deep neural network to generate a Q-value for the action based on the current parameter values ​​of the deep neural network; and Based on the action selection strategy of the work body, the Q value of the action is used to select an action from the predetermined action set.

6. The system according to claim 5, wherein, The reinforcement learning technique is Q-learning, and the iteration includes: When the environment copy is in the next state, determine the maximum output of the target network for any action in the predetermined action set, wherein the target network is a neural network that is the same as the Q network but has potentially different parameter values; The error is determined based on the maximum output of the target network, the actual feedback, and the Q-value of the current action; and The current gradient is determined using the error.

7. The system according to claim 5, wherein, The reinforcement learning technique is SARSA, and the iteration includes: Based on the corresponding action selection strategy of the work body, select the next action to be performed by the action body associated with the work body in response to the next observation; The next action and the next observation are processed using a target network to determine the target network output for the next action, wherein the target network is a neural network that is the same as the Q network but has potentially different parameter values; The error is determined from the target network output of the next action, the actual feedback, and the Q-value of the current action; and The current gradient is determined using the error.

8. The system according to any one of claims 1-7, wherein, The environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment, wherein the corresponding actor is a mechanical actor and interacts with a corresponding copy of the real-world environment during the training of the deep neural network, and wherein each current action includes control input for controlling the mechanical agent.

9. A non-transitory computer storage medium storing one or more instructions, said instructions, when executed by one or more computers, causing said one or more computers to implement a system for training a deep neural network, said deep neural network for selecting actions to be performed by an agent, said agent interacting with an environment by performing actions selected from a predetermined set of actions, said system comprising: A shared memory accessible to all workers in a plurality of workers, wherein the shared memory stores the parameter values ​​of the deep neural network; Multiple processors; Multiple worker entities, wherein each worker entity is configured to operate independently of and in parallel with other worker entities on one of the multiple processors, wherein each worker entity is associated with a corresponding actor that interacts with a corresponding copy of the environment during the training of the deep neural network, and wherein each worker entity is further configured to repeatedly perform operations including: Determine the current parameter values ​​of the deep neural network; Receive the current observation representing the current state of a copy of the environment interacted with by the actors associated with the work unit; The action selection strategy based on the corresponding action of the worker uses one or more outputs generated by the deep neural network based on the current parameter value to select the current action to be performed by the actor associated with the worker in response to the current observation; When the environment copy is in the current state, identify the actual feedback generated by the actor performing the current action; Receive a next observation representing the next state of a copy of the environment interacting with the actor, wherein the copy of the environment changes from the current state to the next state in response to the actor performing the current action; Perform iterations of reinforcement learning techniques to determine the current gradient using the actual feedback and the next observation; The current gradient is used to update the accumulated gradient to determine the updated accumulated gradient; Whether the criteria for determining whether the parameter values ​​used to update the deep neural network stored in the shared memory have been met; and When the criterion used to update the current parameter values ​​of the deep neural network stored in the shared memory has been met: The updated accumulated gradient and the parameter values ​​of the deep neural network stored in the shared memory are used to determine the updated parameter values ​​of the deep neural network; The updated parameter values ​​of the deep neural network are stored in a shared memory that can be accessed by all of the plurality of workers.

10. The computer storage medium according to claim 9, wherein, Each worker's action selection strategy differs from that of each other worker.

11. The computer storage medium according to claim 9, wherein the operation further comprises: When the criterion for updating the parameter values ​​of the deep neural network stored in the shared memory has been met, the accumulated gradient of the update is cleared.

12. The computer storage medium according to claim 9, wherein, Determining whether the criteria for updating the parameter values ​​of the deep neural network stored in the shared memory have been met includes: determining that a specified number of iterations of the reinforcement learning technique have been performed since the parameter values ​​of the deep neural network stored in the shared memory were previously updated.

13. The computer storage medium according to claim 9, wherein, The deep neural network is a Q-network, configured to receive input observations and input actions characterizing the input state of the environment, and to generate a Q-value, which is an estimate of the long-term feedback generated by performing the input action when the environment is in the input state, wherein selecting the current action to be performed by the actor in response to the current observation includes: For each action in the predetermined set of actions, the current observation and the action are processed using the deep neural network to generate a Q-value for the action based on the current parameter values ​​of the deep neural network; and Based on the action selection strategy of the work body, the Q value of the action is used to select an action from the predetermined action set.

14. The computer storage medium according to claim 13, wherein, The reinforcement learning technique is Q-learning, and the iteration includes: When the environment copy is in the next state, determine the maximum output of the target network for any action in the predetermined action set, wherein the target network is a neural network that is the same as the Q network but has potentially different parameter values; The error is determined based on the maximum output of the target network, the actual feedback, and the Q-value of the current action; and The current gradient is determined using the error.

15. The computer storage medium according to claim 13, wherein, The reinforcement learning technique is SARSA, and the iteration includes: Based on the corresponding action selection strategy of the work body, select the next action to be performed by the action body associated with the work body in response to the next observation; The next action and the next observation are processed using a target network to determine the target network output for the next action, wherein the target network is a neural network that is the same as the Q network but has potentially different parameter values; The error is determined from the target network output of the next action, the actual feedback, and the Q-value of the current action; and The current gradient is determined using the error.

16. The computer storage medium according to any one of claims 9-15, wherein, The environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment, wherein the corresponding actor is a mechanical actor and interacts with a corresponding copy of the real-world environment during the training of the deep neural network, and wherein each current action includes control input for controlling the mechanical agent.

17. A method for training a deep neural network, the deep neural network being used to select actions to be performed by an agent, the agent interacting with an environment by performing actions selected from a predetermined set of actions, the method being implemented on multiple processors, the method comprising: The parameter values ​​of the deep neural network are repeatedly updated using multiple workers, the parameter values ​​being stored in a shared memory accessible to all of the multiple workers. Each worker is configured to operate independently of and in parallel with other workers on one of the multiple processors. Each worker is associated with a corresponding actor that interacts with a corresponding copy of the environment during the training of the deep neural network. Each worker is further configured to repeatedly perform operations including: Determine the current parameter values ​​of the deep neural network; Receive the current observation representing the current state of a copy of the environment interacted with by the actors associated with the work unit; The action selection strategy based on the corresponding action of the worker uses one or more outputs generated by the deep neural network based on the current parameter value to select the current action to be performed by the actor associated with the worker in response to the current observation; When the environment copy is in the current state, identify the actual feedback generated by the actor performing the current action; Receive a next observation representing the next state of a copy of the environment interacting with the actor, wherein the copy of the environment changes from the current state to the next state in response to the actor performing the current action; Perform iterations of reinforcement learning techniques to determine the current gradient using the actual feedback and the next observation; The current gradient is used to update the accumulated gradient to determine the updated accumulated gradient; Whether the criteria for determining whether the parameter values ​​used to update the deep neural network stored in the shared memory have been met; and When the criterion used to update the current parameter values ​​of the deep neural network stored in the shared memory has been met: The updated accumulated gradient and the parameter values ​​of the deep neural network stored in the shared memory are used to determine the updated parameter values ​​of the deep neural network; The updated parameter values ​​of the deep neural network are stored in a shared memory that can be accessed by all of the plurality of workers.

18. The method according to claim 17, wherein, Each worker's action selection strategy differs from that of each other worker.

19. The method of claim 17, wherein, The deep neural network is a Q-network, configured to receive input observations and input actions characterizing the input state of the environment, and to generate a Q-value, which is an estimate of the long-term feedback generated by performing the input action when the environment is in the input state, wherein selecting the current action to be performed by the actor in response to the current observation includes: For each action in the predetermined set of actions, the current observation and the action are processed using the deep neural network to generate a Q-value for the action based on the current parameter values ​​of the deep neural network; and Based on the action selection strategy of the work body, the Q value of the action is used to select an action from the predetermined action set.

20. The method according to any one of claims 17-19, wherein, The environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment, wherein the corresponding actor is a mechanical actor and interacts with a corresponding copy of the real-world environment during the training of the deep neural network, and wherein each current action includes control input for controlling the mechanical agent.

21. A system for training a deep neural network for controlling an agent to interact with an environment by performing actions selected from a predetermined set of actions, the deep neural network comprising a policy neural network having a plurality of policy parameters and a baseline neural network having a plurality of baseline parameters, and the system comprising one or more computers configured to implement: One or more working bodies, in, Each worker is associated with a corresponding actor that interacts with a corresponding copy of the environment during the training of the deep neural network, and wherein each worker is further configured to repeatedly perform operations including: Determining the current parameter values ​​of the deep neural network includes: Receive the current values ​​of the baseline parameters of the baseline neural network from a memory accessible by each of the one or more workers; and Receive the current value of the policy parameters of the policy neural network from the memory; Until the environment copy interacting with the actor associated with the work body changes to a state that meets a specific criterion: Receive observations characterizing the state of the environment copy, and in response to each observation, select an action to be performed by the actor associated with the worker in response to the observation, according to the worker's corresponding action selection strategy, wherein the selection for each observation includes: The observation is processed using the policy neural network and based on the current values ​​of the policy parameters of the policy neural network to generate a corresponding score for each action in a predetermined set of actions based on the current values ​​of the policy parameters of the policy neural network. Use the corresponding score to select the current action to be performed by the actor; For each observation received before meeting the specific criteria: A baseline score representing the estimated feedback received by the agent starting from the state characterized by the observation is generated based on the current value of the baseline parameters of the baseline neural network. Identify the actual feedback generated as the actor performs the selected action in response to the observation; The actual long-term feedback is determined at least based on the actual feedback and baseline scores of observations representing states that meet the specific criteria; and The reinforcement learning technique is iterated to determine the corresponding current gradient for the baseline neural network and the policy neural network based on the baseline score and the actual long-term feedback. Based on the corresponding current gradient, the corresponding accumulated gradient is updated for the baseline neural network and the policy neural network to determine the corresponding updated accumulated gradient; Whether the criteria for determining the current parameter values ​​used to update the deep neural network have been met; and When the criterion used to update the current parameter values ​​of the deep neural network has been met: Update the parameters of the deep neural network, wherein the update includes: The updated accumulated gradient is used to determine the updated parameter values; and The updated parameter values ​​are stored in a memory that can be accessed by each of the one or more working bodies.

22. The system according to claim 21, wherein, Each of the one or more workpieces is executed on the same computer.

23. The system of claim 21, wherein the operation further comprises: When the criterion for updating the current parameter values ​​of the deep neural network has been met, the accumulated gradient of the update is cleared.

24. The system according to claim 21, wherein, Determining whether the criteria for updating the current parameter values ​​of the deep neural network have been met includes: determining that a specified number of iterations of the reinforcement learning technique have been performed since the parameter values ​​of the deep neural network were last updated.

25. The system according to claim 21, wherein, Determining the actual long-term feedback includes: For the latest observation received before the criterion is met, the actual long-term feedback is set based on the actual feedback for that observation and the baseline score for the observation characterizing the state of meeting the specific criterion; and For each other observation received before the criterion is met, the actual long-term feedback for that observation is set to the actual feedback plus the discounted sum of the actual long-term feedbacks for observations received after that observation.

26. The system according to claim 21, wherein, The gradient update for the policy neural network is determined by the following: Wherein, π(a t |s t ;θ′) represents the response to the observation s determined based on the current value of the policy parameters of the policy neural network. t The score of the action performed, R t This is the actual feedback, and b t It is the baseline score.

27. The system according to claim 26, wherein, The gradient update for the baseline neural network is determined by the following: Where, b(s) t ;θ′ b ) refers to the observation s t The baseline score.

28. A non-transitory computer storage medium storing one or more instructions, said instructions, when executed by one or more computers, causing said one or more computers to implement a system for training a deep neural network, said deep neural network for controlling an agent that interacts with an environment by performing actions selected from a predetermined set of actions, said deep neural network including a policy neural network having a plurality of policy parameters and a baseline neural network having a plurality of baseline parameters, and said system comprising: One or more working bodies, Each worker is associated with a corresponding action that interacts with a corresponding copy of the environment during the training of the deep neural network, and each worker is further configured to repeatedly perform operations including: Determining the current parameter values ​​of the deep neural network includes: Receive the current values ​​of the baseline parameters of the baseline neural network from a memory accessible by each of the one or more workers; and Receive the current value of the policy parameters of the policy neural network from the memory; Until the environment copy interacting with the actor associated with the work body changes to a state that meets a specific criterion: Receive observations characterizing the state of the environment copy, and in response to each observation, select an action to be performed by the actor associated with the worker in response to the observation, according to the worker's corresponding action selection strategy, wherein the selection for each observation includes: The observation is processed using the policy neural network and based on the current values ​​of the policy parameters of the policy neural network to generate a corresponding score for each action in a predetermined set of actions based on the current values ​​of the policy parameters of the policy neural network. Use the corresponding score to select the current action to be performed by the actor; For each observation received before meeting the specific criteria: A baseline score representing the estimated feedback received by the agent starting from the state characterized by the observation is generated based on the current value of the baseline parameters of the baseline neural network. Identify the actual feedback generated as the actor performs the selected action in response to the observation; The actual long-term feedback is determined at least based on the actual feedback and baseline scores of observations representing states that meet the specific criteria; and The reinforcement learning technique is iterated to determine the corresponding current gradient for the baseline neural network and the policy neural network based on the baseline score and the actual long-term feedback. Based on the corresponding current gradient, the corresponding accumulated gradient is updated for the baseline neural network and the policy neural network to determine the corresponding updated accumulated gradient; Whether the criteria for determining the current parameter values ​​used to update the deep neural network have been met; and When the criterion used to update the current parameter values ​​of the deep neural network has been met: Update the parameters of the deep neural network, wherein the update includes: The updated accumulated gradient is used to determine the updated parameter values; and The updated parameter values ​​are stored in a memory that can be accessed by each of the one or more working bodies.

29. The non-transitory computer storage medium according to claim 28, wherein, Each of the one or more workpieces is executed on the same computer.

30. The non-transitory computer storage medium of claim 28, wherein the operation further comprises: When the criterion for updating the current parameter values ​​of the deep neural network has been met, the accumulated gradient of the update is cleared.

31. The non-transitory computer storage medium according to claim 28, wherein, Determining whether the criteria for updating the current parameter values ​​of the deep neural network have been met includes: determining that a specified number of iterations of the reinforcement learning technique have been performed since the parameter values ​​of the deep neural network were last updated.

32. The non-transitory computer storage medium according to claim 28, wherein, Determining the actual long-term feedback includes: For the latest observation received before the criterion is met, the actual long-term feedback is set based on the actual feedback for that observation and the baseline score for the observation characterizing the state of meeting the specific criterion; and For each other observation received before the criterion is met, the actual long-term feedback for that observation is set to the actual feedback plus the discounted sum of the actual long-term feedbacks for observations received after that observation.

33. The non-transitory computer storage medium according to claim 28, wherein, The gradient update for the policy neural network is determined by the following: Wherein, π(a t |s t ;θ′) represents the response to the observation s determined based on the current value of the policy parameters of the policy neural network. t The score of the action performed, R t This is the actual feedback, and b t It is the baseline score.

34. The non-transitory computer storage medium according to claim 33, wherein, The gradient update for the baseline neural network is determined by the following: Wherein, b(s) t ;θ′ b ) is in response to the observation s t The baseline score.

35. A method for training a deep neural network, executed by one or more computers, the deep neural network for controlling an agent to interact with an environment by performing actions selected from a predetermined set of actions, the deep neural network comprising a policy neural network having a plurality of policy parameters and a baseline neural network having a plurality of baseline parameters, the method comprising: The current parameter values ​​of the deep neural network are determined by a first worker among one or more workers, wherein each worker is associated with a corresponding actor who interacts with a corresponding copy of the environment during the training of the deep neural network, the determination including: Receive the current values ​​of the baseline parameters of the baseline neural network from a memory accessible by each of the one or more workers; and Receive the current value of the policy parameters of the policy neural network from the memory; Until the environment copy interacting with the actor associated with the work body changes to a state that meets a specific criterion: The first worker receives observations characterizing the state of the environment copy, and in response to each observation, selects an action to be performed by the actor associated with the first worker in response to the observation, according to the first worker's corresponding action selection strategy, wherein the selection for each observation includes: The observation is processed using the policy neural network and based on the current values ​​of the policy parameters of the policy neural network to generate a corresponding score for each action in a predetermined set of actions based on the current values ​​of the policy parameters of the policy neural network. Use the corresponding score to select the current action to be performed by the actor; For each observation received before meeting the specific criteria: The first worker generates a baseline score representing the estimated feedback received by the agent starting from the state represented by the observation, based on the current value of the baseline parameters of the baseline neural network. The first working body identifies the actual feedback generated as the acting body performs the selected action in response to the observation; The actual long-term feedback is determined by the first working body based at least on the actual feedback and baseline scores of observations representing states that meet the specific criteria; and The first worker performs iterations of reinforcement learning techniques to determine the corresponding current gradients for the baseline neural network and the policy neural network based on the baseline score and the actual long-term feedback. The first worker updates the corresponding accumulated gradient for the baseline neural network and the policy neural network based on the corresponding current gradient to determine the corresponding updated accumulated gradient; The first worker determines whether the criteria for updating the current parameter values ​​of the deep neural network have been met; and When the criterion used to update the current parameter values ​​of the deep neural network has been met: The parameters of the deep neural network are updated by the first worker, wherein the update includes: The updated accumulated gradient is used to determine the updated parameter values; and The updated parameter values ​​are stored in a memory that can be accessed by each of the one or more working bodies.

36. The method according to claim 35, wherein, Each of the one or more workpieces is executed on the same computer.

37. The method of claim 35, further comprising: When the criterion for updating the current parameter values ​​of the deep neural network has been met, the accumulated gradient of the update is cleared.

38. The method according to claim 35, wherein, Determining whether the criteria for updating the current parameter values ​​of the deep neural network have been met includes: determining that a specified number of iterations of the reinforcement learning technique have been performed since the parameter values ​​of the deep neural network were last updated.

39. The method according to claim 35, wherein, Determining the actual long-term feedback includes: For the latest observation received before the criterion is met, the actual long-term feedback is set based on the actual feedback for that observation and the baseline score for the observation characterizing the state of meeting the specific criterion; and For each other observation received before the criterion is met, the actual long-term feedback for that observation is set to the actual feedback plus the discounted sum of the actual long-term feedbacks for observations received after that observation.

40. The method of claim 35, wherein, The gradient update for the policy neural network is determined by the following: Wherein, π(a t |s t ;θ′) represents the response to the observation s determined based on the current value of the policy parameters of the policy neural network. t The score of the action performed, R t This is the actual feedback, and b t It is the baseline score.

41. A system comprising one or more computers, wherein, The one or more computers are configured to implement a policy neural network for controlling an agent that interacts with the environment by performing actions selected from a predetermined set of actions, wherein: The policy neural network has multiple policy parameters and is configured to process observations characterizing the state of the environment to generate a corresponding score for each action in the predetermined set of actions. The policy neural network has been trained together with a baseline neural network, which has multiple baseline parameters and is configured to process input observations characterizing the state of the environment to generate baseline scores representing the estimated long-term feedback received by the agent starting from the state characterized by the input observations; and The policy neural network and the baseline neural network have been trained using multiple workers, wherein: Each work body is configured to operate independently of each other work body; Each worker is associated with a corresponding actor that interacts with a corresponding copy of the environment during the training; and During the training, each worker is configured to repeatedly perform operations, including: The current baseline parameter values ​​of the baseline neural network and the current policy parameter values ​​of the policy neural network are received from a shared memory accessible by each of the workers. Until the environment copy interacting with the actor associated with this work body changes to a state that meets a specific criterion: Receive observations characterizing the state of the environment replica, and in response to each observation, select an action to be performed by the actor associated with the worker in response to the observation, according to the worker's corresponding action selection strategy, wherein the selection for each observation includes: Based on this observation and the current policy parameter values ​​of the policy neural network, a corresponding score is generated for each action in the predetermined action set, and Use the corresponding score to select the current action to be performed by the actor; For each observation received before the environment copy interacting with the actor associated with the work body is transformed into a state that meets the specific criterion: Based on the observation and the current baseline parameter values ​​of the baseline neural network, a corresponding baseline score is generated representing the estimated long-term feedback received by the agent starting from the state characterized by the observation. Determine the actual long-term feedback corresponding to this observation; and Iterative reinforcement learning techniques are performed to determine the corresponding current gradient for the baseline neural network and the policy neural network based on the baseline score and the actual long-term feedback corresponding to the observation; The corresponding accumulated gradients of the baseline neural network and the policy neural network are updated based on their respective current gradients. Determine whether the criteria used to update the current policy parameter values ​​of the policy neural network and the current baseline parameter values ​​of the baseline neural network have been met; and When the aforementioned criteria have been met: Update the current policy parameter values ​​of the policy neural network and the current baseline parameter values ​​of the baseline neural network, wherein the update includes: The updated accumulated gradient is used to determine the updated parameter values ​​of the policy neural network and the baseline neural network; and The updated parameter values ​​of the policy neural network and the baseline neural network are stored in the shared memory that can be accessed by each of the plurality of workers.

42. The system according to claim 41, wherein, Each workpiece executes independently of every other workpiece on the same computer.

43. The system according to claim 41, wherein, Each worker's action selection strategy differs from that of each other worker.

44. The system according to claim 41, wherein, The operations performed by each work body further include: The updated accumulated gradient is cleared when the criteria for updating the current policy parameter value of the policy neural network and the current baseline parameter value of the baseline neural network have been met.

45. The system according to claim 41, wherein, Determining whether the criteria for updating the current policy parameter values ​​of the policy neural network and the current baseline parameter values ​​of the baseline neural network have been met includes: determining that a specified number of iterations of the reinforcement learning technique have been performed since the parameter values ​​of the policy neural network and the baseline neural network were previously updated.

46. ​​The system according to claim 41, wherein, Determining the actual long-term feedback corresponding to the observations includes: Identify the actual feedback generated as a result of the actor performing a selected action in response to the observation; and The actual long-term feedback corresponding to the observation is determined at least based on the actual feedback of the observation and the baseline score of the observation representing the state that meets the specific criteria.

47. The system according to claim 46, wherein, The actual long-term feedback corresponding to the observation is determined, at least based on the actual feedback from the observation and the baseline score of the observation representing the state that meets the specific criteria, including: For the latest observation received before the criterion is met, the actual long-term feedback is set based on the actual feedback for that observation and the baseline score for the observation characterizing the state of meeting the specific criterion; and For each other observation received before the criterion is met, the actual long-term feedback for that observation is set to the actual feedback plus the discounted sum of the actual long-term feedbacks for observations received after that observation.

48. The system according to claim 41, wherein, The current gradient for the policy neural network is determined as follows: Wherein, π(a t |s t ;θ′) represents the response to the observation s determined based on the current policy parameter value θ′ of the policy neural network. t The action α performed t The score, R t This is the actual long-term feedback corresponding to the observation, and b t It is the baseline score corresponding to the observation.

49. The system according to claim 41, wherein, The current gradient for the baseline neural network is determined as follows: Among them, R t This is the actual long-term feedback corresponding to the observation, b t It is the baseline score corresponding to the observation, and θ′ b This is the current value of the baseline parameter.

50. A non-transitory computer storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to implement a policy neural network for controlling an agent that interacts with an environment by performing actions selected from a predetermined set of actions, wherein: The policy neural network has multiple policy parameters and is configured to process observations characterizing the state of the environment to generate a corresponding score for each action in the predetermined set of actions. The policy neural network has been trained together with a baseline neural network, which has multiple baseline parameters and is configured to process input observations characterizing the state of the environment to generate baseline scores, which represent the estimated long-term feedback received by the agent starting from the state characterized by the input observations. and The policy neural network and the baseline neural network have been trained using multiple workers, wherein: Each work body is configured to operate independently of each other work body; Each worker is associated with a corresponding actor that interacts with a corresponding copy of the environment during the training; and During the training, each worker is configured to repeatedly perform operations, including: The current baseline parameter values ​​of the baseline neural network and the current policy parameter values ​​of the policy neural network are received from a shared memory accessible by each of the workers. Until the environment copy interacting with the actor associated with this work body changes to a state that meets a specific criterion: Receive observations characterizing the state of the environment replica, and in response to each observation, select an action to be performed by the actor associated with the worker in response to the observation, according to the worker's corresponding action selection strategy, wherein the selection for each observation includes: Based on this observation and the current policy parameter values ​​of the policy neural network, a corresponding score is generated for each action in the predetermined action set, and Use the corresponding score to select the current action to be performed by the actor; For each observation received before the environment copy interacting with the actor associated with the work body is transformed into a state that meets the specific criterion: Based on the observation and the current baseline parameter values ​​of the baseline neural network, a corresponding baseline score is generated representing the estimated long-term feedback received by the agent starting from the state characterized by the observation. Determine the actual long-term feedback corresponding to this observation; and Iterative reinforcement learning techniques are performed to determine the corresponding current gradient for the baseline neural network and the policy neural network based on the baseline score and the actual long-term feedback corresponding to the observation; The corresponding accumulated gradients of the baseline neural network and the policy neural network are updated based on their respective current gradients. Determine whether the criteria used to update the current policy parameter values ​​of the policy neural network and the current baseline parameter values ​​of the baseline neural network have been met; and When the aforementioned criteria have been met: Update the current policy parameter values ​​of the policy neural network and the current baseline parameter values ​​of the baseline neural network, wherein the update includes: The updated accumulated gradient is used to determine the updated parameter values ​​of the policy neural network and the baseline neural network; and The updated parameter values ​​of the policy neural network and the baseline neural network are stored in the shared memory that can be accessed by each of the plurality of workers.

51. The non-transitory computer storage medium according to claim 50, wherein, Each workpiece executes independently of every other workpiece on the same computer.

52. The non-transitory computer storage medium according to claim 50, wherein, Each worker's action selection strategy differs from that of each other worker.

53. The non-transitory computer storage medium according to claim 50, wherein, The operations performed by each work body further include: The updated accumulated gradient is cleared when the criteria for updating the current policy parameter value of the policy neural network and the current baseline parameter value of the baseline neural network have been met.

54. The non-transitory computer storage medium according to claim 50, wherein, Determining whether the criteria for updating the current policy parameter values ​​of the policy neural network and the current baseline parameter values ​​of the baseline neural network have been met includes: determining that a specified number of iterations of the reinforcement learning technique have been performed since the parameter values ​​of the policy neural network and the baseline neural network were previously updated.

55. A method executed by one or more data processing devices for controlling an agent to interact with an environment by performing actions selected from a predetermined set of actions, the method comprising: A policy neural network with multiple policy parameters is used to process observations representing the state of the environment to generate a corresponding score for each action in the predetermined set of actions. as well as The agent selects the action to be performed using the corresponding score for each action in the predetermined set of actions; The policy neural network has been trained together with a baseline neural network, which has multiple baseline parameters and is configured to process input observations that characterize the state of the environment to generate a baseline score, which represents the estimated long-term feedback received by the agent starting from the state characterized by the input observation. The policy neural network and the baseline neural network have been trained using multiple workers, wherein: Each work body is configured to operate independently of each other work body; Each worker is associated with a corresponding actor that interacts with a corresponding copy of the environment during the training; and During the training, each worker is configured to repeatedly perform operations, including: The current baseline parameter values ​​of the baseline neural network and the current policy parameter values ​​of the policy neural network are received from a shared memory accessible by each of the workers. Until the environment copy interacting with the actor associated with this work body changes to a state that meets a specific criterion: Receive observations characterizing the state of the environment replica, and in response to each observation, select an action to be performed by the actor associated with the worker in response to the observation, according to the worker's corresponding action selection strategy, wherein the selection for each observation includes: Based on this observation and the current policy parameter values ​​of the policy neural network, a corresponding score is generated for each action in the predetermined action set, and Use the corresponding score to select the current action to be performed by the actor; For each observation received before the environment copy interacting with the actor associated with the work body is transformed into a state that meets the specific criterion: Based on the observation and the current baseline parameter values ​​of the baseline neural network, a corresponding baseline score is generated representing the estimated long-term feedback received by the agent starting from the state characterized by the observation. Determine the actual long-term feedback corresponding to this observation; and Iterative reinforcement learning techniques are performed to determine the corresponding current gradient for the baseline neural network and the policy neural network based on the baseline score and the actual long-term feedback corresponding to the observation; The corresponding accumulated gradients of the baseline neural network and the policy neural network are updated based on their respective current gradients. Determine whether the criteria used to update the current policy parameter values ​​of the policy neural network and the current baseline parameter values ​​of the baseline neural network have been met; and When the aforementioned criteria have been met: Update the current policy parameter values ​​of the policy neural network and the current baseline parameter values ​​of the baseline neural network, wherein the update includes: The updated accumulated gradient is used to determine the updated parameter values ​​of the policy neural network and the baseline neural network; and The updated parameter values ​​of the policy neural network and the baseline neural network are stored in the shared memory that can be accessed by each of the plurality of workers.

56. The method according to claim 55, wherein, Each workpiece executes independently of every other workpiece on the same computer.

57. The method of claim 55, wherein, Each worker's action selection strategy differs from that of each other worker.

58. The method according to claim 55, wherein, The operations performed by each work body further include: The updated accumulated gradient is cleared when the criteria for updating the current policy parameter value of the policy neural network and the current baseline parameter value of the baseline neural network have been met.

59. The method according to claim 55, wherein, Determining whether the criteria for updating the current policy parameter values ​​of the policy neural network and the current baseline parameter values ​​of the baseline neural network have been met includes: determining that a specified number of iterations of the reinforcement learning technique have been performed since the parameter values ​​of the policy neural network and the baseline neural network were previously updated.

60. The method of claim 55, wherein, Determining the actual long-term feedback corresponding to the observations includes: Identify the actual feedback generated as a result of the actor performing a selected action in response to the observation; and The actual long-term feedback corresponding to the observation is determined at least based on the actual feedback of the observation and the baseline score of the observation representing the state that meets the specific criteria.

61. A method for training a deep neural network having multiple parameters, the deep neural network being used to select an action to be performed by an agent, the agent interacting with an environment by performing an action selected from a predetermined set of actions, the method comprising: Multiple workers are used to generate training data for training the parameters of the deep neural network, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a corresponding actor interacting with a corresponding copy of the environment according to a corresponding action selection strategy that differs from the action selection strategy of each other worker, and wherein each worker is further configured to generate training data by repeatedly performing operations, said operations including: Determine the current parameter values ​​of the deep neural network; Receive the current observation representing the current state of a copy of the environment interacted with by the actors associated with the work unit; The action selection strategy of the worker uses one or more outputs generated by the deep neural network based on the current parameter values ​​of the deep neural network to select the current action to be performed by the actor associated with the worker in response to the current observation. When the environment copy is in the current state, identify the actual feedback generated by the actor performing the current action; Receive a next observation representing the next state of a copy of the environment interacting with the actor, wherein the copy of the environment transitions from the current state to the next state in response to the actor performing the current action; and The current action, the actual feedback, and the next observation are added to the training data generated by this worker. Reinforcement learning techniques are applied to the training data generated by each of the plurality of workers to determine one or more current gradients; and The current gradient is used to determine the updated parameter values ​​of the deep neural network.

62. The method according to claim 61, wherein, The action selection strategy associated with each worker is a ∈-greedy policy parameterized by different ∈ probabilities, and wherein, for each worker, selecting the current action to be performed by the actor associated with that worker in response to the current observation, based on the action selection strategy of that worker and using one or more outputs generated by the deep neural network according to the current parameter values ​​of the deep neural network, includes: The current action is randomly selected from the predetermined action set with probability ∈, where ∈ is the probability ∈ of the action selection strategy of the work body.

63. The method according to claim 62, wherein, Each work unit performs operations that further include: In response to the criterion for determining the ∈ probability of updating the action selection strategy associated with the work body being met, a new ∈ probability value is sampled from the probability distribution of the possible values ​​of the ∈ probability.

64. The method according to claim 61, wherein, The action selection strategy associated with each workpiece is parameterized by different temperature parameters, and wherein, for each workpiece, selecting the current action to be performed by the action agent associated with that workpiece in response to the current observation using one or more outputs generated by the deep neural network based on the current parameter values ​​of the deep neural network, according to the action selection strategy of that workpiece, includes: The softmax nonlinearity, parameterized by the temperature parameter of the workpiece, is applied to the action score set generated by the deep neural network to determine the probability distribution of the action set; and The current action is sampled from the predetermined action set according to the probability distribution of the predetermined action set.

65. The method according to claim 61, wherein, The reinforcement learning technique is applied to the training data generated by each of the plurality of workers to determine one or more current gradients for each worker: The reinforcement learning technique is applied to the training data generated by the worker through operations performed by the worker to generate one or more current gradients.

66. The method according to claim 65, wherein, Determining the updated parameter values ​​of the deep neural network using the current gradient includes, for each working body: By performing operations by the worker, the current gradient generated by the worker is used to update the accumulated gradient maintained by the worker to determine the updated accumulated gradient; By performing operations by this worker, it is determined whether the criteria for updating the parameter values ​​of the deep neural network stored in a shared memory accessible to all workers in the worker have been met. as well as When the criterion for updating the parameter values ​​of the deep neural network stored in the shared memory has been met: The updated parameter values ​​of the deep neural network are determined by the operations performed by the worker using the updated accumulated gradient and the parameter values ​​of the deep neural network stored in the shared memory. The updated parameter values ​​of the deep neural network are stored in the shared memory.

67. The method according to claim 61, wherein, Each workpiece executes independently of every other workpiece on the same computer.

68. The method according to claim 61, wherein, The deep neural network is a Q-network, configured to receive input observations and input actions characterizing the input state of the environment, and to generate a Q-value, which is an estimate of the long-term feedback generated by performing the input action when the environment is in the input state, wherein selecting the current action to be performed by the actor in response to the current observation includes: For each action in the predetermined set of actions, the current observation and the action are processed using the deep neural network to generate a Q-value for the action based on the current parameter values ​​of the deep neural network; and Based on the action selection strategy of the work body, the Q value of the action is used to select an action from the predetermined action set.

69. A system comprising: One or more computers; and One or more storage devices are communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations to train a deep neural network with multiple parameters, the deep neural network being used to select actions to be performed by an agent, the agent interacting with the environment by performing actions selected from a predetermined set of actions, the operations including: Multiple workers are used to generate training data for training the parameters of the deep neural network, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a corresponding action agent interacting with a corresponding copy of the environment based on a corresponding action selection strategy that differs from the action selection strategies of each other worker, and wherein each worker is further configured to generate training data by repeatedly performing operations including: Determine the current parameter values ​​of the deep neural network; Receive the current observation representing the current state of a copy of the environment interacted with by the actors associated with the work unit; The action selection strategy of the worker uses one or more outputs generated by the deep neural network based on the current parameter values ​​of the deep neural network to select the current action to be performed by the actor associated with the worker in response to the current observation. When the environment copy is in the current state, identify the actual feedback generated by the actor performing the current action; Receive a next observation representing the next state of a copy of the environment interacting with the actor, wherein the copy of the environment transitions from the current state to the next state in response to the actor performing the current action; and The current action, the actual feedback, and the next observation are added to the training data generated by this worker. Reinforcement learning techniques are applied to the training data generated by each of the plurality of workers to determine one or more current gradients; and The current gradient is used to determine the updated parameter values ​​of the deep neural network.

70. The system according to claim 69, wherein, The action selection strategy associated with each worker is a ∈-greedy policy parameterized by different ∈ probabilities, and wherein, for each worker, selecting the current action to be performed by the actor associated with that worker in response to the current observation, based on the action selection strategy of that worker and using one or more outputs generated by the deep neural network according to the current parameter values ​​of the deep neural network, includes: The current action is randomly selected from the predetermined action set with probability ∈, where ∈ is the probability ∈ of the action selection strategy of the work body.

71. The system according to claim 70, wherein, Each work unit performs operations that further include: In response to the criterion for determining the ∈ probability of updating the action selection strategy associated with the work body being met, a new ∈ probability value is sampled from the probability distribution of the possible values ​​of the ∈ probability.

72. The system according to claim 69, wherein, The action selection strategy associated with each workpiece is parameterized by different temperature parameters, and wherein, according to the action selection strategy for that workpiece, using one or more outputs generated by the deep neural network based on the current parameter values ​​of the deep neural network to select the current action to be performed by the action associated with that workpiece in response to the current observation includes: The softmax nonlinearity, parameterized by the temperature parameter of the workpiece, is applied to the action score set generated by the deep neural network to determine the probability distribution of the action set; and The current action is sampled from the predetermined action set according to the probability distribution of the predetermined action set.

73. The system according to claim 69, wherein, The reinforcement learning technique is applied to the training data generated by each of the plurality of workers to determine one or more current gradients for each worker: The reinforcement learning technique is applied to the training data generated by the worker through operations performed by the worker to generate one or more current gradients.

74. The system according to claim 73, wherein, Determining the updated parameter values ​​of the deep neural network using the current gradient includes, for each working body: By performing operations by the worker, the current gradient generated by the worker is used to update the accumulated gradient maintained by the worker to determine the updated accumulated gradient; By performing operations by this worker, it is determined whether the criteria for updating the parameter values ​​of the deep neural network stored in a shared memory accessible to all workers in the worker have been met. as well as When the criterion for updating the parameter values ​​of the deep neural network stored in the shared memory has been met: The updated parameter values ​​of the deep neural network are determined by the operations performed by the worker using the updated accumulated gradient and the parameter values ​​of the deep neural network stored in the shared memory. The updated parameter values ​​of the deep neural network are stored in the shared memory.

75. The system according to claim 69, wherein, Each workpiece executes independently of every other workpiece on the same computer.

76. The system according to claim 69, wherein, The deep neural network is a Q-network, configured to receive input observations and input actions characterizing the input state of the environment, and to generate a Q-value, which is an estimate of the long-term feedback generated by performing the input action when the environment is in the input state, wherein selecting the current action to be performed by the actor in response to the current observation includes: For each action in the predetermined set of actions, the current observation and the action are processed using the deep neural network to generate a Q-value for the action based on the current parameter values ​​of the deep neural network; and Based on the action selection strategy of the work body, the Q value of the action is used to select an action from the predetermined action set.

77. A non-transitory computer storage medium storing one or more instructions, said instructions, when executed by one or more computers, causing said one or more computers to perform a method for training a deep neural network having multiple parameters, said deep neural network for selecting actions to be performed by an agent, said agent interacting with an environment by performing actions selected from a predetermined set of actions, said method comprising: Multiple workers are used to generate training data for training the parameters of the deep neural network, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a corresponding action agent interacting with a corresponding copy of the environment based on a corresponding action selection strategy that differs from the action selection strategies of each other worker, and wherein each worker is further configured to generate training data by repeatedly performing operations including: Determine the current parameter values ​​of the deep neural network; Receive the current observation representing the current state of a copy of the environment interacted with by the actors associated with the work unit; The action selection strategy of the worker uses one or more outputs generated by the deep neural network based on the current parameter values ​​of the deep neural network to select the current action to be performed by the actor associated with the worker in response to the current observation. When the environment copy is in the current state, identify the actual feedback generated by the actor performing the current action; Receive a next observation representing the next state of a copy of the environment interacting with the actor, wherein the copy of the environment transitions from the current state to the next state in response to the actor performing the current action; and The current action, the actual feedback, and the next observation are added to the training data generated by this worker. Reinforcement learning techniques are applied to the training data generated by each of the plurality of workers to determine one or more current gradients; and The current gradient is used to determine the updated parameter values ​​of the deep neural network.

78. The non-transitory computer storage medium according to claim 77, wherein, The action selection strategy associated with each worker is a ∈-greedy policy parameterized by different ∈ probabilities, and wherein, for each worker, selecting the current action to be performed by the actor associated with that worker in response to the current observation, based on the action selection strategy of that worker and using one or more outputs generated by the deep neural network according to the current parameter values ​​of the deep neural network, includes: The current action is randomly selected from the predetermined action set with probability ∈, where ∈ is the probability ∈ of the action selection strategy of the work body.

79. The non-transitory computer storage medium according to claim 78, wherein, Each work unit performs operations that further include: In response to the criterion for determining the ∈ probability of updating the action selection strategy associated with the work body being met, a new ∈ probability value is sampled from the probability distribution of the possible values ​​of the ∈ probability.

80. The non-transitory computer storage medium according to claim 77, wherein, The action selection strategy associated with each workpiece is parameterized by different temperature parameters, and wherein, according to the action selection strategy for that workpiece, using one or more outputs generated by the deep neural network based on the current parameter values ​​of the deep neural network to select the current action to be performed by the action associated with that workpiece in response to the current observation includes: The softmax nonlinearity, parameterized by the temperature parameter of the workpiece, is applied to the action score set generated by the deep neural network to determine the probability distribution of the action set; and The current action is sampled from the predetermined action set according to the probability distribution of the predetermined action set.

Images