Asynchronous Deep Reinforcement Learning

Abstract

To provide techniques for asynchronously training a deep neural network used by a reinforcement learning system to select actions to be performed by an agent when interacting with an environment.SOLUTION: One of systems includes one or more computers configured to implement a plurality of workers, each worker being configured to operate independently of each other and being associated with a respective actor that interacts with a respective replica of an environment during training of a deep neural network. Aspects of the present disclosure have a technical effect of faster training of a neural network and / or reduced memory requirements for the training.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This specification relates to reinforcement learning. [Background technology]

[0002] In a reinforcement learning system, an agent interacts with the environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

[0003] Some reinforcement learning systems select actions to be performed by an agent in response to receiving a given observation depending on the output of a neural network.

[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output given a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. Summary of the Invention [Means for solving the problem]

[0005] This specification describes technologies related to reinforcement learning. In particular, this specification describes techniques for asynchronously training deep neural networks used by reinforcement learning systems to select actions to be performed by agents when interacting with an environment. Aspects of this specification have the technical effect of faster training of neural networks and / or reduced memory requirements for training.

[0006] To say that one or more computer systems are configured to perform particular operations or actions means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operations or actions. To say that one or more computer programs are configured to perform particular operations or actions means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operations or actions.

[0007] The subject matter described herein may be implemented in particular embodiments to achieve one or more of the following advantages: By parallelizing training, neural networks used by reinforcement learning systems may be trained faster. More specifically, by parallelizing training using multiple workers operating independently on a single machine, communication costs incurred as a result of parallelization may be reduced. Additionally, by eliminating the need for workers to store experience tuples in a replay memory or other storage, as is typically required in parallelization techniques involving multiple workers on multiple machines, memory requirements for training may be reduced, and on-policy reinforcement learning methods may be used.

[0008] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. For example, this specification provides methods, systems, and apparatuses, including a computer program encoded on a computer storage medium, for performing any of the techniques disclosed herein. In some embodiments, the computer program includes instructions that, when executed by a computing device, cause the computing device to perform any of the techniques disclosed herein. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram of an exemplary neural network training system. [Figure 2] 1 is a flow diagram of an exemplary process for training a deep neural network. [Figure 3] 1 is a flow diagram of an exemplary process for performing iterations of a Q-learning technique. [Figure 4] 1 is a flow diagram of an exemplary process for performing iterations of the SARSA technique. [Figure 5] 1 is a flow diagram of an exemplary process for training a Policy neural network. DETAILED DESCRIPTION OF THE INVENTION

[0010] Like reference numbers and designations in the various drawings indicate like elements.

[0011] This document generally describes techniques related to Asynchronous Deep Reinforcement Learning. In particular, this document describes how multiple workers can asynchronously train deep neural networks used by a reinforcement learning system to select actions to be performed by agents interacting with an environment.

[0012] In some implementations, the environment is a simulated environment and the agent is implemented as one or more computer programs that interact with the simulated environment. For example, the simulated environment could be a video game and the agent could be a simulated user playing the video game. As another example, the simulated environment could be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent is a simulated vehicle that moves through the motion simulation. In these implementations, the behavior could be a control input for controlling the simulated user or the simulated vehicle.

[0013] In some other implementations, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real world. For example, the agent may be a robot that interacts with the environment to perform a specific task. As another example, the agent may be an autonomous or semi-autonomous vehicle that moves through the environment. In these implementations, the behavior may be a control input for controlling the robot or autonomous vehicle.

[0014] In general, a reinforcement learning system receives observations that characterize the current state of the environment and uses the observations to select an action to be performed by an agent. In response to the agent performing the selected action, the environment transitions to a new state and the reinforcement learning system receives a reward. The reward is a numerical value that is a function of the state of the environment. While interacting with the environment, the reinforcement learning system attempts to maximize the long-term reward received by the agent in response to the actions performed by the agent.

[0015] In particular, in response to a given observation, the reinforcement learning system selects an action to be performed by the agent according to the output generated by the deep neural network.

[0016] To enable the reinforcement learning system to effectively select actions to be performed by the agent, the neural network training system trains the deep neural network to adjust the values ​​of the parameters of the deep neural network from the initial values ​​of the parameters.

[0017] In particular, the neural network training system includes multiple workers that asynchronously train a deep neural network by selecting actions to be performed by actors that interact with a replica of the environment.

[0018] Each actor is an instance of an agent that operates independently of each other instance. For example, when the agent is a mechanical agent, each actor is also a separate mechanical agent that is the same as or nearly the same as the agent, e.g., has the same or nearly the same characteristics, can perform the same actions as the agent, and operates independently of each other actor.

[0019] 1 illustrates an exemplary neural network training system 100. Neural network training system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0020] The neural network training system 100 includes a plurality of workers 102A-N and a shared memory 110. Each of the workers 102A-N is associated with a respective actor 104A-N that interacts with a corresponding environment replica 106A-N. For example, the worker 102A is associated with the actor 104A that interacts with the environment replica 106A.

[0021] In some implementations, each of workers 110A-N is implemented on the same computer, but executes within a separate thread, process, or other hardware or software within the computer that can independently perform computations for the workers. In these implementations, shared memory 110 is computer memory accessible to each of workers 102A-102N. In implementations where the environment is a virtualized environment, replicas of the actors and environment associated with a given worker also execute within the same thread or other hardware or software as the given worker.

[0022] Each of workers 102A-102N holds an instance of the deep neural network being trained and accesses shared memory 110 to obtain current values ​​of the deep neural network parameters stored in shared memory 110. Each worker then uses the worker's instance of the deep neural network to select an action to be performed by the actor associated with the worker depending on the current values ​​of the neural network parameters obtained from shared memory 110.

[0023] In particular, during training, each worker receives observations characterizing the state of the replica of the environment associated with the worker and, in response to each observation, uses a deep neural network to select an action to be performed by an actor associated with the replica of the environment. The worker then receives a next observation characterizing the next state of the replica of the environment, i.e., the state to which the replica of the environment will transition as a result of the actor performing the selected action, and a reward that results as a result of the actor performing the selected action. The worker determines a gradient using the reward and the next observation and periodically uses the gradient determined by the worker to update parameter values ​​stored in shared memory 110. Determining gradients and updating parameter values ​​stored in shared memory is described in more detail below with reference to FIG. 2.

[0024] By each of the workers 102A-102N repeatedly and asynchronously updating parameter values ​​stored in the shared memory 110, the workers 102A-102N collectively train the deep neural network to determine trained values ​​for the parameters of the deep neural network.

[0025] 2 is a flow diagram of an example process 200 for training a deep neural network. For convenience, process 200 is described as being performed by a worker, e.g., one of workers 102A-N of FIG. 1.

[0026] The worker determines the current values ​​of the parameters of the deep neural network (step 202).

[0027] In some implementations, a worker obtains the current value from a memory that is shared among all of the workers, such as shared memory 110 in FIG.

[0028] In some other implementations, the workers retrieve the current values ​​from memory and write those current values ​​to local memory only after updating the shared memory as described below. In these implementations, the workers determine that the current values ​​are the values ​​to be stored in local memory.

[0029] The worker receives current observations that characterize the current state of the replica of the environment being interacted with by the actor associated with the worker (step 204).

[0030] The worker uses a deep neural network to select a current action to be performed by the actor when the environment is in its current state (step 206).

[0031] In particular, in some implementations, the deep neural network is a Q-network configured to receive observations and actions as inputs and process the inputs according to current values ​​of the Q-network's parameters to generate a neural network output. The neural network output, which will be referred to herein as a Q-value, is an estimate of the long-term reward that would result from the agent or actor performing the input actions when the environment is in a state characterized by the input observations.

[0032] In these implementations, for each action in a predetermined set of actions, a worker processes the action and the current observation using a Q-network to generate a respective current Q-value for each action.

[0033] The worker then uses the current Q value to select an action from a predetermined set of actions as the current action to be performed by the actor, according to the action selection policy for the worker.

[0034] In general, the action selection policy for a given worker is different from the action selection policy for each other worker. For example, in some implementations, the action selection policy for each worker is an ε-greedy policy in which the worker randomly selects an action from a predetermined set of actions with probability ε and selects the action with the highest Q-value with probability 1-ε, where the value of ε is different for each worker. For example, each worker may periodically sample a value for ε for the worker from a distribution of possible values ​​of ε for the worker to determine which ε to use in the action selection policy. As another example, a worker might select an action by applying a Softmax nonlinearity to the Q-value and sampling an action from the resulting distribution. Each worker might then use a different Temperature parameter in Softmax to ensure that each worker had a different action selection policy.

[0035] The worker receives a next observation and an actual reward (step 208). The next observation characterizes the next state of the environment replica, i.e., the state the environment replica transitioned to as a result of the actor associated with the worker taking the current action. The actual reward is a numerical value received from the environment replica as a result of the transition from the current state to the next state.

[0036] The worker runs an iteration of the reinforcement learning procedure to determine the current gradient using the actual reward and the next observation (step 210), which is the gradient of the loss function being optimized to train the deep neural network.

[0037] The loss function and how the workers determine the gradients depend on the reinforcement learning procedure being used to train the deep neural network, which in turn depends on how the deep neural network is constructed.

[0038] For example, when the deep neural network is a Q-network, the workers may use the Q-learning technique or the SARSA technique. Performing iterations of the Q-learning technique is described below with reference to Figure 3. Performing iterations of the SARSA technique is described below with reference to Figure 4.

[0039] The worker updates the accumulated gradient with the current gradient to determine an updated accumulated gradient (step 212). That is, the worker stores the gradient in local memory and updates the stored gradient with the new gradient each time a new gradient is calculated, e.g., by adding the new gradient to the accumulated gradient.

[0040] The worker determines whether a criterion for updating the current value of the neural network's parameter is met (step 214). For example, the criterion for updating the current value may specify that the worker updates the current value after the worker has performed a specified number of reinforcement learning iterations since the worker most recently updated the parameter value. As another example, the criterion may specify that the worker updates the current value when the total number of reinforcement learning iterations performed by all of the multiple workers since the worker most recently updated the parameter value exceeds a specified threshold.

[0041] If the criteria are met, the worker determines updated parameter values ​​using the accumulated gradients (step 216) and writes the updated parameter values ​​to shared memory (step 218). In particular, the worker determines updated parameter values ​​from the accumulated gradients, accesses the current values ​​in shared memory, and then updates those values ​​using the updated parameter value information. In some examples, the worker multiplies the accumulated gradients by a learning rate parameter and then sums the current parameter values ​​with the updated parameter value information to determine updated parameter values. In some other examples, the worker updates the parameter values ​​using an asynchronous variation of the RMSProp optimization procedure.

[0042] After a worker writes the updated parameter values ​​to shared memory, the worker clears the accumulated gradients, so that no gradients are retained by the worker when the next iteration of process 200 begins.

[0043] If the criteria are not met, the worker refrains from writing anything to the shared memory (step 220), i.e., the worker begins executing another iteration of the process 200 without updating the current parameter values.

[0044] Each of the multiple workers may asynchronously and repeatedly execute process 200 to collectively determine trained values ​​for the parameters of the deep neural network. Once the trained values ​​are determined, the reinforcement learning system may select values ​​to be executed by the agent using the deep neural network by the agent interacting with the environment, i.e., by using the deep neural network to process the neural network's inputs according to the trained values ​​of the neural network's parameters. In some other cases, the reinforcement learning system selects values ​​to be executed by the agent using the deep neural network while the workers continue to execute process 200, i.e., by periodically retrieving the values ​​stored in the shared memory and using those values ​​to select actions to be executed by the agent.

[0045] In some cases, once the trained values ​​are determined, they are stored and then transmitted over the network to another system for use in instantiating a trained version of the deep neural network, i.e., to enable the agent to effectively select actions as it interacts with its environment.

[0046] 3 is a flow diagram of an example process 300 for performing iterations of the Q-learning technique. For convenience, process 300 is described as being performed by a worker, e.g., one of workers 102A-N of FIG. 1.

[0047] The worker obtains the current observation, current action, actual reward, and next observation (step 302), i.e., as described above with reference to FIG.

[0048] The workers determine the maximum output of the Target Network when the replica of the environment is in a state characterized by the following observations (step 304). The Target Network is the same neural network as the Q-Network, i.e., it has the same neural network architecture / morphology, but possibly different parameter values. In particular, each worker periodically synchronizes the values ​​of the parameters of the Target Network with the current values ​​of the parameters of the Q-Network stored in shared memory. However, workers generally synchronize the values ​​of the parameters of the Target Network less frequently than they update the parameter values ​​stored in shared memory, resulting in the Target Network and the Q-Network often having different parameter values.

[0049] For each action in the predetermined set of actions, the worker determines the maximum output of the goal network by processing the action in combination with the next observation using the goal network to determine the output of the goal network for the action. The system then selects the maximum of the outputs of the goal network for the actions in the predetermined set of actions as the maximum output of the goal network.

[0050] The worker determines an error from the maximum output of the goal network, the actual reward, and the current Q value previously determined for the current action when the environment was in the current state, i.e., the current Q value used to select the current action as the action to be executed when the environment was in the current state (step 306). In some implementations, the error E is E = r +γmax(Q target ) - Q current where r is the actual reward, γ is a predetermined discount factor, max(Q target ) is the maximum output of the target network, and Q current is the current Q value previously determined for the current action when the environment was in its current state.

[0051] The worker uses the determined error to determine the current gradient (step 308), i.e., the worker updates the value using backpropagation, where the backpropagated error is the determined error for the selected action and zero for all other actions.

[0052] 4 is a flow diagram of an exemplary process 400 for performing iterations of the SARSA technique. For convenience, process 400 is described as being performed by a worker in a neural network training system, for example, one of workers 102A-N in neural network training system 100 of FIG.

[0053] The worker receives the current observation, the current action, the actual reward, and the next observation (step 402).

[0054] The worker selects a next action to be performed by the actor when the replica of the environment is in the next state (step 404). The worker selects the next action as described above with reference to step 206. That is, for each action in the predetermined set of actions, the worker processes the action and the next observation using a Q-network to generate a respective next Q-value for each action. The worker then uses the next Q-value to select an action from the predetermined set of actions as the next action to be performed by the actor in accordance with the action selection policy for the worker.

[0055] The worker determines the output of the goal network for the next action (step 406), i.e., the worker processes the next action and the next observation using the goal network to determine the outcome of the goal network for the next action.

[0056] The worker determines an error from the output of the goal network for the next action, the actual reward, and the current Q value previously determined for the current action (step 408). In some implementations, the error E is E = r + γQ target - Q current where r is the actual reward, γ is a predetermined discount rate, Q target is the output of the goal network for the next action, and Q current is the current Q value previously determined for the current action when the environment was in its current state.

[0057] The worker uses the determined error to determine the current gradient (step 410).

[0058] When performing either process 300 or process 400, the worker further determines whether a criterion for synchronizing the parameters of the target network with the parameters in the shared memory is met. For example, the criterion may be the same as the criterion described above in connection with step 216, except that the specified threshold or specified number of times is greater than the value or number used to determine whether to update the parameters stored in the shared memory. Thus, the worker synchronizes the values ​​of the parameters of the target network less frequently than the worker updates the parameter values ​​stored in the shared memory, resulting in the target network and the Q network having different parameter values ​​more often.

[0059] The above description describes an implementation in which the deep neural network is a Q-network. However, in some other implementations, the deep neural network is a policy neural network that receives input observations and outputs a respective score for each action in a predetermined set of actions, each score representing the likelihood that the corresponding action is the action that should be taken to maximize the long-term reward that results from the environment being in a state characterized by the input observations.

[0060] In these implementations, to select an action in response to a given observation, a reinforcement learning system processes the given observation using a policy neural network and selects an action to be performed by the agent by sampling from the actions according to their scores.

[0061] To ensure that each worker has a different action selection policy, each worker may use a different random seed when sampling from actions, or alternatively, each worker may have a different temperature for the output layer of the policy neural network that generates the score for the action.

[0062] 5 is a flow diagram of an example process 500 for training a policy neural network. For convenience, process 500 is described as being performed by a worker in a neural network training system, for example, one of workers 102A-N in neural network training system 100 of FIG.

[0063] The worker determines the current values ​​of the parameters of the policy neural network (step 502). In particular, the worker obtains the current values ​​from a memory shared among all of the workers, such as shared memory 110 in FIG.

[0064] The workers also retrieve from the shared memory the current values ​​of the parameters of a baseline neural network. The baseline neural network is a neural network configured to receive input observations and generate baseline scores that represent estimated long-term rewards that will be received by the agent starting from a state characterized by the input observations. In some implementations, the baseline and policy networks share some of their parameters.

[0065] The worker receives observations characterizing the state of the replica of the environment and selects an action to be performed by the actor according to the current values ​​of the parameters of the policy neural network (step 504) until the replica of the environment transitions to a state that satisfies a certain criterion. That is, for each observation, the worker processes the observation using the policy neural network according to its current values ​​and selects an action to be performed by the agent by sampling from the actions according to their scores. For example, the certain criterion may be a predetermined number t max may be filled after an observation is received or until a predetermined end state is reached.

[0066] In particular, for each received observation characterizing the state of the replica of the environment, the worker processes the observation using the policy neural network according to the current values ​​of the parameters of the policy neural network to generate a respective score for each action in a predetermined set of actions, and selects an action to be executed by the actor by sampling from the distribution.

[0067] For each received observation, the worker processes the action and also the observation using the underlying neural network according to the current values ​​of the parameters of the underlying neural network to generate an underlying score for the observation.

[0068] In some implementations, one or both of the basis neural network and the policy neural network may be recurrent neural networks.

[0069] The worker determines the actual long-term reward for each received observation (step 506).

[0070] That is, for the last observation received, the system sets the actual long-term reward to, for example, the underlying score predicted by the underlying neural network.

[0071] For a given observation other than the last observation, the system sets the actual long-term reward for the given observation as the reward received when the replica of the environment transitioned to the state characterized by the last observation plus the discounted sum of the actual long-term rewards for the observations received after the actor performed the action selected in response to receiving the given observation.

[0072] For each observation, the worker determines the respective gradient updates for the policy neural network and the respective gradient updates for the basis neural networks (step 510).

[0073] In particular, the observations t The latest information on the gradient of the policy neural network is

number

number

[0074] Observation results t The latest information on the gradients of the underlying neural network is

number

number

[0075] The worker updates the accumulated gradients for the policy neural network with the latest gradient information for the policy neural network to determine updated accumulated gradients for the policy neural network, e.g., by adding the new gradients for each neural network to the accumulated gradients for the neural network, and updates the accumulated gradients for the base neural networks with the latest gradient information for the base neural networks to determine updated accumulated gradients for the base neural networks (step 512).

[0076] The worker determines whether the criteria for updating the current values ​​of the parameters of the basis neural network and the policy neural network are met (step 514).

[0077] For example, the criteria for updating the current value may specify that the worker updates the current value after the worker has performed a specified number of iterations of step 504 since the worker most recently updated the parameter value.

[0078] As another example, the criteria may specify that a worker updates the current value when the total number of iterations of step 504 performed by all of the multiple workers since the worker most recently updated the parameter value exceeds a specified threshold.

[0079] If the criteria are met, the worker determines updated parameter values ​​for the basis neural network and the policy neural network using the corresponding accumulated gradients (step 516), e.g., as described above in connection with steps 216 and 218, and writes the updated parameter values ​​to shared memory (step 518).

[0080] After a worker writes the updated parameter values ​​to shared memory, the worker clears the accumulated gradients, so that no gradients are retained by the worker when the next iteration of process 500 begins.

[0081] If the criteria are not met, the worker refrains from writing anything to the shared memory (step 520), i.e., the worker begins executing another iteration of the process 500 without updating the current parameter values.

[0082] Embodiments and functional operations of the subject matter described herein may be implemented in digital electronic circuitry, including the structures disclosed herein and their structural equivalents, or in tangibly embodied computer software or firmware, computer hardware, or a combination of one or more of these. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0083] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include special-purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Optionally, an apparatus may include, in addition to hardware, code that creates an execution environment for a computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0084] A computer program (which may also be called or referred to as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and may be arranged in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple organized files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program may be arranged to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and connected together by a communications network.

[0085] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs that perform functions by operating on input data and generating output, and the processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0086] A computer suitable for running a computer program may include, and may be based, for example, on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to these mass storage devices, or both. However, a computer need not include such devices. Furthermore, a computer may be incorporated into another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0087] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0088] To provide for interaction with a user, embodiments of the subject matter described herein may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0089] Embodiments of the subject matter described herein may be implemented in a computing system that includes back-end components, e.g., as data servers, or that includes middleware components, e.g., application servers, or that includes front-end components, e.g., a client computer having a Relationship Graphical User Interface or a web browser through which a user can interact with an implementation of the subject matter described herein, or that includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (“LANs”) and wide area networks (“WANs”), e.g., the Internet.

[0090] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0091] While this specification contains many specific implementation details, these should not be considered limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features that are described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as operating in a particular combination, and may even be initially claimed as such, one or more features of a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of a subcombination.

[0092] Similarly, while operations are shown in a particular order in the figures, this should not be understood as requiring such operations to be performed in the particular order shown, or in sequential order, or that all of the operations shown be performed to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the division of various system modules and components in the above-described embodiments should not be understood as requiring such division in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0093] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0094] 100 Neural Network Training System 102A~N Worker 104A~N Actors 106A~106N Environment duplication 110 Shared Memory 200 processes 300 processes 400 processes 500 processes

Claims

1. A method of training a deep neural network having a plurality of parameters for use in selecting an action to be performed by an agent that interacts with an environment by performing an action selected from a predetermined set of actions, comprising: generating training data for training the parameters of the deep neural network using a plurality of workers; For each worker, the workers are configured to operate independently of each other worker; the workers are associated with respective actors that interact with respective copies of the environment according to respective exploration strategies; the exploration strategy is parameterized by a set of exploration strategy parameters, values ​​of the exploration strategy parameters being specific to the worker and different from values ​​of the exploration strategy parameters of each of one or more other workers of the plurality of workers; Each worker is configured to generate training data by repeatedly performing a plurality of operations; The plurality of operations: determining current values ​​of the parameters of the deep neural network; receiving current observations characterizing a current state of a replica of the environment interacting with the actor associated with the worker; and selecting a current action to be performed by the actor associated with the worker in response to the current observation, according to the exploration strategy for the worker, using one or more outputs generated by the deep neural network according to current values ​​of the parameters of the deep neural network; determining an actual reward that would result from the actor performing the current action when the replica of the environment is in the current state; receiving a next observation characterizing a next state of the replica of the environment interacting with the actor, the replica of the environment transitioning from the current state to the next state in response to the actor performing the current behavior; adding the current action, the actual reward, and the next observation to the training data generated by the worker; generating training data, applying reinforcement learning techniques to the training data generated by each of the plurality of workers to determine one or more current gradients; determining updated values ​​for the parameters of the deep neural network using the current gradients; and A method comprising:

2. The exploration strategy associated with each worker is an ε-greedy exploration strategy parameterized by a different probability ε, where ε is an exploration strategy parameter; selecting, for each worker, a current action to be performed by the actor associated with the worker in response to the current observation, in accordance with the exploration strategy for the worker, using one or more outputs generated by the deep neural network in accordance with current values ​​of the parameters of the deep neural network; selecting the current action randomly from the predetermined set of actions with probability ε, where ε is an ε probability parameter that parameterizes the exploration strategy of the worker.

2. The method of claim 1, comprising:

3. Each worker: sampling a new value for the ε probability parameter from a probability distribution over possible values ​​of the probability ε in response to determining that a criterion for updating the ε probability parameter of the exploration strategy associated with the worker is met. The method of claim 2 , further comprising:

4. The exploration strategy associated with each worker is parameterized by a different temperature parameter τ, τ being an exploration strategy parameter; selecting, for each worker, a current action to be performed by the actor associated with the worker in response to the current observation, in accordance with the exploration strategy for the worker, using one or more outputs generated by the deep neural network in accordance with current values ​​of the parameters of the deep neural network; applying a softmax nonlinearity parameterized by the temperature parameter τ for the worker to the set of action scores generated by the deep neural network to determine a probability distribution over the set of actions; sampling the current behavior from the predetermined set of behaviors according to the probability distribution for the predetermined set of behaviors; 2. The method of claim 1, comprising:

5. The method of claim 1, wherein applying reinforcement learning techniques to the training data generated by each of the plurality of workers to determine one or more current gradients comprises, for each worker: applying the reinforcement learning technique to training data generated by the worker to generate one or more current gradients according to actions performed by the worker.

2. The method of claim 1, comprising:

6. The method of claim 5, wherein, for each worker, determining updated values ​​of the parameters of the deep neural network using the current gradients comprises: updating a cumulative gradient maintained by the worker using the current gradient generated by the worker to determine an updated cumulative gradient due to operations performed by the worker; determining whether the actions performed by the workers satisfy criteria for updating the values ​​of the parameters of the deep neural network, the criteria being stored in a shared memory accessible by all of the workers; when the criteria for updating the values ​​of the parameters of the deep neural network stored in the shared memory are met; determining, by operations performed by the workers, updated values ​​of the parameters of the deep neural network using the updated cumulative gradients and the values ​​of the parameters of the deep neural network stored in the shared memory; and storing updated values ​​of the parameters of the deep neural network in the shared memory; 6. The method of claim 5, comprising:

7. The method of claim 1, wherein each worker operates independently of each other worker on the same computer.

8. The deep neural network is a Q-network, and the Q-network comprises: receiving input observations and input actions characterizing an input state of the environment; Generate a Q-value that is an estimate of the long-term reward that will result from performing the input behavior when the environment is in the input state. It is configured as follows: selecting the current action to be performed by the actor in response to the current observation; processing the current observations and the actions using the deep neural network to generate, for each action in the predetermined set of actions, a Q-value for the action according to current values ​​of the parameters of the deep neural network; selecting an action from the predetermined set of actions according to the exploration strategy parameterized by the set of exploration strategy parameters for the worker using the Q-value for the action; 2. The method of claim 1, comprising:

9. A system comprising: one or more computers; one or more storage devices communicatively coupled to the one or more computers; the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a deep neural network having a plurality of parameters used to select an action to be performed by an agent that interacts with an environment by performing an action selected from a predetermined set of actions, the operations comprising: generating training data for training the parameters of the deep neural network using a plurality of workers; For each worker, the workers are configured to operate independently of each other worker; the workers are associated with respective actors that interact with respective copies of the environment according to respective exploration strategies; the exploration strategy is parameterized by a set of exploration strategy parameters, values ​​of the exploration strategy parameters being specific to the worker and different from values ​​of the exploration strategy parameters of each of one or more other workers of the plurality of workers; Each worker is configured to generate training data by repeatedly performing a plurality of operations; The plurality of operations: determining current values ​​of the parameters of the deep neural network; receiving current observations characterizing a current state of a replica of the environment interacting with the actor associated with the worker; and selecting a current action to be performed by the actor associated with the worker in response to the current observation, according to the exploration strategy for the worker, using one or more outputs generated by the deep neural network according to current values ​​of the parameters of the deep neural network; determining an actual reward that would result from the actor performing the current action when the replica of the environment is in the current state; receiving a next observation characterizing a next state of the replica of the environment interacting with the actor, the replica of the environment transitioning from the current state to the next state in response to the actor performing the current behavior; adding the current action, the actual reward, and the next observation to the training data generated by the worker; generating training data, applying reinforcement learning techniques to the training data generated by each of the plurality of workers to determine one or more current gradients; determining updated values ​​for the parameters of the deep neural network using the current gradients; and Including, the system.

10. The exploration strategy associated with each worker is an ε-greedy exploration strategy parameterized by a different probability ε, where ε is an exploration strategy parameter; selecting, for each worker, a current action to be performed by the actor associated with the worker in response to the current observation, in accordance with the exploration strategy for the worker, using one or more outputs generated by the deep neural network in accordance with current values ​​of the parameters of the deep neural network; selecting the current action randomly from the predetermined set of actions with probability ε, where ε is an ε probability parameter that parameterizes the exploration strategy of the worker. The system of claim 9, comprising:

11. Each worker: sampling a new value for the ε probability parameter from a probability distribution over possible values ​​of the probability ε in response to determining that a criterion for updating the ε probability parameter of the exploration strategy associated with the worker is met. The system of claim 10 , further comprising:

12. The exploration strategy associated with each worker is parameterized by a different temperature parameter τ, τ being an exploration strategy parameter; selecting, for each worker, a current action to be performed by the actor associated with the worker in response to the current observation, in accordance with the exploration strategy for the worker, using one or more outputs generated by the deep neural network in accordance with current values ​​of the parameters of the deep neural network; applying a softmax nonlinearity parameterized by the temperature parameter τ for the worker to the set of action scores generated by the deep neural network to determine a probability distribution over the set of actions; sampling the current behavior from the predetermined set of behaviors according to the probability distribution for the predetermined set of behaviors; The system of claim 9, comprising:

13. The method of claim 12, wherein applying reinforcement learning techniques to the training data generated by each of the plurality of workers to determine one or more current gradients comprises, for each worker: applying the reinforcement learning technique to training data generated by the worker to generate one or more current gradients according to actions performed by the worker. The system of claim 9, comprising:

14. The method of claim 13, wherein determining updated values ​​of the parameters of the deep neural network using the current gradients comprises, for each worker: updating a cumulative gradient maintained by the worker using the current gradient generated by the worker to determine an updated cumulative gradient due to operations performed by the worker; determining whether the actions performed by the workers satisfy criteria for updating the values ​​of the parameters of the deep neural network, the criteria being stored in a shared memory accessible by all of the workers; when the criteria for updating the values ​​of the parameters of the deep neural network stored in the shared memory are met; determining, by operations performed by the workers, updated values ​​of the parameters of the deep neural network using the updated cumulative gradients and the values ​​of the parameters of the deep neural network stored in the shared memory; and storing updated values ​​of the parameters of the deep neural network in the shared memory; The system of claim 13, comprising:

15. The system of claim 9, wherein each worker operates independently of each other worker on the same computer.

16. The deep neural network is a Q-network, wherein the Q-network comprises: receiving input observations and input actions characterizing an input state of the environment; Generate a Q-value that is an estimate of the long-term reward that will result from performing the input behavior when the environment is in the input state. It is configured as follows: selecting the current action to be performed by the actor in response to the current observation; processing the current observations and the actions using the deep neural network to generate, for each action in the predetermined set of actions, a Q-value for the action according to current values ​​of the parameters of the deep neural network; selecting an action from the predetermined set of actions according to the exploration strategy parameterized by the set of exploration strategy parameters for the worker using the Q-value for the action; The system of claim 9, comprising:

17. One or more non-transitory computer storage media for storing instructions, comprising: The instructions, when executed by one or more computers, cause the one or more computers to perform operations for training an industrial plant controller that controls the operation of an industrial plant; The operations for training the industrial plant controller include a method for training a deep neural network having a plurality of parameters used to select actions to be performed by an agent that interacts with an environment by performing actions selected from a predetermined set of actions, the method comprising: generating training data for training the parameters of the deep neural network using a plurality of workers; For each worker, the workers are configured to operate independently of each other worker; the workers are associated with respective actors that interact with respective copies of the environment according to respective exploration strategies; the exploration strategy is parameterized by a set of exploration strategy parameters, values ​​of the exploration strategy parameters being specific to the worker and different from values ​​of the exploration strategy parameters of each of one or more other workers of the plurality of workers; Each worker is configured to generate training data by repeatedly performing a plurality of operations; The plurality of operations: determining current values ​​of the parameters of the deep neural network; receiving current observations characterizing a current state of a replica of the environment interacting with the actor associated with the worker; and selecting a current action to be performed by the actor associated with the worker in response to the current observation, according to the exploration strategy for the worker, using one or more outputs generated by the deep neural network according to current values ​​of the parameters of the deep neural network; determining an actual reward that would result from the actor performing the current action when the replica of the environment is in the current state; receiving a next observation characterizing a next state of the replica of the environment interacting with the actor, the replica of the environment transitioning from the current state to the next state in response to the actor performing the current behavior; adding the current action, the actual reward, and the next observation to the training data generated by the worker; generating training data, applying reinforcement learning techniques to the training data generated by each of the plurality of workers to determine one or more current gradients; determining updated values ​​for the parameters of the deep neural network using the current gradients; and non-transitory computer storage media, including 18. The exploration strategy associated with each worker is an ε-greedy exploration strategy parameterized by a different probability ε, where ε is an exploration strategy parameter; selecting, for each worker, a current action to be performed by the actor associated with the worker in response to the current observation, in accordance with the exploration strategy for the worker, using one or more outputs generated by the deep neural network in accordance with current values ​​of the parameters of the deep neural network; selecting the current action randomly from the predetermined set of actions with probability ε, where ε is an ε probability parameter that parameterizes the exploration strategy of the worker.

20. The non-transitory computer storage medium of claim 17, comprising:

19. Each worker: sampling a new value for the ε probability parameter from a probability distribution over possible values ​​of the probability ε in response to determining that a criterion for updating the ε probability parameter of the exploration strategy associated with the worker is met.

20. The non-transitory computer storage medium of claim 18, further comprising:

20. The exploration strategy associated with each worker is parameterized by a different temperature parameter τ, τ being an exploration strategy parameter; selecting, for each worker, a current action to be performed by the actor associated with the worker in response to the current observation, in accordance with the exploration strategy for the worker, using one or more outputs generated by the deep neural network in accordance with current values ​​of the parameters of the deep neural network; applying a softmax nonlinearity parameterized by the temperature parameter τ for the worker to the set of action scores generated by the deep neural network to determine a probability distribution over the set of actions; sampling the current behavior from the predetermined set of behaviors according to the probability distribution for the predetermined set of behaviors; 20. The non-transitory computer storage medium of claim 17, comprising:

Images