Continuous control with deep reinforcement learning

Abstract

The present disclosure relates to continuous control with deep reinforcement learning. Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network for selecting actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a mini-batch of experience tuples, and updating current values of parameters of the actor neural network, including, for each experience tuple in the mini-batch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; using an error between the target neural network output and the neural network output to update current values of parameters of the critic neural network; and using the critic neural network to update current values of parameters of the actor neural network.

Description

[0001] Divisional

[0002] This application is a divisional of Chinese Patent Application No. 201680043561.7, filed on July 22, 2016, which has an application date of July 22, 2016 under 35 U.S.C. § 119(a). This application claims priority to U.S. Patent Application No. 15 / 940, 1 10, filed on April 5, 2018, which claims priority to U.S. Provisional Patent Application No. 62 / 485, 1 10, filed on April 6, 2017, each of which is incorporated by reference in its entirety. TECHNICAL FIELD

[0003] This disclosure relates to continuous control with deep reinforcement learning. BACKGROUND

[0004] This specification relates to selecting actions to be performed by a reinforcement learning agent.

[0005] A reinforcement learning agent interacts with an environment by receiving observations that characterize a current state of the environment and performing actions in response. Some reinforcement learning agents use a neural network to select an action to perform in response to receiving any given observation.

[0006] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., as input to a next hidden layer or to the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. SUMMARY

[0007] This specification describes techniques related to reinforcement learning.

[0008] Generally, one innovative aspect can be embodied in a method for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations that characterize current states of the environment and performing actions selected from a continuous action space, where the actor neural network maps an observation to a next action in accordance with current values of parameters of the actor neural network, and where the method includes: obtaining a mini-batch of experience tuples from a replay memory, each experience tuple including a training observation that characterizes a training state of the environment, a training action from the continuous action space performed by the agent in response to receiving the training observation, a training reward received by the agent for performing the training action, and a next training observation that characterizes a next training state of the environment; and updating the current values of the parameters of the actor neural network using the mini-batch of experience tuples, including: for each experience tuple in the mini-batch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple in accordance with current values of parameters of the critic neural network; and determining a target neural network output for the experience tuple in accordance with the training reward in the experience tuple and the next training observation in the experience tuple; updating the current values of the parameters of the critic neural network using an error between the target neural network output and the neural network output for the experience tuple in the mini-batch; and updating the current values of the parameters of the actor neural network using the critic neural network.

[0009] Determining the target neural network output for the experience tuple can include processing the next training observation using a target actor neural network to determine a predicted next action for the experience tuple in accordance with current values of parameters of the target actor neural network, where the target actor neural network is identical to the actor neural network but the current values of the parameters of the target actor neural network are different from the current values of the parameters of the actor neural network; processing the next training observation and the predicted next action for the experience tuple using a target critic neural network to generate a predicted next neural network output in accordance with current values of parameters of the target critic neural network, where the target critic neural network is identical to the critic neural network but the current values of the parameters of the target critic neural network are different from the current values of the parameters of the critic neural network; and determining the target neural network output for the experience tuple in accordance with the training reward for the experience tuple and the predicted next neural network output. The method can further include updating the current values of the parameters of the target actor neural network using the updated values of the parameters of the actor neural network; and updating the current values of the parameters of the target critic neural network using the updated values of the parameters of the critic neural network. The current values of the parameters of the target actor neural network and the target critic neural network can be constrained to change slowly during training of the actor neural network.

[0010] The evaluator neural network, the actor neural network, or both can include one or more batch normalization neural network layers. The observations received by the agent can characterize the state of the environment using a low-dimensional feature vector that characterizes the state of the environment. Values of different dimensions of the low-dimensional feature vector can have different ranges. The observations received by the agent can characterize the state of the environment using high-dimensional pixel inputs from one or more images that characterize the state of the environment.

[0011] The method can further include, independent of updating the current values of the parameters of the actor neural network using the mini-batch of experience tuples: generating a new experience tuple, including: receiving a new training observation; processing the new training observation using the actor neural network to select a new training action to be performed by the agent according to the current values of the parameters of the actor neural network; receiving a new training reward in response to the agent performing the new training action; receiving a new next training observation; and generating the new experience tuple including the new training observation, the new training action, the new training reward, and the new next training observation; and adding the new experience tuple to the replay memory. Processing the new training observation using the actor neural network to the new training action can include processing the new training observation using the actor neural network to generate an initial new training action; sampling from a noise process to obtain a noise factor; and adjusting the initial new action by the noise factor to generate the new training action.

[0012] The method can further include outputting the actor neural network arranged to select actions to be performed by the agent.

[0013] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages. A reinforcement learning system can efficiently and directly learn an effective action selection policy for an agent in a high-dimensional continuous action space, i.e., by training an actor neural network as described in this specification. In particular, by training an actor neural network as described in this specification, a reinforcement learning system can efficiently learn an effective action selection policy and then efficiently explore even for tasks that require precise control of actions and when discretization of the action space is difficult. Furthermore, a reinforcement learning system can learn an effective policy both with observations that are low-dimensional observations and with observations that are high-dimensional pixel inputs. The subject matter can provide for improved training of a reinforcement learning system to provide an improved reinforcement learning system that can, for example, be able to learn an action selection policy for tasks that have previously been challenging.

[0014] To understand, each aspect can be implemented in any convenient form. For example, aspects and embodiments can be implemented by suitable computer programs that can be carried on suitable carrier media, which can be tangible (e.g., a disc) or intangible (e.g., a communications signal). Aspects can also be implemented by using suitable apparatus, which can be in the form of programmable computers running computer programs. The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 An example reinforcement learning system is shown.

[0016] Figure 2 is a flowchart of an example process of adding an experience tuple to a replay memory.

[0017] Figure 3 is a flowchart of an example process of determining an update to a current value of a parameter of an actor neural network.

[0018] Figure 4 is a flowchart of an example process of determining a target neural network output for an experience tuple.

[0019] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0020] This specification generally describes a reinforcement learning system that selects actions to be performed by a reinforcement learning agent that interacts with an environment. To interact with the environment, the agent receives data that characterizes a current state of the environment, and performs an action in a continuous action space in response to the received data. In this specification, data that characterizes a state of an environment is referred to as an observation.

[0021] In some embodiments, the environment is a simulated environment, and the agent is implemented as one or more computer programs that interact with the simulated environment. For example, the simulated environment can be a video game, and the agent can be a simulated user that plays the video game. As another example, the simulated environment can be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent can be a simulated vehicle that navigates in the motion simulation environment. In these embodiments, the actions can be points in a possible control input space that control the simulated user or simulated vehicle.

[0022] In some other implementations, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent can be a robot that interacts with the environment to complete a particular task. As another example, the agent can be an autonomous or semi-autonomous vehicle that navigates in the environment. In these implementations, the action can be a point in a possible control input space that controls the robot or autonomous vehicle.

[0023] In some cases, the observation values characterize the state of the environment by using a low-dimensional feature vector that characterizes the state of the environment. In these cases, the values of different dimensions of the low-dimensional feature vector can have different ranges.

[0024] In some other cases, the observation values characterize the state of the environment by using a high-dimensional pixel input from one or more images that characterize the state of the environment, e.g., images that simulate the environment or images captured by sensors of the mechanical agent as the mechanical agent interacts with the real-world environment.

[0025] Figure 1 An example reinforcement learning system 100 is shown. The reinforcement learning system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0026] The reinforcement learning system 100 selects actions to be performed by a reinforcement learning agent 102 that interacts with an environment 104. That is, the reinforcement learning system 100 receives observation values, where each observation value characterizes a respective state of the environment 104, and in response to each observation value, selects an action from a continuous action space of actions to be performed by the reinforcement learning agent 102 in response to the observation value.

[0027] In particular, the reinforcement learning system 100 selects actions by using an actor neural network 110. The actor neural network 110 is a neural network that is configured to receive an observation value and process the observation value to map the observation value to a next action, i.e., to a point in a continuous action space that defines an action that should be performed by the agent in response to the observation value.

[0028] To allow the agent 102 to effectively interact with the environment 104, the reinforcement learning system 100 trains the actor neural network 110 to determine training values for the parameters of the actor neural network 110.

[0029] Once the actor neural network 110 has been trained, the reinforcement learning system 100 can effectively use the actor neural network 110 to select actions to be performed by the agent 102. Specifically, when an observation is received, the reinforcement learning system 100 can process the observation by using the actor neural network 110 to map the observation to a new action in accordance with the trained values of the parameters of the actor neural network 110, and then direct the agent 102 to perform the new action in response to the observation, i.e., by sending instructions to the agent 102 that cause the agent to perform the new action.

[0030] To assist in training the actor neural network 110, the reinforcement learning system 100 maintains a training component 120 that includes a replay memory 130, a critic neural network 140, a target actor neural network 150, and a target critic neural network 160.

[0031] The replay memory 130 stores experience tuples generated as a result of the agent 102 interacting with the environment 104 for use in training the actor neural network 110.

[0032] Specifically, each experience tuple in the replay memory includes a training observation that characterizes a training state of the environment, an action performed by the agent 102 in response to the training observation, a training reward received by the system 100 in response to the agent 102 performing the action, and a next observation that characterizes a next state of the environment, i.e., a state to which the environment transitions after the agent performs the action.

[0033] The reinforcement learning system 100 generates experience tuples from interactions of the agent 102 with the environment 104 during training of the actor neural network 110. An example process of generating experience tuples during training is described in more detail below with reference to Figure 2 The example process of generating experience tuples during training is described in more detail below with reference to

[0034] The critic neural network 140 is a neural network that is configured to receive an action and an observation as input and process the action and the observation to generate a neural network output. As will be described in more detail below, during training, the reinforcement learning system 100 adjusts values of parameters of the critic neural network 140, and uses the critic neural network 140 in updating values of parameters of the actor neural network 110.

[0035] In some implementations, the evaluator neural network 140, the actor neural network 110, or both include one or more batch normalization layers to minimize covariate shift during training. Batch normalization layers are described in more detail in Ioffe, Sergey, and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.03167, 2015.

[0036] The target actor neural network 150 is the same as the actor neural network 110 - i.e., has the same neural network architecture as the actor neural network 110, but has parameter values that can be different from the parameter values of the actor neural network 110.

[0037] Likewise, the target evaluator neural network 160 is the same as the evaluator neural network 140 but has parameter values that can be different from the parameter values of the evaluator neural network 140.

[0038] To train the neural networks using the training component 120, the reinforcement learning system 100 repeatedly selects a small batch of experience tuples from the replay memory 130. Each small batch of experience tuples includes a predetermined number of experience tuples. The predetermined number of experience tuples can be randomly selected experience tuples.

[0039] For each experience tuple in a given selected small batch, the reinforcement learning system 100 determines updates to the current values of the parameters of the actor neural network 110 and the current values of the parameters of the evaluator neural network 140 using the evaluator neural network 140, the target actor neural network 150, and the target evaluator neural network 160, and then adjusts the current values of the parameters of the actor neural network 110 and the current values of the parameters of the evaluator neural network 140 by using the updates. The following will refer to the adjustments to the current values of the parameters of the actor neural network 110 and the current values of the parameters of the evaluator neural network 140 as the adjustments to the actor neural network 110 and the evaluator neural network 140, respectively. Figure 3 Generating these updates and adjusting the current values of the parameters of the evaluator neural network 140 and the actor neural network 110 are described in more detail.

[0040] During training, the reinforcement learning system 100 also periodically updates the values of the parameters of the target evaluator neural network 160 and the values of the parameters of the target actor neural network 150 such that the values slowly track changes in the values of the parameters of the evaluator neural network 140 and the values of the parameters of the actor neural network 110, respectively.

[0041] Once the mini-batch of experience tuples has been used for training, the reinforcement learning system 100 can remove the experience tuples in the mini-batch from the replay memory 130.

[0042] In general, during training, the reinforcement learning system 100 generates experience tuples and adds the generated tuples to the replay memory 130 independently of sampling experience tuples from the replay memory 130 and adjusting the parameters of the actor neural network 110 - i.e., asynchronously to sampling experience tuples from the replay memory 130 and adjusting the parameters of the actor neural network 110.

[0043] Figure 2 is a flowchart of an example process 200 of adding experience tuples to a replay memory. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100, appropriately programmed in accordance with this specification, can perform the process 200. Figure 1

[0044] The system receives a current observation that characterizes a current state of the environment (step 202).

[0045] The system processes the observation using the actor neural network in accordance with current values of the parameters of the actor neural network (step 204). As described above, the actor neural network is configured to map the current observation to a next action, i.e., a point in the continuous action space, in accordance with the current values of the parameters.

[0046] The system selects an action to be performed by the agent by using the next action (step 206).

[0047] In some implementations, the system selects the next action as the action to be performed by the agent.

[0048] In some other implementations, to encourage exploration of the action space during training, the system samples a noise process to obtain a noise factor and then adjusts the next action by the noise factor to generate the action to be performed by the agent.

[0049] The noise process used to obtain the noise factor can be selected to suit the environment. For example, for some environments, the noise process can be an Ornstein-Uhlenbeck process that generates exploration that is temporally correlated. The Ornstein-Uhlenbeck process is described in more detail in George E. Uhlenbeck and Leonard S. Ornstein. "On the theory of the Brownian motion". In: Physical review 36.5 (1930), p. 823.​

[0050] The system receives a reward and a next observation (step 206). The next observation characterizes a next state of the environment, i.e., a state to which the environment transitions as a result of the agent performing the selected action, and the reward is a numerical value that the system receives from the environment as a result of the agent performing the selected action.

[0051] The system generates an experience tuple that includes the current observation, the selected action, the reward, and the next observation, and stores the generated experience tuple in the replay memory for use in training the actor neural network (step 208).

[0052] Figure 3 is a flowchart of an example process 300 of determining an update to current values of parameters of an actor neural network. For convenience, process 300 will be described as performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., reinforcement learning system 100, appropriately programmed in accordance with this specification, can perform process 300. Figure 1

[0053] The system receives an experience tuple (step 302). The experience tuple is one of a small batch of experience tuples that the system samples from the replay memory.

[0054] The experience tuple includes a training observation that characterizes a training state of the environment, a training action from a continuous action space that is performed by the agent in response to receiving the training observation, a training reward received by the agent for performing the training action, and a next training observation that characterizes a next training state of the environment.

[0055] The system determines a neural network output for the experience tuple from the training observation in the experience tuple and the selected action using the critic neural network in accordance with current values of parameters of the critic neural network (step 304).

[0056] The system determines a target neural network output for the experience tuple from the training reward in the experience tuple and the next training observation in the experience tuple (step 306). Typically, the system determines the target neural network output using a target actor neural network and a target critic neural network. Determining the target neural network output is described in more detail below with reference to Figure 4

[0057] ​​The system determines an update to the current values of the parameters of the evaluator neural network using the error between the target neural network output for the experience tuple and the neural network output generated by the evaluator neural network for the experience tuple (step 308). That is, the system can determine an update to the current values of the parameters that reduces the error by using conventional machine learning training techniques, e.g., by performing iterations of gradient descent with backpropagation. From the determination of the update to the current values of the parameters of the evaluator neural network, the system updates the current values of the parameters of the evaluator neural network (step 310). Figure 4 As will be clear from the description of the system, by updating the current values of the parameters in this way, the system trains the evaluator neural network to generate a neural network output that represents the time-discounted total future reward that would be received as a response of the agent to performing a given action in response to a given observation.

[0058] The system determines an update to the current values of the parameters of the actor neural network using the actor neural network (step 310).

[0059] In particular, to determine the update, the system processes the training observation in the tuple using the actor neural network to generate a next action for the training observation in accordance with the current values of the parameters.

[0060] The system then determines an update to the current values of the parameters of the actor neural network that depends on (i) a gradient of the evaluator neural network with respect to the next action input pair for the training observation and in accordance with the current values of the parameters of the evaluator neural network and (ii) a gradient of the actor neural network with respect to the action taken for the training observation and in accordance with the current values of the parameters of the actor neural network, i.e., is a product of the two gradients or a different combination thereof. The system can determine the gradients (i) and (ii) by backpropagating the respective gradients through the respective networks.

[0061] In general, the system performs the process 300 for each experience tuple in a given mini-batch to determine an update to the parameters of the evaluator neural network and an update to the parameters of the actor neural network for each tuple. Once the updates for each tuple in the mini-batch have been determined, the system updates the current values of the parameters of the actor neural network and the current values of the parameters of the evaluator neural network using the updates for the tuples in the mini-batch. For example, for each network, the system can add each update to the current values of the parameters of the network to update these values.

[0062] Once the updated values of the parameters of the actor neural network and the critic neural network have been determined, the system updates the current values of the target critic neural network parameters and the target actor neural network parameters, causing these values to slowly track changes in the values of the parameters of the critic neural network and the values of the parameters of the actor neural network. In particular, the system constrains the values of the target critic neural network parameters and the target actor neural network parameters to change slowly during training to improve the stability of the training process.

[0063] For example, the updated value of one of the target networks can be a linear interpolation between the updated value of the corresponding actor or critic network and the current value of the target network, with the current value of the target network being more heavily weighted in the interpolation.

[0064] By repeatedly performing the process 300 on multiple different mini-batches of experience tuples, the system can train the actor neural network to determine trained values of the parameters of the actor neural network and allow the actor neural network to be effectively used to select actions to be performed by an agent interacting with an environment.

[0065] Figure 4 is a flowchart of an example process 400 of determining a target neural network output for an experience tuple. For convenience, the process 400 will be described as performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100, appropriately programmed in accordance with this specification, can perform the process 400. Figure 1

[0066] The system processes the next training observation in the experience tuple using the target actor neural network in accordance with the current values of the parameters of the target actor neural network to generate a predicted next action (step 402). As described above, the target actor neural network is identical to the actor neural network but has possibly different parameter values.

[0067] The system processes the next training observation and the predicted next action using the target critic neural network in accordance with the current values of the parameters of the target critic neural network to generate a predicted next neural network output (step 404). As described above, the target critic neural network is identical to the critic neural network but has possibly different parameter values.

[0068] The system determines the target neural network output for the experience tuple from the training reward of the experience tuple and the predicted neural network output. In particular, the system multiplies the predicted neural network output by a predetermined temporal discount factor and then adds the resulting product to the training reward to generate the target neural network output for the experience tuple.

[0069] ​"one or more computer programs" are intended to include a computer program or a portion of a computer program that is implemented as code that is executed by a data processing apparatus. The program code is executed by the data processing apparatus to perform the operations described herein.

[0070] Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0071] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0072] A computer program, which can also be referred to or described as a program, software, a software application, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and

[0073] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit), and that apparatus can also be implemented as the special purpose logic circuitry.

[0074] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, for example, magnetic, magneto optical, or optical disks, or a computer can be operatively coupled to receive data from or transfer data to or both, a communications network in

[0075] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0076] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's user device in response to requests received from the web browser.

[0077] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), e.g., the Internet.

[0078] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0079] Although this description contains many specifics, these should not be construed as limiting the scope of any invention or application in which such inventions or applications can be embodied, but merely as providing information to enable a specific embodiment thereof. Some features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features can be described above as acting in certain combinations and initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a sub-combination or a variation of a sub-combination.

[0080] Also, although operations can be shown in a particular order in the drawings, this should not be understood as requiring the operations to be performed in that particular order, or in sequential order, or as requiring all illustrated operations to be performed to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0081] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

Claims

1. A method for training an actor neural network, the actor neural network being used for selecting an action to be performed by an agent interacting with an environment by receiving an observation representing a current state of the environment and performing an action selected from a continuous action space, wherein, the actor neural network maps observation values to next actions according to current values of parameters of the actor neural network, wherein the observation values received by the agent characterize the state of the environment using high-dimensional pixel inputs from one or more images that characterize the state of the environment, or the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment, the mechanical agent comprising a robot that interacts with the real-world environment to achieve a particular task, or an autonomous or semi-autonomous vehicle that navigates in the real-world environment, or the environment is a simulated environment and the agent is implemented as one or more computer programs that interact with the simulated environment, the agent comprising a simulated user that plays a video game, or a simulated vehicle that navigates in a sports simulation environment, and wherein the method comprises: obtaining a mini-batch of experience tuples from a replay memory, each experience tuple comprising a training observation that characterizes a training state of the environment, a training action from the continuous action space that was performed by the agent in response to receiving the training observation, a training reward received by the agent for performing the training action, and a next training observation that characterizes a next training state of the environment; and updating the current values of the parameters of the actor neural network using the mini-batch of experience tuples, comprising: for each experience tuple in the mini-batch: processing the training observation and the training action in the experience tuple using an evaluator neural network to determine a neural network output for the experience tuple according to current values of parameters of the evaluator neural network, and determining a target neural network output for the experience tuple from the training reward in the experience tuple and the next training observation in the experience tuple, updating the current values of the parameters of the evaluator neural network using an error between the target neural network output and the neural network output for the experience tuple in the mini-batch; and updating the current values of the parameters of the actor neural network using the evaluator neural network.

2. The method of claim 1, wherein, determining the target neural network output for the experience tuple comprises: processing the next training observation using a target actor neural network to determine a predicted next action for the experience tuple according to current values of parameters of the target actor neural network, wherein the target actor neural network is identical to the actor neural network but the current values of the parameters of the target actor neural network are different from the current values of the parameters of the actor neural network; processing the next training observation and the predicted next action for the experience tuple using a target evaluator neural network to generate a predicted next neural network output according to current values of parameters of the target evaluator neural network, wherein the target evaluator neural network is identical to the evaluator neural network but the current values of the parameters of the target evaluator neural network are different from the current values of the parameters of the evaluator neural network; and determining the target neural network output for the experience tuple from the training reward and the predicted next neural network output for the experience tuple.

3. The method of claim 2, further comprising: updating a current value of a parameter of the target actor neural network using the updated value of the parameter of the actor neural network; and updating a current value of a parameter of the target critic neural network using the updated value of the parameter of the critic neural network.

4. The method of claim 3, wherein, The current values of the parameters of the target actor neural network and the target critic neural network are constrained to change slowly during training of the actor neural network.

5. The method of claim 1, wherein, The critic neural network, the actor neural network, or both include one or more batch normalization neural network layers.

6. The method of claim 1, wherein, The observation values received by the agent represent the state of the environment using low-dimensional feature vectors that characterize the state of the environment.

7. The method of claim 6, wherein, The values of different dimensions of the low-dimensional feature vectors have different ranges.

8. The method of any one of claims 1-7, further comprising: independent of updating a current value of a parameter of the actor neural network using the mini-batch of experience tuples: generating new experience tuples, including: receiving a new training observation value, processing the new training observation value using the actor neural network to select a new training action to be performed by the agent from a current value of a parameter of the actor neural network, receiving a new training reward in response to the agent performing the new training action, receiving a new next training observation value, and generating a new experience tuple including the new training observation value, the new training action, the new training reward, and the new next training observation value; and adding the new experience tuple to the replay memory.

9. The method of claim 8, wherein, Processing the new training observation value using the actor neural network to select a new training action includes: processing the new training observation value using the actor neural network to generate an initial new training action; sampling from a noise process to obtain a noise factor; and adjusting the initial new training action by the noise factor to generate the new training action.

10. The method of any one of claims 1 to 7, wherein, Updating a current value of a parameter of the actor neural network using the critic neural network includes: for each experience tuple in the mini-batch: processing the training observation in the experience tuple using the actor neural network to generate a next action for the training observation from a current value of a parameter of the actor neural network; and determining a parameter update for the actor neural network based on (i) a gradient of the critic neural network with respect to a next action input pair for the training observation and from a current value of a parameter of the critic neural network and (ii) a gradient of the actor neural network with respect to a parameter of the actor neural network for the training observation and from a current value of a parameter of the actor neural network.

11. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations according to the method of any one of claims 1-10.

12. A computer-readable storage medium comprising instructions stored thereon that are executable by a processing device, and upon such execution cause the processing device to perform operations according to the method of any one of claims 1-10.

13. An actor neural network implemented by one or more computers and used to select an action to be performed by an agent that interacts with an environment by receiving an observation that characterizes a current state of the environment and performing an action selected from a continuous action space, wherein, the actor neural network maps the observation to a next action according to current values of parameters of the actor neural network, wherein the actor neural network is trained according to the method of any one of claims 1-10.

Images