Method, program product, and storage medium for distributed reinforcement learning

Abstract

Methods, systems, and devices, including computer programs encoded on computer storage media, to select an action to be performed by a reinforcement learning agent interacting with an environment. A current observation is received that characterizes a current state of the environment. For each action of a set of multiple actions that can be performed by the agent interacting with the environment, a probability distribution over possible Q returns for the action-current observation pair is determined. For each action, a measure of central tendency of the possible Q returns relative to the probability distribution for the action-current observation pair is determined. Using the measures of central tendency, an action to be performed by the agent in response to the current observation is selected.

Description

[0001] This application is a divisional application of the application patent application with the application number 201880025072.8, the application name "Method, Program Product and Storage Medium for Distributed Reinforcement Learning", the application date 16 April 2018. BACKGROUND

[0002] This specification relates to reinforcement learning.

[0003] In a reinforcement learning system, an agent interacts with an environment by performing actions selected by the reinforcement learning system in response to receiving observations characterizing a current state of the environment.

[0004] Some reinforcement learning systems select an action to be performed by an agent in response to receiving a given observation in accordance with an output of a neural network.

[0005] A neural network is a machine learning model that employs one or more layers of non-linear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with a current value of a respective set of parameters. SUMMARY

[0006] This specification describes reinforcement learning systems that are implemented as computer programs on one or more computers in one or more locations.

[0007] According to a first aspect, there is provided a method of selecting an action to be performed by a reinforcement learning agent interacting with an environment. The method comprises receiving a current observation characterizing a current state of the environment. For each action of a set of actions to be performed by the agent interacting with the environment, the action and the current observation are processed using a distributed Q-network having a plurality of network parameters. The distributed Q-network is a deep neural network configured to process the action and the current observation in accordance with current values of the network parameters to generate a network output defining a probability distribution over possible Q-rewards for the action-current observation pair. Each possible Q-reward is an estimate of a reward that would result from the agent performing the action in response to the current observation. For each action, a measure of central tendency of the possible Q-rewards relative to the probability distribution for the action-current observation pair is determined. Using the measures of central tendency of the actions, the action is selected to be performed by the agent in response to the current observation.

[0008] In some implementations, selecting the action to be performed by the agent comprises selecting the action having the highest measure of central tendency.

[0009] In some implementations, selecting the action to be performed by the agent includes selecting an action having a highest measure of central tendency of the probabilities with probability 1 - e, and randomly selecting an action with probability e from the plurality of actions.

[0010] In some implementations, the network output includes a respective output value for each of a plurality of possible Q returns for the action - current observation pair.

[0011] In some implementations, the measure of central tendency is a mean of the possible Q returns.

[0012] In some implementations, determining the mean of the possible Q returns with respect to the probability distribution includes determining a respective probability of each of the plurality of possible Q returns from the output values. Each of the possible Q returns is weighted by the probability of the possible Q return. The mean is determined by summing the weighted possible Q returns.

[0013] According to a second aspect, there is provided a method of training a distributed Q network. The method includes obtaining an experience tuple comprising (i) a current training observation, (ii) a current action performed by an agent in response to the current training observation, (iii) a current reward received in response to the agent performing the current action, and (iv) a next training observation representing a state into which an environment is transformed as a result of the agent performing the current action. A respective current probability of each of a plurality of possible Q returns is determined. More specifically, the current training observation and the current action are processed using the distributed Q network and in accordance with current values of network parameters of the distributed Q network to generate a current network output, the current network output defining a current probability distribution over the possible Q returns for the current action - current training observation pair. For each action, the action and the next training observation are processed using a target distributed Q network (or in some cases, the distributed Q network) and in accordance with current values of target network parameters of the distributed Q network to generate a next network output for the action - next training observation pair. The next network output defines a next probability distribution over the possible Q returns for the action - next training observation pair. The target distributed Q network has the same neural network architecture as the distributed Q network, but the current values of the target network parameters are different from the current values of the network parameters. For each action, a measure of central tendency of the possible Q returns with respect to the respective next probability distribution for the action - next training observation pair is determined. An argmax action is determined, where the argmax action is the one from the plurality of actions having the highest measure of central tendency of the possible Q returns. A respective projected sample update of each of the possible Q returns is determined using the current reward and the argmax action. A gradient is determined with respect to the network parameters of a loss function that depends on the projected sample updates of the possible Q returns and the current probabilities of the possible Q returns. The current values of the network parameters are updated using the gradient.

[0014] In some implementations, determining the respective projected sample update for each of the possible Q returns using the current reward and the argmax action includes determining, from the current reward, a respective sample update for each of the possible Q returns. The respective projected sample update for each of the possible Q returns is determined from the respective sample update and probabilities in the next probability distribution for the argmax action - next training observation pair.

[0015] In some implementations, the respective projected sample update for each of the possible Q returns is equal to a product of the discount factor and the possible Q return plus the current reward. The respective sample update can be determined subject to a constraint that the respective sample update is not less than a minimum possible Q return of the plurality of possible Q returns. And not greater than a maximum possible Q return of the plurality of possible Q returns.

[0016] In some implementations, determining the respective projected sample update for each of the possible Q returns from the respective sample update and probabilities in the next probability distribution for the argmax action - next training observation pair includes, for each possible Q return: assigning probabilities in the next probability distribution for the argmax action - next training observation pair to at least some of the projected sample updates. The probabilities are distributed by intensity that is based on a distance between the sample update for the possible Q return and a corresponding Q return of the projected sample update for each projected sample update.

[0017] In some implementations, the loss function is a Kullback-Leibler divergence between (i) the respective projected sample updates and (ii) the current probability distribution.

[0018] In some implementations, the possible Q returns are indexed from 0 to N-1. Determining the respective projected sample updates includes, for each particular possible Q return of the plurality of possible Q returns, remapping the sample update for the particular possible Q return to fall within a range of 0 to N-1. A lower bound and an upper bound of the remapped sample update are determined based on a probability of the particular possible Q return in the next probability distribution for the argmax action - next training observation pair. A projected sample update for a possible Q return having an index matching the lower bound is updated based on a distance between the remapped sample update and the lower bound. A projected sample update for a possible Q return having an index matching the upper bound is updated based on a distance between the remapped sample update and the upper bound.

[0019] In some implementations, the loss function is a negative sum of a projection of each possible Q-return onto the possible Q-return and a logarithm of a current probability of the possible Q-return.

[0020] In some implementations, the method further comprises periodically updating the value of the target network parameter to match the value of the network parameter.

[0021] In some implementations, the method further comprises updating the current value of the target network parameter by interpolating between the current value of the target network parameter and the updated value of the network parameter.

[0022] According to a third aspect, there is provided a system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the previously described method.

[0023] According to a fourth aspect, there is provided a computer program product storing instructions that when executed by one or more computers cause the one or more computers to perform the previously described method.

[0024] According to another aspect of the application, there is provided a method performed by one or more data processing apparatus for training a distributed Q network, the method comprising: obtaining an experience tuple, the experience tuple comprising (i) a current training observation, (ii) a current action performed by an agent in response to the current training observation, (iii) a current reward received in response to the agent performing the current action, and (iv) a next training observation, the next training observation characterising a state into which an environment is transformed as a result of the agent performing the current action; processing the current training observation and the current action using the distributed Q network and in dependence on current values of network parameters of the distributed Q network to generate a current network output, the current network output comprising a plurality of values which together define a current probability distribution over possible Q-returns for the current action-current training observation pair; processing the possible next action and the next training observation using a target distributed Q network and in dependence on current values of target network parameters of the target distributed Q network to generate a next network output for the action-next training observation pair, the next network output comprising a plurality of values which together define a next probability distribution over possible Q-returns for the possible next action-next training observation pair, wherein the target distributed Q network has the same neural network architecture as the distributed Q network but the current values of the target network parameters are different to the current values of the network parameters; determining a gradient with respect to the network parameters of a loss function in dependence on: (i) the current probability distribution over possible Q-returns for the current action-current training observation pair, (ii) the next probability distribution over possible Q-returns for the possible next action-next training observation pair, and (iii) the current reward; and updating the current values of the network parameters using the gradient.

[0025] According to yet another aspect of the application, there is provided a system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a distributed Q-network, the operations comprising: obtaining an experience tuple, the experience tuple containing (i) a current training observation, (ii) a current action performed by an agent in response to the current training observation, (iii) a current reward received in response to the agent performing the current action, and (iv) a next training observation, the next training observation characterizing a state into which an environment is transformed as a result of the agent performing the current action; processing the current training observation and the current action using the distributed Q-network and in accordance with current values of network parameters of the distributed Q-network to generate a current network output, the current network output comprising a plurality of numerical values that collectively define a current probability distribution over possible Q-returns for the current action-current training observation pair; processing a possible next action and the next training observation using a target distributed Q-network and in accordance with current values of target network parameters of the target distributed Q-network to generate a next network output for the action-next training observation pair, the next network output comprising a plurality of numerical values that collectively define a next probability distribution over possible Q-returns for the possible next action-next training observation pair, wherein the target distributed Q-network has a same neural network architecture as the distributed Q-network but the current values of the target network parameters are different from the current values of the network parameters; determining, with respect to the network parameters, a gradient of a loss function in dependence on: (i) the current probability distribution over possible Q-returns for the current action-current training observation pair, (ii) the next probability distribution over possible Q-returns for the possible next action-next training observation pair, and (iii) the current reward; and updating the current values of the network parameters using the gradient.

[0026] According to yet another aspect of the present disclosure, there is provided a non-transitory computer storage medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a distributed Q-network, the operations comprising: obtaining an experience tuple, the experience tuple containing (i) a current training observation, (ii) a current action performed by an agent in response to the current training observation, (iii) a current reward received in response to the agent performing the current action, and (iv) a next training observation, the next training observation characterizing a state into which an environment is transformed as a result of the agent performing the current action; processing the current training observation and the current action using the distributed Q-network and in accordance with current values of network parameters of the distributed Q-network to generate a current network output, the current network output comprising a plurality of numerical values that collectively define a current probability distribution over possible Q-rewards for the current action-current training observation pair; processing a possible next action and the next training observation using a target distributed Q-network and in accordance with current values of target network parameters of the target distributed Q-network to generate a next network output for the action-next training observation pair, the next network output comprising a plurality of numerical values that collectively define a next probability distribution over possible Q-rewards for the possible next action-next training observation pair, wherein the target distributed Q-network has a same neural network architecture as the distributed Q-network but the current values of the target network parameters are different from the current values of the network parameters; determining, with respect to the network parameters, a gradient of a loss function in dependence on: (i) the current probability distribution over possible Q-rewards for the current action-current training observation pair, (ii) the next probability distribution over possible Q-rewards for the possible next action-next training observation pair, and (iii) the current reward; and updating the current values of the network parameters using the gradient.

[0027] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0028] The reinforcement learning system as described in this specification uses a distributed Q neural network to select actions to be performed by an agent. For a given action-observation pair, the distributed Q network generates an output that defines a probability distribution over a set of possible Q returns (i.e., a numerical value that estimates a result produced by the agent performing an action in response to an observation). The probability distribution can be defined based on multiple numerical values, at least two of which are used to select an action to be performed by the agent. In contrast, some conventional systems use a conventional Q neural network to select actions, which generates a single expected Q return for a given action-observation pair. By modeling a probability distribution over Q returns for an action-observation pair, rather than a single expected Q return (i.e., as in some conventional systems), the system as described in this specification can learn a richer (and thus more efficient) model of action selection. For example, the system as described in this specification can select actions to be performed by an agent interacting with an environment more efficiently than some conventional systems (e.g., allowing the agent to perform a particular task more efficiently).

[0029] Furthermore, the system as described in this specification can be trained based on a loss function that depends on the full Q return probability distribution, rather than a single expected Q return (i.e., as in some conventional systems), thereby resulting in more stable parameter value updates. Parameter value updates are said to be stable if they consistently cause the system to select actions that enable the agent to perform a task efficiently. By enabling more stable parameter value updates, the system as described in this specification can consume less computational resources (e.g., computational power and memory) during training than some conventional systems. For example, the system as described in this specification can require fewer iterations of a training process and less training data to achieve an acceptable level of performance than some conventional systems.

[0030] In some cases, systems as described in this specification can be used to select actions to be performed by a robotic agent that interacts with a real-world environment. In these cases, systems as described in this specification can enable a robotic agent to achieve acceptable performance more quickly, to perform actions that more effectively accomplish a task, and to more easily adapt to previously unknown environments than if actions to be performed by the robotic agent were selected by a conventional system. For example, the agent can be a robotic agent that performs a task such as moving objects between locations (e.g., in a shipping warehouse), assembling components (e.g., electronic components in a manufacturing environment), navigating between locations (e.g., as an automated or semi-automated vehicle). By selecting actions to be performed by the agent based on a full Q-return distribution rather than a single expected Q-return, systems as described in this specification can enable a robotic agent to perform a task more quickly and more safely, even under previously unknown environmental conditions (e.g., if the agent is an automated vehicle operating in heavy traffic or during a snowstorm).

[0031] The details of one or more implementations of the subject matter in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and potential advantages of the subject matter will become apparent from the description, the drawings, and the claims. The subject matter can be implemented as a computer-implemented method, a computer system configured to perform the method, or a computer program product (e.g., one or more computer storage media) storing program instructions to cause a computer system to perform one of the methods. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a block diagram of an example reinforcement learning system.

[0033] Figure 2 is a flow diagram of an example process of selecting actions to be performed by an agent using a reinforcement learning system.

[0034] Figure 3 is a flow diagram of an example process of training a reinforcement learning system.

[0035] Figure 4 is a flow diagram of an example process of determining a projected sample update of Q-returns during training of a reinforcement learning system.

[0036] The same numbers in different drawings have the same meanings. DETAILED DESCRIPTION

[0037] Figure 1is a block diagram of an example reinforcement learning system 100. The reinforcement learning system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, wherein the systems, components, and techniques described below are implemented.

[0038] The reinforcement learning system 100 selects actions 102 to be performed by an agent 104 that interacts with an environment 106 at each of a plurality of time steps. At each time step, the system 100 receives data characterizing a current state of the environment 106, such as an image of the environment 106, and selects an action 102 to be performed by the agent 104 in response to the received data. The data characterizing the state of the environment 106 will be referred to in this specification as an observation 108.

[0039] The systems 100 described herein are widely applicable and are not limited to one particular implementation. However, for illustrative purposes, a few example implementations are described below.

[0040] In some implementations, the environment 106 is a real-world environment, and the agent 104 is a mechanical agent that interacts with the real-world environment. For example, the agent 104 can be a robot that interacts with the environment 106 to accomplish a particular task. As another example, the agent 104 can be an automated or semi-automated vehicle that navigates through the environment 106. In these implementations, the actions 102 can control inputs to control the robot or the automated vehicle. In some of these implementations, the observations 108 can be generated by or derived from sensors of the agent 104. For example, the observations 108 can be captured by a camera of the agent 104. As another example, the observations 108 can be derived from data captured by a laser sensor of the agent 104. As another example, the observations 108 can be hyperspectral images captured by a hyperspectral sensor of the agent 104.

[0041] In some implementations, the environment 106 can be a simulated environment, and the agent 104 can be implemented as one or more computer programs that interact with the simulated environment. For example, the simulated environment 106 can be a video game, and the agent 104 can be a simulated user that plays the video game. As another example, the simulated environment 106 can be a sports simulation environment, such as a driving simulation or a flight simulation, and the agent 104 is a simulated vehicle that navigates through the sports simulation environment. In these implementations, the actions 102 can control inputs to control the simulated user or the simulated vehicle.

[0042] At each time step, the state of the environment 106 at that time step (as characterized by the observation 108) depends on the state of the environment 106 at the previous time step and the action 102 performed by the agent 104 at the previous time step.

[0043] At each time step, the system 100 can receive a reward 110 based on the current state of the environment 106 at that time step and the action 102 performed by the agent 104. In general, the reward 110 is a numerical value. The reward 110 can be based on any event or aspect of the environment 106. For example, the reward 110 can indicate whether the agent 104 has completed a task (e.g., navigated to a target location in the environment 106), or indicate the agent’s 104 progress toward completing a task.

[0044] The system 100 includes a distributed Q neural network 112 configured to process an input (referred to in this specification as an action-observation pair) that includes (i) the action 102 from the set of actions that can be performed by the agent 104, and (ii) the observation 108. The distributed Q network 112 is configured to process the action-observation pair according to current values of distributed Q network parameters to generate an output that defines a probability distribution 114 over a set of possible Q returns for the action-observation pair. For a given action-observation pair, a Q return (which can also be referred to as a Q value) is a numerical value that is an estimate of a return produced by the agent 104 that performs the action in response to the observation. A return refers to a cumulative measure of rewards received by the system 100 as the agent 104 interacts with the environment 106 over multiple time steps. For example, a return can refer to a long-term time-discounted reward received by the system 100.

[0045] The set of possible Q returns for an action-observation pair is a set of numerical values. The set of possible Q returns can be a continuous set or a discrete set. For example, the possible Q returns for an action-observation pair can be a continuous range: [-2, 2], or a discrete set: {-2, -1, 0, 1, 2}.

[0046] The output of the distributed Q-network 112 is an ordered collection of numerical values (e.g., vectors or matrices). In some cases, particularly when the set of possible Q-rewards is a discrete set, the output of the distributed Q-network 112 includes a respective output for each Q-reward in the set of possible Q-rewards. In these cases, if the respective output values for each Q-reward are processed, e.g., by a soft-max function, then the output of the distributed Q-network can define a probability distribution 114 over the set of possible Q-rewards, thereby determining a respective probability value for each Q-reward. In some cases, the output of the distributed Q-network 112 includes respective output values that define a parametric probability distribution over the set of possible 0-rewards. For example, the output of the distributed Q-network 112 can include respective output values that define a mean and a standard deviation of a normal distribution over the set of possible Q-rewards.

[0047] The distributed Q-network 112 can be implemented as any suitable deep neural network model. For example, the distributed Q-network 112 can be implemented as a feedforward neural network (e.g., a multi-layer perceptron or a convolutional neural network) or a recurrent neural network (e.g., a long short-term memory network).

[0048] The system 100 uses the distributed Q-network 112 to select actions 102 to be performed by the agent 104 in response to the observations 108 at each time step. In particular, at each time step, the system 100 provides an action-current observation pair as input to the distributed Q-network 112 for each action from a set of actions that can be performed by the agent. The distributed Q-network 112 processes each action-current observation pair to generate an output that defines a respective probability distribution 114 over a set of possible Q-rewards for each action-current observation pair.

[0049] For example, the output of the distributed Q-network 112 can define: (i) a first probability distribution 116 over a set of possible Q-rewards for a first action-current observation pair, (ii) a second probability distribution 118 over a set of possible Q-rewards for a second action-current observation pair, and (iii) a third probability distribution 120 over a set of possible Q-rewards for a third action-current observation pair.

[0050] For each action, the system 100 determines a measure 122 of central tendency (i.e., a central or typical value) of the corresponding set of possible Q-rewards relative to the probability distribution defined by the output of the distributed Q-network 112 for the action-current observation pair. For example, as will be described further below, the measure of central tendency can be a mean, a median, or a mode. Figure 2 As described, the measure of central tendency can be a mean, a median, or a mode.

[0051] The system 100 selects the action 102 to be performed by the agent 104 at a time step based on the centralized tendency measure 122 corresponding to the action. In some implementations, the system 100 selects the action having the highest corresponding centralized tendency measure 122 from among all actions in a set of actions that can be performed by the agent 104. In some implementations, the system 100 selects the action according to an exploration policy. For example, the system 100 can use an -greedy exploration policy. In this example, the system 100 can select the action having the highest corresponding centralized tendency measure with a probability of 1 - e, and randomly select an action with a probability of e, where e is a number between 0 and 1.

[0052] The system 100 includes a training engine 124 configured to train the distributed Q-network 112 over multiple training iterations using reinforcement learning techniques. The training engine 124 trains the distributed Q-network 112 by iteratively (i.e., with each training iteration) adjusting current values of the distributed Q-network parameters. By training the distributed Q-network 112, the training engine 124 can, for example, cause the distributed Q-network 112 to generate outputs that result in the selection of actions 102 to be performed by the agent 104 that improve a cumulative measure of rewards received by the system 100. Through the cumulative measure of rewards received by the system 100 (e.g., long-term time-discounted rewards), the training engine 124 can (by extension) cause the agent 104 to more effectively perform a given task.

[0053] The training engine 124 trains the distributed Q-network 112 based on training data comprising a set of multiple experience tuples 126. Each experience tuple includes data indicative of: (i) a training observation, (ii) an action performed by the agent in response to the training observation, (iii) a reward received in response to the agent performing the action, and (iv) a next training observation characterizing a state into which the environment transitions as a result of the agent performing the action. The set of experience tuples 126 can include online experience tuples, offline experience tuples, or both. Online experience tuples refer to experience tuples in which the action included in the experience tuple is selected based on outputs generated by the distributed Q-network 112 according to current values of the distributed Q-network parameters. Offline experience tuples refer to experience tuples in which the action included in the experience tuple is selected based on any suitable action selection policy (e.g., a random action selection policy).

[0054] As will be further described with reference to FIG. 2, the system 100 can train the distributed Q-network 112 using a set of experience tuples 126 that includes both online experience tuples and offline experience tuples. Figure 3As described, at each training iteration, the training engine 124 can obtain and process experience tuples, while determining, for each possible Q return, a numerical value of a projected sample update referred to in this specification as a Q return. The training engine 124 can determine, with respect to the distributed Q network parameters, a gradient of a loss function (e.g., which depends on the projected sample updates), and use the gradient to update current values of the distributed Q network parameters. The loss function may, for example, encourage the distributed Q network 112 to generate outputs that define probability distributions (i.e., in response to processing action-training observation pairs), where each Q return’s probability value is similar to the Q return’s projected sample update. In other words, the loss function can encourage the distributed Q network 112 to generate outputs that define probability distributions over possible Q returns, where at least one numerical measure of similarity (e.g., a Kullback-Leibler divergence measure) between the probability distributions and the projected sample updates is improved. Thus, in some cases, the set of projected sample updates can be interpreted as target values for probability distributions defined by the distributed Q network 112’s outputs.

[0055] In some implementations, the system 100 includes a target distributed Q network 128. In general, the target distributed Q network 128 has the same neural network architecture (e.g., number of layers, neural topologies, etc.) as the distributed Q network 112. Moreover, the target distributed Q network 128 (similar to the distributed Q network 112) is also configured to process action-observation pairs to generate outputs that define probability distributions over a set of possible Q returns. However, values of the target distributed Q network parameters can be different from values of the corresponding distributed Q network parameters. As further described below, the training engine 124 can use the target distributed Q network 128 to determine projected sample updates of Q returns during training. Figure 3 As described, the training engine 124 can use the target distributed Q network 128 to determine projected sample updates of Q returns during training.

[0056] The system 100 can periodically (e.g., after a given number of training iterations) update values of the target distributed Q network parameters based on current values of the distributed Q network parameters. For example, the system 100 can update values of the target distributed Q network parameters to match current values of the distributed Q network parameters. As another example, the system 100 can update values of the target distributed Q network parameters by interpolating between current values of the target distributed Q network parameters and current values of the distributed Q network parameters. For example, the system 100 can set a value of each target distributed Q network parameter to an average of the target distributed Q network parameter’s current value and the corresponding distributed Q network parameter’s value.

[0057] Figure 2A flowchart of an example process of selecting an action to be performed by an agent using a reinforcement learning system. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., reinforcement learning system 100, appropriately programmed in accordance with this specification, can perform process 200. Figure 1

[0058] The system receives a current observation that characterizes a current state of the environment (202). The current observation can be generated by or derived from a sensor of the agent. For example, the current observation can be captured by a camera of the agent. As another example, the current observation can be derived from data captured by a laser sensor of the agent. As another example, the current observation can be a hyperspectral image captured by a hyperspectral sensor of the agent.

[0059] For each action from the set of actions performed by the agent, the system determines a corresponding probability distribution over the set of possible Q-rewards (204). More specifically, for each action, the system provides the action-current observation pair as input to the distributed Q neural network. The distributed Q network processes each action-current observation pair in accordance with current values of the distributed Q network parameters to generate a respective output. For each action-current observation pair processed by the distributed Q network, the output generated by the distributed Q network defines a respective probability distribution over the set of possible Q-rewards.

[0060] The set of possible Q-rewards for an action-observation pair is a set of numerical values. The set of possible Q-rewards can be a continuous set or a discrete set. For example, the possible Q-rewards for an action-observation pair can be the continuous range: [-2, 2], or the discrete set: {-2, -1, 0, 1, 2}.

[0061] For a given action-observation pair, the output of the distributed Q network is an ordered collection of numerical values (e.g., a vector or a matrix). In some cases, particularly when the set of possible Q-rewards is a discrete set, the output of the distributed Q network includes a respective output for each Q-reward in the set of possible Q-rewards. In these cases, if the respective output values for each Q-reward are processed, e.g., by a soft-max function, then the output of the distributed Q network can define a probability distribution over the set of possible Q-rewards, determining a respective probability value for each Q-reward. In some cases, the output of the distributed Q network includes respective output values that define a parametric probability distribution over the set of possible Q-rewards. For example, the output of the distributed Q network can include respective output values that define a mean and a standard deviation of a normal distribution over the set of possible Q-rewards.

[0062] ​The distributed Q-network can be implemented as any suitable deep neural network model. For example, the distributed Q-network can be implemented as a feedforward neural network (e.g., a multilayer perceptron or a convolutional neural network) or a recurrent neural network (e.g., a long short-term memory network).

[0063] For each action, the system determines a measure of central tendency (i.e., a central or typical value) of the set of possible Q-rewards with respect to the probability distribution defined by the output of the distributed Q-network for the action-current observation pair (206). The measure of central tendency can be, for example, a mean, a median, or a mode. In a particular example, the measure of central tendency is a mean and the set of possible Q-rewards is a discrete set. In this example, for each action, the system can weight each Q- reward in the set of possible Q-rewards by the probability of the Q-reward according to the probability distribution defined by the output of the distributed Q-network for the action-current observation pair. The system can then determine the mean by summing the weighted Q-rewards. In other words, the system can determine the measure of central tendency m for an action from:

[0064]

[0065] where i indexes the possible Q-rewards {q i}, and p i is the probability value for Q-reward q i .

[0066] The system selects the action to be performed by the agent based on the measure of central tendency corresponding to the action (208). In some implementations, the system selects the action with the highest corresponding measure of central tendency from among all of the actions in the set of actions that can be performed by the agent 104. In some implementations, the system selects the action according to an exploration policy. For example, the system can use an ε-greedy exploration policy. In this example, the system can select the action with the highest corresponding measure of central tendency with a probability of 1 - ε, and select an action with a probability of ε, where ε is a number between 0 and 1.

[0067] Figure 3 is a flowchart of an example process of training a reinforcement learning system. For convenience, the process 300 will be described as performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100, appropriately programmed in accordance with this specification, can perform the process 300. Figure 1

[0068] ​The system obtains an experience tuple (302). The experience tuple includes data indicative of: (i) a current training observation, (ii) a current action performed by the agent in response to the training observation, (iii) a current reward received in response to the agent performing the action, and (iv) a next training observation that characterizes a state into which the environment transitions as a result of the agent performing the action. The experience tuple can be an online experience tuple or an offline experience tuple. An online experience tuple refers to an experience tuple in which the action included in the experience tuple is selected based on output generated by the distributed Q-network according to current values of the distributed Q-network parameters (e.g., with reference to Figure 2 An offline experience tuple refers to an experience tuple in which the action included in the experience tuple is selected based on any suitable action selection policy (e.g., a random action selection policy).

[0069] The system determines a respective current probability of each Q-return in a set of possible Q-returns (304). More specifically, the system provides the current action-current training observation pair (i.e., from the experience tuple obtained in 302) as input to the distributed Q-network. The distributed Q-network processes the current action-current training observation pair according to current values of the distributed Q-network parameters to generate output. The output generated by the distributed Q-network defines a probability distribution over the set of possible Q-returns, and in particular, a respective current probability of each Q-return. For example, if the output of the distributed Q-network includes a respective output value for each Q-return, the system can determine the current probability of each Q-return by processing the respective output values using a soft-max function.

[0070] The system determines a respective next probability distribution over the set of possible Q-returns for each action that can be performed by the agent (306). More specifically, for each action, the system provides the action-next training observation pair as input to the distributed Q-network. The distributed Q-network processes each action-next training observation pair according to current values of the distributed Q-network parameters to generate respective output. For each action-next training observation pair processed by the distributed Q-network, the output generated by the distributed Q-network defines a next probability distribution over the set of possible Q-returns.

[0071] In some implementations, to determine the corresponding next probability distribution, the system processes each action-next training observation pair using a target distributed Q network, rather than using the distributed Q network to process each action-next training observation pair. In general, the target distributed Q network has the same neural network architecture (e.g., number of layers, neural topologies, etc.) as the distributed Q network. Moreover, the target distributed Q network is also configured (similar to the distributed Q network) to process action-observation pairs to generate output defining a probability distribution over a set of possible Q returns. However, the values of the target distributed Q network parameters can be different from the values of the corresponding distributed Q network parameters.

[0072] For each action, the system determines a measure of central tendency of the corresponding set of possible Q returns with respect to the probability distribution defined by the output of the distributed Q network (or the target distributed Q network) for the action-next training observation pair (308). The measure of central tendency can be, for example, a mean, a median, or a mode. Examples of determining a measure of central tendency of a set of possible Q returns with respect to a probability distribution are described with reference to 206.

[0073] The system determines an action corresponding to the measure of central tendency (e.g., determined in 308) for the action, which is referred to as the argmax action (310). For example, the system determines the argmax action from among all actions in the set of actions that can be performed by the agent 104 as the action having the highest corresponding measure of central tendency.

[0074] For each Q return in the set of possible Q returns, the system determines a respective value of what is referred to in this specification as a projected sample update of the Q return (312). In general, the system determines the projected sample update of the Q return based on the current reward (i.e., from the experience tuple obtained in 302) and the argmax action (i.e., determined in 310). Examples of determining a projected sample update of a Q return are described with reference to 206. Figure 4 An example process of determining a projected sample update of a Q return is described.

[0075] The system determines a gradient of a loss function with respect to the distributed Q network parameters (314). In general, the loss function depends on the projected sample update of the Q return (i.e., determined in 312) and the current probability of the Q return (i.e., determined in 304). In some cases, the loss function encourages the distributed Q network to generate output defining a probability distribution (i.e., in response to processing the current action-current training observation pair) in which the probability value for each Q return is similar to the projected sample update of the Q return. For example, the loss function can be a divergence (e.g., a Kullback-Leibler divergence) between (i) the projected sample update of the Q return and (ii) the current probability of the Q return. In other words, the loss function may be given by

[0076]

[0077] where M denotes the projected sample update of the Q return, and P denotes the current probability of the Q return. As another example, when the set of possible Q returns is a discrete set, the loss function can be the negative of the sum over each Q return of the logarithm of the projected sample update of the Q return and the current probability of the Q return. In other words, the loss function may be given by:

[0078]

[0079] where i indexes the set of possible Q returns, and for each i, m i is the projected sample update of the i-th Q return, and p i is the current probability value of the i-th Q return.

[0080] In general, the system can determine the gradient of the loss function with respect to the distributed Q network parameters by any suitable method. For example, the system can determine the gradient by a backpropagation process.

[0081] The system updates the current values of the distributed Q network parameters using the gradient of the loss function (316). For a given parameter θ of the distributed Q network, the system can update the current value of the parameter by:

[0082]

[0083] where r is a positive learning rate hyperparameter (i.e., a system hyperparameter that governs how quickly parameter values can change during training) and is the gradient of the loss function with respect to the parameter θ (e.g., as determined in 314).

[0084] Figure 4 is a flowchart of an example process of determining a projected sample update of a Q return during training of a reinforcement learning system. For convenience, the process 400 will be described as performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, such as the reinforcement learning system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400. Figure 1

[0085] The subsequent description of the process 400 is described with reference to a particular Q return. To determine a projected sample update, the system can perform the steps of the process 400 for each Q return in a set of possible Q returns. The description of the process 400 assumes that the set of possible Q returns is a discrete set.

[0086] ​The system determines a sample update to the Q-return (402). The sample update to the Q-return is a numerical value. The system can determine the sample update to the Q-return based on a current reward (i.e., the current reward of the experience tuple as described in 302), a discount factor (i.e., a numerical value between 0 and 1), and the Q-return itself. For example, the system can determine the sample update to the Q-return q is:

[0087]

[0088] where r is the current reward and γ is the discount factor. In some cases, the system can determine the sample update to the Q-return subject to the constraint that (i) the sample update is greater than a minimum Q-return in a set of possible Q-returns and (ii) the sample is less than a maximum Q-return in the set of possible Q-returns. For example, the system can clip the sample update to be in the range [a, b], where a is the minimum Q-return in the set of possible Q-returns and b is the maximum Q-return in the set of possible Q-returns.

[0089] The system re-maps (i.e., transforms) the sample update to fall in the range of 0 to N-1, where the Q-returns are indexed from 0 to N-1, and N is a positive integer (404). For example, the system can determine the re-mapped sample update b from the relationship:

[0090]

[0091] where the sample update, V MIN is the minimum Q-return in the set of possible Q-returns and Δq is an increment to separate the Q-returns (i.e., for each i ∈ {1,..., N-1}, q i - q i-1 = Δq).

[0092] The system determines a lower bound and an upper bound for the re-mapped sample update (406). The lower bound for the re-mapped sample update is defined as less than the maximum integer value of the re-mapped sample update. The upper bound for the re-mapped sample update is defined as greater than the minimum integer value of the re-mapped sample update.

[0093] The system updates the projected sample update (408). As previously described, to determine the projected sample update, the system can perform the steps of process 400 for each Q-return in the set of possible Q-returns. The value of the projected sample update can be initialized (i.e., before the system performs the steps of process 400 for any Q-return in the set of possible Q-returns), for example, to 0. Each time the system performs the steps of process 400, particularly step 408, the value of the projected sample update for one or more Q-returns is updated.

[0094] The system can update the projected sample updates by assigning the probability of the Q-return in the next probability distribution for the argmax action-next training observation pair (i.e., as determined in 306) to the one or more projected sample updates. In other words, the system can increase the values of the one or more projected sample updates based on the probability of the Q-return in the next probability distribution for the argmax action-next training observation pair. In some cases, the system can assign the probability of the Q-return in the next probability distribution for the argmax action-next training observation pair with an intensity that is based on a distance between the sample update for the Q-return (i.e., as determined in 402) and the corresponding Q-return of the projected sample update for each projected sample update. Assigning the probability of the Q-return in the next probability distribution for the argmax action-next training observation pair refers to increasing the values of the one or more projected sample updates based on the probability of the Q-return.

[0095] For example, the system can update the projected sample updates for the Q-returns that have indices (i.e., in the range of {0,..., N-1}) that match the lower bound of the remapped sample updates (i.e., as determined in 406). In this example, the system can update the projected sample updates for the Q-returns that have indices that match the lower bound of the remapped sample updates based on (i) the probability of the Q-return in the next probability distribution for the argmax action-next training observation pair (i.e., as determined in 306) and (ii) a distance between the remapped sample update and the upper bound of the remapped sample update. For example, in this example, the system can update the projected sample updates for the Q-returns that have indices that match the lower bound of the remapped sample updates based on the following relationship:

[0096] m l ←m l +p · (u - b) (7)

[0097] where m l is the projected sample update for the Q-return that has an index that matches the lower bound of the remapped sample update, p is the probability of the Q-return in the next probability distribution for the argmax action-next training observation pair, u is the upper bound of the remapped sample update, and b is the remapped sample update.

[0098] As another example, the system can update the projected sample update of the Q- return that has an index matching the upper bound of the remapped sample update (i.e., as determined at 406) (i.e., in the range of {0,..., N-1}). In this example, the system can update the projected sample update of the Q-return that has an index matching the upper bound of the remapped sample update based on: (i) the probability of the Q-return in the next probability distribution for the argmax action-next training observation pair (i.e., as determined at 306), and (ii) the distance between the remapped sample update and the lower bound of the remapped sample update. For example, in this example, the system can update the projected sample update of the Q-return that has an index matching the upper bound of the remapped sample update based on the following relationship:

[0099] m u ←m u +p · (b - l) (8)

[0100] The specification uses the term“configured to” in connection with systems and computer program components. For a system of one or more computers to be“configured to” perform particular operations or actions, there must be installed on the system the software, firmware, hardware, or combination thereof that in operation causes or cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions, we mean that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0101] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be machine-readable storage, machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0102] The term“data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0103] A computer program, which can also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a data communication network.

[0104] In this specification, the term“engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components that are installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0105] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0106] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and digital and / or quantum data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from and / or transfer data to, or in

[0107] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0108] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device of the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device of the user, e.g., a smart phone that is running a messaging application, and receiving responsive messages from the user as a result.

[0109] Data processing devices for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing the common and compute-intensive portions of machine learning training or production (i.e., inference) workloads.

[0110] The machine learning model can be implemented and deployed using a machine learning framework, such as a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.

[0111] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0112] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., data files, such as HTML pages, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at a server from the device.

[0113] While this specification contains many specific implementation details, these should not be construed as limitations on the scope or range of any invention, but rather as descriptions of features that can be specific to certain embodiments. Certain features that are described in the context of separate embodiments can also be implemented in combination or in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a sub-combination or variation of a sub-combination.

[0114] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0115] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential order illustrated, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous.

Claims

1. A method for training a distributed Q-network performed by one or more data processing apparatus, the method comprising: obtaining an experience tuple, the experience tuple including (i) a current training observation, (ii) a current action performed by an agent in response to the current training observation, (iii) a current reward received in response to the agent performing the current action, and (iv) a next training observation, the next training observation characterizing a state into which an environment transitions as a result of the agent performing the current action; processing the current training observation and the current action using the distributed Q-network and in accordance with current values of network parameters to generate a current network output, the current network output including a plurality of numerical values that collectively define a current probability distribution over possible Q-returns for the current action-current training observation pair; processing possible next actions and the next training observation using a target distributed Q-network and in accordance with current values of target network parameters of the distributed Q-network to generate a next network output for action-next training observation pairs, the next network output including a plurality of numerical values that collectively define a next probability distribution over possible Q-returns for possible next action-next training observation pairs, wherein the target distributed Q-network has a same neural network architecture as the distributed Q-network but the current values of the target network parameters are different from the current values of the network parameters; determining, with respect to the network parameters, a gradient of a loss function in dependence on: (i) the current probability distribution over possible Q-returns for the current action-current training observation pair, (ii) the next probability distribution over possible Q-returns for possible next action-next training observation pairs, and (iii) the current reward, including: determining, using the current reward and the possible next actions, a respective projected sample update for each of the possible Q-returns; and wherein the loss function depends on the projected sample updates for the possible Q-returns; and updating the current values of the network parameters using the gradient.

2. The method of claim 1, wherein determining, using the current reward and the possible next actions, a respective projected sample update for each of the possible Q-returns includes: determining, from the current reward, a respective sample update for each of the possible Q-returns; and determining, from the respective sample update and the next probability distribution for possible next action-next training observation pairs, the respective projected sample update for each of the possible Q-returns. ​ ​ 3. The method of claim 2, wherein the respective sample update for each of the possible Q returns is equal to the current reward plus a product of a discount factor and a possible Q return subject to the constraint that the respective sample update is not less than a minimum possible Q return of a plurality of possible Q returns and not greater than a maximum possible Q return of the plurality of possible Q returns.

4. The method of claim 2, wherein determining the respective projected sample update for each of the possible Q returns from the respective sample update and a next probability distribution for possible next action-next training observation pairs comprises, for each of the possible Q returns: assigning probabilities of possible Q returns in the next probability distribution for possible next action-next training observation pairs to at least some of the projected sample updates with intensities that are based on a distance between the sample update for the possible Q return and a corresponding possible Q return of the projected sample update for each projected sample update.

5. The method of claim 2, wherein the loss function comprises a Kullback-Leibler divergence between (i) the respective projected sample updates and (ii) the current probability distribution.

6. The method of claim 2, wherein the possible Q returns are indexed from 0 to N-1, and wherein determining the respective projected sample updates comprises, for each of the possible Q returns of a plurality of possible Q returns: remapping the sample update for the possible Q return to fall in a range of 0 to N-1 ; determining a lower bound and an upper bound of the remapped sample update; updating a projected sample update for the possible Q return having an index matching the lower bound based on probabilities of the possible Q returns in the next probability distribution for possible next action-next training observation pairs and based on a distance between the remapped sample update and the lower bound; and updating a projected sample update for the possible Q return having an index matching the upper bound based on probabilities of the possible Q returns in the next probability distribution for possible next action-next training observation pairs and based on a distance between the remapped sample update and the lower bound.

7. The method of claim 6, wherein the loss function comprises a negative sum of logarithms of the projected sample update for the possible Q return and the probability of the possible Q return under the current probability distribution for each of the possible Q returns.

8. The method of claim 1, further comprising: periodically updating values of the target network parameters to match values of the network parameters.

9. The method of claim 1, further comprising: updating a current value of the target network parameters by interpolating between the current value of the target network parameters and the updated values of the network parameters.

10. A system for training a distributed Q network comprising: one or more computers that include one or more processors; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a distributed Q-network, the operations comprising: obtaining an experience tuple containing (i) a current training observation, (ii) a current action performed by an agent in response to the current training observation, (iii) a current reward received in response to the agent performing the current action, and (iv) a next training observation representing a state into which an environment transitions as a result of the agent performing the current action; processing the current training observation and the current action using the distributed Q-network and in accordance with current values of network parameters of the distributed Q-network to generate a current network output, the current network output comprising a plurality of numerical values that collectively define a current probability distribution over possible Q-returns for the current action-current training observation pair; processing possible next actions and the next training observation using a target distributed Q-network and in accordance with current values of target network parameters of the target distributed Q-network to generate a next network output for action-next training observation pairs, the next network output comprising a plurality of numerical values that collectively define a next probability distribution over possible Q-returns for possible next action-next training observation pairs, wherein the target distributed Q-network has a same neural network architecture as the distributed Q-network but the current values of the target network parameters are different from the current values of the network parameters; determining, with respect to the network parameters, a gradient of a loss function in dependence on: (i) the current probability distribution over possible Q-returns for the current action-current training observation pair, (ii) the next probability distribution over possible Q-returns for possible next action-next training observation pairs, and (iii) the current reward, including: determining, using the current reward and the possible next actions, a respective projected sample update for each of the possible Q-returns; and wherein the loss function depends on the projected sample updates for the possible Q-returns; and updating the current values of the network parameters using the gradient.

11. The system of claim 10, wherein determining, using the current reward and the possible next actions, a respective projected sample update for each of the possible Q-returns comprises: determining, from the current reward, a respective sample update for each of the possible Q-returns; and determining, from the respective sample update and the next probability distribution for possible next action-next training observation pairs, the respective projected sample update for each of the possible Q-returns.

12. The system of claim 11, wherein the respective sample update for each of the possible Q returns is equal to the current reward plus a product of a discount factor and a possible Q return subject to the constraint that the respective sample update is not less than a minimum possible Q return of a plurality of possible Q returns and not greater than a maximum possible Q return of the plurality of possible Q returns.

13. The system of claim 11, wherein determining the respective projected sample update for each of the possible Q returns from the respective sample update and a next probability distribution over possible next action-next training observation pairs comprises, for each of the possible Q returns: assigning probabilities of possible Q returns in the next probability distribution over possible next action-next training observation pairs to at least some of the projected sample updates with intensities that are based on a distance between the sample update for the possible Q return and a corresponding possible Q return of the projected sample update for each projected sample update.

14. A non-transitory computer storage medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a distributed Q network, the operations comprising: obtaining an experience tuple containing (i) a current training observation, (ii) a current action performed by an agent in response to the current training observation, (iii) a current reward received in response to the agent performing the current action, and (iv) a next training observation representing a state into which an environment transitions as a result of the agent performing the current action; processing the current training observation and the current action using the distributed Q network and in accordance with current values of network parameters to generate a current network output, the current network output including a plurality of values that collectively define a current probability distribution over possible Q returns for a current action-current training observation pair; processing a possible next action and the next training observation using a target distributed Q network and in accordance with current values of target network parameters of the distributed Q network to generate a next network output for an action-next training observation pair, the next network output including a plurality of values that collectively define a next probability distribution over possible Q returns for a possible next action-next training observation pair, wherein the target distributed Q network has a same neural network architecture as the distributed Q network but the current values of the target network parameters are different from the current values of the network parameters; determining, with respect to the network parameters, a gradient in dependence on: (i) the current probability distribution over possible Q returns for the current action-current training observation pair, (ii) the next probability distribution over possible Q returns for the possible next action-next training observation pair, and (iii) the current reward, including: determining, using the current reward and the possible next action, a respective projected sample update for each of the possible Q returns; and wherein the loss function depends on the projected sample updates for the possible Q returns; and updating the current values of the network parameters using the gradient.

15. The non-transitory computer storage medium of claim 14, wherein determining, using the current reward and the possible next action, a respective projected sample update for each of the possible Q returns comprises: determining, from the current reward, a respective sample update for each of the possible Q returns; and determining, from the respective sample update and the next probability distribution for possible next action-next training observation pairs, the respective projected sample update for each of the possible Q returns.

16. The non-transitory computer storage medium of claim 15, wherein the respective sample update for each of the possible Q returns is equal to the current reward plus a product of a discount factor and a possible Q return, subject to the constraint that the respective sample update is not less than a minimum possible Q return of a plurality of possible Q returns and not greater than a maximum possible Q return of the plurality of possible Q returns.

17. The non-transitory computer storage medium of claim 15, wherein determining, from the respective sample update and the next probability distribution for possible next action-next training observation pairs, the respective projected sample update for each of the possible Q returns comprises, for each possible Q return: assigning probabilities of possible Q returns in the next probability distribution for possible next action-next training observation pairs to at least some of the projected sample updates by intensity, the intensity being based on a distance between the sample update for the possible Q return and the corresponding possible Q return of the projected sample update for each projected sample update.

18. The non-transitory computer storage medium of claim 15, wherein, the loss function comprises a Kullback-Leibler divergence between (i) the respective projected sample updates and (ii) the current probability distribution.

19. The non-transitory computer storage medium of claim 15, wherein the possible Q returns are indexed from 0 to N-1, and wherein determining the respective projected sample updates comprises, for each possible Q return of a plurality of possible Q returns: remapping the sample update for the possible Q return to fall within a range of 0 to N-1; determining a lower bound and an upper bound of the remapped sample update; updating the projected sample update for the possible Q return having an index matching the lower bound based on probabilities of the possible Q return in the next probability distribution for possible next action-next training observation pairs and based on a distance between the remapped sample update and the upper bound; and updating the projected sample update of the possible Q-return having an index matching the upper limit based on a probability of the possible Q-return in a next probability distribution for a possible next action-next training observation pair and based on a distance between the remapped sample update and the lower limit.

20. The non-transitory computer storage medium of claim 19, wherein, the loss function comprises a negative sum of a logarithm of the projected sample update of the possible Q-return and the probability of the possible Q-return under a current probability distribution for each possible Q-return.

Images