Training reinforcement learning agents using augmented temporal difference learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network for selecting actions to be performed by an agent interacting with an environment by performing actions that cause states of the environment to transition. One of the methods includes maintaining a replay memory storing a plurality of transitions, selecting a plurality of transitions from the replay memory, and training the neural network on the plurality of transitions, including, for each transition: generating an initial Q-value for the transition, determining a scaled Q-value for the transition, determining a scaled temporal difference learning target for the transition, determining an error between the scaled temporal difference learning target and the scaled Q-value, determining an update to current values of Q-network parameters, and determining an update to current values of scaling terms.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 087,156, filed October 2, 2020. The disclosure of that earlier application is considered part of the disclosure of this application and is incorporated herein by reference. Technical Field

[0003] This manual relates to reinforcement learning. Background Technology

[0004] In a reinforcement learning system, an agent interacts with the environment by performing actions selected by the reinforcement learning system in response to received observations representing the current state of the environment.

[0005] Some reinforcement learning systems select the action to be performed by the agent in response to a given observation, based on the output of the neural network.

[0006] A neural network is a machine learning model that uses one or more non-linear units to predict the output in response to a given input. Some neural networks are deep neural networks, which include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input for the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0007] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which uses a control neural network system to control an agent to perform one or more tasks.

[0008] Generally, an innovative aspect of the subject matter described in this specification can be embodied in a method for training a Q-neural network having multiple Q-network parameters and for selecting actions performed by a reinforcement learning agent interacting with the environment by executing actions that cause a state transition in the environment. The method includes: maintaining a playback memory storing multiple transitions generated as a result of the reinforcement learning agent's interaction with the environment, each transition including a corresponding current observation representing the corresponding current state of the environment, a corresponding current action performed by the agent in response to the current observation, a corresponding next observation representing the corresponding next state of the environment, and a reward received in response to the agent performing the current action; selecting multiple transitions from the playback memory. The method involves training a Q-neural network on multiple transformations, including, for each of the multiple transformations: processing the current observation and current action based on the current values ​​of the Q-network parameters to generate an initial Q-value for the transformation; determining a scaled Q-value for the transformation based on (i) the current value of the scaling term and (ii) the initial Q-value; determining a scaling temporal difference learning objective for the transformation based on (i) the current value of the scaling term and (ii) the next observation included in the transformation; determining the error between the scaling temporal difference learning objective and the scaled Q-value; determining an update to the current value of the Q-network parameters based on the gradient of the error relative to the Q-network parameters; and determining an update to the current value of the scaling term based on the gradient of the error relative to one or more scaling term parameters.

[0009] The learning objective for determining the scaling time difference of the transition based on (i) the current value of the scaling term and (ii) the next observation may include: processing the next observation and each action in the set of possible actions that the agent can perform when interacting with the environment using a Q-neural network to output a corresponding Q-value for the action as an estimate of the reward the agent will receive if it performs the action in response to including the next observation in the transition; selecting the selected action from the set of possible actions based on the corresponding Q-value; determining the next expected reward with adjusted initial time based on the value of the discount factor and the Q-value of the selected action if the selected action is performed in response to the next observation; determining the scaled, adjusted time next expected reward based on (i) the current value of the scaling term and (ii) the adjusted time next expected reward; and calculating the sum of the reward included in (i) the transition and (ii) the scaled, adjusted time next expected reward.

[0010] The learning objective, which determines the scaling time difference of the transition based on (i) the current value of the scaling term and (ii) the next observation, may also include: using a target instance of a Q-neural network to process the next observation and each action in the set of possible actions that the agent can perform when interacting with the environment based on the target parameter values ​​of the neural network, to output a corresponding Q value for the action as an estimate of the reward that the agent will receive if it performs an action in response to including the next observation in the transition.

[0011] Determining the scaling Q value for the transformation based on (i) the current value of the scaling term and (ii) the initial Q value may include: calculating the product of the scaling term and the initial Q value.

[0012] The method may also include initializing the current value of the scaling term to one.

[0013] The scaling term can be of the form exp(ρ), where ρ is a trainable parameter.

[0014] Determining an update to the current value of the Q network parameters based on the gradient of the error relative to the Q network parameters may include: adding a ranking regularization term to the error, wherein the ranking regularization term depends on the maximum of the following: (i) a constant plus the difference between (a) and (b): (a) the initial Q value of the transformation, (b) the Q value generated by processing the current observation and the remaining actions in the set of possible actions different from those included in the transformation according to the current value of the Q network parameters; and (ii) zero; and determining an update to the current value of the Q network parameters based on the gradient of the ranking regularization term and the error relative to the Q network parameters.

[0015] The method may also include adding a ranking regularization term to the error based on a filtering scheme that determines whether the ranking regularization term should be added to the error based on the rewards included in the transformation.

[0016] Determining an update to the current value of a scaling term based on the gradient of the error relative to one or more scaling term parameters may include: adding a scaling term regularization term to the error, wherein the scaling term regularization term depends on the difference between (i) the scaling Q value of the transformation and (ii) the sum of adjustment times for the corresponding rewards included in one or more subsequent transformations for a transformation selected from the playback memory; and determining an update to the current value of the scaling term based on the gradient of the regularization term and the error relative to one or more scaling term parameters.

[0017] Determining an update to the current value of the Q network parameters may also include determining the value of the Q network parameter learning rate; and determining an update to the current value of the scaling term may also include determining the value of the scaling term learning rate which is less than the value of the Q network parameter learning rate.

[0018] The value of the scaling term learning rate can depend on the number of multiple transformations selected from the playback memory.

[0019] Each transition in the playback memory may also include a corresponding next action performed by the agent in response to the next observation; and training the Q-neural network on multiple transitions may also include, for each of the multiple transitions: processing the current observation and the current action based on the current values ​​of the Q-network parameters to generate an initial Q-value for the transition; determining a temporal difference learning objective for the transition based on the next observation and the next action included in the transition; determining the error between the temporal difference learning objective and the initial Q-value; determining an update to the current values ​​of the Q-network parameters based on calculating the gradient of the error relative to the Q-network parameters; and determining an update to the current values ​​of the scaling terms based on calculating the error gradient relative to one or more scaling term parameters.

[0020] None of the multiple transformations can be generated by controlling the agent using a Q-neural network.

[0021] The environment can be a real-world environment, and the observations are generated by sensors arranged to sense the state of the real-world environment.

[0022] Another innovative aspect of the subject matter described in this specification can be embodied in a method comprising: receiving a new observation, the new observation representing a new state of an environment with which an agent is interacting; processing the new observation using a Q-neural network to generate a corresponding Q-value for each action in a set of possible actions that the agent can perform in response to the new observation, the Q-value for each action being an estimate of the reward that the agent would receive if it performed the action in response to the new observation, wherein the Q-neural network has been trained using the method of any of the preceding claims; selecting an action from the set of possible actions based on the corresponding Q-value; and causing the agent to perform the selected action. The environment may be a real-world environment, the observation may be generated by sensors arranged to sense the state of the real-world environment, and the agent may be a robot navigating through the environment or an autonomous or semi-autonomous land, air, or sea vehicle. The Q-neural network may be trained using a simulated environment.

[0023] Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. A system of one or more computers may be configured to perform specific operations or actions by software, firmware, hardware, or any combination thereof installed on the system, which, in operation, may cause the system to perform the actions. One or more computer programs may be configured to perform specific operations or actions by including instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0024] The subject matter described in this specification can be implemented in particular embodiments to achieve one or more of the following advantages.

[0025] Offline RL training (i.e., RL training based on a pre-existing training data corpus, without generating additional training data to supplement the training data used for training) is an efficient algorithm for training neural networks that select actions to be performed by an agent, as the network can be trained without controlling the agent's interaction with the real-world environment. During training, this avoids executing risky actions due to suboptimal policies and prevents mechanical wear, tear, or other damage to the real-world agent. However, traditional offline RL algorithms inevitably introduce extrapolation errors because the training process involves estimating state-action pairs that are not well covered by the training data. When mixed with bootstrapping, this can lead to unbounded learning objectives and thus hinder successful training.

[0026] However, the disclosed technique mitigates these problems by allowing the use of training data from the playback memory in a manner that increments the values ​​of selected data during offline RL training. Specifically, by using a scaling term to limit the temporal difference learning objective value to a reasonable range through additional training, the impact of extrapolation errors caused by overestimation of the learning objective in the training data on neural network training can be mitigated, thereby allowing the learning of high-quality action selection policies in an extremely stable and data-efficient manner. Furthermore, by employing a ranking-based regularization scheme, the disclosed technique encourages a more thorough update of neural network parameter values ​​when learning from already collected training data (especially training data already generated by a well-performing agent), such as the remaining actions in the set of possible actions not yet selected by the agent during the interaction, and the corresponding rewards received by the agent in response to performing the unselected actions. This further prevents extrapolation errors from hindering successful training.

[0027] Therefore, the disclosed techniques can improve the training speed of neural networks used in selecting actions to be performed by an agent and reduce the amount of training data required to effectively train those neural networks. Consequently, the amount of computational resources required to train neural networks can be reduced. For example, the amount of memory required to store training data can be reduced, the amount of processing resources used in the training process can be reduced, or both. This increased speed of neural network training is particularly important for more complex neural networks that are more difficult to train or for training neural networks to select actions to be performed by agents performing complex reinforcement learning tasks.

[0028] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0029] Figure 1 An example reinforcement learning system is shown.

[0030] Figure 2 This is a diagram illustrating an example of extrapolation error that hinders the traditional training of reinforcement learning systems.

[0031] Figure 3 This is a flowchart of an example process for training a Q-neural network.

[0032] Figure 4 This is a flowchart of an example process for determining the scaling time difference (TD) learning objective.

[0033] Figure 5 This is a flowchart of another example process for training a Q-neural network.

[0034] Figure 6 yes Figure 5 A diagram comparing the example training process with the traditional training process.

[0035] The same reference numerals and figure marks in the various figures indicate the same elements. Detailed Implementation

[0036] This specification describes a reinforcement learning system that controls an agent interacting with the environment by processing data (i.e., "observations") characterizing the current state of the environment at each of multiple time steps to select an action to be performed by the agent.

[0037] At each time step, the state of the environment at that time step depends on the state of the environment at the previous time step and the action performed by the agent at the previous time step.

[0038] In some implementations, the environment can be a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment, such as a robot navigating through the environment or an autonomous or semi-autonomous land, air, or sea vehicle.

[0039] In these implementations, observations may include one or more of the following: images, object location data, and sensor data, to capture observations as the agent interacts with the environment, such as sensor data from images, distance or position sensors, or from actuators.

[0040] For example, in the case of a robot, observations may include data characterizing the robot’s current state, such as one or more of the following: joint positions, joint velocities, joint forces, torques or accelerations, such as gravity-compensated torque feedback, and the global or relative pose of an object held by the robot.

[0041] In the case of robots or other mechanical intelligent agents or vehicles, observation can similarly include the position, linear or angular velocity, force, torque or acceleration of one or more parts of the agent, and one or more of its global or relative attitude. Observation can be defined in 1, 2, 3 or more dimensions and can be absolute and / or relative.

[0042] Observations may also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or, image or video data, such as data from a camera or LIDAR sensor, data from a sensor of an agent or data from a sensor located separately from the agent in the environment.

[0043] In these implementations, the action can be a control input used to control: a robot, such as the torque of a robot joint or a higher-level control command; or an autonomous or semi-autonomous land, air, or sea vehicle, such as the torque of a vehicle's control surface or other control element or a higher-level control command.

[0044] In other words, actions can include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a component of another mechanical agent. Action data may additionally or alternatively include electronic control data such as motor control data, or more generally include data for controlling one or more electronic devices within an environment whose control affects the observed environmental state. For example, in the case of autonomous or semi-autonomous land, air, or sea vehicles, actions can include actions controlling navigation, such as steering and movement, such as braking and / or acceleration of the vehicle.

[0045] In some other applications, intelligent agents can control actions in real-world environments, including, for example, equipment items in a data center, an electricity / water distribution system, or a manufacturing plant or service facility. Observations may then relate to the operation of the plant or facility. For example, observations may include observations of electricity or water consumption by equipment, or observations of power generation or distribution control, or observations of resource use or waste generation. These actions may include actions that control equipment items in the plant / facility or impose operating conditions on equipment items in the plant / facility, and / or actions that cause changes to settings in the operation of the plant / facility, such as adjusting or turning on / off components of the plant / facility.

[0046] In the case of an electronic agent, observations may include data from one or more sensors that monitor a part of a plant or service facility, such as current, voltage, power, temperature, and other sensors and / or electronic signals representing electronic and / or mechanical equipment items. For example, the real-world environment might be a manufacturing plant or service facility, and observations might relate to the operation of the plant or facility, such as resource usage (e.g., power consumption), and the agent might control actions or operations within the plant / facility to, for example, reduce resource usage. In some other implementations, the real-world environment could be a renewable energy plant, and observations might relate to the operation of the plant to, for example, maximize current or future planned power generation, and the agent could control actions or operations within the plant to achieve this.

[0047] As another example, the environment can be a chemical synthesis or protein folding environment, such that each state is a corresponding state of a protein chain or one or more intermediate or precursor chemicals, and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, these actions might be folding actions for folding the protein chain or actions for assembling the precursor chemical / intermediate, and the desired outcome might include, for example, folding the protein to stabilize it and enable it to perform a specific biological function or providing an efficient synthetic route for the chemical. As another example, the agent can be a mechanical agent that executes or controls protein folding actions or chemical synthesis steps automatically selected by the system without human interaction. Observations can include direct or indirect observation of the state of the protein or chemical / intermediate / precursor, and / or may be derived from simulation.

[0048] In some implementations, the environment can be a simulated environment and the agent can be implemented as one or more computers that interact with the simulated environment.

[0049] The simulation environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the intelligent agent can be a simulated vehicle navigating through motion simulation. In these implementations, actions can be control inputs used to control the simulated user or the simulated vehicle.

[0050] In some implementations, the simulated environment can be a simulation of a specific real-world environment. For example, the system can be used to select actions in the simulated environment during the training or evaluation of a control neural network, and after training or evaluation, or both, it can be deployed to control a real-world agent in the real-world environment simulated by the simulated environment. This avoids unnecessary wear and tear or other damage to the real-world environment or the real-world agent, and allows control of the neural network to be trained and evaluated in situations that rarely occur or are difficult to recreate in the real-world environment.

[0051] Typically, in a simulated environment, observations may include simulated versions of one or more of the previously described observations or observation types, and actions may include simulated versions of one or more of the previously described actions or action types.

[0052] Alternatively, in any of the above implementations, observations at any given time step may include data from previous time steps that may be helpful in characterizing the environment, such as actions performed at previous time steps, rewards received at previous time steps, etc.

[0053] Figure 1 An example reinforcement learning system 100 is shown. The reinforcement learning system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the following systems, components and techniques are implemented.

[0054] The reinforcement learning system 100 selects actions to be performed by the reinforcement learning agent 102 interacting with the environment 104. That is, the reinforcement learning system 100 receives observations 108, each representing a corresponding state of the environment 104, and in response to each observation, selects an action 106 from a predetermined set of actions that the reinforcement learning agent 102 performs in response to the observations 108. In response to some or all of the actions performed by the agent 102, the reinforcement learning system 100 receives a reward. Each reward is a numerical value received from the environment 104 as a result of the agent performing the action; that is, the reward will vary depending on the state to which the environment 104 transitions as a result of the agent 102 performing the action.

[0055] Specifically, the reinforcement learning system 100 selects actions 106 to be performed by the agent 102 using the Q-neural network 110 and the training engine 120.

[0056] The Q-neural network 110 is a neural network that receives observations 108 representing the state of the environment 104 as input and generates a corresponding Q-value 112 as output for each action in the action set. The Q-value 112 of an action is an estimate of the "reward," which arises from the agent performing the action in response to the current observation 108, and then selecting future actions for the agent 102 to perform based on the current values ​​of the parameters of the Q-neural network. The reward refers to a cumulative measure of the "gift" received by the agent, such as the sum of time-discounted rewards.

[0057] Therefore, the function of the Q-neural network 110 is to generate data indicating the Q-value 112 for each possible action in the action set. In other words, its function is to evaluate the function Q(observation, action) for each possible action. Accordingly, in the following text, we will refer to the process of receiving an observation and determining a Q-value for a certain action as "processing the observation and a certain action according to the current values ​​of the Q-neural network parameters to generate the Q-value of the action". The Q-neural network 110 can perform this operation on all actions in the action set simultaneously. In other words, in this document, expressions such as "processing the observation and a certain action according to the current values ​​of the Q-neural network parameters to generate the Q-value of the action" should be understood as including, in a form, "processing the observation according to the current values ​​of the Q-neural network parameters to generate the Q-values ​​of the entire action set, and processing a certain action to select the Q-value generated corresponding to that particular action".

[0058] Note that in some embodiments, the Q-network 110 can be implemented by a computer process (e.g., a multilayer perceptron, MLP) that receives observations and data for a single action in a set of actions (e.g., a one-hot vector with components for each action) and generates a Q-value only for that single action (i.e., the Q-network with the current values ​​of its Q-network parameters will generate the same Q-value for that action upon receiving the observation). This is also considered below as an example of “processing observations and an action based on the current values ​​of the Q-network parameters to generate the Q-value of the action.” Note that if the computer process receives (i) an observation and (ii) data indicating a corresponding action in a set of possible actions at each of multiple consecutive occurrences, the computer process will consecutively generate Q-values ​​for all possible actions in that set.

[0059] The Q-neural network 110 can be implemented using any suitable neural network architecture that enables it to perform the functions described therein. In one example, the Q-neural network 110 may include an “embedding” subnetwork, a “core” subnetwork, and a “selection” subnetwork. A subnetwork of a neural network refers to a set of one or more neural network layers within the neural network. When the observation is an image, the embedding subnetwork may be a convolutional subnetwork, i.e., it includes one or more convolutional neural network layers configured to process the observation for a time step. When the observation is low-dimensional data, the embedding subnetwork may be a fully connected subnetwork. The core subnetwork may be a recurrent subnetwork, for example, it includes one or more long short-term memory (LSTM) neural network layers configured to process the output of the embedding subnetwork, and in some cases, process action data (or data derived from action data or both) that defines each action from a set of possible actions. The selection subnetwork may be configured to process the output of the core subnetwork to generate a Q-value output for the action.

[0060] To select the action 106 that the agent should perform, the system can process the Q-value (e.g., using a soft-max function) to generate a corresponding probability value for each possible action, and then sample the actions based on their probability values. Alternatively, the system can select the action with the highest Q-value as the action the agent should perform.

[0061] In order to allow the agent 102 to interact effectively with the environment 104, the reinforcement learning system 100 includes a training engine 120 for training a Q-neural network 110 to determine the training values ​​of the parameters 118 of the Q-neural network.

[0062] To assist in the training of the Q-neural network 110, the training engine 120 maintains a playback memory 130. The playback memory 130 stores data items (“transformations”) generated as a result of interactions between agent 102 or another agent and the environment 104 or another instance of the environment used to train the Q-neural network 110.

[0063] In some implementations, each transition is a tuple comprising: (1) a current observation representing the current state of the environment at a given time; (2) a current action performed by the agent in response to the current observation; (3) a next observation representing the next state of the environment after the agent performs the current action, i.e., the state to which the environment is transformed as a result of the agent performing the current action; (4) a reward received by the agent in response to the agent performing the current action; and, optionally, (5) a corresponding next action performed by the agent in response to the next observation.

[0064] The training engine 120 trains the action selection neural network 110 by repeatedly selecting transitions from the playback memory 130 (e.g., randomly) and training the Q neural network 110 on the selected transitions.

[0065] This training setup, which can be called offline reinforcement learning, focuses the learning process on the recorded transformation data without using a Q-neural network to actually control the agent 102 to interact with the environment 104. In other words, the training engine 120 can train the Q-neural network 110 even when multiple transformations in the playback memory 130 are not generated by using a Q-neural network to control the agent.

[0066] Offline reinforcement learning enables the training of Q-neurons using recorded data for controlling agents across a wide range of technological use cases, including real-world applications such as healthcare and autonomous vehicles, where repeated interactions with the environment are costly and potentially unsafe or unethical, and where historical data can be abundant. However, offline reinforcement learning also presents unique challenges for training the system's RL capabilities. A major challenge is handling errors caused by overestimation of the values ​​of state-action pairs that are not well covered by the transition data. These errors are amplified during training due to bootstrapping and can lead to divergence, hindering successful training. Another challenge stems from the staleness of the training data. Simply put, unlike online reinforcement learning, in offline reinforcement learning, the agent cannot explore new actions or states that were not collected during the generation of the training dataset, even if these new actions appear optimal. This tends to manifest as an overestimation of the Q-value for actions not present in the training dataset, which is reinforced by the bootstrapping mechanism of the learning, leading to a policy (parameterized by the Q-neuron) that tends to move away from the state space region covered by the training dataset and into regions where the learned policy (i.e., the trained Q-neuron) is unpredictable.

[0067] Figure 2 This is a diagram illustrating the extrapolation error of a typical training example that hinders reinforcement learning systems. Q is the actual Q-value of the action a performed in response to an observation s representing the state of the environment, and Q... θ It is the estimated (or more accurately, extrapolated) Q-value generated for action a by processing observations representing the state using the Q-network during system training based on the current values ​​of the Q-network parameters θ.

[0068] exist Figure 2 In the example, a1 is observed in the playback memory (i.e., included in at least one transition in playback memory 130 that also includes observations s), while a2 and a3 are not observed (i.e., not included in any transition that also includes observations s). In this example, to evaluate the loss of the Q-neural network on a given task during training, the Q-neural network (or, equivalently, the current values ​​of the Q-network parameters θ) is used to extrapolate beyond the training data, i.e., extrapolated Q-values ​​are generated for the unobserved actions a2 and a3. This extrapolation may lead to overestimation of the values.

[0069] More specifically, overestimation occurs when the Q-neural network outputs an estimated Q-value that is larger than the actual Q-value of the action in the transition. (As previously stated) When used to select the action a that an agent performs in response to observation s, and correspondingly to evaluate the loss of the Q-neural network during training, these estimated Q-values ​​can lead to an overconfident estimate of the true value of the state propagated during learning. For example, in Figure 2In this context, an estimated Q-value might lead to the selection of a3 (instead of a1). While in online RL, such overestimation might lead to optimistic exploration, which is then corrected through additional learning, in offline RL settings, overestimation typically causes the system training to diverge, thus hindering effective training or reducing the performance of the trained Q-network.

[0070] When estimating arbitrarily high values, these extrapolation errors will propagate further through bootstrapping into the values ​​of other states (e.g., states in preceding or succeeding transitions). Furthermore, in low-data conditions, the impact of unobserved actions becomes even greater, where the chance of encountering unobserved actions is higher, and Q-neurons are more prone to extrapolation errors.

[0071] Therefore, to overcome the aforementioned drawbacks and improve overall training efficiency (e.g., in terms of processing cycles), the training engine 120 employs a regularized offline learning scheme, which effectively mitigates the impact of extrapolation errors caused by inaccurate estimations of interaction information lost from playback memory on offline training. Specifically, the regularized offline learning scheme may include (i) reparameterization using the Q-value of the scaling term, (ii) transformation rank regularization, and (iii) behavioral value estimation, each of which can be used individually or in combination, for example, in conjunction with Q-learning or Sarsa-learning. References will follow. Figure 3 and 4 The method of training the Q-neural network 110 using this regularized offline learning scheme is described in more detail.

[0072] Figure 3 This is a flowchart of an example process 300 for training a Q-neural network. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed system, such as... Figure 1 The reinforcement learning system 100 can execute process 300.

[0073] The system maintains a replay memory (302). As described above, the replay memory stores multiple transitions generated as a result of the interaction between the reinforcement learning agent (or another agent) and the environment (or another instance of the environment). In various cases, the agent may be controlled by another control system (including: by human operation or another machine learning-based control system; by a hard-coded policy that selects actions to be performed by the agent according to pre-programmed logic; or simply by a stochastic policy that selects actions with uniform randomness).

[0074] exist Figure 3In the example, each transition can be a tuple comprising: (1) the current observation s, which represents the current state of the environment at a time; (2) the current action a performed by the agent in response to the current observation; (3) the next observation s′, which represents the next state of the environment after the agent performs the current action, i.e. the state to which the environment is transformed due to the agent performing the current action; and (4) the reward r received in response to the agent performing the current action a′.

[0075] The system selects multiple transitions from the playback memory (304). The system may select transitions randomly or according to a priority strategy (e.g., based on the value of the associated temporal difference learning error or some other learning progress metric).

[0076] To train a Q-neural network on a selection of multiple transformations, the system may repeat steps 306-316 for each of the multiple transformations.

[0077] The system processes the current observation s and the current action a based on the current value of the Q-network parameter θ to generate the initial Q-value Q of the transformation. θ (s, a)(306). The Q value is an estimate of the reward an agent would receive if it took an action in response to the current observation.

[0078] In some implementations, the system can use Q-value reparameterization to reduce overestimation. In these implementations, the system determines the scaled Q-value for transformation based on (i) the current value of the scaling term α and (ii) the initial Q-value (308). Typically, the system uses the scaling term α to prevent severe overestimation during training by limiting the initial Q-value predicted by the Q-neural network.

[0079] In various cases, such as depending on the actual configuration of the Q-neuron, the initial Q-values ​​can be bounded or unbounded. When they are unbounded, the system can process the initial Q-values ​​(e.g., by using a tanh activation function) to generate “normalized” Q-values ​​that are bounded within a predetermined range. For example

[0080] Specifically, the system can determine the scaled Q value by calculating the product of the scaling term α and the initial Q value (or the standardized initial Q value), that is, determine the scaled Q value as αQ. θ (s, a) or The scaling Q value determined from the standardized Q value is always defined within the absolute value of the scaling term α, for example...

[0081] In some of these implementations, the current value of the scaling term α can be independent of the current action and current observation included in the transformation. In these implementations, the system can separate the value of the scaling term α from the relative magnitude of the initial Q-value predicted by the Q-neural network, while also being able to impose constraints on the scaling term α.

[0082] In some of these implementations, the current value of the scaling term α can be defined by one or more scaling term parameters, i.e., it can be determined by evaluating a predetermined function on one or more scaling term parameters learned during the training of the Q-neural network. For example, the scaling term α = exp(ρ) such that α > 0, where ρ is a trainable parameter whose current value can be determined as part of the system's training of the Q-neural network.

[0083] In these implementations, the system determines the scaling time difference learning objective of the transformation based on (i) the current value of the scaling term α and (ii) the next observation included in the transformation (310). See below for reference. Figure 4 As further described, the scaling time difference learning objective can be the sum of: (a) the scaled, time-adjusted (e.g., time-discounted) next expected reward if the next action is performed in response to the next observation in the transition, and (b) the reward in the transition. Determining this scaling time difference learning objective differs from determining a conventional time difference learning objective in that, in addition to using a discount factor γ to discount (i.e. reduce) the expected future reward received from the environment, the system also uses a scaling term α to adjust the next expected reward that has already been adjusted in time; that is, the scaled, time-adjusted next expected reward is determined from the next expected reward of the initial time adjustment. As mentioned above, the system uses the scaling term α to prevent a significant overestimation of the time difference learning objective during training.

[0084] In these implementations, the system determines the error between the scaling time difference learning target and the scaling Q value (312).

[0085] Specifically, the system calculates the difference between the scaling time difference learning objective and the scaling Q-value:

[0086]

[0087] in, It is the scaled Q value, and This is the scaling-time difference learning objective. The value θ′ is called the “target network parameter.” They are updated at intervals (e.g., after each C update step during Q-neural network training) to be equal to the current value of the Q-neural network parameter θ, otherwise they remain fixed between individual update steps. As mentioned earlier, the value of the scaling term α can be iteratively adjusted during training, and if so, the algorithm will take the target scaling term α’, which can be updated at intervals to be equal to the current value α. In versions where α is not iteratively adjusted, α’ is equal to α. More generally, the error can be any suitable relative measure between two values, such as one of the following: difference, square of difference, cube of difference, square root of difference, etc.

[0088] In some other implementations, in addition to or instead of using Q-value reparameterization, the system can utilize ranking regularization to encourage the Q-network to generate predictable Q-values ​​for Q-network inputs, including unseen (or rarely seen) actions, observations, or both. In these implementations, the system can add a ranking regularization term to the error, which, when used to evaluate the loss of the Q-network, encourages the Q-network to output lower Q-values ​​from processing Q-network inputs (including actions, observations, or both that were not (or rarely) used during training). Specifically, this error can be the error determined from step 312 or a conventional temporal difference learning error associated with the selected transition. The conventional temporal difference learning error can be determined based on calculating the difference between: (i) the initial Q-value determined from step 306; and (ii) the sum of: (a) the expected reward of the next time discount if the next action is performed in response to the next observation in the transition, and (b) the reward in the transition.

[0089] Specifically, the ranking regularization term can be computed for each remaining action in the set of possible actions that differs from the actions included in the transformation as the sum of the maximum values ​​of the following two parts: (i) a constant plus the difference between (a) the initial Q-value of the transformation and (b) the Q-value generated by processing the current observation and remaining actions according to the current values ​​of the Q-network parameters; and (ii) zero. Mathematically, the system can compute the ranking regularization term as:

[0090]

[0091] Wherein, the constant v can be set to have a small positive value (e.g., 5e-1, 5e-2, or 5e-3), and |A| is the set of possible actions, where a t These are the actions included in the transformation.

[0092] When used by the system, this ranking regularization encourages the Q-neuron to output lower Q-values ​​from processing Q-neuron inputs, which include actions, observations, or both that are not (or rarely) used during training, because the error (which includes the ranking regularization and drives the training of the Q-neuron) is usually larger when the Q-neuron generates higher Q-values ​​by processing such Q-neuron inputs.

[0093] In some of these implementations, instead of adding this ranking regularization term to the error associated with each transformation, the system can add the ranking regularization term to the error based on a filtering scheme. The filtering scheme determines whether the ranking regularization term should be added to the error based on the reward included in the transformation. For example, the filtering scheme could specify that the ranking regularization term is added only to the error associated with "reward" transformations (i.e., transformations that include rewards greater than a predetermined threshold), rather than to the error associated with all selected transformations, including those with rewards below the predetermined threshold, such as transformations generated due to the agent performing a suboptimal action.

[0094] In some implementations that use Q-value reparameterization, the system can impose additional soft constraints to encourage the Q-neural network to generate appropriate Q-values ​​for the set of possible actions, selecting actions that are close to those included in the transformation. This further prevents overall overestimation of the Q-neural network during training. In these implementations, the system can add scaling regularization terms to the error.

[0095] Specifically, the scaling term regularization term can depend on the difference between two parts: (i) the scaling Q-value of the transition and (ii) the sum of adjustment times for the corresponding rewards included in one or more subsequent transitions of a transition selected from the playback memory. Mathematically, the scaling term regularization term can be calculated as:

[0096]

[0097] For each of the multiple transformations selected from the playback memory, It is the scaling Q value of the selected transformation, and This is the sum of the adjustment time for the corresponding rewards included in one or more transformations following the selection of the transformation. In this example, the system only... The scaling term α is constrained by the softplus function. Softplus is a smooth approximation of the rectified linear (ReLU) function and can be defined as follows: Here, β is a real value, which may be equal to 1.

[0098] The system determines the update of the current value of the Q network parameters based on the gradient of a loss function that includes the error relative to the Q network parameters (314). Specifically, the loss function may include the error (i.e., the error determined in step 312, or the conventional temporal difference learning error), and in some implementations, it may also include an additional loss term that depends on a ranking regularization term or a scaling regularization term or both, and the system can determine the update via backpropagation.

[0099] The system determines the update of the current value of the scaling term based on the gradient of the loss function, which includes the error relative to one or more scaling term parameters that define the scaling term, and through backpropagation (316).

[0100] In some implementations, the current values ​​of the Q-network parameters are updated with a learning rate that is different from (e.g., larger) the learning rate of one or more scaling term parameters. The system can set different Q-network parameter learning rates and different scaling term learning rates based on the mini-batch size of the selected transformations used to train the Q-neural network, i.e., based on the exact number of multiple transformations selected from the playback memory.

[0101] The system then continues to update the current values ​​of the Q-network parameters and scaling terms based on determined updates, according to an appropriate machine learning training algorithm (e.g., stochastic gradient descent, Adam, or rmsProp). Alternatively, once steps 306-316 have been performed on the selected multiple transformations for the entire batch, the system continues to update only the current values ​​of the Q-network parameters and scaling terms. In other words, the system combines, for example, the corresponding updates determined during a fixed number of iterations in steps 306-316 by weighted or unweighted averages and continues to update the current values ​​of the Q-network parameters and scaling terms based on the combined updates.

[0102] Figure 4 This is a flowchart of an example process 400 for determining a scaling-time difference learning objective. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed system, such as... Figure 1 The reinforcement learning system 100 can execute process 400.

[0103] For each transition, the scaling-time difference learning objective can be the sum of: (a) the scaled, time-adjusted (e.g., time-discounted) next expected reward if the next action is performed in response to the next observation in the transition, and (b) the reward in the transition. The way the system selects the next action and determines the next expected reward depends on the reinforcement learning algorithm used to train the Q-neural network.

[0104] For example, in Q-learning techniques, the system selects the action as the next action for which the Q-neuron (or target Q-neuron, i.e., the target instance of the Q-neuron) outputs the highest Q-value when the next observation is provided as input, and uses the Q-value generated by providing the next observation as input to the Q-neuron (or target Q-neuron) as the next reward. In other words, in this example, the system uses the Q-neuron (or target Q-neuron) and processes each action in the set of possible actions that each agent can perform when interacting with the environment based on the parameter values ​​of the Q-neuron (or target Q-neuron) to output the corresponding Q-value of that action (402), and then selects a chosen action from the set of possible actions based on the corresponding Q-value (404), i.e., selects the action with the highest Q-value. The corresponding Q-value for each action is an estimate of the reward that the agent would receive if it performed the action in response to the next observation included in the transition.

[0105] In this example, the system can use a target Q-neural network to mimic a Q-neural network because, at every fixed number of training steps, the parameter values ​​from the Q-neural network are copied to the target Q-neural network. The target Q-neural network is used to determine the next expected reward, which is then used to determine the learning objective from the time-difference learning of the training that drives the Q-neural network. This helps to stabilize the learning. In some implementations, instead of copying the parameter values ​​to the target Q-neural network, the parameter values ​​of the target Q-neural network slowly track the Q-neural network (“learning” the neural network) according to θ′←τθ+(1-τ)θ′, where θ′ represents the parameter values ​​of the target Q-neural network, and θ represents the parameter values ​​of the Q-neural network, and τ << 1.

[0106] As another example, in Sarsa learning techniques, the next action is the action the agent actually performs in response to the next observation, and the next reward is the Q-value generated by providing the next observation and, in some cases, the next action as input to the Q-neural network (or the target instance of the Q-neural network). See below for reference. Figure 5 The method of training a Q-neural network using a regularized offline learning scheme in conjunction with the Sarsa learning technique is further described.

[0107] If the selected action is performed in response to the next observation, the system determines the next expected reward for adjusting the initial time based on the value of the discount factor γ and the Q-value of the selected action (406). Specifically, to determine the next expected reward for adjusting the initial time, the system can calculate the product of the discount factor γ and the Q-value of the selected action.

[0108] The system determines the next expected return for scaling and adjusting time based on (i) the current value of the scaling term and (ii) the next expected return for adjusting the initial time (408). Specifically, for example, the system can calculate the product of the target scaling term α' and the next expected return for adjusting the initial time, and correspondingly determine the next expected return for scaling and adjusting time as follows: In this example, similar to the Q-network parameter value update scheme in dual-deep Q-learning techniques, the system uses a target scaling term α' to mimic the scaling term α, because the values ​​of the scaling term are copied to the target scaling term at intervals (or, because the values ​​of the target scaling term slowly track the values ​​of the scaling term according to a predetermined schedule). The target scaling term is used to determine the next expected reward after scaling and adjusting for time, which is then used to determine the scaling time difference learning objective, from which the training of the Q-neural network is driven. This helps stabilize the learning process.

[0109] The system then determines the scaled time difference learning objective (410) by calculating the sum of (i) the reward included in the transition and (ii) the scaled, adjusted next expected reward. In other words, the system can determine the scaled time difference learning objective as follows:

[0110]

[0111] The following is an example algorithm for training a Q-neural network using a regularized offline learning scheme.

[0112]

[0113]

[0114] In the example algorithm shown above, the scaling term α is initialized to 1 and then updated during training using the same technique (i.e., stochastic gradient descent) as updating the Q-network parameters θ, but preferably with a smaller learning rate (e.g., by selecting a transformed mini-batch B2 from the playback memory that is larger than the transformed mini-batch B1) to stabilize learning and reduce the variance of the estimate. This example algorithm also utilizes a dual deep Q-learning technique, which employs a target Q-neuron and a target scaling term, where the values ​​of the target Q-network parameters θ′ and the target scaling term α′ are set at each t... target The step size has been updated.

[0115] Figure 5 This is a flowchart of another example process 500 for training a Q-neural network. For convenience, process 500 will be described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed system, such as... Figure 1 The reinforcement learning system 100 can execute process 500.

[0116] Figure 5 Examples and Figure 3 The difference is that each transition in the replay memory also includes a corresponding next action performed by the agent in response to the next observation included in the transition. In other words, in Figure 5 In the example, the system maintains multiple transitions in the playback memory, where each transition can be a tuple comprising: (1) a current observation s representing the current state of the environment at a time; (2) a current action a performed by the agent in response to the current observation; (3) a next observation s′ representing the next state of the environment after the agent performs the current action, i.e., the state to which the environment is transformed as a result of the agent performing the current action; (4) a reward r received in response to the agent performing the current action; and (5) a corresponding next action a′ performed by the agent in response to the next observation.

[0117] The system selects multiple transformations from the playback memory and then trains the Q-neural network by repeating steps 502-508 for each of the selected multiple transformations.

[0118] The system processes the current observation and current action based on the current values ​​of the Q network parameters to generate the initial Q value (502) for the transformation.

[0119] The system determines the temporal difference learning objective of the transition based on the next observation and the next action included in the transition (504). This step can be called behavior value estimation because the next action is the action actually performed by an agent controlled in response to the next observation when generating the transition to be included in the playback memory, using a "behavior" control system (as described above, which may be, for example, manually operated or another machine learning-based control system). The next reward is a Q-value generated by providing the next observation and, in some cases, the next action as input to a Q-neural network, which is configured to process the next observation based on the current values ​​of the Q-network parameters and, in some cases, the next action, to generate a Q-value for the next action. This Q-value is an estimate of the reward the agent will receive if it performs the next action in the transition in response to the next observation in the transition. To determine the next expected reward for the adjusted time, the system can calculate the product of a discount factor γ and the Q-value generated by the Q-neural network. The system can then calculate the temporal difference learning objective as a sum of (a) the next expected reward for the adjusted time (e.g., time discount) and (b) the reward in the transition.

[0120] The system determines the error between the time difference learning objective and the initial Q value (506). Specifically, the system calculates the difference between the time difference learning objective and the initial Q value:

[0121] Q θ (s,a)-(r+γQ θ′(s′,a′)),

[0122] Among them, Q θ (s, a) are the initial Q values, and (r + γQ) θ′ (s′,a′)) is the time difference learning objective.

[0123] More generally, error can be any appropriate relative measure between two values, such as one of the following: difference, square of difference, cube of difference, square root of difference, etc.

[0124] The system determines the update of the current values ​​of the Q network parameters based on the gradient of the calculated loss function and through backpropagation (508). The loss function includes (i) the error and, in some implementations, (ii) a ranking regularization term that can be added to the error according to a filtering scheme relative to the Q network parameters.

[0125] Mathematically, when a ranking regularization term is added to the error according to the filtering scheme, the loss function... This can include error-dependent (Q) θ (s,a)-(r+γQ θ′ The first loss term of (s′, a′) and the ranking regularization term The second loss item.

[0126] Figure 6 yes Figure 5 A diagram comparing an example training process with a traditional training process. In an offline reinforcement learning setting, the action selection policy is improved (by means of adjusting the Q-network parameters Q). θ Greedily select actions based on the current value, i.e., calculate the maximum π(s) = arg max for all actions. a Q θ (s, a)) may be risky because it relies on extrapolation from outside the data.

[0127] However, some traditional methods perform many iterations of policy improvement during training when calculating the target Q-value of the transition in the training data (e.g.) Figure 6As shown at the top, this leads to extrapolation errors and divergence. Conversely, as shown at the bottom, by executing process 500, the system is able to estimate the value of the behavioral policy during training without performing any policy improvement steps, i.e., without performing maximization on all actions. The system is also able to perform policy improvement (by computing maximization on all actions) only when controlling the agent to interact with the environment using the trained Q-neural network, i.e., only when selecting actions that the agent has performed after the Q-neural network has been trained. The system thus mitigates the effects of these extrapolation errors during training. Furthermore, ranking regularization is used to train the Q-neural network to generate Q-values ​​that result in actions in the dataset that lead to successful outcomes being more likely to be selected than any unobserved actions.

[0128] This specification uses the term "configured to" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that software, firmware, hardware, or a combination thereof have been installed on the system, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0129] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in one or more combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0130] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0131] A computer program (which may also be referred to as a program, software, software application, app, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored as a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0132] In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or not at all, and it can be stored on storage devices in one or more locations. Therefore, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0133] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same computer or on multiple computers.

[0134] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry (such as an FPGA or ASIC) or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0135] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for receiving data from or transferring data to, one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0136] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0137] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball), through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and, in turn, receiving response messages from the user.

[0138] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0139] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0140] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component as an application server, or a frontend component as a client computer having, for example, a graphical user interface, web browser, or app that a user can interact with through an implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication in any form or medium—for example, a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0141] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0142] While this specification includes numerous details of specific implementations, these should not be construed as limiting the scope of any invention or the scope of what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this specification within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0143] Similarly, although operations are depicted in the accompanying drawings and recited in the claims in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or to perform all shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0144] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for training a Q-neural network, the Q-neural network having a plurality of Q-network parameters, and for selecting actions performed by a reinforcement learning agent interacting with the environment by performing actions that cause a state transition in the environment, the method comprising: A replay memory is maintained, storing multiple transitions generated as a result of the reinforcement learning agent's interaction with the environment. Each transition includes a corresponding current observation representing the corresponding current state of the environment, a corresponding current action performed by the agent in response to the current observation, a corresponding next observation representing the corresponding next state of the environment, and a reward received in response to the agent performing the current action, wherein: The environment is a real-world environment, the intelligent agent is a mechanical intelligent agent that interacts with the real-world environment, and the current observation includes one or more of image data, object location data, or sensor data, or The environment is a simulated environment, the agent is implemented as one or more computer programs that interact with the simulated environment, and the current observation includes a simulated version of one or more of image data, object location data, or sensor data; Select multiple conversions from the playback memory; and Training the Q-neural network on the plurality of transformations includes, for each of the plurality of transformations: The current observation and the current action are processed based on the current values ​​of the Q network parameters to generate the initial Q value of the transformation; The scaling Q value of the transformation is determined based on (i) the current value of the scaling term and (ii) the initial Q value; Based on (i) the current value of the scaling term and (ii) the next observation included in the transformation, determine the scaling time difference learning target of the transformation; Determine the error between the scaling time difference learning target and the scaling Q value; Based on the gradient of the error relative to the Q-network parameters, an update to the current value of the Q-network parameters is determined; and An update to the current value of the scaling term is determined based on the gradient of the error relative to one or more scaling term parameters.

2. The method according to claim 1, wherein, The learning objective for determining the scaling time difference of the transformation based on the current value of the scaling term (i) and the next observation (ii) includes: The Q-neural network is used to process the next observation and each action in the set of possible actions that the agent can perform when interacting with the environment, to output a corresponding Q value for the action as an estimate of the reward that the agent will receive if it performs the action in response to including the next observation in the transition. The selected action is chosen from the set of possible actions based on the corresponding Q value; If the selected action is performed in response to the next observation, the next expected return for adjusting the initial time is determined based on the value of the discount factor and the Q value of the selected action. Based on the current value of the scaling term (i) and (ii) the next expected return after adjusting the initial time, determine the scaled, adjusted time's next expected return; and Calculate the sum of (i) the rewards included in the transformation and (ii) the next expected return after scaling and adjusting the time.

3. The method according to claim 1, wherein, The learning objective for determining the scaling time difference of the transformation based on the current value of the scaling term (i) and the next observation (ii) also includes: The target instance of the Q-neural network processes the next observation and each action in the set of possible actions that the agent can perform when interacting with the environment, based on the target parameter values ​​of the neural network, to output a corresponding Q value for the action as an estimate of the reward that the agent will receive if it performs the action in response to including the next observation in the transition.

4. The method according to claim 1, wherein, Determining the scaling Q value of the transformation based on (i) the current value of the scaling term and (ii) the initial Q value includes: Calculate the product of the scaling term and the initial Q value.

5. The method of claim 1, further comprising initializing the current value of the scaling item to one.

6. The method according to claim 1, wherein, The scaling term is In the form of, These are trainable parameters.

7. The method according to claim 1, wherein, Determining the update of the current value of the Q-network parameters based on calculating the gradient of the error relative to the Q-network parameters includes: A ranking regularization term is added to the error, wherein the ranking regularization term depends on the maximum of the following: (i) a constant plus the difference between (a) and (b): (a) the initial Q-value of the transformation; (b) the Q-value generated by processing the current observation and the remaining actions in the set of possible actions different from those included in the transformation according to the current values ​​of the Q-network parameters; and (ii) zero; and An update to the current value of the Q network parameters is determined based on the gradient of the ranking regularization term and the error relative to the Q network parameters.

8. The method of claim 7, further comprising adding the ranking regularization term to the error according to a filtering scheme, the filtering scheme determining whether the ranking regularization term should be added to the error based on the reward included in the transformation.

9. The method according to claim 1, wherein, Determining an update to the current value of the scaling term based on calculating the gradient of the error relative to one or more scaling term parameters includes: A scaling regularization term is added to the error, wherein the scaling regularization term depends on the difference between (i) the scaling Q value of the transformation and (ii) the sum of the adjustment times for the corresponding rewards included in one or more subsequent transformations of a transformation selected from the playback memory; and An update to the current value of the scaling term is determined based on the gradient of the scaling term regularization term and the error relative to the one or more scaling term parameters.

10. The method according to claim 1, wherein: Determining an update to the current values ​​of the Q-network parameters further includes: determining the value of the learning rate for the Q-network parameters; and Determining the update of the current value of the scaling term further includes: determining the value of the scaling term learning rate that is less than the value of the Q network parameter learning rate.

11. The method according to claim 10, wherein, The value of the scaling learning rate depends on the number of transformations selected from the playback memory.

12. The method according to claim 1, wherein: Each transition in the playback memory also includes a corresponding next action performed by the agent in response to the next observation; and Training the Q-neural network on the plurality of transformations further includes, for each of the plurality of transformations: The current observation and the current action are processed based on the current values ​​of the Q network parameters to generate the initial Q value of the transformation; Based on the next observation and the next action included in the transition, determine the temporal difference learning objective of the transition; Determine the error between the time difference learning objective and the initial Q value; and The update of the current value of the Q network parameters is determined based on the gradient of the error relative to the Q network parameters.

13. The method according to any one of claims 1-12, wherein, None of the multiple transformations were generated by controlling the agent using the Q neural network.

14. A method comprising: Receive new observations, which represent a new state of the environment with which the agent is interacting; The new observation is processed using a Q-neural network to generate a corresponding Q-value for each action in a set of possible actions that the agent can perform in response to the new observation, the Q-value for each action being an estimate of the reward that the agent will receive if it performs the action in response to the new observation, wherein the Q-neural network has been trained using the method of any one of claims 1 to 13. Based on the corresponding Q value, an action is selected from the set of possible actions; and The agent is made to perform the selected action.

15. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform a corresponding operation of the method of any one of claims 1 to 14.

16. A system comprising one or more computers and a storage device containing one or more storage instructions, wherein the instructions, when executed by the one or more computers, cause the one or more computers to perform a corresponding operation of the method of any one of claims 1 to 14.

Images