Determine control strategies by minimizing the impact of delusions

Abstract

The present invention relates to methods, systems and apparatus for determining a control policy for an agent interacting with an environment, including a computer program encoded on a computer storage medium. One of the methods includes updating a control policy using a policy-consistent backup by using Q-learning. To determine the policy-consistent backup, the system determines a policy-consistent backup for the control policy at a current observation-current action pair, including: for each of a plurality of actions in a set of possible actions that the agent can perform, identifying a Q-value assigned by the control policy to the next observation-action pair of the control policy and justified by at least one of the information sets; pruning from the identified Q-values ​​any Q-value justified only by an information set that is not policy-consistent; and determining the policy-consistent backup from the reward and only the identified Q-values ​​that were not pruned.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. application Ser. No. 62 / 752,306, filed Oct. 29, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This specification relates to reinforcement learning. Background Art

[0004] In a reinforcement learning system, an agent interacts with an environment by performing actions selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

[0005] Some reinforcement learning systems select an action to be performed by the agent in response to receiving a given observation based on the output of a neural network.

[0006] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters. Summary of the invention

[0007] This specification generally describes a reinforcement learning system that controls an agent that interacts with an environment, and in particular determines a control policy for controlling the agent.

[0008] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0009] Conventional systems that use Q-learning to learn control policies for agents may update policies by using backup value estimates derived from action choices that are not possible in the base policy class. That is, in conventional Q-learning, backups of state-action pairs are generated by independently selecting actions at corresponding next states using the max operator, i.e., by using the Q-value for the argmax action when the argmax action is selected at the next state to generate the target Q-value. This assumes that independently selected maxima are feasible, i.e., selecting the action that yields the maximum Q-value at the next state is consistent with other action choices taken to reach the next state. In other words, this assumes that there is a control policy that both selects the action that yields the maximum Q-value at the next state and makes other action choices taken to reach the next state.

[0010] When this assumption is violated, which it may be from time to time, this can cause problems in the control policy learning process. In particular, violations of these assumptions (as occurs in systems implementing a conventional Q-learning variant) can cause the learning process to diverge, can cause the control policy to be learned to not perform well on the task, or cause the learning process to run an excessive number of iterations. On the other hand, the described techniques avoid violations of this assumption or minimize violations of this assumption, thereby improving the control policy learned for the agent, which in turn improves the performance of the agent on the desired task. Additionally, the control policy can be learned with fewer iterations to reduce the computational resources consumed by the learning process.

[0011] In particular, the described techniques determine updates (and backups) to the control policy by using a variety of techniques that avoid explicitly violating this assumption, for example, by maintaining sets of information and updating them only with policy-consistent Q-values, or by employing heuristics that reduce the likelihood of violating the assumption, such as by choosing a next action that is locally consistent within a batch of training tuples.

[0012] The described techniques, when used in conjunction with real-world environments and agents such as mechanical agents / robots or factory / service facilities, can result in improvements to control strategies learned for controlling the agents, such as improvements in energy efficiency, accuracy, speed and / or output of performing tasks using the learned control strategies.

[0013] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become more apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A An example reinforcement learning system is shown.

[0015] Figure 1B Showing an example environment susceptible to "delusional bias".

[0016] Figure 2 is a flow chart of an example process for learning a control policy by using Q-learning with policy-consistent backup.

[0017] Figure 3 is a flow chart of an example process for learning a control policy by using value iteration with policy-consistent backups.

[0018] Figure 4 is a flow chart of an example process for learning a control policy by using locally consistent backups.

[0019] Like reference numbers and designations in the various drawings represent like elements. DETAILED DESCRIPTION

[0020] This specification describes a reinforcement learning system that controls an agent that interacts with an environment by processing, at each of a plurality of time steps, data characterizing the current state of the environment at that time step (i.e., an "observation") to select an action to be performed by the agent.

[0021] At each time step, the state of the environment at that time step depends on the state of the environment at the previous time step and the action performed by the agent at the previous time step.

[0022] In some embodiments, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle that navigates the environment.

[0023] In these embodiments, the observations may, for example, include one or more of: images, object position data, and sensor data used to capture the observations as the agent interacts with the environment, such as sensor data from images, distance or position sensors, or from actuators.

[0024] For example, in the case of a robot, the observations may include data characterizing the current state of the robot, such as one or more of: joint positions, joint velocities, joint forces, torques or accelerations (e.g., gravity-compensated torque feedback), and the global or relative pose of an item held by the robot.

[0025] In the case of a robot or other mechanical agent or vehicle, the observations may similarly include one or more of: position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. Observations may be defined in 1, 2, or 3 dimensions and may be absolute and / or relative.

[0026] The observations may also include, for example: sensing electronic signals, such as motor current or temperature signals; and / or image or video data, for example from a camera or LIDAR sensor, such as data from sensors of the agent, or data from sensors located separately from the agent in the environment.

[0027] In these embodiments, the actions may be control inputs for controlling a mechanical agent / robot, such as torques applied to a robot's joints or higher-level control commands, or control inputs for controlling an autonomous or semi-autonomous land, air, or sea vehicle, such as torques applied to a vehicle's control surfaces or other control elements, or higher-level control commands.

[0028] In other words, an action can, for example, include position, velocity, or force / torque / acceleration data of one or more joints of a robot or part of another mechanical agent. The action data may additionally or alternatively include electronic control data, such as motor control data, or more generally data for controlling one or more electronic devices in the environment, where controlling such electronic devices has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land or air or sea vehicle, the action may include actions to control navigation (e.g., steering) and movement (e.g., braking and / or acceleration) of the vehicle.

[0029] In the case of an electronic agent, the observations may include data from one or more sensors (e.g., current, voltage, power, temperature, and other sensors) used to monitor a portion of a plant or service facility and / or electronic signals used to represent the functionality of electronic and / or mechanical items of equipment. For example, the real-world environment may be a manufacturing plant or service facility, the observations may be about the operation of the plant or facility, e.g., about resource usage, such as power consumption, and the agent may control actions or operations in the plant / facility, e.g., to reduce resource usage. In some other embodiments, the real-world environment may be a renewable energy plant, the observations may be about the operation of the plant, e.g., about maximizing current or future planned power generation, and the agent may control actions or operations in the plant to achieve this goal.

[0030] In some other applications, the agent may control actions in a real-world environment (including equipment items), such as in a data center, in a power / water distribution system, or in a manufacturing plant or service facility. Thus, the observations may be about the operation of the plant or facility. For example, the observations may include observations of the power or water used by equipment, or observations of power generation or distribution control, or observations of resource use or waste generation. The actions may include actions to control or apply operating conditions to equipment items of the plant / service facility, and / or actions that cause settings to be changed in the operation of the plant / service facility, such as to adjust or turn on / off components of the plant / facility.

[0031] As another example, the environment can be a chemical synthesis or protein folding environment, so that each state is a respective state of a protein chain or one or more intermediates or precursor chemicals, and the agent is a computer system for determining how to fold a protein chain or synthesize a chemical. In this example, the action is a possible folding action for folding a protein chain or an action for polymerizing a precursor chemical / intermediate, and the result to be achieved can, for example, include folding a protein so that the protein is stable and achieves a specific biological function, or provides an effective synthesis pathway for a chemical. As another example, the agent can be a mechanical agent that automatically performs or controls a protein folding action or a chemical synthesis step selected by the system without human intervention. The observation can include directly or indirectly observing the state of a protein or chemical / intermediate / precursor, and / or can be derived from a simulation.

[0032] In some implementations, the environment may be a simulated environment, and the agent may be implemented as one or more computers that interact with the simulated environment.

[0033] The simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle that navigates through the motion simulation. In these embodiments, the action may be a control input that controls a simulated user or a simulated vehicle.

[0034] In some embodiments, the simulation environment can be a simulation of a particular real-world environment. For example, the system can be used to select actions in the simulation environment during training or evaluation of a control neural network, and after completing the training or evaluation or both, the system can be deployed to control a real-world agent in a real-world environment simulated by the simulation environment. This can avoid unnecessary wear and tear and damage to the real environment or the real-world agent, and can allow the control neural network to be trained and evaluated for situations that rarely occur or are difficult to reproduce in the real world.

[0035] Typically, in the case of a simulated environment, the observations may include simulated versions of one or more previously described observations or types of observations, and the actions may include simulated versions of one or more previously described actions or types of actions.

[0036] Figure 1A An example reinforcement learning system 100 is shown. Reinforcement learning system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0037] The system 100 controls the interaction of the agent 102 with the environment 104 by selecting actions 106 to be performed by the agent 102 and then causing the agent 102 to perform the selected actions 106 .

[0038] The agent 102 performs the selected action 106 to generally transition the environment 104 to a new state. By repeatedly causing the agent 102 to perform actions in the environment 104, the system 100 is able to control the agent 102 to complete a specified task.

[0039] The system 100 includes a control strategy 110 , a training engine 150 , and one or more memories for storing a set of control strategy parameters 118 for the control strategy 110 .

[0040] At each time step in a plurality of time steps, control strategy 110 maps current observation 120 , which characterizes a current state of environment 104 , to action 106 according to control strategy parameters 118 to generate action selection output 122 .

[0041] In particular, the control policy 110 generates a corresponding Q-value for each action in the action set and then selects one of the actions. The Q-value of an action is an estimate of the "reward" that would result from the agent performing the action in response to the current observation 120, and thereafter selecting future actions performed by the agent 102 according to a greedy control policy that selects the action with the highest Q-value in response to each observation, i.e., the Q-value is generated according to the control policy parameters 118.

[0042] A reward is a cumulative measure of the "rewards" 124 received by an agent, such as a sum of rewards or a time-discounted sum of rewards. An agent can receive a corresponding reward 124 at each time step, where the reward 124 is specified by a scalar value and, for example, characterizes the agent's progress toward completing a specified task.

[0043] For example, during training, the control policy 110 can select actions to be performed by the agent according to an exploration strategy. For example, the exploration strategy can be an ∈-greedy exploration strategy, in which the system 100 selects the action with the highest Q value with probability 1-∈ and randomly selects an action with probability ∈. In this example, ∈ is a scalar value between 0 and 1.

[0044] After training, the control policy 110 can adopt a greedy action selection strategy, that is, by always selecting the action with the highest Q-value.

[0045] The system 100 then controls the agent, ie, by causing the agent to perform the action 106 selected by the control policy 110 .

[0046] The control strategy 110 can map the observations to Q-values ​​using any of a variety of techniques.

[0047] As an example, the control strategy 110 can maintain a tabular representation of a Q-function that maps (observation-action) pairs to Q-values. In this example, during training, the system 100 directly updates the Q-values ​​in the tabular representation, i.e., the control strategy parameters are the Q-values ​​in the tabular representation.

[0048] As another example, the control strategy 110 can maintain a function approximator for approximating the Q function. When the observations include high-dimensional data such as an image of the environment, the function approximator can be, for example, a neural network (also known as a Q neural network). Alternatively, the function approximator can be a linear model or a generalized linear model. In these cases, during training, the system 100 learns the parameter values ​​of the function approximator, that is, the control strategy parameters are the parameters of the function approximator.

[0049] The training engine 150 is configured to train the control strategy 110 by repeatedly updating the control strategy parameters 118 (ie, parameters of the function approximator or Q-values ​​in a tabular representation) of the control strategy 110 .

[0050] In particular, the training engine 150 trains the control policy 110 by using reinforcement learning that uses the observations 120 and rewards 124 generated as a result of the agent's interactions with the environment during training.

[0051] In general, the training engine 150 can train the control policy 110 to increase the rewards (ie, the cumulative measure of reward) received by the agent by using Q-updates (eg, by using Q-learning or value iteration).

[0052] In other words, the training engine 150 repeatedly updates the control strategy parameters 118 using the observations 120 , the actions performed in response to those observations, and the rewards 124 .

[0053] The training engine 150 updates the control policy parameters 118 in a manner that eliminates or mitigates the effects of "delusional bias" on the learned control policy parameters 118. In particular, "delusional bias" occurs whenever a backup value estimate is derived from an action choice that is not achievable in the base policy class. In other words, if all past (implicit) action choices cannot be jointly expressed by any policy in the admissible class, then the backup value does not correspond to a Q-value that can be achieved by any expressible policy.

[0054] Figure 1B An environment 170 susceptible to "paranoia bias" is illustrated.

[0055] exist Figure 1BIn an environment of , an episode (i.e., a series of interactions during which the agent attempts to perform a specified task) starts in state s1, and there are two actions: a1 causes the episode to terminate, except at s1, where if a1 is executed, the environment can enter state s4 with probability q. Another action, a2, moves the environment deterministically to the next state in the sequence s1 to s4, and when a2 is executed at s4, episode termination occurs. The reward for executing an action is 0, but there is a non-zero positive reward R(s1,a1) when a1 is executed at s1, and there is another non-zero positive reward R(s4,a2) when a2 is executed at s4.

[0056] Specifically, let q=0.1, R(s1, a1)=0.3, and R(s4, a2)=2.

[0057] At this time, consider the linear function approximator f used to operate on the features of the state-action pair θ , that is, the observation is the feature value of the state, and the input of the approximator is the feature value of the state and the feature value of the action, and generates a Q value for the state-action pair. In particular, the linear function approximator operates on a two-dimensional feature vector, where the first dimension is the feature of the state and the second dimension is the feature of the action, i.e., the state-action feature Therefore, the linear function approximator has the form where θ1 and θ2 are the approximator parameters learned via Q-updates (e.g., via Q-learning or value iteration).

[0058] exist Figure 1B In the specific example of , the two state-action features are as follows:

[0059] Given this, a linear function approximator cannot (i) Map to Higher Q value, (ii) will Map to Higher Q. That is, no combination of values ​​of θ1 and θ2 can produce an approximator that satisfies both (i) and (ii).

[0060] Therefore, no greedy action selection policy π can simultaneously satisfy π(s2) = a2 and π(s3) = a2, that is, no policy can make the system choose action a2 in state s2 and action a2 in state s3. In other words, since the greedy policy chooses the action with the highest Q value, any greedy policy that would choose action a2 in state s2 will choose action a1 in state s3, and any greedy policy that would choose action a2 in state s3 will choose action a1 in state s2. Therefore, the optimal unconstrained policy (taking a2 in all cases, with an expected value of 2) is also unachievable. Therefore, the Q update can never converge to the optimal unconstrained policy. Instead, the optimal achievable policy will take a1 in s1 and a2 in s4, resulting in a value of 0.5.

[0061] In this example, conventional Q-updates fail to find the optimal admissible policy π. For example, online Q-learning on data generated using the ε-greedy behavior policy (ε=0.5) converges to a fixed point that gives a "compromise" admissible policy that takes a1 (value 0.3) at both s1 and s4.

[0062] This example shows how paranoia bias can prevent Q-learning from reaching a reasonable fixed point.

[0063] For example, consider the backups at (s2, a2) and (s3, a2). Assume that the current function approximator assigns a "high" value to (s3, a2) (i.e., such that the Q value for (s3, a2) is greater than the Q value for (s3, a1)), as required by the optimal control strategy.

[0064] Intuitively, this requires θ1 to be less than 0 and to generate a "high" bootstrap value for (s2,a2). But any updates to θ1 and θ2 used to try to fit this value (i.e., making the Q-value of (s2,a2) greater than the Q-value of (s2,a1)) will force θ1 to be greater than 0, which is inconsistent with the assumption required to generate high bootstrap values ​​(i.e., θ1<0). In other words, any update that moves the Q-value for (s2,a2) higher will weaken the proof that it is higher.

[0065] The result is that the Q updates compete with each other, and the Q-value of (s2, a2) converges to a compromise value that cannot be achieved by any possible policy. This is because the backups generated by conventional Q-learning are independent of the previous actions and do not consider whether the actions required to generate the backup are consistent with the previous actions.

[0066] By modifying the way backups are generated (e.g., for Q-learning or value iteration), the described system is able to achieve better performance than conventional updating techniques, i.e., the rewards compiled by the generated control policies are closer to the optimal rewards than those generated by using conventional techniques, and therefore result in an agent that performs better on the specified task.

[0067] In some cases, the system maintains information sets and uses these information sets when determining backups for Q learning. That is, the system updates the control policy by using model-free Q learning. Figure 2 This process is described in detail.

[0068] In some other cases, the system maintains a transition model that models the dynamics of the environment. The transition model maps observation-action pairs to probabilities for each of a plurality of next states. The probabilities of the next states represent the probabilities that the next state will be the state to which the environment transitions when the agent performs the actions in the pair in response to the observations in the pair. The system can use the transition model and the maintained information set to determine backups using value iteration. Figure 3 This process is described in detail.

[0069] In other cases, the system does not maintain multiple information sets, but ensures that backups are computed by using locally consistent Q values ​​when performing batch Q learning. Figure 4 This process is described in detail.

[0070] Figure 2 is a flow chart of an example process 200 for learning a control policy by using Q-learning to learn a backup consistent with the policy. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed reinforcement learning system, such as Figure 1A The reinforcement learning system 100 is capable of performing process 200.

[0071] The system can repeatedly perform process 200 to update the control strategy, ie, repeatedly update the control strategy parameters.

[0072] The system maintains data defining one or more information sets (step 202). Each information set corresponds to a corresponding policy constraint set and identifies the Q-values ​​assigned by the control policy to observation-action pairs under the policy constraint set.

[0073] Where the policy is represented using a table, the system can maintain an additional Q-value table for each information set.

[0074] Where the strategy uses a function approximator, the system can maintain an additional function approximator for each information set (or equivalently, a separate set of parameters or weights for the same function approximator for each information set).

[0075] Each policy constraint set generally corresponds to some set of action-observation pairs and specifies that the policy must be able to select an action in the pair in response to an observation in the pair for all corresponding action-observation pairs. In other words, each policy constraint set specifies one or more paths through the environment that must be achieved by the policy. As a specific example, in Figure 1B In the example, one information set may constrain the policy to those policies that are able to select action a1 in state s1 and action a2 in state s4, another information set may constrain the policy to those policies that are able to select action a1 in state s1 and action a1 in state s4, and yet another information set may constrain the policy to those policies that are able to select action a2 in state s1 and action a2 in state s2.

[0076] However, in some cases, the pairs corresponding to any given set of policy constraints can also be based on constraints on agent operation that affect which actions can be selected in a given environmental state, such as safety constraints or other operational constraints.

[0077] In particular, let Θ be the parameter class used to define the Q function. Information Set It is a set of policy constraints used to justify assigning a specific Q-value q to a pair (s, a).

[0078] In general, the information set can be viewed as a finite partition of Θ, i.e., a set of non-empty subsets P = {X1, ..., Xk} such that for all i ≠ j, X1 ∪ · · · ∪ Xk = Θ, and

[0079] Additionally, if for all X'∈P', there exists X∈P such that Then partition P' is a refinement of P.

[0080] The system receives a current observation characterizing a current state of the environment, a current action performed by the agent in response to the current observation, a next observation characterizing a next state of the environment, and a reward received as a result of the agent performing the current action (step 204).

[0081] In other words, the system receives training tuples for use in determining updates to a control policy. In some cases, such as when updates are being computed on-policy, actions are selected based on current values ​​of policy parameters. In other cases, such as when updates are being computed off-policy, training tuples are sampled from a replay memory and actions may have been selected by using different older values ​​of policy parameters.

[0082] The system determines a policy-consistent Q-backup for the control policy at the current observation s-current action a pair (step 206).

[0083] In particular, for each of a plurality of actions in a set of possible actions that the agent can perform, the system identifies a Q-value assigned by the control policy to the next observation-action pair and justified by at least one of the information sets. If the control policy operates under the constraints imposed by the information set, the Q-value is justified by the information set when the control policy has assigned the Q-value to the next observation-action pair.

[0084] The system then prunes from the identified Q-values ​​any Q-values ​​that are justified only by an information set that is not policy class consistent. An information set is not policy class consistent when a greedy policy operating under the constraints imposed by the information set does not select a current action in response to a current observation. Thus, any Q-value justified only by an information set that does not result in the current action being performed in response to the current observation is pruned from the set of identified Q-values.

[0085] The system then determines a policy-consistent backup from the rewards and only the unpruned identified Q-values. In particular, the policy-consistent backup includes a backup for each set of information used to justify the unpruned Q-values, i.e., the target output. In particular, for a given one of these information sets, the update can be based on the reward in the tuple and the identified Q-value justified by the information, e.g., the sum of (i) the reward, (ii) the product of the discount factor and the identified Q-value.

[0086] The system updates the control policy for the agent by using Q-learning using the policy-consistent backup (step 208). In particular, the system can update each information set used to prove the unpruned Q-value by using the update of the information set.

[0087] When the system uses a tabular representation of the Q-function, the system can update the Q-value for each information set by computing the weighted sum of the current Q-value and the backup for that information set.

[0088] When the system uses a function approximator to approximate the Q function, the system can calculate an update to the function approximator corresponding to the information set by calculating a supervised learning update based on the error gradient (e.g., mean squared error) between the current Q value and a backup for the information set.

[0089] Therefore, from Figure 2 As can be seen from the description of , the system does not independently identify the argmax action when computing the backup (as is done in regular Q-updates), but instead maintains information sets and updates each information set only by using Q-values ​​consistent with that information set.

[0090] Because the information sets impose policy constraints, the number of information sets increases as the agent explores the environment, so in complex environments, the total number of information sets may grow too large. In some embodiments, the system can prune information sets during training, merge information sets during training, or both when certain criteria are met. For example, when the number of information sets exceeds a threshold number, the system can prune the information sets with the lowest Q values ​​or can combine the information sets with the lowest Q values.

[0091] Once training is complete, the system is able to select the policy parameters represented by one of the information sets as the final policy parameter set. In particular, the system is able to select the policy parameters that result in the highest Q-value being assigned to the action selected by the control policy at the initial state of any given episode of the control agent as the policy parameters that will be employed to control the agent in that episode.

[0092] Table 1 below shows pseudo code for learning a control policy through process 200 when the system uses a tabular representation of the Q function. In particular, in Table 1, the symbols are as follows:

[0093] -(s,a,r,s') is the training tuple,

[0094] - Q[sa] is a table such that Q[sa](X) is the Q-value of taking action a at time s and then following a greedy policy parameterized by θ∈X, where the parameter θ represents the constraints imposed by the information set X,

[0095] - ConQ[sa] is a table such that ConQ[sa](X) is the Q-value of taking action a at time s and then following a greedy policy parameterized by θ∈X, with the additional constraint π θ (s) = a. That is, the greedy strategy, when parameterized by the parameter θ, must choose action a in state s so that the Q value of taking action a in state s appears in the table ConQ[sa].

[0096] - ConQ[s] is a table formed by the concatenation of all tables ConQ[sa] that take action a in state s,

[0097] -s→a is the set of policy parameters that makes the greedy policy choose action a in state s,

[0098] - is the intersection sum and is defined such that Defined by:

[0099] as well as

[0100] -dom(.) is the domain of the function.

[0101]

[0102] Table 1

[0103] Figure 3 is a flow chart of an example process 300 for learning a control policy by using value iteration to back up consistent policy. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed reinforcement learning system, such as Figure 1A The reinforcement learning system 100 is capable of performing process 300.

[0104] As described above, the system maintains one or more sets of information (step 302).

[0105] The system obtains the current observation, and the action performed in response to the observation, and the reward (step 304).

[0106] The system determines a corresponding probability for each of the plurality of next states by using the transition model (step 306 ).

[0107] In particular, the system uses a transition model that models the dynamics of the environment to predict a respective probability for each of a plurality of next states, which probability represents the likelihood that the next state is the state to which the environment transitions as a result of the agent performing the action in response to the current observation.

[0108] The system determines a policy-consistent Bellman backup of the control policy at the current observation s-current action a pair (step 308).

[0109] In general, the system performs the following operations independently for each next state assigned a non-zero probability by the transition model to generate a set of Q-values ​​for the next state, and then combines the results to generate a policy-consistent Bellman backup of the control policy.

[0110] In particular, for each of a plurality of actions in a set of possible actions that the agent can perform, and for any given next state, the system identifies a Q-value assigned by the control policy to the next observation-action pair and justified by at least one of the information sets (where "next observation" is the one used to characterize the given next state).

[0111] The system then generates a set of Q-values ​​for the next state by pruning from the identified Q-values ​​any Q-values ​​that are justified only by an information set that is not policy class consistent. Thus, any Q-values ​​justified only by an information set that does not result in the current action being performed in response to the current observation are pruned from the set of identified Q-values ​​for the next state.

[0112] A policy-consistent backup includes a backup of each information set that proves an unpruned Q-value for any next state, i.e., the target output. In particular, for a given one of these information sets, the update can be based on the reward in the tuple, and for each next state for which the given information set proves an unpruned Q-value, the update is based on the Q-value proved by the information set and the probability assigned to the next state. For example, to compute the backup of a given information set, the system can compute, for each next state, the product of the probability assigned to the next state and the Q-value proved by the information set for the next state, and then sum these products to determine the initial update. The system can then compute the backup of the information set as the sum of (i) and (ii): (i) the reward, (ii) the product of the discount factor and the initial backup of the next state.

[0113] The system updates the agent's control policy using Q-learning using policy-consistent backups (step 310). In particular, the system can use updates to the information set to update each information set that justifies an unpruned Q-value. In some cases, the system also updates information sets that do not justify any unpruned Q-values, for example, by using only rewards.

[0114] When the system uses a tabular representation of the Q-function, the system is able to update the Q-value of each information set by computing the weighted sum of the current Q-value and the backup of the information set.

[0115] When the system uses a function approximator to approximate the Q function, the system can compute updates to the function approximator corresponding to the information set by computing supervised learning updates based on the error gradient (e.g., mean squared error) between the current Q value and a backup of the information set.

[0116] As described above, in some embodiments, the system can prune information sets during training, merge information sets during training, or both when certain criteria are met. For example, when the number of information sets exceeds a threshold number, the system can prune the information set with the lowest Q value or can combine the information sets with the lowest Q value.

[0117] Once training is complete, the system is able to select the policy parameters represented by one of the information sets as the final policy parameter set. In particular, the system is able to select the policy parameters that result in the highest Q-value being assigned to the action selected by the control policy at the initial state of any given episode of the control agent as the policy parameters that will be employed to control the agent in that episode.

[0118] Table 2 below shows pseudo code for learning a control policy using process 300 when the system uses a tabular representation of the Q function. The symbols used in Table 2 are the same as those used in Table 1 above, with the addition that:

[0119] - p(s'|s,a) is the probability assigned by the transition model to the next state s', i.e., the probability that the next state s' is the state that the environment transitions to as a result of the agent performing the current action a when in the current state s.

[0120]

[0121] Table 2

[0122] In both process 200 and process 300, the system maintains information sets. When the action space is large, when the space of possible constraints (or possible policy parameters) is large, or when the amount of computational resources available to perform the training process is limited, the system can avoid using multiple information sets and instead use a single function approximator for estimating observation-action pairs. In these cases, the system can use updates for enforcing local consistency to mitigate the effects of paranoia bias on the training process.

[0123] Figure 4 is a flow chart of an example process 400 for learning a control policy by using locally consistent backups. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed reinforcement learning system, such as Figure 1A The reinforcement learning system 100 is capable of performing process 400.

[0124] The system receives a batch of training tuples (step 402). As described above, each training tuple includes a current observation representing the current state of the environment, a current action performed by the agent in response to the current observation, a next observation representing the next state of the environment, and a reward received as a result of the agent performing the current action.

[0125] The system selects a corresponding next action for each next observation in the batch (step 404). In particular, the system selects the corresponding next actions so that these next actions are locally consistent. More specifically, the system does not select the argmax action independently for each next observation as in conventional Q-learning, but selects the next action based on a criterion that ensures that the next actions are locally consistent with each other. The criterion specifies that (i) all next actions can be selected using the same control strategy, that is, there is a certain set of policy parameters, and the control strategy will select all next actions according to these parameters to respond to their corresponding next observations, and (ii) each next action must be consistent with the current action in the same tuple, that is, there is a certain set of policy parameters, and the control strategy will select the current action according to these parameters to respond to the current observation and select the next action to respond to the next observation. In other words, for each next observation, when only actions that meet both criteria (i) and criteria (ii) are considered, the system selects the argmax action as the next action, that is, any action that does not meet (i), (ii) or both is ignored.

[0126] The system calculates a corresponding target Q-value for each of the batch of training tuples (step 406). In particular, the system determines the target Q-value for each tuple from the reward in the tuple and the Q-value generated by the control policy for the next observation-next action pair, and then determines an update based on the target Q-value, where the next observation is the next observation in the tuple and the next action is the next action selected for the tuple in step 404. For example, the target Q-value can be the sum of the product of the reward and the discount factor and the Q-value generated by the control policy for the next observation-next action pair.

[0127] The system uses the target Q-values ​​for the training tuples to update the control policy parameters (step 408). In particular, the system computes supervised learning updates based on the error gradient (e.g., mean squared error) between the current Q-value for each tuple (i.e., the Q-value currently generated by the function approximator for the current observation-current action pair) and the target Q-value for the tuple.

[0128] This specification uses the term "configured" with respect to system and computer program components. For a system of one or more computers, being configured to perform a particular operation or action means that the system has software, firmware, hardware, or a combination thereof installed thereon that, when in operation, causes the system to perform the operation or action. For one or more computer programs, being configured to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0129] The embodiments of the subject matter and function operations described in this specification can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structure disclosed in this specification and its structural equivalents, or one or more combinations thereof. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, which are encoded on a tangible non-temporary storage medium for execution or control of its operation by a digital processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage matrix, a random or serial access memory device, or a combination of one or more thereof. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as an electrical, optical, or electromagnetic signal generated by a machine, which is generated to encode information, which is used to transmit to a suitable receiver device for execution by a data processing device.

[0130] The term "data processing equipment" refers to data processing hardware and covers all types of equipment, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The equipment can also be or further include special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits). In addition to hardware, the equipment can also optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0131] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for a computing environment. A program may but need not correspond to a file in a file system. A program can be stored in a file portion that stores other programs or data, for example, one or more scripts are stored in a markup language document, in a single file dedicated to the above program, or in multiple collaborative files, for example, files for storing one or more modules, subroutines or code portions. A computer program can be deployed to execute on a computer or on multiple computers located in one location or distributed across multiple locations and interconnected by a data communication network.

[0132] In this specification, the term "database" refers broadly to any collection of data: the data need not be structured in any particular way or at all, and the data can be stored on storage devices in one or more locations. Thus, for example, an index database can include multiple data collections, each of which can be organized and accessed differently.

[0133] Similarly, in this specification, the term "engine" refers generally to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0134] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. These processes and logic flows can also be performed by a dedicated logic circuit such as an FPGA or ASIC, or by a combination of a dedicated logic circuit and one or more programmed computers.

[0135] A computer suitable for executing a computer program can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or any other kind of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit, or incorporated into a special logic circuit. Generally, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or be operably coupled thereto to receive data from it, transmit data to it, or both. However, a computer does not necessarily have such a device. Moreover, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0136] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as built-in hard disks or removable disks; magneto-optical disks; and CDROM disks and DVD-ROM disks.

[0137] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointer device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending files to and receiving files from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. Moreover, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smart phone running a messaging application) and receiving response messages from the user.

[0138] The data processing device for the machine learning model can also, for example, include a dedicated hardware accelerator unit, a common and computationally intensive part for processing machine learning training or generation (i.e., inference) workloads.

[0139] Ability to implement and deploy machine learning models by using a machine learning framework (e.g., TensorFlow framework, Microsoft Cognitive Toolkit framework, Apache Singa framework, or Apache MXNet framework).

[0140] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, for example, as a data server, a middleware component, for example, an application server, or a front-end component, for example, a client computer with a graphical user interface, a web browser, or an App by which a user can interact with an implementation of the subject matter described in this specification, or any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, for example, a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), for example, the Internet.

[0141] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs that run on corresponding computers and have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a client device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at a user device, such as the result of a user interaction, can be received from the device on the server.

[0142] Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of what may be claimed, but should be understood as a description of features that may be directed to a specific embodiment of a specific invention. It is also possible to combine certain features described in this specification in the context of separate embodiments and implement them in a single embodiment. Conversely, it is also possible to implement various features described in the context of a single embodiment separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as acting in certain combinations, and even initially claimed as such, it is possible to delete one or more features from the claimed combination from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0143] Similarly, although operations are depicted and described in a particular order in the drawings and claims, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, but it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged as multiple software products.

[0144] Specific embodiments of the subject matter are described herein. Other embodiments fall within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for determining a control strategy for an agent interacting with an environment, wherein: The agent comprises at least one of a mechanical agent, a simulated version of an agent, and an electronic agent, and the method comprises: maintaining data defining a plurality of information sets, each information set corresponding to a respective policy constraint set and identifying a Q-value assigned by the control policy to an observation-action pair under the policy constraint set; receiving a current observation characterizing a current state of the environment, a current action performed by the agent in response to the current observation, a next observation characterizing a next state of the environment, and a reward received as a result of the agent performing the current action, wherein the observations characterizing the current state and / or next state of the environment include or are generated from sensor data and the current action includes an input to control the agent; Determining a policy-consistent backup for the control policy at the current observation-current action pair, including: identifying, for each of a plurality of actions in a set of possible actions that can be performed by the agent, a Q-value assigned by the control policy to a next observation-action pair of the control policy and justified by at least one of the information sets, wherein the set of possible actions includes inputs to control the agent; pruning from the identified Q-values ​​any Q-values ​​justified only by an information set that is not policy class consistent when a greedy policy operating under the policy constraint set would not select the current action in response to the current observation; and determining the policy-consistent backup from the reward and only the identified Q-values ​​that have not been pruned; and The policy-consistent backup is used to update the agent's control policy by using Q-learning.

2. The method according to claim 1, wherein: Updating the control policy of the agent using the policy-consistent backup using Q-learning includes updating the control policy using model-free Q-learning, and wherein determining the policy-consistent backup from the reward and only the unpruned identified Q-values ​​includes determining a Q-backup.

3. The method according to claim 1, wherein: The policy-consistent backup includes a corresponding backup for each information set used to prove the unpruned Q-value.

4. The method according to claim 3, wherein: The corresponding backup is based on (i) the reward and (ii) the Q-value that has not been pruned and is proven by the information set.

5. The method according to any one of claims 1 to 4, wherein: Information sets that are not policy class consistent are those that impose policy constraints that cause the control policy to not select the current action in response to the current observation.

6. A method of determining a control strategy for an agent interacting with an environment, wherein: The agent comprises at least one of a mechanical agent, a simulated version of an agent, and an electronic agent, and the method comprises: maintaining data defining a plurality of information sets, each information set corresponding to a respective policy constraint set and identifying a Q-value assigned by the control policy to an observation-action pair under the policy constraint set; receiving a current observation characterizing a current state of the environment, a current action performed by the agent in response to the current observation according to a current control policy, and a reward received as a result of the agent performing the current action, wherein the observation characterizing the current state and / or a next state of the environment comprises or is generated from sensor data and the current action comprises an input to control the agent; Determining a policy-consistent backup for the control policy at the current observation-current action pair, including: For each of multiple next states: identifying, for each of a plurality of actions in a set of possible actions that can be performed by the agent, a Q-value assigned by the control policy to a next observation-action pair of the control policy and justified by at least one of the information sets, wherein the next observation is an observation used to characterize the next state; and pruning from the identified Q-values ​​any Q-values ​​justified only by an information set that is not policy class consistent when a greedy policy operating under the policy constraint set would not select the current action in response to the current observation; and determining the policy-consistent backup from the reward and the identified Q-values ​​that are unpruned only for each of the next states; and The policy-consistent backup is used to update the agent's control policy by using Q-learning.

7. The method according to claim 6, further comprising: A dynamic transition model of the environment is maintained, wherein determining the policy-consistent backup from the reward and only the unpruned identified Q-values ​​comprises determining a Bellman backup using the reward and the unpruned identified Q-values ​​for the next state.

8. The method according to claim 7, wherein: The transition model maps the current observation and the current action to corresponding probabilities for each of the next states, and wherein determining a Bellman backup comprises determining a Bellman backup using the reward, the corresponding probabilities for the next state, and the identified Q-value that is not pruned for the next state.

9. The method according to claim 6, wherein: The control policy selects actions to be performed by the agent using a neural network, and wherein updating the control policy comprises training a corresponding neural network for each information set used to justify an unpruned Q-value.

10. The method according to claim 6, wherein: The control policy selects an action to be performed by the agent by using a linear function approximator, and wherein updating the control policy comprises updating a weight of a corresponding linear function approximator for each information set used to justify an unpruned Q-value.

11. The method according to claim 6, wherein: The control policy selects an action to be performed by the agent by using a tabular representation that maps observation-action pairs to Q-values, and wherein updating the control policy comprises updating the Q-value for the current observation-action pair in the corresponding tabular representation for each information set used to justify the unpruned Q-value.

12. The method according to any one of claims 6 to 11, wherein: said agent comprises said electronic agent; Observations characterizing a current state and / or a next state of the environment include or are generated from sensor data used to monitor a portion of a plant or service facility; The current action and / or set of possible actions include actions for controlling and / or imposing operating conditions on items of equipment in the plant or service facility.

13. One or more computer-readable storage media for storing instructions that, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the method according to any one of claims 1 to 12.

14. A system comprising one or more computers and one or more storage devices, the one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform corresponding operations of the method according to any one of claims 1 to 12.

Images