Sample-Efficient Reinforcement Learning

By using model set and combination strategies in reinforcement learning systems, we can deal with dynamic model uncertainty in complex dynamic environments, and solve the problems of low efficiency of learning strategies and proxy wear in the existing technology, and realize efficient strategy learning and Computational optimization.

CN112119404BActive Publication Date: 2024-11-19GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980032627.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-05-18
Filing Date
2019-05-20
Publication Date
2024-11-19
Estimated Expiration
2039-05-20

AI Technical Summary

Technical Problem

In complex dynamic environments, existing model-based reinforcement learning schemes are difficult to learn high-performance strategies, and due to the deviation and uncertainty of the dynamic model, errors and sample efficiency in the learning process are low, increasing the wear of the agent and Calculate overhead.

Method used

Using a ensemble and combinatorial strategy of the model, taking into account uncertainty and Q network bias of the dynamic model through the target output of multiple trajectories, reducing the number of samples required for learning, and allowing for less accurate but computationally efficient models.

Benefits of technology

Efficient learning of effective strategies in complex environments, reducing sample size and calculation overhead, reducing agent wear and training time, suitable for continuous control tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112119404B_ABST
    Figure CN112119404B_ABST
Patent Text Reader

Abstract

Methods, systems and apparatus for sample efficient reinforcement learning, including computer programs encoded on computer storage media. One of these methods includes: maintaining a set of Q networks, a set of transition models and a set of reward models; obtaining transitions; using the set of transition models to generate M trajectories; for each time step in each of the trajectories: using the set of reward models to generate N rewards for the time step, using the set of Q networks to generate L Q values ​​for the time step, and determining L*N candidate target Q values ​​for the trajectory and for the time step based on the rewards, Q values ​​and training rewards; for each of the time steps, combining the candidate target Q values; determining a final target Q value; and using the final target Q value to train at least one of the Q networks in the set.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] The present application relates to reinforcement learning, and more particularly to reinforcement learning for agents interacting with real-world environments.

[0002] In a reinforcement learning system, an agent interacts with an environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

[0003] Some reinforcement learning systems select an action to be performed by the agent in response to receiving a given observation based on the output of a neural network.

[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input according to the current values ​​of the corresponding set of parameters. Summary of the invention

[0005] The present application generally describes a reinforcement learning system for training a Q-neural network. In some cases, the Q-neural network is used to train a policy neural network, which is then used to select actions to be performed by a reinforcement learning agent (e.g., a robot or other mechanical agent) interacting with a physical real-world environment. In other cases, the Q-neural network is used directly to select actions to be performed by the reinforcement learning agent interacting with the environment.

[0006] During training, the system maintains an ensemble of Q-networks, each Q-network being a neural network configured to: receive a Q-network input comprising (i) an input observation characterizing a state of an environment interacted with by the agent, and (ii) data identifying an action from a set of actions, and process the Q-network input to generate a Q-value for an input observation-action pair.

[0007] The system also maintains a collection of transition models, each of which is configured to: receive a transition input, the transition input comprising (i) an input observation and (ii) an action performed by the agent in response to the input observation, and process the transition input to generate a predicted next observation, the predicted next observation being used to characterize the state to which the environment transitions as a result of the agent performing the action in response to the observation.

[0008] The system also maintains a collection of reward models, each reward model being configured to: receive a reward input comprising (i) an input observation, (ii) an action performed by the agent in response to the input observation, and (iii) a next observation representing a state to which the environment transitions as a result of the agent performing the action in response to the observation, and process the reward input to generate a predicted reward received by the agent in response to performing the action.

[0009] The system then uses the set of Q-networks, transition models, and reward models to generate target Q-values ​​for the transitions, and then uses those target Q-values ​​to train the set of Q-networks. In particular, the system generates multiple different trajectories from a single transition, and then interpolates between target Q-values ​​from multiple different time steps within the multiple trajectories to determine a final target Q-value for the transition.

[0010] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0011] Model-based reinforcement learning schemes use a model of the environment's dynamics to predict the future state of the environment during the training of a policy neural network. These schemes can produce high-performance strategies while greatly reducing the complexity and time required to learn the strategy. However, in an environment with complex dynamics, for example, in a real-world environment interacted with by a robot or other mechanical agent, the predictions made by the dynamics model will be prone to error and introduce biases into the learning process. This often causes existing schemes using dynamics models to fail to learn high-performance strategies. However, the described technology takes into account the biases and uncertainties in the dynamics model that appear in complex environments, allowing effective strategies to be learned with much higher sample efficiency even for very complex tasks. In particular, by adopting a collection of models and combining target outputs from both trajectories of different lengths and multiple trajectories of the same length, the described technology can take into account the uncertainty of the dynamics model and the Q-network bias, while still reducing the number of actual samples required to learn effective strategies from the environment. This is particularly advantageous in the case where the agent is a robot or other mechanical agent interacting with a real environment, because collecting actual samples from the environment increases the wear of the agent, increases the chance of mechanical failure of the agent, and is very time-intensive. Additionally, even when it is possible to maintain a highly accurate model of the environment, the described techniques can allow for the use of less computationally intensive models that may be less accurate, thereby reducing the computational overhead of the training process. As a specific example, the described techniques can be used to learn in a computationally efficient manner effective policies for controlling an agent in a continuous control task, i.e., a task in which the space of actions to be performed by the agent is continuous.

[0012] The details of one or more embodiments of the subject matter described in this application are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 An example reinforcement learning system is shown.

[0014] Figure 2 An example of using a collection of trajectory models to generate a trajectory is shown.

[0015] Figure 3 is a flow chart of an example process for training an ensemble of Q networks.

[0016] Figure 4 is a flow chart of an example process for generating combined candidate target Q values.

[0017] Figure 5 is a flow chart of an example process for generating candidate target Q-values ​​for time steps in a trajectory.

[0018] Like reference numbers and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION

[0019] The present application describes a reinforcement learning system that controls an agent that interacts with an environment by processing, at each of a plurality of time steps, data characterizing the current state of the environment at that time step (i.e., an "observation") to select an action to be performed by the agent.

[0020] At each time step, the state of the environment at that time step depends on the state of the environment at the previous time step and the action performed by the agent at the previous time step.

[0021] In some embodiments, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle that navigates through the environment.

[0022] In these embodiments, the observations may include, for example, one or more of: images, object orientation data, and sensor data used to capture observations as the agent interacts with the environment, such as sensor data from images, range or orientation sensors, or from actuators.

[0023] For example, in the case of a robot, the observations may include data characterizing the current state of the robot, such as one or more of: joint orientation, joint velocity, joint force, torque or acceleration, such as gravity-compensated torque feedback, and a global or relative pose of an item held by the robot.

[0024] In the case of a robot or other mechanical agent or vehicle, the observations may similarly include one or more of the position, linear or angular velocity, forces, torques or accelerations, and global or relative pose of one or more parts of the agent. The observations may be defined in 1, 2, or 3 dimensions, and may be absolute and / or relative observations.

[0025] The observations may also include, for example, sensed electronic signals, such as motor currents or temperature signals; and / or image or video data, for example from a camera or LIDAR sensor, for example, data from sensors of the agent or from sensors positioned separately from the agent in the environment.

[0026] In the case of an electronic agent, the observations may include data from one or more sensors used to monitor a portion of a plant or service facility, such as current, voltage, power, temperature, and other sensor and / or electronic signals representing the functionality of electronic and / or mechanical items of equipment.

[0027] In these embodiments, the action can be a control input for controlling the robot, such as a torque or a higher-level control command for a joint of the robot, or a control input for controlling an autonomous or semi-autonomous land, air, or sea vehicle, such as a torque or a higher-level control command to a control surface or other control element of the vehicle.

[0028] In other words, the actions can include, for example, the position, velocity, or force / torque / acceleration data of one or more joints of a robot or a part of another mechanical agent. The action data may additionally or alternatively include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment, the control of which has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land or air or sea vehicle, the actions may include actions for controlling navigation (e.g., steering) and movement (e.g., braking and / or acceleration of the vehicle).

[0029] In certain other applications, the agent may control actions in a real-world environment including items of equipment, such as in a data center, in a power / water distribution system, or in a manufacturing plant or service facility. The observations may then be related to the operation of the plant or facility. For example, the observations may include observations of power or water use by equipment, or observations of power generation or distribution control, or observations of resource use or waste generation. The actions may include actions for controlling or imposing operating conditions on items of equipment of the plant / facility, and / or actions for causing changes in operating settings of the plant / facility, such as for adjusting or turning on / off components of the plant / facility.

[0030] Figure 1 An example reinforcement learning system 100 is shown. Reinforcement learning system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, or techniques described below are implemented.

[0031] The system 100 controls an agent 102 interacting with an environment 104 by selecting actions 106 to be performed by the agent 102 that cause the state of the environment to transition to a new state.

[0032] System 100 includes a policy neural network 110 , a training engine 116 , and a set of model parameters 118 for the policy neural network 110 .

[0033] At each of a plurality of time steps, the policy neural network 110 is configured to process inputs including a current observation 120 characterizing a current state of the environment 104 in accordance with model parameters 118 to generate an action selection output 122 (“action selection policy”).

[0034] The system 100 uses the action selection output 122 to select an action 106 to be performed by the agent at the current time step. Some examples of using the action selection output 122 to select an action 106 to be performed by the agent are described next.

[0035] In one example, the action selection output 122 can include a corresponding numerical probability value for each action in a set of possible actions that can be performed by the agent. The system 100 can select an action to be performed by the agent, for example, by sampling actions according to their probability values ​​or by selecting the action with the highest probability value.

[0036] In another example, the action selection output 122 may directly define an action to be performed by the robotic agent, for example by defining values ​​of torque that should be applied to a joint of the agent.

[0037] In another example, the action selection output 122 can include a corresponding Q-value for each action in a set of possible actions that can be performed by the agent. The system 100 can process the Q-values ​​(e.g., using a soft-max function) to generate corresponding probability values ​​for each possible action, which can be used to select an action to be performed by the agent (as described earlier). The system 100 can also select the action with the highest Q-value as the action to be performed by the agent.

[0038] The Q-value for an action is an estimate of the "return" that would result from the agent performing the action in response to the current observation 120, and then selecting future actions performed by the agent 102 in accordance with the current values ​​of the policy neural network parameters.

[0039] Return refers to a cumulative measure of "rewards" 124 received by the agent, e.g., a time-discounted sum of rewards. The agent can receive a corresponding reward 124 at each time step, where the reward 124 is specified by a scalar value and represents, for example, the agent's progress toward completing an assigned task.

[0040] In some cases, the system 100 can select an action to be performed by the agent according to an exploration strategy. For example, the exploration strategy can be a "∈-greedy exploration" strategy, in which the system 100 selects an action to be performed by the agent according to the action selection output 122 with probability 1-∈, and randomly selects the action with probability ∈. In this example, ∈ is a scalar value between 0 and 1.

[0041] The policy neural network 110 can have any of a variety of neural network architectures. For example, the architecture of the policy neural network 110 can include a sequence of one or more convolutional layers, followed by a recurrent layer (e.g., a long short-term memory (LSTM) layer) and an output layer for generating the action selection policy. As another example, the architecture can be that of a feedforward neural network, e.g., with a stack of four, six, or eight fully connected layers.

[0042] The training engine 116 is configured to train the policy neural network 110 by repeatedly updating model parameters 118 of the policy neural network 110 based on the agent's interactions with the environment.

[0043] In particular, during training, transitions generated as a result of the agent's interaction with the environment are stored in a transition buffer 114. Each transition includes an initial observation, an action performed by the agent in response to the initial observation, a reward, and a next observation characterizing the next state of the environment.

[0044] The training engine 116 can train the policy neural network 110 to increase the rewards (i.e., cumulative measures of rewards) received by the agent using reinforcement learning techniques by sampling transitions from the transition buffer 114. Because rewards measure the agent's progress in completing a task, training the policy neural network 110 to increase rewards results in the agent being able to successfully complete a designated task while being controlled by the policy neural network 110.

[0045] In particular, the training engine 116 maintains a set of L Q networks 160A-L, a set of M transition models 150A-M, and a set of N reward models 140A-N. Typically, L, M, and N are all fixed integer values ​​greater than one and may be the same value or may be different from each other.

[0046] Each Q-network in the set is a neural network configured to receive a Q-network input comprising (i) an input observation characterizing a state of an environment interacted with by the agent and (ii) data identifying an action from a set of actions, and to process the Q-network input to generate a Q-value for the input observation-action pair.

[0047] In some cases, the set of Q networks together is the policy neural network 110, i.e., the system 100 uses outputs generated by the Q networks in the set to control the agent, and the model parameters 118 are parameters of the set of Q networks.

[0048] In other cases, the system uses the collection of Q networks when updating the model parameters 118 of the separate policy neural network 110.

[0049] As a specific example, the policy neural network 110 can generate an action selection output 122 that directly defines an action to be performed by the robotic agent, such as by defining the value of the torque that should be applied to the joints of the agent, and the training engine 116 can use the Q-network 160 to apply a policy gradient reinforcement learning technique (e.g., a deep deterministic policy gradient (DDPG) reinforcement learning technique) to update the model parameters 118 of the policy neural network 110. In other words, the training engine 116 can use the policy gradient technique to train the Q-network jointly with the policy neural network 110. This approach can be employed when the action space is continuous.

[0050] As another specific example, the policy neural network 110 can generate an action selection output 122 for defining a probability distribution over possible actions to be performed by the agent, and the training engine 116 can use the Q-network 160 to apply an actor-critic reinforcement learning technique (e.g., an asynchronous advantage actor-critic (A3C) reinforcement learning technique) to update the model parameters 118 of the policy neural network 110. In other words, the training engine 116 can train the Q-network jointly with the policy neural network 110 using an actor-critic technique.

[0051] Typically, the system independently initializes parameter values ​​for each Q network in the set at the start of training, and, as described below, trains each Q network on different training data so that the values ​​of the parameters of the Q networks will all be different during the training.

[0052] Each transition model in the set is configured to: receive a transition input, the transition input comprising (i) an input observation and (ii) an action performed by the agent in response to the input observation; and process the transition input to generate a predicted next observation representing a state to which the environment transitions as a result of the agent performing the action in response to the observation. In other words, the transition model is configured to predict the effect of the agent performing the input action on the environment when the environment is in the state represented by the input observation.

[0053] Each reward model in the set is configured to: receive a reward input, the reward input comprising (i) an input observation, (ii) an action performed by the agent in response to the input observation, and (iii) a next observation characterizing a state to which the environment transitions as a result of the agent performing the action in response to the observation; and, process the reward input to generate a predicted reward received by the agent in response to performing the action.

[0054] The Q network, transition model, and reward model can have any of a variety of model architectures. For example, all of the models in the set can be neural networks. The architecture of any or all of these neural networks can include a sequence of one or more convolutional layers, followed by a recursive layer (e.g., a long short-term memory (LSTM) layer) and an output layer for generating appropriate outputs. As another example, the architecture can be that of a feedforward neural network, for example, a stack with four, six, or eight fully connected layers. In another example, the Q network can be a neural network with one of the above architectures, while the reward model, transition model, or both are other types of machine learning models, such as random forests, support vector machines, decision tree-based models, and the like.

[0055] The training engine 116 uses the set of Q networks, transition models, and reward models to generate target Q values ​​for transitions sampled from the transition buffer 114, and then uses those target Q values ​​to train the set of Q networks that collectively make up the policy neural network 110 or are used to repeatedly update model parameters of the policy neural network 110, i.e., using one of the techniques described above.

[0056] In particular, the training engine 116 generates a plurality of different trajectories using the policy neural network 110 and the set of trajectory models while sampling transitions from the transition buffer 114 .

[0057] That is, the engine 116 generates multiple different trajectories from a single transition.

[0058] Figure 2 An example of using a collection of trajectory models to generate a trajectory is shown.

[0059] exist Figure 2 In the example, a transition 210 is received, which includes an initial training observation s for characterizing the state of the environment, a training action a performed by the agent in response to the initial observation s, a next training observation s' for characterizing the state to which the environment transitions after the agent performs action a in response to the observation s, and a training reward r received in response to the agent performing the action a.

[0060] To train the set of Q networks using the transitions, the system generates trajectory data 220 using the set of transition models and the transitions 210 .

[0061] exist Figure 2 In the example of , the trajectory data includes three trajectories 230 , 240 , and 250 , each of which is generated using a transition model different from the set of transition models.

[0062] Each trajectory 230, 240, and 250 includes observations at the same predetermined number of time steps, i.e., each trajectory is of the same length. Figure 2 In the example of , each trajectory includes the next training observation s' at time step 0, and then three more predicted observations at time steps 1, 2, and 3.

[0063] In particular, to generate the next observation in a given trajectory after the first observation s', the system uses a control policy (i.e., using the action selection policy described above) to select an action. If the policy is random, then action a 0,1 、a0,2 and a 0,3 can be different. If the policy is deterministic, then given that each trajectory starts from the same next training observation, the action a 0,1 、a 0,2 and a 0,3 will all be the same.

[0064] The system then processes the transition input including the next observation and the selected action using the transition model corresponding to the trajectory to generate a predicted observation, i.e., observation s 1,1 , Observation results 1,2 Or observations 1,3 Since the transition models will typically have different parameter values, even if action a 0,1 、a 0,2 and a 0,3 are the same, the observations 1,1 , Observation results 1,2 Or observations 1,3 will also usually all be different.

[0065] Taking trajectory 240 as an example, the system then uses the control strategy again and takes the observations s 1,1 Start selecting action a 1,1 And using the transition model corresponding to trajectory 240 to generate a predicted observation s 2,1 The system then predicts the observation s from 2,1 Start repeating this process to select action a 2,1 and generate the predicted observations s 3,1 In other words, the system continues to generate new observations for each trajectory until each trajectory includes corresponding observations at a predetermined number of time steps.

[0066] The system also generates a reward for each time step in each trajectory. Figure 2 Only a single reward per time step and per trajectory is shown, e.g., for ease of illustration, the reward r for time step 2 in trajectory 240 is 2,1 , but the system actually generates N rewards per time step in each trajectory, that is, one reward for each of the N reward models in the set of reward models.

[0067] Although in Figure 2 Not shown in , but the system also generates L Q-values ​​per time step and per trajectory, ie, one Q-value for each of the L Q-networks in the set of Q-networks.

[0068] The system then calculates a target Q-value for the transition 210 using the L Q-values ​​and N rewards for each of the time steps in each of the trajectories 240 , 250 , and 260 , and then uses the target Q-value to train one or more of the Q-networks in the set.

[0069] Typically, the system computes target Q-values ​​by dynamically interpolating between the target Q-values ​​from various different time steps within multiple trajectories, i.e., dynamically interpolating between trajectories of various horizontal lengths for each individual transition. In doing so, the system ensures that the training process maintains low sampling complexity and other benefits of model-based reinforcement learning, while also ensuring that using the transition model and reward model to predict future states of the environment does not introduce significant error into the target Q-values, despite the fact that the transition model and reward model are inherently imperfect predictors. In other words, the system ensures that errors caused by the transition model and reward model do not prevent the system from learning an effective control policy for the agent.

[0070] Reference below Figure 3-5 Describe this training in more detail.

[0071] Figure 3 is a flow chart of an example process 300 for training a collection of Q networks. For convenience, process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed reinforcement learning system (e.g. Figure 1 The reinforcement learning system 100) is capable of performing the process 300.

[0072] The system can perform the process 300 for each transition in a minimal batch of transitions sampled from a memory storing transitions generated during the agent's interaction with the environment.

[0073] By repeatedly performing process 300 for mini-batches, the system trains an ensemble of Q-networks so that the ensemble can be used to effectively control the agent (when the Q-networks are used directly to control the agent) or to improve the training of a policy neural network that is being trained jointly with the ensemble of Q-networks.

[0074] The system obtains a transition (step 302). The transition includes (i) an initial training observation, (ii) a training action, (iii) a training reward, and (iv) a next training observation. As described above, the training action is an action performed by the agent in response to the initial observation, the training reward is a reward received in response to the agent performing the training action, and the next training observation represents the state to which the environment transitions after the agent performs the training action.

[0075] The system generates M trajectories using the set of transition models (step 304). That is, the system generates a corresponding trajectory using each of the M transition models in the set. Each of the M trajectories starts from the next training observation in the transition and includes a corresponding observation at each time step in a predetermined number of time steps.

[0076] In particular, to generate trajectories using a transition model, the system uses some action selection strategy to select an action to be performed in response to the next observation.

[0077] The system then processes a transition input comprising (i) a next observation and (ii) data identifying a selected action using a transition model to generate a predicted next observation characterizing a state to which the environment transitions as a result of the agent performing the selected action in response to the next observation.

[0078] The system can continue selecting actions and then using the transition model to generate predicted observations until the trajectory includes an observation at each of a predetermined number of time steps.

[0079] In some embodiments, the action selection strategy employed by the system is to select each action using the policy neural network that is trained using an ensemble of Q networks.

[0080] In some other embodiments, the action selection strategy employed by the system is to select each action using: (i) a corresponding one of the Q networks, (ii) a Q network having parameter values ​​that are an older copy of the parameters of the Q networks in the set, (iii) an average of the outputs of the Q networks in the set. In these embodiments, the system is capable of selecting the action that has the highest Q value when processed along with the observation.

[0081] The system generates L*N candidate target Q values ​​for each time step in each trajectory (step 306). In particular, for a given time step in a trajectory, the system uses each possible combination of L Q networks and N reward models to generate a corresponding target Q value, resulting in L*N candidate target Q values ​​generated for a given time step in a given trajectory. Figure 5 Generating candidate target Q values ​​is described in more detail.

[0082] The system combines the M*L*N candidate target Q values ​​for the time step from the M trajectories to generate a combined candidate target Q value for the time step (step 308). That is, the system combines the M*L*N candidate target Q values ​​for the time step across all M trajectories to generate a combined candidate target Q value for the time step. Figure 4 Generating combined candidate target Q-values ​​for a time step is described in more detail.

[0083] The system determines a final target Q value for the transition from the combined target Q values ​​for the time steps (step 310). For example, the system can calculate the sum or average of the combined target Q values ​​for the time steps to generate the final target Q value.

[0084] The system trains at least one of the Q networks using the final target Q value (step 312).

[0085] As a specific example, each transition in the mini-batch can be assigned to a corresponding one of the Q-networks in the set. That is, the transitions in the mini-batch can be split into random partitions, where each partition is assigned to a corresponding one of the Q-networks in the set. In this example, the system trains the Q-networks to which the transitions are assigned using the final target Q-value, and by performing the process 300 for all transitions in the mini-batch, the system trains all Q-networks in the set.

[0086] To train the Q network using the final target Q value, the system uses the Q network to process the initial observation and the initial action in the transition to generate a Q value, and then determines the gradient of the error (e.g., the square of the difference) between the Q value and the final target Q value with respect to the Q network parameters by back-propagation. The system can then use the calculated gradients to update the values ​​of the Q network parameters for all transitions assigned to the Q network in a mini-batch, for example, in accordance with an update rule for an optimizer (e.g., an Adam update rule, an rmsProp update rule, or a stochastic gradient descent update rule).

[0087] In some cases, the system uses an older copy of the parameters of the Q network to perform the steps necessary to generate a final target Q value. That is, the system maintains current and older values ​​for the parameters of the Q networks in the set, and only periodically updates the older values ​​to match the current values ​​during the training. The system can use current values ​​when updating the policy neural network, and use older values ​​when generating target Q values ​​for training the set of Q networks.

[0088] Figure 4is a flow chart of an example process 400 for generating a combined candidate target Q value. For convenience, the process 400 is described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed reinforcement learning system, such as Figure 1 The reinforcement learning system 100 is capable of performing the process 400.

[0089] The system determines a measure of central tendency of the M*L*N candidate target Q values ​​for a time step (step 402). In other words, the system calculates a measure of central tendency of the M*L*N candidate target Q values ​​for a time step across all M trajectories, such as a mean or median.

[0090] The system determines a measure of variation of the M*L*N candidate target Q values ​​for a time step (step 404). In other words, the system calculates a measure of variation of the M*L*N candidate target Q values ​​for a time step across all M trajectories, such as variance, standard deviation, or numerically stable standard deviation.

[0091] The system generates a combined candidate target Q value for the time step based on the measure of central tendency and the measure of variation for the time step (step 406). In particular, the combined candidate target Q value can satisfy:

[0092]

[0093] in, is a measure of central tendency, is a measure of change, and j ranges over all time steps in the trajectory.

[0094] Figure 5 is a flow chart of an example process 500 for generating candidate target Q values ​​for time steps in a trajectory. For convenience, the process 500 is described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed reinforcement learning system, such as Figure 1 The reinforcement learning system 100 is capable of performing the process 500.

[0095] The system can perform the process 500 for each time step in each trajectory to generate L*N candidate target Q values ​​for each time step in each trajectory.

[0096] The system generates N rewards for a time step using a set of reward models (step 502). In particular, the system uses each of the N reward models in the set to process a reward input including an observation at a previous time step, an action performed in response to the observation at the previous time step, and an observation at the time step to generate N rewards for the time step.

[0097] The system generates L Q-values ​​for a time step using a collection of Q-networks (step 504). In particular, the system processes a Q-network input comprising (i) an observation at the time step and (ii) data identifying an action from an action set to be performed in response to the observation at the time step, and processes the Q-network input using each of the L Q-networks to generate the L Q-values.

[0098] The system determines L*N candidate target Q values ​​for a time step based on the N rewards, L Q values, and training rewards (step 506).

[0099] To generate a candidate target Q-value for a time step, the system computes a corresponding model-based value extended target for each combination of (i) a reward model from N reward models and (ii) a Q network from L Q networks. In particular, the candidate target Q-value for a given time step H in a given trajectory according to reward model n and Q network l satisfies:

[0100]

[0101] Where r is the training reward, D i is the probability that the trajectory has not reached the terminal state since time step i, i.e., the state to which the environment is transformed when the task has been successfully completed or some other termination criterion has been satisfied, γ is a fixed discount factor between zero and one, and r n (s i-1 , a i-1 ,s i ) is the reward generated by reward model n for time step i, and Q l (s H , a H ) is the Q value for the time step H.

[0102] The system can calculate the probability D in any of a variety of ways. iFor example, the system can access data identifying certain states as terminal states. The system can use this data to determine whether any given state in the trajectory is a terminal state, and can assign a probability of one to each state in the trajectory before a first state being a terminal state, and a probability of zero to the first terminal state and each state in the trajectory after the first terminal state.

[0103] As another example, the system can maintain a termination model that receives observations as input and returns the probability that the state represented by the observations is a terminal state. In this example, the system can calculate Where d(s j ) is the output of the termination model for the observation at time step j.

[0104] Because the system uses every possible combination of reward model and Q-network to compute the candidate target Q-values ​​as described above, the result is L*N candidate target Q-values ​​for each time step in a given trajectory.

[0105] In some cases, the parameter values ​​of the reward model and transition model are fixed, i.e., the reward model and transition model are pre-trained before training the ensemble of Q networks. In other cases, the system trains each reward model and each transition model on a subset of transitions in a mini-batch, like the Q networks in the ensemble.

[0106] In particular, the system can use supervised learning to train the reward model, transition model, and termination model (when used) to minimize the following loss function:

[0107] E(||T(s,a)-s′|| 2 +H(de(s′),d(T(s,a)))+(r(s,a,s′)-r) 2 ),

[0108] where E is the expectation function, T(s,a) is the predicted observation by processing the initial observation and the initial action in the transition, s′ is the next observation in the transition, H is the cross entropy function, and de(s′) is an indicator function that returns 1 when the next observation represents a terminal state and 0 otherwise.

[0109] The above description indicates that a collection of models is used to generate uncertainty and variation between different trajectories (i.e., different transitions generated by the transition model, different rewards generated by the reward model, and different Q values ​​generated by the Q network). However, it should be understood that other techniques for incorporating uncertainty into predictions made by machine learning models can replace any or all of the collections in this specification. For example, one or more of these collections can be replaced with a single Bayesian neural network instead, and the system can sample multiple times from the same Bayesian neural network to generate outputs described as being generated by the collection. As another example, one or more of these collections can be replaced with a single neural network instead, and the system can use Dropout to discard, i.e., disable, different sets of nodes in the neural network to generate the necessary multiple outputs.

[0110] Although the subject technology has been described primarily in the context of a physical real-world environment, it is understood that the technology described herein may also be utilized with non-real-world environments. For example, in some embodiments, the environment may be a simulated environment, and the agent may be implemented as one or more computers interacting with the simulated environment.

[0111] The simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle that navigates through the motion simulation. In these embodiments, the action may be a control input for controlling a simulated user or a simulated vehicle.

[0112] In another example, the simulated environment may be a video game, and the agent may be a simulated user playing the video game. Typically, in the case of a simulated environment, the observations may include simulated versions of one or more previously described observations or types of observations, and the actions may include simulated versions of one or more previously described actions or types of actions.

[0113] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0114] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-temporary storage medium to be executed by a data processing device or to control the operation of the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage matrix, a random or serial access storage device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device to be executed by a data processing device.

[0115] The term "data processing apparatus" refers to data processing hardware, and includes various devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0116] A computer program may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may also be referred to or described as a program, software, software application, application, module, software module, script, or code; it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file used to store other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one or more computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0117] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and may be stored in a storage device at one or more locations. Thus, for example, an index database may include multiple data collections, each of which may be organized and accessed differently.

[0118] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0119] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit (e.g., an FPGA or ASIC), or by a combination of a dedicated logic circuit and one or more programmed computers.

[0120] A computer suitable for executing a computer program may be based on a general or special microprocessor or both, or on a central processing unit of any other kind. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for exercising or executing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory may be supplemented by or incorporated into a dedicated logic circuit. Typically, a computer will also include one or more mass storage devices (e.g., magnetic, magneto-optical disks or optical disks) for storing data, or may be operably coupled to one or more mass storage devices (e.g., magnetic, magneto-optical disks or optical disks) to receive data from the one or more mass storage devices or to transfer data to the one or more mass storage devices, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0121] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM optical disks.

[0122] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user; and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. Moreover, a computer may interact with a user by sending a text message or other form of message to a personal device (e.g., a smartphone running a messaging application) and then receiving a response message from the user.

[0123] A data processing device for implementing a machine learning model may also include, for example, a dedicated hardware accelerator unit for processing common and computationally intensive portions of machine learning training or production (i.e., inference, workloads).

[0124] The machine learning model can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).

[0125] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification), or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0126] The computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data to the user device, such as an HTML page, for example, for displaying data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, such as the result of the user interaction, can be received from the device at the server.

[0127] Although this specification contains many specific implementation details, these details should not be interpreted as limiting the scope of any invention or the scope of the content claimed, but should be interpreted as a description of the features of a specific embodiment specific to a specific invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments, or in any suitable sub-combination. Moreover, although the features may be described above as working in certain combinations and even initially claimed, in some cases, one or more features in the claimed combination may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variant of a sub-combination.

[0128] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed, to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product, or packaged into multiple software products.

[0129] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

[0130] The claims are as stated above.

Claims

1. A computer-implemented method comprising: Maintaining a set of Q networks, a set of transition models, and a set of reward models, wherein each Q network in the set of Q networks is a neural network, and each Q network is configured to: receive a Q network input, the Q network input comprising (i) an input observation characterizing a state of an environment interacting with the agent and (ii) data identifying an action from a set of actions, and process the Q network input to generate a Q value for the input observation-action pair, and each transition model in the set of transition models is configured to: receive a transition input, the transition input comprising (i) an input observation and (ii) an action performed by the agent in response to the input observation, and process the transition input to generate a predicted next observation characterizing a state to which the environment transitions as a result of the agent performing the action in response to the observation, and each reward model in the set of reward models is configured to: receive a reward input comprising (i) an input observation, (ii) an action performed by the agent in response to the input observation, and (iii) a next observation characterizing a state to which the environment transitions as a result of the agent performing the action in response to the observation, and process the reward input to generate a predicted reward received by the agent in response to performing the action, and wherein: The environment is a real-world environment, the agent is a mechanical agent configured to interact with the real-world environment, and the input observations to each Q-network, the input observations to each transition model, and the input observations to each reward model each include one or more of: an image of the environment, object orientation data describing the orientation of one or more objects in the environment, or sensor data for capturing observations as the agent interacts with the environment, or The environment is a simulated environment, the agent is implemented as one or more computers that interact with the simulated environment, and the input observations of each Q network, the input observations of each transition model, and the input observations of each reward model each include a simulated version of one or more of: an image of the environment, object position data describing the position of one or more objects in the environment, or sensor data for capturing observations as the agent interacts with the environment; obtaining a transition comprising (i) an initial training observation, (ii) a training action, (iii) a training reward, and (iv) a next training observation; generating, using the set of transition models, M trajectories starting from the next training observation in the transition, each of the M trajectories including a corresponding observation at each time step of a predetermined number of time steps; For each of the M trajectories and for each time step of the predetermined number of time steps: Using the ensemble of reward models to generate N rewards for the time step, using the ensemble of Q networks to generate L Q values ​​for the time step, and Determine L*N candidate target Q values ​​for the trajectory and for the time step based on the N rewards, the L Q values, and the training rewards; for each time step in the predetermined number of time steps, combining the M*L*N candidate target Q-values ​​for the time step from the M trajectories to generate a combined candidate target Q-value for the time step; determining a final target Q value based on the combined candidate target Q values ​​for the predetermined number of time steps; and At least one Q network in the set of Q networks is trained using the final target Q value.

2. The method according to claim 1, wherein: The transition is one of a transition in a minimal batch of transitions, wherein each transition in the minimal batch is assigned to a corresponding one of the set of Q networks, and wherein training at least one Q network in the set of Q networks using the final target Q value comprises training the Q network to which the transition is assigned using the final target Q value.

3. The method according to claim 1, further comprising: At least one of the set of transition models and at least one of the set of reward models are trained on the transition.

4. The method according to claim 1, further comprising: The policy neural network is trained using the ensemble of Q networks.

5. The method according to claim 4, wherein: Using the ensemble of transition models to generate M trajectories starting from the next training observation in the transition includes: for each trajectory and at each time step, using the policy neural network to select the action at the time step.

6. The method according to claim 1, wherein: Using the set of transition models to generate M trajectories starting from the next training observation in the transition includes: for each trajectory and at each time step, selecting the action at the time step using: (i) a corresponding one of the set of Q networks, or (ii) a Q network with parameter values ​​that are an older copy of the parameters in the set of Q networks.

7. The method according to claim 1, wherein: For each time step in the predetermined number of time steps, combining the M*L*N candidate target Q-values ​​for the time step from the M trajectories to generate a combined candidate target Q-value for the time step comprises: Determining a measure of central tendency of the M*L*N candidate target Q values ​​for the time step; Determining a measure of change in the M*L*N candidate target Q values ​​for the time step; and The combined candidate target Q-value is generated for the time step based on the measure of central tendency and the measure of variation.

8. The method according to claim 7, wherein: The combined candidate target Q value for the time step i satisfies: in is the measure of central tendency, is a measure of the variation, and j ranges over all the predetermined number of time steps.

9. The method according to claim 1, wherein: The observation at the first time step in each of the M trajectories is the next training observation.

10. The method according to claim 1, wherein: Determining L*N candidate target Q values ​​for the trajectory and for the time step according to the N rewards, the L Q values, and the training rewards includes: For each combination of (i) reward model and (ii) Q-network, determine the corresponding model-based value expansion target.

11. The method according to claim 1, wherein: The agent is configured to control the actions of an item of equipment in a factory or service facility, the observations being related to the operation of the factory or service facility, and the actions comprising actions for controlling or imposing operating conditions on the item of equipment in the factory or service facility and / or actions resulting in changes in operating settings of the factory or service facility.

12. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: Maintaining a set of Q networks, a set of transition models, and a set of reward models, wherein each Q network in the set of Q networks is a neural network, and each Q network is configured to: receive a Q network input, the Q network input comprising (i) an input observation characterizing a state of an environment interacting with the agent and (ii) data identifying an action from a set of actions, and process the Q network input to generate a Q value for the input observation-action pair, and each transition model in the set of transition models is configured to: receive a transition input, the transition input comprising (i) an input observation and (ii) an action performed by the agent in response to the input observation, and process the transition input to generate a Q value for the input observation-action pair. The reward model comprises a reward input including (i) an input observation, (ii) an action performed by the agent in response to the input observation, and (iii) a next observation representing the state to which the environment transitions as a result of the agent performing the action in response to the observation, and processing the reward input to generate a predicted reward received by the agent in response to performing the action, and wherein: The environment is a real-world environment, the agent is a mechanical agent configured to interact with the real-world environment, and the input observations to each Q-network, the input observations to each transition model, and the input observations to each reward model each include one or more of: an image of the environment, object orientation data describing the orientation of one or more objects in the environment, or sensor data for capturing observations as the agent interacts with the environment, or The environment is a simulated environment, the agent is implemented as one or more computers that interact with the simulated environment, and the input observations of each Q network, the input observations of each transition model, and the input observations of each reward model each include a simulated version of one or more of: an image of the environment, object position data describing the position of one or more objects in the environment, or sensor data for capturing observations as the agent interacts with the environment; obtaining a transition comprising (i) an initial training observation, (ii) a training action, (iii) a training reward, and (iv) a next training observation; generating, using the set of transition models, M trajectories starting from the next training observation in the transition, each of the M trajectories including a corresponding observation at each time step of a predetermined number of time steps; For each of the M trajectories and for each time step of the predetermined number of time steps: Using the ensemble of reward models to generate N rewards for the time step, using the ensemble of Q networks to generate L Q values ​​for the time step, and Determine L*N candidate target Q values ​​for the trajectory and for the time step based on the N rewards, the L Q values, and the training rewards; for each time step in the predetermined number of time steps, combining the M*L*N candidate target Q-values ​​for the time step from the M trajectories to generate a combined candidate target Q-value for the time step; determining a final target Q value based on the combined candidate target Q values ​​for the predetermined number of time steps; and At least one Q network in the set of Q networks is trained using the final target Q value.

13. A system comprising one or more computers and one or more storage devices storing instructions, wherein the instructions, when executed by the one or more computers, cause the one or more computers to perform operations comprising: Maintaining a set of Q networks, a set of transition models, and a set of reward models, wherein each Q network in the set of Q networks is a neural network, and each Q network is configured to: receive a Q network input, the Q network input comprising (i) an input observation characterizing a state of an environment interacting with the agent and (ii) data identifying an action from a set of actions, and process the Q network input to generate a Q value for the input observation-action pair, and each transition model in the set of transition models is configured to: receive a transition input, the transition input comprising (i) an input observation and (ii) an action performed by the agent in response to the input observation, and process the transition input to generate a Q value for the input observation-action pair. The reward model comprises a reward input including (i) an input observation, (ii) an action performed by the agent in response to the input observation, and (iii) a next observation representing the state to which the environment transitions as a result of the agent performing the action in response to the observation, and processing the reward input to generate a predicted reward received by the agent in response to performing the action, and wherein: The environment is a real-world environment, the agent is a mechanical agent configured to interact with the real-world environment, and the input observations to each Q-network, the input observations to each transition model, and the input observations to each reward model each include one or more of: an image of the environment, object orientation data describing the orientation of one or more objects in the environment, or sensor data for capturing observations as the agent interacts with the environment, or The environment is a simulated environment, the agent is implemented as one or more computers that interact with the simulated environment, and the input observations of each Q network, the input observations of each transition model, and the input observations of each reward model each include a simulated version of one or more of: an image of the environment, object position data describing the position of one or more objects in the environment, or sensor data for capturing observations as the agent interacts with the environment; obtaining a transition comprising (i) an initial training observation, (ii) a training action, (iii) a training reward, and (iv) a next training observation; generating, using the set of transition models, M trajectories starting from the next training observation in the transition, each of the M trajectories including a corresponding observation at each time step of a predetermined number of time steps; For each of the M trajectories and for each time step of the predetermined number of time steps: Using the ensemble of reward models to generate N rewards for the time step, using the ensemble of Q networks to generate L Q values ​​for the time step, and Determine L*N candidate target Q values ​​for the trajectory and for the time step based on the N rewards, the L Q values, and the training rewards; for each time step in the predetermined number of time steps, combining the M*L*N candidate target Q-values ​​for the time step from the M trajectories to generate a combined candidate target Q-value for the time step; determining a final target Q value based on the combined candidate target Q values ​​for the predetermined number of time steps; and At least one Q network in the set of Q networks is trained using the final target Q value.

14. The system according to claim 13, wherein: The transition is one of a transition in a minimal batch of transitions, wherein each transition in the minimal batch is assigned to a corresponding one of the set of Q networks, and wherein training at least one Q network in the set of Q networks using the final target Q value comprises training the Q network to which the transition is assigned using the final target Q value.

15. The system of claim 13, the operations further comprising: At least one of the set of transition models and at least one of the set of reward models are trained on the transition.

16. The system of claim 13, the operations further comprising: The policy neural network is trained using the ensemble of Q networks.

17. The system of claim 16, wherein: Using the ensemble of transition models to generate M trajectories starting from the next training observation in the transition includes: for each trajectory and at each time step, using the policy neural network to select the action at the time step.

18. The system of claim 13, wherein: Using the set of transition models to generate M trajectories starting from the next training observation in the transition includes: for each trajectory and at each time step, selecting the action at the time step using: (i) a corresponding one of the set of Q networks, or (ii) a Q network with parameter values ​​that are an older copy of the parameters in the set of Q networks.

19. The system of claim 13, wherein: For each time step in the predetermined number of time steps, combining the M*L*N candidate target Q-values ​​for the time step from the M trajectories to generate a combined candidate target Q-value for the time step comprises: Determining a measure of central tendency of the M*L*N candidate target Q values ​​for the time step; Determining a measure of change in the M*L*N candidate target Q values ​​for the time step; and The combined candidate target Q-value is generated for the time step based on the measure of central tendency and the measure of variation.