Meta-gradient updates of the function returned by training a reinforcement learning system
The meta-learning method updates the return function and policy parameters in the reinforcement learning system, and uses meta-objective function and gradient optimization to solve the problems of low training efficiency and overfitting, and achieve faster and more accurate policy parameter convergence and reward functions. train.
Patent Information
- Application Number
- CN201980033531.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-05-18
- Filing Date
- 2019-05-20
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2039-05-20
AI Technical Summary
The existing reinforcement learning system has problems of overfitting and low efficiency during the training process, making it difficult to quickly converge to the optimal strategy parameters, and is not effective enough for the training of reward functions.
The meta-learning method is adopted to update the return function and policy parameters through the meta-objective function and gradient rise or fall method, and iteratively update the parameters using different experience sets to avoid overfitting and improve training efficiency. The meta-parameters are used to adjust the return function to improve reward calculations.
This enables faster and more efficient training of reinforcement learning systems, and can converge to the optimal strategy parameters faster with less training data, improving the accuracy and efficiency of learning behavior.
Smart Images

Figure CN112292693B_ABST
Abstract
Description
Technical Field
[0001] This specification relates to reinforcement learning. Background Art
[0002] In a reinforcement learning system, an agent interacts with an environment by performing actions selected by the reinforcement learning system in response to receiving observations characterizing the current state of the environment.
[0003] Some reinforcement learning systems select actions to be performed by the agent based on the output of a neural network in response to receiving given observations.
[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values of the corresponding set of parameters. Summary of the invention
[0005] This specification generally describes a reinforcement learning system that selects actions to be performed by a reinforcement learning agent that interacts with an environment. In order for the agent to interact with the environment, the system receives data representing the current state of the environment and selects actions to be performed by the agent in response to the received data. The data representing the state of the environment is referred to as observation data in this specification.
[0006] In some implementations, the environment is a simulated environment, and the agent is implemented as one or more computer programs that interact with the simulated environment. For example, the simulated environment can be a video game, and the agent can be a simulated user playing the video game. As another example, the simulated environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent can be a simulated vehicle navigating through the motion simulation. In these implementations, the action can be a control input that controls the simulated user or the simulated vehicle.
[0007] In some other implementations, the environment is a real-world environment and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent can be a robot that interacts with the environment to complete a specific task. As another example, the agent can be an autonomous or semi-autonomous vehicle that navigates the environment. In these implementations, the action can be a control input that controls the robot or autonomous vehicle.
[0008] In one innovative aspect, the subject matter described in this specification can be implemented in a reinforcement learning system, the system comprising one or more processors configured to retrieve multiple experiences from a reinforcement learning neural network, the reinforcement learning neural network being configured to control an agent interacting with an environment to perform a task in an attempt to achieve a specified result based on one or more policy parameters of the reinforcement learning neural network, each experience comprising observation data representing an environmental state, an action performed by the agent in response to the observation data, and a reward received in response to the action. The one or more processors are also configured to update one or more policy parameters of the reinforcement learning neural network based on a first set of experiences using a return function calculated based on the reward, and to update one or more return parameters of the return function based on the one or more updated policy parameters and a second set of experiences. The one or more return parameters are updated via a gradient ascent or descent method using a meta-objective function differentiated relative to the one or more return parameters, wherein the meta-objective function depends on the one or more policy parameters.
[0009] The embodiments described herein apply meta-learning (particularly meta-gradient reinforcement learning) to learn an optimal return function G to improve the training of the system. This provides a more effective and efficient means of training a reinforcement learning system, because the system can converge to an optimal set of one or more policy parameters θ more quickly by gradually training the return function G. In particular, the return function G is made dependent on one or more policy parameters θ, and a meta-objective function J' that is differentiated with respect to one or more return parameters η is used to improve the training of the return function G.
[0010] Meta-learning can be thought of as the act of training a system to learn more efficiently. The meta-goal J' can therefore be thought of as a goal to improve the learning function of the reinforcement learning neural network. Specifically, the meta-objective function J' can be a function used to optimize the return parameter η of the reinforcement learning neural network. The meta-objective function J' can serve the purpose of identifying the return function that maximizes the overall performance in the agent. This can be directly measured by a meta-objective that focuses on optimizing the return (in other words, a policy gradient objective). For example, the meta-objective function can calculate the error (e.g., mean squared error) between the return function and the value function used by the agent to determine the action, and the system can be configured to update the return parameter to reduce (e.g., minimize) the error.
[0011] Retrieving experience τ may comprise the system generating experience (i.e. the reinforcement learning neural network may form part of the system) or, for example, accessing experience from a storage device or an external system. That is, the experience may be generated online by the reinforcement learning system itself, or may be obtained from an external reinforcement learning neural network. Thus, updating the policy parameters θ may comprise sending updated policy parameters θ′ to an external reinforcement neural network or updating a reinforcement neural network forming part of the overall system.
[0012] Based on various types of reinforcement learning, various types of return functions G can be used. For example, the return function G can be the return function in the stochastic gradient ascent method, or can be used as the target in the Q learning method.
[0013] The first experience set and the second experience set may be the same. Alternatively, the first experience set and the second experience set may include different experiences. Using different experiences for updating one or more return parameters for those experiences for updating one or more policy parameters (retaining training data for training the return function) improves training by avoiding overfitting.
[0014] Updating one or more return parameters may utilize the differential of one or more updated policy parameters relative to one or more return parameters. The differential of the meta-objective function J' relative to one or more return parameters η may utilize partial derivatives, dividing the differential into two components, the first component being the differential of the meta-objective function J' relative to one or more updated policy parameters θ', and the second component being the differential of one or more updated policy parameters θ' relative to one or more return parameters η. Therefore, this allows the system to utilize updated (improved) policy parameters θ' when updating one or more return parameters η, thereby improving the effectiveness of training.
[0015] The one or more policy parameters may be one or more parameters that define the functionality of the reinforcement learning neural network (one or more parameters that define the actions taken by the neural network). The one or more return parameters may be parameters that define how the return is determined based on the reward.
[0016] The one or more processors may also be configured to iteratively: retrieve updated experiences generated by the reinforcement neural network using the one or more updated policy parameters and the one or more updated return parameters; further update the one or more policy parameters based on the updated first experience set using the one or more updated return parameters; and further update the one or more return parameters based on the further updated policy parameters and the updated second experience set by a gradient ascent or descent method until an end condition is reached.
[0017] Thus, the system can iteratively update one or more policy parameters and one or more return parameters to converge to an optimal policy. By updating one or more return parameters during training of one or more policy parameters, the system is able to improve the calculated reward and can therefore train the policy more accurately and (more efficiently) use fewer training episodes.
[0018] Each time a policy parameter is updated, one or more of the return parameters can be updated. This provides a computationally simpler and more efficient mechanism for training the system.
[0019] Alternatively, one or more return parameters may remain fixed across multiple updates of one or more policy parameters. In this case, one or more return parameters may then be updated over time by back propagation.
[0020] Updating one or more return parameters may include applying a further return function as part of a meta-objective function, and evaluating the updated policy according to the return from the further return function when applied to a second set of experiences. This may include the goal of maximizing the total return from the further return function relative to the one or more return parameters. The further return function G' may be a function returned from the reward calculation based on one or more further return parameters η'. The further return function may be considered as a meta-return function with a meta-return parameter (or more generally, a meta-parameter). The further return function may be used as a means of training a reinforcement learning neural network to improve the return function G. The further return function may be different from the return function. The further return parameter may remain fixed during training.
[0021] Updates to one or more policy parameters may apply one or more of a policy function and a value function conditioned on one or more return parameters. Conditioning the policy and / or value function based on one or more return parameters makes the training method more stable. As the return function changes during training, the policy or value function may become invalid. This may cause a collapse of the value evaluation policy. By conditioning the policy and / or value function on one or more return parameters, the agent is forced to learn a common policy and / or value function for various sets of return parameters. This allows the method to freely shift meta-parameters without waiting for the approximator to "catch up".
[0022] The definition can be performed via embedding of one or more return parameters. Embedding can form one or more embedded return parameters by inputting one or more parameters into the embedding function. One or more embedded return parameters can use hidden (latent) variables to represent one or more parameters. Specifically, the policy function and the value function can be defined as:
[0023]
[0024] in:
[0025] V θ () is a value function, and is a conditional value function;
[0026] π θ () is a strategy, and It is a conditional strategy;
[0027] e η is an embedding of one or more return parameters η;
[0028] [S;e η ] represents the vector S (corresponding state) and e η concatenation; and
[0029] w η is an embedding function with η as input, where w η Can be learned via back-propagation in agent training.
[0030] The one or more parameters may include one or more of a discount factor for the return function and a bootstrapping factor for the return function. The return function may apply a discount factor y to provide a discounted return. The discounted return may include a weighted sum of rewards, where the discount factor defines the decay of the weighted sum. Optimizing the discount factor has been found to be a particularly effective method for improving the efficiency and accuracy of reinforcement learning. Similarly, the return function may apply a bootstrap factor λ to a geometrically weighted combination of returns (bootstrap parameter return function or λ return). The return function may compute a weighted combination of returns, each return being estimated over multiple steps (e.g., a decaying weighted sum of rewards). Optimizing the bootstrap factor (possibly in combination with the discount factor) results in more efficient and more accurate reinforcement learning.
[0031] One or more processors may also be configured to: update one or more policy parameters of the reinforcement learning neural network based on the second experience set; and update one or more return parameters of the return function based on the one or more updated policy parameters and the first experience set, wherein the one or more return parameters are updated via a gradient ascent or descent method. This improves the efficiency and effectiveness of training by repeating the update step but swapping the experience set used for each update. As described above, using different experience sets avoids overfitting. However, this reduces the amount of training data that can be used for each update. In order to improve data efficiency, the first experience set and the second experience set can be exchanged after the update, so that the second experience set can be used to train the policy, and the second experience set can be used to train the return function.
[0032] That is, the first experience set can be used to update one or more policy parameters, and the performance of the update can be verified by evaluating the meta-objective function using the second experience set. Then, the roles of the first experience set and the second experience set can be swapped so that the second experience set is used to update one or more policy parameters, and the performance of the update can be verified by evaluating the meta-objective function on the first experience set. In this way, the proposed method does not require any additional data other than the data used to train the one or more policy parameters in order to meta-learn updates to the one or more return parameters.
[0033] The meta-objective function of differentiation can be:
[0034]
[0035] in:
[0036] n is one or more return parameters; and
[0037] J′(τ′, θ′, η′) is a meta-objective function that is conditioned on a second set of experiences τ′, one or more updated policy parameters θ′, and one or more return parameters η′ of a further return function that forms part of the meta-objective function.
[0038] The above formula is equivalent to formula (2) described later in this specification.
[0039] The system can be configured to calculate a differential meta-objective function based on the differential dθ′ / dη of the updated policy parameter θ′ with respect to the return parameter η′, the differential dθ′ / dη being calculated by adding the differential df(τ, θ, η) / dη of the update function with respect to the return parameter to the differential dθ / dη of the policy parameter θ with respect to the return parameter η, the update function being used to update the policy.
[0040] The derivative dθ′ / dη of the updated policy parameter θ′ with respect to the return parameter η can be calculated using the cumulative trace as ≈dθ / dη, such that:
[0041]
[0042] in:
[0043] μ is the decay parameter; and
[0044] f(τ, θ, η) is the update function used to update the policy.
[0045] Update the derivative of the function with respect to the returned argument It can be calculated as follows:
[0046]
[0047] in:
[0048] α is the learning rate applied when updating one or more policy parameters from one or more previous policy parameters;
[0049] G η (τ) is based on one or more return parameters η t , the return function calculated from the first experience set τ;
[0050] π θ is the strategy used by the reinforcement learning neural network to determine the action Aa from the state S, and the strategy π θ Operate according to one or more policy parameters θ;
[0051] c is the coefficient; and
[0052] v θ (S) is a value function that determines the value of state S based on one or more policy parameters θ.
[0053] The above formula is equivalent to formula (13) described later in this specification.
[0054] In other words, the derivative of the updated strategy parameter with respect to the returned parameter can be calculated as follows:
[0055]
[0056] Updates to one or more return parameters can be made using:
[0057]
[0058] in:
[0059] G η′(τ′) is a further return function calculated and returned from a second experience set τ′ based on one or more further return parameters η′;
[0060] π θ′ is the strategy for the reinforcement learning neural network to determine the action A′ from the state S′ taken from the second experience set τ′. The strategy π θ′ Operate according to one or more updated policy parameters θ′;
[0061] v θ′ (S′) is a value function that determines the value of state S′ based on one or more updated policy parameters θ′.
[0062] The above formula is equivalent to formula (14) described later in this specification.
[0063] This will evaluate the updated policy according to the total return in the case of η′ when measured on the second experience set τ′.
[0064] One or more further return parameters may be kept fixed. This allows obtaining a gradient with respect to one or more return parameters.
[0065] Updating one or more return parameters may include calculating:
[0066]
[0067] in
[0068] η′ is one or more updated return parameters; and
[0069] β is a learning factor used to update one or more return parameters.
[0070] The above formula is equivalent to formulas (2) to (5) described later in this specification.
[0071] Thus, the update of the one or more returned parameters may be based on the gradient of the meta-objective function with respect to the one or more returned parameters by applying a gradient ascent method.
[0072] In one innovative aspect, the subject matter described in this specification can be embodied in a computer-implemented method for reinforcement learning. The method includes: retrieving multiple experiences from a reinforcement learning neural network, the reinforcement learning neural network is configured to control an agent interacting with an environment to perform a task to attempt to achieve a specified result based on one or more policy parameters of the reinforcement learning neural network, each experience including observation data representing the state of the environment, an action performed by the agent in response to the observation data, and a reward received in response to the action; using a return function returned based on the reward calculation, updating one or more policy parameters of the reinforcement learning neural network based on a first set of experiences; and updating one or more return parameters of the return function based on the one or more updated policy parameters and a second set of experiences, wherein the one or more return parameters are updated via a gradient ascent or descent method using a meta-objective function differentiated relative to the one or more return parameters, wherein the meta-objective function depends on the one or more policy parameters.
[0073] Another aspect of the present invention may be embodied in one or more computer storage media storing instructions which, when executed by one or more computers, cause the one or more computers to perform the operations of any of the methods described herein.
[0074] Certain novel aspects of the subject matter of this specification are set forth in the following claims.
[0075] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.
[0076] The implementation of the system facilitates improved training of a reinforcement learning system by applying meta-learning to train a reward function of the reinforcement learning system. The method described herein utilizes a meta-objective function that depends on one or more policy parameters of the system being trained. The meta-objective function is differentiated to obtain a meta-gradient that can be used to adapt the properties of the return online while interacting with and learning from the environment.
[0077] More specifically, the update to the return function is applied after one or more policy parameters are updated. The gradient of the policy parameter with respect to the one or more return parameters indicates how the meta-parameter affects the policy parameter(s). Thus, the method is able to measure the performance of the updated policy parameter(s) on a second experience sample to improve the parameter(s) of the return function.
[0078] This improves the performance of the system during training and allows the system to be trained more efficiently using a smaller amount of training data. For example, by learning an improved return function while the system is training, the system is able to converge to the optimal set of policy parameters faster using fewer updates. Due to the use of improved rewards during training, the final trained system also shows more accurate and effective learning behavior.
[0079] Implementations of the system can be trained online on a data stream, offline using stored data, or both. The system can automatically adapt its training to specific training tasks, learning to perform those tasks better. This helps automate the training process and enables implementations of the system to be used for a wider range of different tasks without having to be tuned or adapted for a specific task. The proposed method is applicable to almost all current reinforcement learning systems because the return is always used for agent updates and almost all current reinforcement learning updates include a differentiable function of the return. For example, this includes value-based methods (such as Q(λ)), policy gradient methods, or action-critic algorithms such as A3C (Advantage Actor Critic) proposed by Mnih et al. in 2016 or IMPALA (Importance-Weighted Actor-Learner) proposed by Espholt et al. in arXiv:1802.01561.
[0080] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0081] Figure 1 An example reinforcement learning system for training an agent to interact with an environment is shown.
[0082] Figure 2 A reinforcement learning system that applies meta-learning according to the present specification is shown.
[0083] Figure 3 Shows Figure 2 The process followed by the system to update policy parameters and meta-parameters based on multiple experiences retrieved.
[0084] Figure 4 The process of updating the meta parameters after updating the policy parameters is shown.
[0085] The same reference numbers and names in different drawings represent the same elements. DETAILED DESCRIPTION
[0086] The implementations described herein involve reinforcement learning systems.
[0087] In broad terms, a reinforcement learning system is a system that selects actions to be performed by a reinforcement learning agent that interacts with an environment. In order for the agent to interact with the environment, the system receives data representing the current state of the environment and selects actions to be performed by the agent in response to the received data. The data representing the state of the environment is referred to as observation data in this specification. Optionally, the observation data at a time step may include data from previous time steps, such as actions performed at previous time steps, rewards received at previous time steps, etc.
[0088] In some implementations, the environment is a real-world environment, and the agent is an electromechanical agent that interacts with the real-world environment. For example, the agent can be a robot or other stationary or moving machine that interacts with the environment to complete a specific task (e.g., locating an object of interest in the environment or moving an object of interest to a specified location in the environment or navigating to a specified destination in the environment); or the agent can be an autonomous or semi-autonomous land or air or sea vehicle that navigates in the environment.
[0089] In these implementations, the observation data may include, for example, one or more of images, object position data, and sensor data to capture the observation data when the agent interacts with the environment, such as sensor data from an image, a distance or position sensor, or from an actuator. In the case of a robot or other mechanical agent or vehicle, the observation data may similarly include one or more of position, linear velocity or angular velocity, force, torque or acceleration, and the overall or relative posture of one or more parts of the agent. The observation data may be defined as one-dimensional, two-dimensional, or three-dimensional, and may be absolute and / or relative observation data. For example, in the case of a robot, the observation data may include data characterizing the current state of the robot, such as one or more of the following: joint position, joint velocity, joint force, torque or acceleration, and the overall or relative posture of a part of the robot (such as an arm, and / or an item held by the robot). The observation data may also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or image or video data, such as from a camera or LIDAR sensor, such as data from a sensor of the agent or data from a sensor placed separately from the agent in the environment.
[0090] In these implementations, the action may be a control input to control a robot, such as a torque or higher-level control command to a robot joint; or a control input to control an autonomous or semi-autonomous land, air or sea vehicle, such as a torque or higher-level control command to a control surface or other control element of the vehicle; or, for example, motor control data. In other words, for example, the action may include position, velocity or force / torque / acceleration data of one or more joints of a robot or part of another mechanical agent. The action data may include data for these actions and / or electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices in an environment, the control of which has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air or sea vehicle, the action may include actions to control navigation, such as steering, and motion, such as braking and / or acceleration, of the vehicle.
[0091] In some implementations, the environment is a simulated environment, and the agent is implemented as one or more computers that interact with the simulated environment.
[0092] For example, the simulation environment can be a simulation of a robot or a vehicle, and the reinforcement learning system can be trained on the simulation. For example, the simulation environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent is a simulated vehicle navigating in the motion simulation. In these implementations, the action can be a control input that controls a simulated user or a simulated vehicle. Before using the reinforcement learning system in the real world, the simulation environment is very useful for training the system. In another example, the simulation environment can be a video game, and the agent can be a simulated user playing the video game. In general, in the case of a simulation environment, the observation data can include simulated versions of one or more of the previously described observation data or observation data types, and the action can include simulated versions of one or more of the previously described actions or action types.
[0093] In another example, the environment can be a protein folding environment, such that each state is a corresponding state of a protein chain, and the agent is a computer system for determining how to fold the protein chain. In this example, the actions are possible folding actions for folding the protein chain, and the results to be achieved can include, for example, folding the protein to stabilize the protein and thereby achieve a specific biological function. As another example, the agent can be a mechanical agent that executes or controls protein folding actions selected by the system automatically without human interaction. The observation data can include direct or indirect observation data of the protein state and / or can be derived from the simulation.
[0094] In a similar manner, the environment can be a drug design environment, such that each state is a corresponding state of a potential pharma chemical drug, and the agent is a computer system for determining elements of the pharma chemical drug and / or a synthetic pathway for the pharma chemical drug. For example, in a simulation, the drug / synthesis can be designed based on a reward derived from the drug's goal. As another example, the agent can be a mechanical agent that performs or controls the synthesis of the drug.
[0095] In the case of an electronic agent, the observation data may include data from one or more sensors that monitor parts of a plant or service facility, such as current, voltage, power, temperature, and electronic signals from other sensors and / or functions of electronic and / or mechanical equipment components represented. In some applications, the agent can control actions in a real-world environment including equipment components, such as in facilities such as data centers, server bodies, grid mains power or water distribution systems, or in a manufacturing plant or service facility. The observation data may be related to the operation of a plant or facility. For example, in addition to or alternatively to those previously described, they may include observation data on the use of electricity or water for equipment, or observation data on power generation or distribution control, or observation data on resource use or waste generation. The agent may control actions in the environment to improve efficiency, such as by reducing resource use, and / or reducing the impact of operations in the environment on the environment, such as by reducing waste. For example, the agent may control the use of electricity or other power consumption or water in a facility and / or the temperature of a facility and / or items within the facility. Actions may include actions to control or impose operating conditions on items of equipment of the plant / facility, and / or actions that result in setting changes in the operation of the plant / facility, such as adjusting or turning on / off components of the plant / facility.
[0096] In some other applications, the environment is a real-world environment, and the agent manages the distribution of tasks across computing resources (e.g., in mobile devices and / or data centers). In these implementations, the action may include assigning a task to a specific computing resource. As another example, the action may include presenting an advertisement, the observed data may include ad impressions or click-through counts or click-through rates, and the reward may characterize one or more users' prior selections of an item or content.
[0097] The reinforcement learning system can be implemented as one or more computer programs on one or more computers at one or more locations in which the systems, components, and techniques described herein are implemented.
[0098] Figure 1An example of a reinforcement learning system is shown. The reinforcement learning system 10 includes an agent 20 that determines an action based on a policy 25. Each time an action is determined, the action is output to an environment 30 controlled by the agent 20. The action updates the state of the environment 30. The updated state is returned to the reinforcement learning system 10 along with the reward associated with the action. These are used by the reinforcement learning system 10 to determine the next action. Generally, the reward is a numerical value. The reward can be based on any event or aspect of the environment 30. For example, the reward can indicate whether the agent 20 has completed a task (e.g., navigated to a target location in the environment 30) or the progress of the agent 20 towards completing the task.
[0099] The interaction of the agent 20 with the environment 30 over one or more time steps may be represented by a "trace" (i.e., sequence) of experience tuples, where each experience tuple corresponds to a respective time step. The experience tuple corresponding to a time step may include: (i) observations characterizing the state of the environment at the time step, (ii) an action selected to be performed by the agent at the time step, (iii) subsequent observations characterizing the subsequent state of the environment after the agent performs the selected action, (iv) a reward received after the agent performs the selected action, and (v) a subsequent action selected to be performed at a subsequent time step.
[0100] The policy 25 defines how the system performs actions based on the state of the environment. When the system 10 is trained based on the experience set 17, the policy 25 followed by the agent 20 is updated by evaluating the value of the action according to an approximate value function or a return function to improve the expected return from the action taken by the policy 25. This is usually achieved by combining prediction and control to evaluate the success of the action performed by the agent 20, sometimes referred to as "return". The return is calculated based on the reward received after a given action. For example, the return can be the accumulation of multiple reward values over multiple time steps.
[0101] Some of the parameters used to define how the system learns are the discount factor γ and the bootstrap parameter λ. These parameters are discussed in more detail below with reference to equations (6) and (7).
[0102] The discount factor γ determines the time scale of the returns. Discount factors close to γ = 1 provide a long-sighted goal of accumulating rewards into the future, while discount factors close to γ = 0 provide a short-sighted goal that prioritizes short-term rewards. Even in problems with long-term expectations, it is often observed that discount factor values γ < 1 give better results, especially during early learning. It is well known that various algorithms converge faster at lower discount factor values, but too low discount factor values can lead to suboptimal policies. Therefore, in practice, it is best to optimize for the short-sighted horizon first, for example, by first setting γ = 0 and then iteratively increasing the discount factor value in later stages.
[0103] The returns can also be bootstrapped to different time horizons. n-step returns accumulate rewards over n time steps and then add the value function at the nth time step. λ-return is a geometrically weighted combination of the n-step returns. In either case, the parameter n or λ may be important for the performance of the algorithm (tradeoff between bias and variance), and therefore effective choices of these parameters are desirable.
[0104] Figure 2 A reinforcement learning system for applying meta-learning according to the present specification is shown. Here, in addition to the policy, the return function itself is learned by treating the return function as a parameter function with an adjustable meta-return parameter or meta-parameter η. For example, such a meta-parameter η may include a discount factor γ or a bootstrap parameter λ. For the avoidance of doubt, the "meta-return parameter" and "return parameter" described so far are equivalent to the "meta-parameter" η described below.
[0105] The meta-parameter η is adjusted during the agent's interaction with the state or environment, allowing the return to be adapted both to the specific problem and dynamically to the changing learning situation over time. Thus, this paper describes a practical gradient-based meta-learning method and shows that it can improve the performance of large-scale deep reinforcement learning applications.
[0106] Back to Figure 2 , the reinforcement learning system 200 is similar to Figure 1 200 is a system of the type described herein because it includes an agent 205 that determines actions based on a policy 210 and experience 250. The policy 210 is defined by policy parameters 215 that may be stored in a local memory. The agent 205 may be implemented using a neural network.
[0107] The reinforcement learning system 200 also includes a policy training module 230 and a return function training module 240. The policy training module 230 is configured to update the policy parameters 215 to train the policy 210. The policy parameters 215 are updated based on the return, which is calculated based on the experience 250 using the return function 220 defined by the return parameter 225. The return parameter 225 can be stored in the local memory.
[0108] The reinforcement learning system 200 also includes a return function training module 240. The return function training module 240 is configured to update the return parameter 225 based on experience 250 using a meta-objective function 260. By updating (training) the return function, the system is able to learn a better return function, thereby improving the training of the policy. This allows a more accurate policy to be reached faster and more efficiently.
[0109] In one embodiment, the return function training module 240 is configured to update the return parameters to reduce the error in the return parameters. This can be relative to the value function that the agent 205 uses to determine the expected return from an action (and thereby determine the action that provides the highest expected return). For example, the meta-objective function can calculate the mean square error between the return of one or more experiences and the value, and the return function training module 240 can be configured to update the return parameters to reduce (minimize) the mean square error.
[0110] As described above, the reinforcement learning system 200 includes a policy 210 including one or more policy parameters 215, a return function 220 including one or more return parameters 225, and a value function. The system 200 retrieves a plurality of experiences 250 from a reinforcement neural network (where the reinforcement neural network may or may not form part of the system 200) that is configured to control an agent interacting with an environment to perform tasks in an attempt to achieve a specified outcome based on the one or more policy parameters 215 of the reinforcement learning neural network.
[0111] Each of the plurality of experiences 250 includes observations characterizing a state of the environment, actions performed by the agent in response to the observations, and rewards received in response to the actions.
[0112] In some implementations, the system 200 may generate the experience 250 (i.e., the reinforcement learning neural network may form part of the system 200). Alternatively, the system 200 may access a plurality of experiences 250, for example from a storage device or from an external system. In the latter implementation, as the policy parameters 215 are updated, these parameters may be shared with the neural network to train the neural network.
[0113] In the present embodiment, the system 200 is configured such that the plurality of experiences 250 are divided into a first experience set and a second experience set, wherein each of the first experience set and the second experience set can be used in conjunction with a return parameter 225 (also referred to as a meta-return parameter or meta-parameter) to update a policy parameter 215 of the policy 210. For example, this can be implemented using a policy training module 230. A meta-objective function 260 is then used to adjust the return parameter 225. For example, this can be implemented using a return function training module 240. This process can be repeated multiple times to iteratively update the policy parameters and the return parameters.
[0114] When implemented, the processor can store the agent 205 receiving the experience 250. The policy training module 230 of the processor can update the policy parameters 215 based on the experience 250, and this in turn updates the policy 210 executed by the agent 205. The return function training module 240 of the processor can then adjust the return parameters 225 stored in the memory. The policy training module 230 can access the updated return parameters. Therefore, this iteratively updates the policy 210 and the return function 220.
[0115] Figure 3 The process 300 taken by the system 200 in optimizing the policy 210 and the return function 220 is shown. In step 302, the system 200 retrieves a plurality of experiences 250. In some implementations, the system 200 may generate the experiences 250 (i.e., the reinforcement learning neural network 205 may form part of the system 200). Alternatively, the system 200 may retrieve the plurality of experiences from an external system including the policy parameters. In the latter case, each time the policy parameters 215 are updated, they are sent to the external system for generating the next set of experiences. For example, the plurality of experiences 250 may be generated online by the reinforcement learning system 200 itself, or may be obtained from an external reinforcement learning neural network 205.
[0116] In step 304 , the policy parameters 215 are then updated using the update function 240 and based on the first set of experiences 250 a of the plurality of experiences 250 to form updated policy parameters.
[0117] In step 306 , the updated policy parameters are cross-validated based on a second set of experiences of the plurality of experiences 250 , the meta-objective function 260 , and the return function 220 .
[0118] In step 308, the return parameters are updated based on cross validation of the updated policy parameters and the meta-objective function 260 to form one or more meta-parameters 225, which then update the update function and return function 220. In subsequent parameter updates, existing meta-parameters 225 from previous updates are updated based on the most recent cross validation.
[0119] Then, in step 310, the system determines whether the end criteria have been reached. The end criteria may be a maximum number of iterations or a predetermined performance level. The performance level may be the performance of the policy (e.g., a predefined cumulative reward or return) or the performance of the return function (e.g., the error of the return function relative to the ground truth return). If the end criteria are not reached, the system 200 returns to step 302 to perform a new iteration for a new parameter update. If the end criteria are met, the system 200 completes the training and outputs the results in step 312.
[0120] The output may be an optimized policy (set of policy parameters) or an optimized return function (set of return parameters), as determined in the most recent iteration. The policy can be used to implement the trained agent. The return function can be used to help train more reinforcement learning agents more efficiently.
[0121] As mentioned above, the advantage of using the first experience set to update the policy parameters before using the second experience set to cross-validate the updated policy parameters is that overfitting is avoided and the amount of training data used for each iterative update is reduced.
[0122] In more detail, the value function v θ (S) and strategy π θ (A|S) 210 is approximated by a neural network with parameters θ 215. To better approximate the value function and policy 210, the method includes an update function,
[0123] θ′=θ+f(τ,θ,η) (1) The update function is derived from a series of empirical t ={S t , A t , R t+1 , ...} adjust the parameters, where S represents the state, A represents the action, R represents the reward, and t represents the number of parameter updates that have been performed. The nature of the update function f is determined by the return parameter or meta parameter η225.
[0124] The meta-gradient reinforcement learning method described in this specification is based on the principle of online cross-validation using continuous experience samples. The base reinforcement learning method is applied to a first experience set τ, and its performance is measured using a second experience set τ′. Specifically, the method starts with policy parameters θ215 and applies an update function to the first experience set τ, producing new parameters θ′. The gradients dθ / dη of these updates then indicate how the meta-parameters η225 affect these new policy parameters θ′. The method then uses a differentiable meta-objective
[0125] J′(τ′, θ′, η′) measures the performance of the new policy parameters θ′ on a subsequent independent second set of experience τ′. When verifying the performance on the second set of experience τ′, the fixed meta-parameter η′ in J′ is used as a reference value. In this way, a differentiable function of the meta-parameter η is formed, and the gradient of η can be obtained by taking the derivative of the meta-objective J′ with respect to η and applying the chain rule:
[0126]
[0127] The parameters form an additive sequence, so the gradient dθ′ / dη of the policy parameter update can be expressed as
[0128]
[0129] where I is the identity matrix.
[0130] gradient is large and can be approximated using the cumulative trace as z≈dθ / dη, so
[0131]
[0132] That is, the gradient of the updated policy parameter θ′ with respect to the return parameter η (dθ′ / dη is approximately z′) can be obtained by differentiating the update function (evaluated on the first experience set τ using the policy parameter θ and the return parameter η) with respect to the return parameter η It is calculated iteratively by adding it to the previous gradient z (the gradient of the policy parameter θ with respect to the return parameter η).
[0133] The gradient of formula (3) can be defined using fixed meta-parameters η, or adapted online. To perform this adaptation, the parameter μ∈[0,1] attenuates the trace and only focuses on the most recent updates. For example, choosing μ=0 will produce a trace that only considers the immediate impact of the meta-parameter η on the policy parameter θ.
[0134] The meta-parameter η 225 may then be updated to the updated meta-parameter to optimize the meta-objective function J′ 260 .
[0135]
[0136] Here, β is the learning rate used to update the meta-parameter η225.
[0137] For example, such an update can be done by applying stochastic gradient descent to update the meta-parameter η in the direction of the meta-gradient. Alternatively, the meta-objective function J′ can be optimized by any other known gradient ascent or descent method.
[0138] The updated meta-parameters may then be used as the meta-parameters η 225 for the next iteration when retrieving the next set of multiple experiences 250 .
[0139] The following potential implementations consider the case where the meta-parameter η225 is used for prediction using time difference updates, and the case where the meta-parameter 225 is used for control, where this is implemented using a canonical action-criteria update function and a policy gradient meta-objective. Those skilled in the art will appreciate that many other alternative implementations of reinforcement learning using this meta-gradient approach are also possible.
[0140] Figure 4 The step 400 of updating one or more meta-parameters η and applying them to the return function G is shown. After the policy parameters θ have been updated to the updated policy parameters θ′ and cross-validated, the process forms Figure 3 One implementation of step 308 shown previously in FIG.
[0141] In step 402, the gradient of the return function G220 with respect to one or more meta-parameters η225 is determined.
[0142] In step 404, the gradient of the update function f with respect to one or more meta-parameters η225 is determined.
[0143] In step 406 , the gradient of the meta-objective function J′ 260 with respect to one or more meta-parameters η 225 is determined.
[0144] Apart from Figure 4 In addition to the steps shown, and depending on the choice of meta-objective function J(τ, θ, η) selected for system 200, the gradient of the value function with respect to policy parameters 215 can also be calculated (i.e., ) to determine the gradient of the update function f with respect to one or more meta-parameters η225.
[0145] Thus, the meta-gradient may be evaluated for one or more meta-parameters η225, and the one or more meta-parameters η225 may be adjusted accordingly in step 408 to ensure the optimal return function G, thereby forming updated meta-parameters. These updated meta-parameters may then be used as meta-parameters η225 in subsequent update iterations (where it is concluded that the optimal return function has not yet been reached).
[0146] In more detail, the function G returned η (τ t )220 is defined as a function of the empirical fragment or truncated n-step sequence, τ t ={S t , A t , R t+1 , ..., S t+n}. As before, the function G is returned η The properties of are determined by one or more meta-parameters η.
[0147] n steps return function G η 220 accumulates rewards over the sequence and then bootstraps from the value function, so that
[0148] G η (τ t )=R t+1 +γR t+2 +γ 2 R t+3 +…+γ n-1 R t+n +γ n v θ (S t+n ) (6)where η={γ,n}.
[0149] The bootstrap parameter λ returns a function or λ is a geometric mixture of n steps, so the return function G η can be redefined as
[0150] G η (τ t )=R t+1 +γ(1-λ)v θ (S t+1 )+γλG η (τ t+1 ) (7)
[0151] where η = [γ, λ}. λ return has the advantage of being fully differentiable with respect to the meta-parameters γ and λ225.
[0152] In this case, the chosen meta-parameters η, γ, and λ can be thought of as gates or conditions that lead to returning to termination (γ=0), bootstrapping (λ=0), or continuing to the next step (γ=1 or λ=1).
[0153] Traditionally, typical reinforcement learning algorithms will manually select meta-parameters η, such as the discount factor γ and the bootstrap parameter λ, and these will remain fixed throughout the training. However, in the reinforcement learning system described in this article, the return function G is parameterized by the meta-parameters η, which can then be differentiated in order to understand the dependence of the return function G on η. This in turn allows the gradient of the update function f with respect to the meta-parameters η to be determined. Therefore, the meta-gradient of the meta-objective function J′ can also be determined This allows the system to evaluate which return function G 220 results in the best performance and adjust the meta-parameter η 225 accordingly according to equations (2) to (5).
[0154] In a particular implementation of the system 200, a canonical TD(λ) algorithm for prediction may be used to make predictions about the optimal return function based on the selected meta-parameters η 225. The goal of the TD(λ) algorithm is to minimize the value function approximator v θ (S) and λ return G η The square error between (τ),
[0155]
[0156] Here, τ is the first set of experiences starting from the initial state S, and is the semi-gradient. For example, λ returns is treated as a constant.
[0157] The TD(λ) update function f(τ, θ, η) applies stochastic gradient descent to update the agent’s policy parameters θ215 to reduce the gradient of the objective with respect to the policy parameters θ215, thereby
[0158]
[0159] Here, α is the learning rate used to update the policy parameters θ. The update function f here is differentiable with respect to the meta-parameters η225, so
[0160]
[0161] In this implementation, the purpose of the meta-gradient prediction is to adjust the meta-parameter η225 in the direction that achieves the best prediction accuracy. This is measured in the steps described previously and Figure 3 As shown in step 306 of , the updated policy parameter θ′ is cross-validated based on the second experience set τ′ starting from state S′ using the mean square error (MSE) meta-objective function J′ and taking its half gradient, and the form is
[0162] J′(τ′,θ′,η′)=(G η (τ′)-v θ′ (S′) 2
[0163]
[0164] Therefore, the meta-gradient of the meta-objective function Formulas (2) to (5) are combined to determine and implement the necessary updates to the meta-parameters η.
[0165] In this implementation, the meta-objective function J′ may use an unbiased and long-term return function G220, for example, using η′={γ′, λ′}, where γ′=1 and λ′=1.
[0166] In another implementation of the system 200, the meta-gradient can be applied to control, such as the A2C action-criteria algorithm. In this implementation, the action-criteria update function combines both prediction and control into a single update to the policy.
[0167] The semi-gradient of the A2C meta-objective function is defined as:
[0168]
[0169] In this formula, the first term represents the configuration strategy π θ 210 is the control objective to maximize the measured reward of the return function 220. The second term represents the configuration value function approximator v θ To accurately estimate the return function G η The third term is a term for the entropy H of the regularize strategy 210, and c and d are coefficients for appropriately weighting the different terms in the meta-objective function.
[0170] The A2C update function f(τ, θ, η) applies stochastic gradient descent to update the policy parameters θ215. This update function is differentiable with respect to the meta-parameters η225, so
[0171]
[0172] Here, α is the learning rate applied when updating one or more policy parameters from one or more previous policy parameters.
[0173] In this implementation, the choice of meta-objective function 260 is one that ensures that the returned function maximizes the performance of the agent. This can be achieved by using a policy gradient objective of the form:
[0174]
[0175] Here, G η′ is a further return function that, when applied to the second experience set τ′, evaluates the updated policy according to the return from the further return function, v θ′ (S′) is the value function associated with the updated policy for state S′, and π θ′ (A′|S′) is the policy of action A′ in response to the update of state S′.
[0176] Therefore, the formula evaluates the success of the updated policy parameters θ′ using the second experience set τ′, considering the returns calculated in the case of η′.
[0177] In this implementation, when the updated policy parameters θ′ are evaluated by cross-validation using the meta-objective function, a fixed meta-parameter η′ that represents a good approximation of the agent's true goal can be used. This can include choosing a reasonable value for η′ based on values that perform well in practice.
[0178] The meta-gradient learning algorithm in this implementation can be implemented in the following way. First, the policy parameter θ215 is updated based on the first experience set τ using the A2C update function shown in formula (12). This is consistent with the gradient of the update function shown in formula (13): is accumulated into the trace z as shown in Equation (4) above. Then, the policy gradient meta-objective is used as shown in Equation (14) The performance is cross-validated on the second experience set τ′. Finally, the meta-parameter η 225 can then be updated according to equations (2) to (5) according to the gradient of the meta-objective function 260 .
[0179] One aspect of the above implementation is to return the function G η (τ)220 is non-stationary because it is updated along with the meta-parameters η225 during the entire training process. This may cause the value function v θ becomes inaccurate as it may be close to being returned out of date.
[0180] For example, the value function v θ A good approximation may initially be formed at the beginning of the training process with γ=0, but a poorer approximation may be formed later in the training process after γ has been adapted to γ=1.
[0181] The same applies to policy π 220 , which may also be formed based on expired returns.
[0182] Therefore, in order to solve the value function v θ This non-stationary aspect of the strategy π220 can be implemented in a similar way to the universal value function approximation (UVFA). Here, the meta-parameter η is provided as an additional input to define the value function v θ and strategy π220, which is of the form
[0183]
[0184] where e η is the embedding of η, [s; e η ] are vectors s and e η cascade, and W η is the embedding matrix (or row vector, for a scalar η) that is updated by back-propagation during training.
[0185] In this implementation, the agent then explicitly learns the most appropriate value function v for any given value of the meta-parameter η225 θ and strategy π 220. This has the advantage of allowing the meta-parameter η to be adjusted without waiting for the approximator to "catch up".
[0186] The methods and various implementations of the system 200 described so far can be scaled up. For example, to improve efficiency, the A2C objective and meta-objective function can be accumulated over all time steps within the n-step experience set. The A2C objective function can be optimized by RMSProp without momentum. This results in a differentiable function of the meta-parameter η225, and can therefore be replaced similarly to stochastic gradient descent (see Formula 13). As in IMPALA, off-policy corrections based on V-shaped trace returns can be used. For more efficient further implementation, small experience sets can be calculated in parallel, or experience sets can be reused twice for updating functions and cross-validation. For example, in order to reduce the data required for meta-learning, experience can be used for both agent training and meta-learning. For example, experience τ can be used to update θ to θ′, and the performance of this update can be verified by evaluating J′ on the experience τ′. Vice versa, the roles of τ and τ′ can be exchanged so that experience τ′ can be used to update θ to θ′, and the performance of this update can be verified by evaluating J′ on the experience τ. In this way, the proposed method requires no additional data beyond the data used to train the agent parameters θ to meta-learn updates to η.
[0187] For a system consisting of one or more computers, being configured to perform a specific operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof, which in operation causes the system to perform the operation or action. For one or more computer programs being configured to perform a specific operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.
[0188] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them). Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier, for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or additionally, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. However, a computer storage medium is not a propagation signal.
[0189] The term "data processing apparatus" encompasses all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may include special-purpose logic circuitry, for example, a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, for example, code forming processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0190] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and the computer program may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that stores other programs or data (e.g., in a markup language document, in a single file dedicated to the program in question, or in one or more scripts in multiple collaborative files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.
[0191] In this specification, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs that are different from inputs. An engine can be a coded functional block, such as a library, a platform, a software development kit (SDK), or an object. Each engine can be implemented on any suitable type of computing device (e.g., a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smart phone, or other fixed or portable device), and the computing device includes one or more processors and a computer-readable medium. In addition, two or more of the engines can be implemented on the same computing device, or on different computing devices.
[0192] The processes and logic flows described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows may also be performed by dedicated logic circuits, and the apparatus may also be implemented as dedicated logic circuits (e.g., field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs)). For example, the processes and logic flows may be performed by a graphics processing unit (GPU), and the apparatus may also be implemented as a graphics processing unit (GPU).
[0193] Computers suitable for running computer programs include, for example, general or special microprocessors or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic element of a computer is a central processing unit for performing or running an instruction and one or more memory devices for storing instructions and data. Generally, a computer will also include or be operably coupled to one or more large-capacity storage devices (such as magnetic disks, magneto-optical disks or optical disks) for storing data, to receive data from it or to transmit data to it, or both. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device (such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device (such as a universal serial bus (USB) flash drive), just to name a few examples).
[0194] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROMs and DVD-ROMs. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.
[0195] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.
[0196] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.
[0197] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0198] Although this specification contains a number of specific implementation details, these should not be interpreted as limitations on any invention or the scope of the claimed protection, but rather as descriptions of specific features of specific embodiments of specific inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations, and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.
[0199] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order shown or sequentially, or that all of the operations shown be performed to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0200] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes described in the accompanying drawings do not necessarily require the particular order shown or sequentially to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.
Claims
1. A reinforcement learning system, comprising one or more processors, wherein the one or more processors are configured to: retrieving a plurality of experiences generated by a reinforcement learning neural network configured to control an agent interacting with an environment to perform a task in an attempt to achieve a specified outcome by performing actions selected by the reinforcement learning neural network based on one or more policy parameters of the reinforcement learning neural network, each experience comprising observations characterizing a state of the environment, an action performed by the agent in response to the observations, and a reward received in response to the action, in, The observation data includes one or more of an image, object location data, and sensor data other than an image or object location data, or the observation data includes a simulated version of one or more of an image, object location data, and sensor data other than an image or object location data; and The reinforcement learning neural network is trained by the following steps: Using a return function returned based on the reward calculation, updating one or more policy parameters of the reinforcement learning neural network based on the first experience set; as well as Based on the one or more updated policy parameters and the second experience set, one or more return parameters of the return function are updated, wherein the one or more return parameters include one or more of a discount factor of the return function and a bootstrap factor of the return function, and the one or more return parameters are updated via a gradient ascent or descent method using a meta-objective function differentiated relative to the one or more return parameters, wherein the meta-objective function depends on the one or more policy parameters, and the differentiated meta-objective function is: in: n is the one or more return parameters; and J′(τ′, θ′, η′) is the meta-objective function conditioned on the second set of experiences τ′, the one or more updated policy parameters θ′ and one or more further return parameters η′ of a further return function forming part of the meta-objective function, wherein the one or more further return parameters are kept fixed during training and wherein the further return function can be different from the return function returned based on the reward calculation received by the agent.
2. The reinforcement learning system according to claim 1, in, The one or more processors are further configured to iteratively: retrieving updated experience generated by the reinforcement neural network using the one or more updated policy parameters and the one or more updated return parameters; further updating the one or more policy parameters based on the first updated experience set using the one or more updated returned parameters; as well as further updating the one or more return parameters based on the further updated policy parameters and the second updated experience set via a gradient ascent or descent method, Until the end condition is reached.
3. The reinforcement learning system according to claim 1, in, The updating of the one or more policy parameters applies one or more of a policy and a value function conditioned on the one or more return parameters.
4. The reinforcement learning system according to claim 1, in, The one or more processors are further configured to: updating one or more policy parameters of the reinforcement learning neural network based on the second experience set; as well as Based on the one or more updated policy parameters and the first experience set, one or more return parameters of the return function are updated, wherein the one or more return parameters are updated via a gradient ascent or descent method.
5. The reinforcement learning system according to claim 3, in, The system is configured to calculate the differentiated meta-objective function based on the differential dθ′ / dη of the updated policy parameter θ′ with respect to the return parameter η, wherein the differential dθ′ / dη is obtained by differentiating the update function θ′=θ+f(τ,θ,η) with respect to the return parameter The update function is calculated by adding to the differential dθ / dη of the strategy parameter θ with respect to the return parameter η, and is used to update the strategy.
6. The reinforcement learning system according to claim 5, in, The derivative of the update function with respect to the return parameter Calculated by the following formula: in: α is the learning rate applied when updating the one or more policy parameters from one or more previous policy parameters; G η (τ) is a return function that calculates a return from the first experience set τ based on the one or more return parameters η; π θ is the strategy for the reinforcement learning neural network to determine action A from state S, the strategy π θ operating according to the one or more policy parameters θ; c is the coefficient; and v θ (S) is a value function that determines the value of state S based on the one or more policy parameters θ.
7. The reinforcement learning system according to claim 1, in: in: G η′ (τ′) is a further return function for calculating a return from said second experience set τ′ based on said one or more further return parameters η′; π θ′ is a strategy for the reinforcement learning neural network to determine action A′ from the state S′ taken from the second experience set τ′, the strategy π θ′ operating according to the one or more updated policy parameters θ′; and v θ′ (S′) is a value function that determines the value of state S′ based on the one or more updated policy parameters θ′.
8. The reinforcement learning system according to claim 1, in, Updating the one or more returned parameters comprises calculating: in n′ is the one or more updated return parameters; and β is a learning factor used to update the one or more return parameters.
9. A computer-implemented method for reinforcement learning, the method include: retrieving a plurality of experiences generated by a reinforcement learning neural network configured to control an agent interacting with an environment to perform a task in an attempt to achieve a specified outcome by performing actions selected by the reinforcement learning neural network based on one or more policy parameters of the reinforcement learning neural network, each experience comprising observation data representing a state of the environment, an action performed by the agent in response to the observation data, and a reward received in response to the action, wherein the observation data comprises one or more of an image, an object position data, and sensor data other than the image or the object position data, or the observation data comprises a simulated version of one or more of the image, the object position data, and sensor data other than the image or the object position data; and The reinforcement learning neural network is trained by the following steps: Using a return function returned based on the reward calculation, updating one or more policy parameters of the reinforcement learning neural network based on the first experience set; and Based on the one or more updated policy parameters and the second experience set, one or more return parameters of the return function are updated, wherein the one or more return parameters include one or more of a discount factor of the return function and a bootstrap factor of the return function, and the one or more return parameters are updated via a gradient ascent or descent method using a meta-objective function differentiated relative to the one or more return parameters, wherein the meta-objective function depends on the one or more policy parameters, and the differentiated meta-objective function is: in: n is the one or more return parameters; and J′(τ′, θ′, η′) is the meta-objective function conditioned on the second set of experiences τ′, the one or more updated policy parameters θ′ and one or more further return parameters η′ of a further return function forming part of the meta-objective function, wherein the one or more further return parameters are kept fixed during training and wherein the further return function can be different from the return function returned based on the reward calculation received by the agent.
10. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of claim 9.
