Systems and methods for learning reusable options for transferring knowledge between tasks

Abstract

A robot comprising an RL agent configured to learn a strategy to maximize the cumulative reward of a task by determining one or more features that are minimally correlated with each other. The features are then used as pseudo-rewards, referred to as feature rewards, wherein each feature reward corresponds to an option strategy or skill, and the RL agent learns the option strategy or skill to maximize the feature reward. In one example, the RL agent is configured to select the most relevant features to learn the corresponding option strategy. The RL agent is configured to learn, for each of the selected features, the corresponding option strategy that maximizes the corresponding feature reward. Using the learned option strategy, the RL agent is configured to learn a new (second) strategy for a new (second) task, which can be selected from any of the learned option strategies or actions available to the RL agent.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. patent application Ser. No. 16 / 900,291, filed on June 12, 2020, entitled “SYSTEMS AND METHODS FOR LEARNING REUSABLE OPTIONS TOTRANSFER KNOWLEDGE BETWEEN TASKS,” which is incorporated herein by reference as if reproduced in its entirety. Technical Field

[0003] The present invention generally relates to robot learning for solving tasks in an environment using reinforcement learning, and more particularly to a robot that reuses knowledge learned when solving one task in an environment using reinforcement learning to efficiently solve another new task in the same action space and state space of the environment using reinforcement learning. Background Art

[0004] In the context of artificial intelligence (AI), reinforcement learning (RL) has historically been implemented using dynamic programming techniques, which learn the value of a policy and find better policies that maximize that value. The value and policy estimates are iteratively improved until convergence. Typically, an RL agent learns by interacting with its environment by receiving the environment's state and rewards, performing actions on the environment, and receiving how the actions affect the environment's state and rewards. The RL agent learns a policy that maximizes the value of the current state, which is the expected sum of future cumulative rewards, by performing actions based on the current state, receiving the next state and reward, and improving its policy and state-value estimate. The RL agent executes its policy using input from the environment to achieve one or more goals in the environment.

[0005] Training an RL agent on a robot can be a very expensive process. Several difficulties can arise when applying RL to robotics, including:

[0006] The curse of dimensionality: As the dimensionality of the state space and action space grows, the computation required for RL increases exponentially. In robotics, problems are often high-dimensional. For example, the state space of a simple robotic arm has about 20 dimensions, and the action has 7 dimensions.

[0007] The curse of real-world samples: Robots interacting with the real world can lead to numerous problems. One of these is the cost of running experiments, which involves maintenance costs, damage to the robot from unsafe exploration, and human involvement during restarts. Another major issue is the uncertainty introduced by measurement noise and discretization.

[0008] The curse of goal specification: RL algorithms require a well-defined reward function that characterizes the desired behavior. Simply specifying the binary reward that a robotic learning agent receives upon completion of a task is often insufficient, as the robot may never complete the task.

[0009] One exemplary approach to reducing the cost of training RL agents in robotics is to reuse knowledge (skills) from previous tasks to solve new ones. There are two main approaches to learning new tasks from previously learned tasks: (i) explicitly specifying hand-engineered rewards and learning skills that maximize those rewards; and (ii) imitation / inverse RL, in which skills are learned by imitating provided examples. However, both approaches have drawbacks. Manipulating hand-engineered rewards can be expensive and difficult to scale. Imitation / inverse RL can run the risk of overfitting to the collected data, hindering generalization.

[0010] Many RL agents do not use any prior knowledge when learning new tasks. In such RL agents, the RL agent learns to solve each task tabularly. Summary of the Invention

[0011] The present invention generally relates to a method in which a robot reuses knowledge learned from solving one task in an environment using reinforcement learning to efficiently solve another new task in the same action space and state space of the environment using reinforcement learning.

[0012] In some exemplary embodiments, a robot includes an RL agent that learns a first policy to maximize the cumulative reward for a first task. The first policy is learned using an RL algorithm and transition tuples collected in the environment, each transition tuple comprising a state, an action, the reward of the first policy after executing the action, and a next state. In some embodiments, the first policy learned using the RL algorithm is modeled as a neural network that maps states in an action space to actions (hereinafter referred to as the policy network). In some embodiments, the first policy learned using the RL algorithm is an action that maximizes a value function at a given state, where the value function is modeled as a neural network that maps states and actions to values ​​(hereinafter referred to as the value network). A feature neural network (hereinafter referred to as the feature network) is extracted from the policy network or the value network. The feature network generates features for a given state. These features are then used as pseudo rewards, referred to as feature rewards. Each feature reward corresponds to the reward of an alternative policy that the RL agent learns to maximize using transition tuples collected in the environment, where each transition tuple comprises a state, an action, a feature reward, and a next state. The RL agent learns a second policy to maximize the future cumulative reward for a second (different) task. The second policy selects an alternative policy from the learned alternative policies. The selected alternative policy generates an action for a given state. The second policy is learned using an RL algorithm and transition tuples collected in the environment. Each transition tuple consists of a state, an option policy, the reward of the second policy after executing the action generated by the option policy, and the next state. The second policy is modeled as a neural network that maps states to option policies (hereafter referred to as the second policy network).

[0013] In some exemplary embodiments, an RL agent is used to select the most relevant features from which to learn the corresponding selection policy. Features can be selected based on the variance of each feature.

[0014] One advantage of RL agents is that they do not need to use a learning blank slate to learn new policy options for new tasks in the same environment.

[0015] Another advantage is that the RL agent efficiently learns a new (second) policy by reusing the option policies learned from the first policy.

[0016] In some exemplary embodiments, the RL agent selects features with high variance to learn their respective option policies. For example, the RL agent selects features with variance above a specified threshold. In another example, the RL agent sorts features by variance and selects a specified number of features with the highest variance.

[0017] In some exemplary embodiments, the feature network is a subnetwork of the policy neural network, and the feature network is used to minimize a decorrelation regularizer loss function that causes the feature network to generate features that are minimally correlated with each other.

[0018] Another exemplary embodiment is a method that includes learning a first policy to maximize the cumulative reward for a first task. The first policy is learned using an RL algorithm and transition tuples collected in an environment, each tuple including a state, an action, a reward of the first policy after performing the action, and a next state. The method also includes extracting a feature network from the neural network used to learn the first policy. The feature network outputs features at a given state. The method also includes calculating the variance of each feature output by the feature network; and selecting at least one of the features based on the calculated variance. The method also includes: for each selected feature, learning an option policy from transition tuples collected in the environment, each transition tuple including a state, an option policy, a feature reward, and a next state, thereby maximizing the cumulative feature reward of the selected feature; and learning a second policy to maximize the future cumulative reward for a second (different) task. The second policy selects an option policy from the learned option policies. The selected option policy generates an action for a given state. The second policy is learned using an RL algorithm and transition tuples collected in the environment, each transition tuple including a state, an option policy, a reward of the second policy after performing the action generated by the option policy, and a next state. The second policy is modeled as a neural network that maps states to option policies (hereafter referred to as the second policy network).

[0019] Another exemplary embodiment is a RL agent comprising a memory and a processing unit for executing instructions stored in the memory to perform the described method.

[0020] Another exemplary embodiment is a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor of a processing device, cause the processing device to perform the described method. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] For a more complete understanding of exemplary embodiments and their advantages, reference is now made to the following detailed description taken in conjunction with the accompanying drawings, in which:

[0022] Figure 1 is a flowchart of a method for learning and migrating a robot's option strategy provided by an exemplary embodiment.

[0023] Figure 2A A perspective view illustrating a robot learning a first task in an environment provided by an exemplary embodiment is shown.

[0024] Figure 2B The robot provided by the exemplary embodiment is shown in Figure 2A Learning perspective for a second related task in the same environment.

[0025] Figure 3is a schematic diagram of a robot for generating one or more features provided by an exemplary embodiment.

[0026] Figure 4 is a block diagram implementation of a robot provided by an exemplary embodiment for learning such as Figure 3 A selection strategy based on one of the features generated by operating the robot.

[0027] Figure 5 is a block diagram implementation of a robot for learning a new strategy provided by an exemplary embodiment, the new strategy being based on Figure 4 The system is constructed based on one or more learned option policies generated by operating the robot to achieve a new task.

[0028] Like reference numerals are used in different drawings to denote like components. DETAILED DESCRIPTION

[0029] The following terms may be used in exemplary embodiments:

[0030] Actions: Control decisions, which can be generated by an RL agent to enable the robot to interact with the environment;

[0031] Action space: the set of all possible actions;

[0032] Primitive Actions: Low-level actions that can be performed in just one step; an example of a primitive action is the steering wheel angle;

[0033] State space: the set of all possible configurations of the robot;

[0034] State: A description of the environment that is sufficient to predict the future state of the environment given the next action without any other information (i.e., without the need for historical state);

[0035] Transition probability: given an action, the probability of transitioning from one state to another;

[0036] Reward: A real-valued function defined for all possible (state, action, next state) triplets;

[0037] Characteristics: describes an aspect of a state;

[0038] Feature reward: The feature that is rewarded, describing the task to maximize that feature;

[0039] Environment: The world that the agent interacts with; RL agents typically model tasks in the environment as MDPs.

[0040] Task: In RL, this is often modeled as an MDP where the agent’s goal is to maximize the cumulative reward; another way to define a task is to reach a goal state in the environment;

[0041] Related tasks: the set of tasks in the environment, where each MDP differs in the reward definition; the state definition can also vary from task to task, but we assume that we can approximate the state of each task using the same set of observations shared across all tasks;

[0042] RL: reinforcement learning;

[0043] Transfer learning: generalization across RL tasks that share state and action representations;

[0044] Markov decision process (MDP): a collection of state space, action space, transition probabilities, and rewards;

[0045] Policy: A decision rule that specifies the actions to take in a given state;

[0046] Options: An option is defined by a policy, a termination function, and a launch set; the launch set describes when the option can be called, and the termination function outputs the probability of ending; an option is a policy that can execute for a number of time steps before terminating and switching to another option.

[0047] The present invention generally relates to a robot that includes a RL agent that controls its interaction with an environment. To control the robot's interaction with the environment, the RL agent receives the current state of the environment, calculates a reward corresponding to the current state, and generates an action from an action space based on the current state. The RL agent provides the generated action to the robot's controller. The robot's controller generates control signals for the robot's actuators to cause the robot to perform the action in the environment.

[0048] The environment can be a simulation environment, and the robot can be implemented as one or more computer programs that interact with the simulation environment. For example, the simulation environment can be a video game, and the robot, including the RL agent, can be a simulated user playing the video game. As another example, the simulation environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the RL agent can be a simulated driver navigating the robot through the motion simulation. In these implementations, the action can be a point in the space of possible control inputs for controlling the robot.

[0049] Alternatively, the environment can be a real-world environment, and the robot is a mechanical device that interacts with the real-world environment. For example, the RL agent controls the robot that interacts with the environment to complete a specific task. As another example, the robot can be a vehicle, and the RL agent controls the vehicle to navigate the environment autonomously or semi-autonomously. In these implementations, actions are drawn from an action space that includes all possible actions for controlling the robot.

[0050] In some examples, a low-dimensional feature vector representing the environment is used to represent the observed state of the environment. In these examples, values ​​of different dimensions of the low-dimensional feature vector may have different ranges.

[0051] In some other examples, the observed state of the environment is represented using high-dimensional pixel input from one or more images representing the environment, such as images of a simulated environment or images captured by one or more sensors of the robotic agent as it interacts with the real-world environment.

[0052] Now refer to Figure 1 、 Figure 2A and Figure 2B . Figure 1 A flow chart of a method 20 of learning and transferring option policies is shown. Figure 2A and Figure 2B An exemplary robot 100 is shown that includes a RL agent 102 for performing method 20 .

[0053] Now refer to Figure 2A and Figure 2B , shows an example of a robot 100 including an RL agent 102. Figure 2A and Figure 2B In the embodiment, the robot 100 is a robotic arm 30. However, it should be understood that the robot 100 may include any suitable robotic device that can be controlled to perform a specific task in an environment, such as a vehicle robot, a vacuum robot, an assembly robot, etc. Figure 2A 1 shows the RL agent 102 controlling the robot 100 to perform a first task (Task 1) in the environment. Figure 2B RL agent 102 is shown controlling robot 100 to perform a second task (Task 2) related to the first task in the same state space and action space of the environment. The second task is a different task from the first task. Robot 100 may also include a controller (not shown) and one or more actuators (not shown) for controlling robotic arm 30.

[0054] The RL agent 102 of the robot 100 is used to reuse the knowledge required to solve a first task (e.g., Task 1) in order to solve a second, related task (e.g., Task 2). This is often referred to as transfer learning. For example, a task is a problem that needs to be solved in an environment to achieve some goal (which can be measured by maximizing a cumulative reward). Two tasks are related if they are in the same state space and action space of the environment, and solving these tasks may involve similar types of actions and skills to achieve their respective goals (which can be measured by their respective cumulative rewards). The RL agent 102 of the robot 100 is learning transferable skills. Skills that are useful in solving one task can be used to solve another, different task. Skills are defined as options in reinforcement learning, and the strategies that focus on learning corresponding to these options are called option strategies.

[0055] In an exemplary embodiment, RL agent 102 is configured to learn the skills (i.e., option policies) required to solve a first task. The learned skills can be continuously executed in such a manner that RL agent 102 can solve more complex tasks or different tasks. Figure 2A and Figure 2B Examples of two tasks performed by the robot arm 30 of the robot 100 shown in FIG are: Task 1 ( Figure 2A ): Move the robot arm's grasp to the target position 34; Task 2 ( Figure 2B ): The block 32 is pushed to the target position 34 by the robot gripper.

[0056] The RL agent 102 can reuse the knowledge learned from solving Task 1 in solving Task 2. Task 1 and Task 2 are referred to as related tasks because the robotic arm 30 operates in the same state space and action space, and solving these tasks can involve similar types of actions and skills. The difference between the two tasks lies in the specific goals to be achieved (and the associated cumulative rewards). In this example, the robotic arm 30 will achieve different tasks and goals. In an exemplary embodiment, the RL agent 102 is used to learn an option policy for each feature reward selected and generated by a feature neural network extracted from the policy neural network learned to solve Task 1. The RL agent 102 is used to reuse at least one option policy from the set of option policies learned by the RL agent to solve a portion of Task 2.

[0057] In some exemplary embodiments, the set of option strategies learned by the RL agent 102 is minimally redundant and minimally correlated. In one exemplary embodiment, the set of option strategies learned by the RL agent 102 is non-redundant and uncorrelated. For example, the final state of each option strategy learned by the RL agent 102 can be different and unique. A final state occurs when the RL agent 102 has achieved its goal or has not achieved its goal. A final state is the last state that the RL agent 102 is in when one of many possible final conditions is met. Final conditions include goal achievement, death, collision, timeout, boundary violation, etc.

[0058] Reference again Figure 1 According to an exemplary embodiment, a method 20 for learning and transferring an option policy for a robot 100 will now be described in greater detail. Method 20 includes four steps 22, 24, 26, and 28, described in greater detail below. Method 20 is implemented as software executable by a processor (not shown) of the robot 100. The software implementing method 20 may include routines (also referred to as procedures or functions) for implementing each of steps 22, 24, 26, and 28 of method 20. The coding for the software to perform method 20, including the coding for each of the routines corresponding to steps 22, 24, 26, and 28, is well within the scope of the present invention for one of ordinary skill in the art, given the present invention. In some embodiments, the RL agent 102 includes the software implementing method 20, and the computer-readable code or instructions of the software implementing method 20 are executable by the processor of the robot 100. In some embodiments, the RL agent 102 includes the software implementing method 20 and hardware including memory and one or more processors. In this embodiment, computer-readable code or instructions for software implementing method 20 are stored in a memory of RL agent 102 and can be executed by one or more processors of RL agent 102. In some embodiments, computer-readable code or instructions for software implementing method 20 can be stored in a computer-readable medium.

[0059] Method 20 begins at step 22. In step 22, method 20 solves the first task (Task 1) by learning a first policy π(s) to maximize the cumulative reward of the first task (Task 1). The first policy is learned using an RL algorithm and state transition tuples collected in the environment, where each state transition tuple includes a state s, an action a, the reward of the first policy after performing action a, and the next state after performing action a. The learned first policy π(S) is modeled as a neural network (referred to as a policy network) that maps state S to action a. The first policy π(s) is defined by π(S) = F[φ(s|w)|θ], where φ(s|w) is a feature neural network (hereinafter referred to as the feature network). The feature network is a hidden layer of the first policy network and can therefore be considered a subnetwork of the policy network π(s). The feature network φ(s|w) generates features, parameterized by w. The first policy π(s) maps the features generated by the feature network φ(s|w) to actions parameterized by θ. The feature network φ(s|w) generates features and outputs a vector of generated features of length n (called a feature vector). The features generated by the feature network φ(s|w) are constrained to be minimally correlated with each other during the learning of the decorrelation regularizer. The decorrelation regularizer minimizes the covariance of the features generated by the feature network φ(s|w). In some embodiments, the decorrelation regularizer can be omitted, however, this may cause the features generated by the feature network φ(s|w) to be highly correlated.

[0060] In some embodiments, the features generated by the feature network φ(s|w) can be constrained to be minimally correlated using other algorithms, such as those described in HYLKB Mavrin, "Deep Reinforcement Learning with Decorrelation," May 8, 2019, [Online]; Accessed from: https: / / arxiv.org / abs / 1903.07765, and DGACB Mavrin, "Efficient decorrelation of features using Gramian in Reinforcement Learning," November 19, 2019, [Online]; Accessed from: https: / / arxiv.org / abs / 1911.08610, [Accessed January 17, 2020], both of which are incorporated herein by reference.

[0061] When step 22 of method 20 is executed (e.g., when the code or instructions of the routine implementing step 22 are executed by a processor of the robot or one or more processors of the RL agent 102), method 20 receives a feature decorrelation regularization parameter λ, a discount factor γ control, the importance of immediate rewards versus long-term rewards, and a reward function for a first task (Task 1). When learning a first policy π(s) to maximize the cumulative reward for the first task (Task 1), a feature network φ(s|w) is extracted from the policy network π(s). Method 20 then provides as output: (i) the extracted feature network φ(s|w); and (ii) an estimated variance of each feature generated by the extracted feature network φ(s|w) (which may be referred to as a "feature variance"). Method 20 then proceeds to step 24.

[0062] In some embodiments, the learned first policy π(s) is deterministic and is learned using a deep Q-network (DQN) algorithm (when the actions are discrete actions) or a deep deterministic policy gradient (DDPG) algorithm (when the actions are continuous actions). In some embodiments, the learned first policy π(s) is a stochastic policy (e.g., a distribution of actions conditioned on a state), and the first policy π(s) is an action in a given state, and the first policy π(s) is learned using a proximal policy gradient (PPO) algorithm or a soft actuation evaluation algorithm.

[0063] In step 24, method 20 selects features generated by the extracted feature network φ(s|w) based on the feature variance of the features. In step 24, the feature selection is as follows. The features are sorted in descending order according to their feature variance. In some embodiments, features whose feature variance is less than a predefined threshold are filtered out (discarded). For example, only the k features with the largest feature variance are selected. In some embodiments, a human expert can manually select the features generated by the feature network using code or an input interface. In some embodiments, in step 24, method 20 selects all features generated by the feature network φ(s|w), and the selected features are used to generate sub-goals (i.e., when k=n). In an exemplary embodiment, setting the threshold to the top k features (where k is equal to the number of features) selects all features generated by the extracted feature network φ(s|w). In some embodiments, step 24 of method 20 is not performed. In some embodiments, when step 24 is executed (e.g., when code or instructions implementing the routine of step 24 are executed by a processor of a robot or one or more processors of RL agent 102), features are selected based on a heuristic (e.g., sorting features by highest feature variance first and removing features with feature variance below a threshold). After executing step 24, method 20 proceeds to step 26.

[0064] In step 26, a corresponding option policy is learned for each selected feature generated by the extracted feature network φ(s|w). The “feature” pseudo reward (also called feature reward) is defined using the following reward function:

[0065] r i (s t ,a,s t+1 )=φ i (s t+1 )-φ i (s t );

[0066] Where i = 1...k, where maximizing the value of the pseudo-reward for each feature represents a sub-goal. Option policies are learned to maximize the feature with the largest feature variance among k≤n features. In some embodiments, the number of option policies learned when performing step 26 (e.g., when code or instructions implementing the routine of step 26 are executed by a processor of robot 100 or one or more processors of RL agent 102) is constrained by the dimensionality of the extracted feature network φ(s|w).

[0067] In step 28, a second, related task (i.e., Task 2) is solved by learning a second policy that, given the current state, selects one of the learned option policies and maximizes the future cumulative reward of the second task (i.e., Task 2). When the second policy selects a learned option policy, the selected learned option policy is executed for one or more time steps (i.e., through one or more interactions with the environment). When the future cumulative reward of the selected learned option policy is maximized, execution of the selected learned option policy can be discontinued. In some embodiments, when step 28 is executed (e.g., when code or instructions implementing the routine of step 28 are executed by a processor of the robot 100 or one or more processors of the RL agent 102), the learned option policy is selected from the augmented action space. The one or more learned option policies in the augmented action space can be learned option policies or primitive actions. If the second policy selects a primitive action from the augmented action space, the selected option policy is only executed for one time step.

[0068] As understood in the art, in exemplary embodiments, each time step (t) may be a specific time interval, may be non-periodic, may be event-based (e.g., a turn-based game or activity), or may depend on other internal or external criteria.

[0069] Minimal correlation of learned features ensures that the corresponding rewards will result in a learned option policy with a unique final condition and a unique behavior.

[0070] Tasks with higher variance are generally easier to learn. This is particularly beneficial in robotics due to the curse of goal specification mentioned above. On the other hand, features with zero variance yield uninteresting tasks where all states are goal states and, therefore, should not be considered for option learning in exemplary embodiments. In general, in some exemplary embodiments, features can be selected based on a specific variance threshold.

[0071] Figure 3 A block diagram of an exemplary robot 100 provided by an exemplary embodiment is shown, including an RL agent 102 that performs step 22 of method 20 to learn a first policy to maximize the cumulative reward of a first task (Task 1).

[0072] The robot 100 includes a sensor 110 and a state processor 112. The sensor 110 is used to sense the environment of the robot 100 and generate observations of the environment when the robot 100 interacts with the environment. The sensor 110 sends the observations of the environment to the state processor 112. The sensor 110 may include a camera, a detection and ranging sensor, such as a light detection and ranging (LIDAR) sensor, an accelerometer, a thermometer, etc. The state processor 112 receives the observations of the environment from the sensor 110 and processes the observations to generate a state representing the environment of the robot 100. t , and the state s t Provided to RL agent 102.

[0073] like Figure 3 As shown, the RL agent 102 includes a reward processor 114, a policy update processor 116, and a policy network 118 that models a first policy π(s). The policy network 118 includes a hidden layer that forms a feature network 124 (e.g., a feature network φ(s|w) that generates features). The first policy π(s) modeled by the policy network 118 transforms the state s t Mapped to action a t The policy network 118 has learnable parameters (e.g., weights). The RL agent 102 performs step 22 of method 20 to learn a first policy π(s), as shown below. The reward processor 114 receives the state s generated by the state processor 112. t , and based on the state s t Calculate reward r t The reward processor 114 will be based on the state s t The calculated reward r t The policy update processor 116 receives the current state s from the state processor 112. t and the reward r calculated by the reward processor 114 t(i.e., when performing action a t Then the reward r of the first strategy π(s) t ), and receive the next state s t+1 The policy update processor 116 collects the state s t 、Action a t , reward r t and the next state s t+1 , and creates a transition tuple that is used by the RL algorithm to compute updates to the parameters (e.g., weights) of the policy neural network 118. The computed updates to the parameters (e.g., weights) of the policy network 118 depend on the RL algorithm executed by the policy update processor 116. In some embodiments, the policy update processor 116 includes an additional regularization operation when learning updates to the parameters (e.g., weights) of the feature network 124 (e.g., a feature network φ(s|w) that is part of the policy network 118 (i.e., a hidden layer of the policy network 118)) to ensure that the features generated by the feature network 124 (e.g., the feature network φ(s|w) are minimally correlated with each other). In an exemplary embodiment, given the next action without any other information, the state s t Sufficient to predict future states s in the environment t+1、 s t+2 、……、s t+n In other words, no historical states are needed to predict future states in the environment given the next action.

[0074] In some exemplary embodiments, the first policy π(s) learned is t The action a that maximizes the value function t , and the policy network 118 is a value network that maps states and actions to values. Examples of value networks include deep Q-networks (DQNs).

[0075] As described above, the RL agent 102 performs step 22 of method 20 to learn a first policy π(s) to maximize the cumulative reward for the first task. The policy network 118 includes hidden layers forming a feature network 124 (e.g., feature network φ(s|w)) and a feature map action a generated by the feature network 114 (e.g., feature network φ(s|w)). t The other layers 126 of φ (denoted by f(φ)). The features generated by the feature network 124 (eg, feature network φ(s|w)) may be minimally correlated with respect to each other.

[0076] The operation of the robot 100 when step 20 is performed by the RL agent 102 will now be described. The sensor 110 of the robot 100 observes the environment and sends the observations of the environment to the state processor 112. The state processor 112 receives the observations of the environment and generates the current state of the environment s t , and the current state s t Provided to the reward processor 114 and the policy network 118. The reward processor 114 receives the current state s t , calculate the reward r for the first task (task 1) t The policy network 118 is based on the current state s t In generating action a t After a time step (t), due in part to the action a performed by the robot 100 t , the policy update processor 116 receives the next state s of the environment t+1 , and generate the state s t 、Action a t , the reward r of the first policy after executing the action t and the next state s t+1 The policy processor 116 then calculates updates to the parameters of the policy network 118 based on the transition tuples collected in the environment (possibly including historical transition tuples generated in the past). The policy network 118 outputs an action given the current state to maximize the cumulative reward of the first policy.

[0077] The learned policy network 118 (i.e., the policy network 118 that has learned parameters) is based on the current state s t Generate action a t , and the generated action a t is sent to the controller 120. For example, the action a generated by the learned policy network 118 t The controller 120 is used to process each action a received from the policy network 118 of the RL agent 102. t and sends corresponding control signals to one or more actuators 122 to perform action a t (e.g., motor control, electrical activation).

[0078] An example of the algorithm (hereinafter referred to as Algorithm 1) performed by step 22 of method 20 will now be described. The routine implementing step 22 performs Algorithm 1. In this embodiment, the first policy π(s) is parameterized by θ and w, and is mathematically represented by π(s) = f[φ(s|w)|θ], where φ(s|w) is a feature network parameterized by w. Algorithm 1 is used to learn a first policy π(s) that maximizes a value function under a given state. The value function can be obtained from the action-value function Q(s,a). The action-value function Q(s,a) can be modeled as a neural network (e.g., an action-value network) that maps states and actions to values. The action-value function Q(s,a) (e.g., parameters of the action-value network) is learned using a deep Q-learning algorithm. Learned action-value function Q(s,a) = φ(s|w) T W a , where W a is of size (n,n a ) weight matrix, where n a is the number of actions in the action space. Then, the first policy π(s) is determined as follows:

[0079]

[0080] The sample covariance matrix of the features generated by the feature network φ(s|w) is defined as:

[0081]

[0082] Here, Φ=[φ(s1)…φ(s N )] T and {s1…s N} is an example of a state.

[0083] The off-diagonal elements of the covariance matrix correspond to the sample cross-covariance of the features, that is, for i>j, Therefore, when When the off-diagonal elements of are zero, the features are minimally correlated. The features are least correlated when the off-diagonal elements of are minimized or close to zero. The diagonal elements correspond to the variance of the features. Regularization is an additional loss used when learning the update of the parameters (such as weights) of the action-value network. When the non-diagonal elements of , additional loss terms are added, namely:

[0084]

[0085] The decorrelation regularizer objective is combined with the main RL loss. For example, see HYLKBMavrin, “Deep Reinforcement Learning with Decorrelation,” May 8, 2019, [Online]; Accessed from: https: / / arxiv.org / abs / 1903.07765, and DGACBMavrin, “Efficient decorrelation of features using Gramian in Reinforcement Learning,” November 19, 2019, [Online]; Accessed from: https: / / arxiv.org / abs / 1911.08610, [Accessed January 17, 2020], both of which are incorporated by reference into this paper. For DQN, the loss is:

[0086] L RL =E[r+γmax a′ Q(s′,a′;θ target )-Q(s,a;θ i )] 2 .

[0087] Here, Q(s,a;θ) is the Q-value function in DQN.

[0088] As described above, in step 22 of method 20, receiving λ>0,γ∈[0,1] and r of task 1 I Reward. Algorithm 1 includes the following steps:

[0089] In step 1: initialize the depth policy π(s) = f[φ(s|w)|θ].

[0090] In step 2: sample the initial state from the distribution s~v(s).

[0091] In step 2.1: Initialize s′=s.

[0092] In step 3: When s′ is not a final state, execute steps 3.1 to 3.6:

[0093] In step 3.1: select an action a←π(s).

[0094] In step 3.2: perform action a, observe r I ,s′.

[0095] In step 3.3: transform (s,a,r I,s′) is stored in the replay buffer D.

[0096] In step 3.4: sample b transformations (mini-batches) from the replay buffer D, i.e. {(s,a,r I ,s′) b}~D.

[0097] In step 3.5: minimize the loss L on the mini-batch by using gradient descent method RL (π|{(s,a,r I ,s′) b})+λ∑ i>j cov[φ i ,φ j |{s b}] 2 (For example, maximize the cumulative reward r of task 1 I ) to update the policy weights w,θ.

[0098] In step 3.6: update s=s'.

[0099] In step 4: Estimate the variance of each feature i=1...n on the replay buffer D of size N, i.e.

[0100] As described above, the feature network φ(s|w) is a hidden layer of the first policy network π(s), and is therefore a subnetwork of the policy network 118. In step 22, the feature network φ(s|w)i is extracted from the first policy network 118, and the extracted feature network and the estimated feature variance are output. Among them, i=1...n.

[0101] An example of the algorithm (hereinafter referred to as Algorithm 2) executed by step 24 of method 20 will now be described. The routine implementing step 24 performs Algorithm 1. As described above, in step 24, the extracted feature network φ(s|w) and the estimated feature variance are received. Where i = 1...n. The features generated by the feature network φ(s|w) will be based on the corresponding feature variance (where i=1...n) are sorted and selected. For example, a feature with zero variance will generate a zero feature reward, which will generate a trivial sub-goal. In some embodiments, these features are not selected and can be filtered out. In addition, human experts can provide insights into the "usefulness" of features by studying the activation map of each feature using, for example, the Grad-CAM method, as described in the following document: RRaCMaDAaVRaPDaBDSelvaraju "Grad-cam: Visual explanations from deep networks via gradient-based localization", Proceedings of the IEEE International Conference on Computer Vision (2017), which is incorporated herein by reference. In step 24, the RL agent 102 can receive information related to the features to be learned through the received code. Alternatively, the user can select the features to be learned using a user input interface of the computing device, and the RL agent can receive information related to the features to be learned from the computing device.

[0102] Algorithm 2 includes the following steps:

[0103] In step 1: by the corresponding feature variance Sort the features in descending order.

[0104] In step 2: select the variance above a fixed threshold characteristics.

[0105] In step 3: remove all features except the top k features sorted by variance I←{i∈I|i≤k}.

[0106] The output generated by Algorithm 2 is a reduced feature set I, which is sorted by feature variance (largest to smallest).

[0107] In some embodiments, routine 24 is optional. In some embodiments, routine 24 is performed after routine 26 rather than before routine 26. In these embodiments, all features generated by the feature network φ(s|w) are sorted and selected.

[0108] Now refer to Figure 4 , a block diagram of an exemplary robot 100 provided by an exemplary embodiment is shown, the exemplary robot 100 including an RL agent 102 that performs step 26 of method 20 to learn an option policy for each selected feature. Figure 4In step 28, the RL agent 102 learns a selection strategy for each selected feature and stores the learned selection strategy for each selected feature for subsequent use (e.g., when the RL agent 102 performs step 28 (see Figure 5 )). The RL agent 102 learns each selected feature φ generated by the feature network φ(s|w) i Option strategy (e.g., option 1, option 2, ..., option n ). Figure 4 The robot 100 shown in FIG. 1 includes Figure 3 The robot 100 also includes the same components of the robot 100 shown in FIG. 1 (ie, sensors 110 , state processor 112 , reward processor 114 , one or more controllers 120 , and actuators 122 ). Figure 4 The RL agent 102 of the robot 100 is shown as Figure 3 The components shown (for ease of illustration) Figure 4 In addition to the above (not shown), it also includes multiple option strategy update processors 216 and multiple neural networks 218. Each corresponding option strategy (option i ) is modeled by one of the plurality of neural networks 218 (hereinafter generally referred to as option policy network 218 and individually referred to as option policy network 218), which models the state s t Mapped to action a t Each option policy network 218 has learnable parameters (e.g., weights). Option policy update processors 216 are associated with option policy networks 218 in a one-to-one fashion (i.e., each option policy update processor 116 is associated with one of the option policy networks 218). The RL agent 102 stores the feature φ for each choice i Learning option strategy (option i ) for use when RL agent 102 performs step 28 of method 20.

[0109] exist Figure 4 In the example shown, the feature φ with the largest feature variance is first selected i Learn about options strategies i For example, use the argmax function of the feature variances generated in step 22 to identify the feature φ with the highest feature variance i . When learning the feature φ with the highest feature variance i After the option policy, the RL agent 102 then uses a similar argmax function (excluding the option i ) Learn the option strategy for the feature with the next highest feature variance, and so on, until all selected features are used to learn the corresponding option strategy.n ) is the feature with the smallest feature variance t i .

[0110] Each option policy is learned by having the robot 100 interact with the environment. The feature reward processor 214 receives as input the ranked list of selected features generated in step 24 and the feature network φ(s|w) generated in step 22. The feature reward processor 214 receives the current state from the state processor 112 and outputs a pseudo reward associated with the feature selected from the ranked feature list. The policy update processor 216 receives the feature reward and the current state and runs the option policy and updates the policy network 118 by storing the parameters of each individually learned option policy in the policy network 118. For each selected feature, the RL agent 102 learns a single option policy from the state transition tuples collected in the environment of state, action reward, and next state tuples to maximize the cumulative feature reward (pseudo reward).

[0111] Given the feature set I generated in step 24, determine the corresponding feature reward function set {r i |i∈I}, as shown below:

[0112]

[0113] Feature reward function Form a sub-goal to maximize the selected feature φ i For each sub-goal, learn the corresponding option strategy (option i ).

[0114] An example of an algorithm (hereinafter referred to as Algorithm 3) executed by step 26 of method 20 will now be described. The routine implementing step 26 executes Algorithm 3. Algorithm 3 calculates for each selected feature {π i |i∈I} generates option strategies. Algorithm 3 includes the following steps:

[0115] Step 1 is performed for each feature i∈I, learning the corresponding option strategy π i (s), including steps 1.1 to 1.3:

[0116] In step 1.1: Initialize the option policy π i (s|w i ).

[0117] In step 1.2: Sample the initial state from the distribution s~ν(s).

[0118] In step 1.2.1: initialize s′=s.

[0119] In step 1.3: When s′ is not a final state, perform steps 1.3.1 to 1.3.7 as follows:

[0120] In step 1.3.1: select action a←π i (s).

[0121] In step 1.3.2: perform action a and observe s′.

[0122] In step 1.3.3: Calculate the reward r for feature i i (s,a,s′)=φ i (s′)-φ i (s).

[0123] In step 1.3.4: transform (s,a,r i ,s′) is stored in the replay buffer D.

[0124] In step 1.3.5: From the replay buffer {(s,a,r,s′) b}~D samples b transformations (mini-batch).

[0125] In step 1.3.6: By minimizing the loss L of the mini-batch using the gradient descent method RL (π|{(s,a,r,s′) b}) (e.g., maximize reward) to update the option strategy weight w i .

[0126] In step 1.3.7: update s=s'.

[0127] Algorithm 3 For each selected feature {π i |i∈I} generates option strategies.

[0128] Now refer to Figure 5 , shows a block diagram of an exemplary robot 100 including an RL agent 102 that performs step 28 of method 20 to learn a second policy to maximize a second cumulative reward for a second task (Task 2). The second task is different from the first task (Task 1). Figure 5 The robot 100 shown in FIG. 1 includes Figure 3 The robot 100 also includes a reward processor 314. Figure 5 The RL agent 102 of the robot 100 is shown as Figure 3 and Figure 4 The components shown (for ease of illustration) Figure 5In addition to the above (not shown), the system also includes a policy update processor 316, an option policy processor 336 and a policy network 318 for modeling the second policy μ.

[0129] As noted above with respect to step 28, RL agent 102 receives the option policy (e.g., option i ,……,options n ) and form an enhanced action space 130, which includes the learned option policy (e.g., the option policy learned in step 26) and primitive actions 134 that the RL agent 102 can use for the second task (Task 2). The primitive action set includes m primitive actions (e.g., action 1, action 2, ..., action n ).therefore, Figure 5 The RL agent 102 shown in FIG includes an augmented action space 130 from which a policy processor 318 selects n option policies (e.g., option i ,……,options n ) or m primitive actions (e.g., action 1, action 2, ..., action n The second policy μ maps states to actions and is modeled by a neural network (policy network 318) that selects one of n option policies (e.g., option i ,……,options n ), or optionally select one or more of the m primitive actions (e.g., action 1, action 2, ..., action n ).

[0130] The reward processor 314 receives the current state of the environment from the state processor 112. The reward processor 314 calculates the reward for the second task (Task 2) based on the current state, which is used by the RL agent 102 to learn a second policy μ that maximizes the cumulative reward of the second task (Task 2).

[0131] exist Figure 5 , the policy network 318 selects one of the option policies 218 or m primitive actions (e.g., action 1, action 2, ..., action 3) from the enhanced action space 130. n ). When one of the option strategies 218 is selected, the option strategy processor 336 executes the selected option strategy, receives the current state from the state processor 112, and outputs an action to be performed by the robot 100. The option strategy processor 336 generates actions based on the current state until a termination condition is reached. The option strategy processor 336 may terminate generating actions based on the current state in one step, or may terminate generating actions based on the current state when the future reward of the selected option strategy 218 is maximized.

[0132] The option policy processor 336 is called only when an option policy is selected from the enhanced action space 130. If a primitive action is selected from the enhanced action space 130, the option policy processor 336 receives the selected primitive action (e.g., action i ) and outputs the selected primitive action (e.g., action i ). RL agent 102 outputs primitive actions to be executed by robot 100 (e.g., actions i ).

[0133] In this example, given the received state s t and reward r t , the second policy μ learns which option to choose from the available options and primitive actions i For the received state s not shown here t and reward r t For other possible instances, the second policy μ can learn to choose different option strategies, for example, in addition to the action i Any other available option strategy other than Action 1, Action 2, ..., Action n , or you can choose from any primitive action.

[0134] An example of the algorithm (hereinafter referred to as Algorithm 4) executed by step 28 of method 20 will now be described. The routine implementing step 28 executes Algorithm 4. As described above, in step 28, the option policy {π i |i∈I},λ>0,γ∈[0,1]and r II Reward. Step 28 outputs a new (e.g., second) policy μ that maximizes the cumulative reward for the second task (Task 2). As described above, potential option policies (Option 1, Option 2, ..., Option 1) may be selected for learning the second policy μ. n ) may also include preset or pre-programmed primitive actions 134.

[0135] In step 1: Use the set of option strategies learned in step 26 Expand the primitive action space, that is, generate a new enhanced action space A + ,in,

[0136]

[0137] In step 2: Initialize the policy μ(s|w μ ):S→A + .

[0138] In step 3: sample the initial state from the distribution s~v(s).

[0139] In step 3.1: Initialize s′=s.

[0140] In step 4: When s′ is not a final state, execute steps 4.1 to 4.6:

[0141] In step 4.1: select action a←μ(s)

[0142] In step 4.2: perform action a, observe r II ,s′.

[0143] In step 4.3: transform (s,a,r II ,s′) is stored in the replay buffer D.

[0144] In step 4.4: From the replay buffer {(s,a,r II ,s′) b}~D samples b transformations (mini-batch).

[0145] In step 4.5: minimize the loss L on the mini-batch by using gradient descent method RL (μ|{(s,a,r II ,s′) b})(For example, maximize the reward r of task 2 II ) to update the strategy weight w μ .

[0146] In step 4.6: update s=s'.

[0147] Algorithm 4 outputs a new (eg, second) policy μ) that maximizes the cumulative future reward of the new (eg, second) task (Task 2).

[0148] refer to Figure 5 , the RL agent 102 receives the state s generated by the reward processor 314 for the second task (Task 2) t and reward r t , and the option processor 336 selects one of the learned option policies or primitive actions from the augmented action space 130 to perform the corresponding action in order to maximize the cumulative future reward of the second task (Task 2).

[0149] In some embodiments, steps 22, 24, and 26 of method 20 are repeated for one or more other tasks (e.g., Task 3, Task 4, etc.) to increase the number of learned potential option policies that can be used to learn the second policy μ in step 28. In these embodiments, the learned option policies are then appended to the action space (i.e., the space of available actions), and the option policies are used to learn the second policy μ as described above.

[0150] It will be appreciated that exemplary embodiments of the robot 100 may be data-driven, may not require human demonstrations, may not require reward / task engineering, may be sample efficient by reusing skills for new tasks, may have unique learned skills (besides trivial zero-variance features), and may scale to high-dimensional problems.

[0151] It will be appreciated that exemplary embodiments of the RL agent 102 are scalable to complex tasks and are data-driven, obviating the need for manual engineering skills and human demonstrations in exemplary embodiments.

[0152] Software coding for performing the methods and functions described above is within the scope of exemplary embodiments considered by one of ordinary skill in the art. Machine-readable code executed by one or more processors of one or more corresponding devices to perform the above-described methods may be stored in a machine-readable medium, such as a memory of a vehicle control system or a memory of a neural network controller (not shown). The steps and / or operations in the flowcharts and figures described herein are for example purposes only. There may be many variations to these steps and / or operations without departing from the teachings of the described exemplary embodiments. For example, the steps may be performed in a different order, or steps may be added, deleted, or modified.

[0153] All values ​​and subranges within the disclosed ranges are also disclosed. In addition, although the systems, devices, and processes disclosed and illustrated herein may include a specific number of elements / components, these systems, devices, and assemblies may be modified to include more or fewer such elements / components. For example, although any disclosed element / component may be singular, the embodiments disclosed herein may be modified to include multiple such elements / components. The subject matter described herein is intended to cover and encompass all appropriate technical variations.

[0154] Although some exemplary embodiments may be described at least in part in terms of methods, one of ordinary skill in the art will understand that the exemplary embodiments also relate to various components for performing at least some aspects and features of the described methods through hardware (digital signal processor (DSP), application-specific integrated circuit (ASIC) or field programmable gate array (FPGA)), software, or a combination thereof. Therefore, the technical solutions of the exemplary embodiments may be embodied in a non-volatile or non-transitory machine-readable medium (e.g., an optical disk, flash memory, etc.) having executable instructions tangibly stored therein that cause a processing device (e.g., a vehicle control system) to perform the examples of the methods disclosed herein.

[0155] The exemplary embodiments may be embodied in other specific forms without departing from the subject matter of the claims. The described exemplary embodiments are intended in all respects to be illustrative and not restrictive. The described examples are intended to cover and encompass all suitable technical variations. Therefore, the scope of the present invention is to be determined by the appended claims rather than by the foregoing description. The scope of the claims should not be limited by the embodiments described in the examples, but should be given the broadest interpretation consistent with the entire description.

Claims

1. A robot, characterized in that: include: Memory; a processing unit configured to execute instructions of the agent stored in the memory to: learning a first policy to maximize a cumulative reward for a first task in an environment, wherein the first policy is learned using a reinforcement learning algorithm and first transition tuples collected in the environment, wherein each first transition tuple includes a state, an action, and a reward of the first policy after performing the action; Extract feature networks from neural networks; Calculating the variance of each feature output by the feature network; selecting at least one of the features based on the calculated variance; For each selected feature, learning a selection policy based on second transition tuples collected in the environment to maximize the cumulative feature reward of the selected feature, and storing the learned selection policy for the selected feature in an augmented action space, wherein each second transition tuple includes a state, an action, a feature reward, and a next state; A second policy is learned to maximize a second cumulative reward for a second task by selecting one of the learned option policies in the reinforced action space and using a reinforcement learning algorithm and third transition tuples collected in the environment, wherein each third transition tuple includes a state, the selected option policy, a reward of the selected learned option policy after performing the action generated by the selected learned option policy, and a next state.

2. The robot according to claim 1, characterized in that Selecting at least one of the features based on the calculated variance includes selecting at least one feature that exceeds a predetermined threshold variance.

3. The robot according to claim 1 or 2, characterized in that: The selecting at least one of the features includes sorting the features from highest variance to lowest variance and selecting a specified number of features having the highest variance.

4. The robot according to claim 1, characterized in that The feature network generates features that are minimally correlated with each other.

5. The robot according to claim 1, characterized in that Each learned option policy maps states to actions.

6. The robot according to claim 1, characterized in that The first policy is modeled as a neural network that maps states to actions.

7. The robot according to claim 1, characterized in that The first policy is an action that maximizes a value function in a state, and the value function is modeled as a neural network that maps states and actions to values.

8. The robot according to claim 1, wherein: The processing unit is further configured to execute other instructions of the agent stored in the memory to store primitive actions in the augmented action space and learn the second policy by selecting primitive actions and executing the primitive actions for one time step.

9. The robot according to claim 1, characterized in that The second policy is learned by executing the selected learned option policy for one or more time steps until a future cumulative reward of the selected learned option policy is maximized.

10. A method for learning reusable options for transferring knowledge between tasks, characterized in that The robot according to any one of claims 1 to 9 comprises: learning a first policy to maximize a cumulative reward for a first task in an environment, wherein the first policy is learned using a reinforcement learning algorithm and first transition tuples collected in the environment, wherein each first transition tuple includes a state, an action, and a reward of the first policy after performing the action; Extract feature networks from neural networks; Calculating the variance of each feature output by the feature network; selecting at least one of the features based on the calculated variance; For each selected feature, learning a selection policy based on second transition tuples collected in the environment to maximize the cumulative feature reward of the selected feature, and storing the learned selection policy for the selected feature in an augmented action space, wherein each second transition tuple includes a state, an action, a feature reward, and a next state; A second policy is learned to maximize a second cumulative reward for a second task by selecting one of the learned option policies in the augmented action space and using a reinforcement learning algorithm and third transition tuples collected in the environment, wherein each third transition tuple includes a state, the selected option policy, a reward of the selected learned option policy after performing the action generated by the selected learned option policy, and a next state.

11. The method according to claim 10, characterized in that Selecting at least one of the features based on the calculated variance includes selecting at least one feature that exceeds a predetermined threshold variance.

12. The method according to claim 10 or 11, characterized in that The selecting at least one of the features includes sorting the features from highest variance to lowest variance and selecting a specified number of features having the highest variance.

13. The method according to claim 10, characterized in that The feature network generates features that are minimally correlated with each other.

14. The method according to claim 10, characterized in that Each learned option policy maps states to actions.

15. The method according to claim 10, characterized in that The first policy is modeled as a neural network that maps states to actions.

16. The method according to claim 10, characterized in that The first policy is an action that maximizes a value function in a state, and the value function is modeled as a neural network that maps states and actions to values.

17. The method according to claim 10, wherein: It also includes storing primitive actions in the enhanced action space and learning the second policy by selecting primitive actions and executing the primitive actions for one time step.

18. The method according to claim 10, wherein: Learning the second policy includes executing the selected learned option policy for one or more time steps until a future cumulative reward of the selected learned option policy is maximized.

19. A computer-readable medium, characterized in that comprising instructions stored thereon which, when executed by an agent of the robot, cause the agent to perform the method according to any one of claims 10 to 18.

20. A computer program product, characterized in that comprising instructions that, when executed by an agent of the robot, The agent is caused to execute the method according to any one of claims 10 to 18.

Images