Efficient Adaptation of Robot Control Strategies for New Tasks Using Meta-learning Based on Meta-imitation Learning and Meta-reinforcement Learning

Abstract

Techniques are disclosed that can use imitation learning and reinforcement learning to train a meta-learning model for use in causing a robot to perform a task. Some embodiments involve training the meta-learning model using imitation learning based on one or more human-guided demonstrations of the task. Additional or alternative embodiments involve training the meta-learning model using reinforcement learning based on trials of the robot attempting to perform the task. Further embodiments involve using the trained meta-learning model to learn a new task with a few samples (or a single sample) based on human-guided demonstrations of the new task.

Description

Background Art

[0001] Many robots are programmed to grasp one or more objects using one or more end effectors. For example, a robot can utilize a grasping end effector, such as an "impact" gripper or an "intrusive" gripper (e.g., physically penetrating an object using a pin, needle, etc.) to pick up an object from a first location, move the object to a second location, and then drop the object at the second location. Some other examples of robot end effectors that can grasp objects include a "retracting" end effector (e.g., using suction or vacuum to pick up an object) and one or more "adjacent" end effectors (e.g., using surface tension, freezing, or adhesives to pick up an object), to name a few. Summary of the invention

[0002] The technology disclosed herein is directed to training a robotic agent using meta-learning so that the agent can learn to perform a new task based on a few demonstrations of the task (e.g., a single demonstration, two demonstrations, less than 5 demonstrations, less than 10 demonstrations, etc.) and a few trial and error attempts to perform the task (e.g., a single trial attempt, two trial attempts, less than 5 trial attempts, less than 10 trial attempts, etc.). The technology disclosed herein is further directed to utilizing such a robotic agent when performing a new task. Meta-learning enables a robotic agent to learn how to learn a new task. A meta-learning model of a robotic agent can be initially trained using a meta-training process to learn how to learn using training tasks. The trained meta-learning model can then learn a new task through a meta-testing process based on some demonstrations of the new task and some trial and error attempts of the agent performing the task.

[0003] In some embodiments, the meta-learning model may include an experimental strategy for collecting information about a new task through imitation learning. One or more demonstrations of the new task may be used to generate an experimental strategy for the new task. The experimental strategy can then be used to generate one or more trial and error attempts to perform the task. In some embodiments, the experimental strategy can be used to constrain the potential actions taken by the agent when attempting to perform the task in the trial and error attempts. Additionally or alternatively, the meta-learning model may include an adaptive experimental strategy that can be used to extract information from the experiments and combine it with the demonstrations to learn how to perform the new task. The adaptive experimental strategy can be trained using reinforcement learning.

[0004] In some embodiments, the test strategy and the adaptive test strategy can be combined into a model with shared parameters, wherein the test strategy and the adaptive test strategy are trained simultaneously. For example, the meta-learning model can use imitation learning to learn the test strategy of the new task by processing the demonstration of the new task. The test strategy of the new task can be used to shape the actions taken by the agent when learning to perform the task through trial and error reinforcement learning. The test strategy for the new task can be used to generate one or more trials in which the robot attempts to perform the new task. One or more trials can be used to train the adaptive strategy to perform the new task using reinforcement learning. Since the test strategy and the adaptive test strategy share parameters, training the adaptive strategy via reinforcement learning will also update one or more parts of the test strategy. The updated test strategy can then be used to generate additional trials for the new task, and these additional trials can be used to further train the adaptive test strategy. In other words, when training the meta-learning model, trials for new tasks can be continuously generated based on the current test strategy parameters.

[0005] In some embodiments, Q-learning can be used to train a meta-learning model. Q-learning learns a policy network that can be used to determine what actions an agent (e.g., a robot) should take under what circumstances (e.g., based on current robot state data and / or current environment state data). In some embodiments, a finite Markov decision process can be used to represent a task. Q-learning can be used to find an optimal policy that maximizes the expected value of a reward by taking consecutive steps starting from the current state in the finite Markov decision process. In other words, Q-learning can be used to find an optimal policy for performing a new task.

[0006] In some embodiments, actor-critic reinforcement learning techniques can be used during meta-training, where off-policy task demonstrations (e.g., demonstrations of new tasks where the meta-learning model has not yet been trained to perform the new task) are utilized. A normalized advantage function can provide a single computational graph and / or objective for representing and optimizing the actor and the evaluator. The normalized advantage function can decompose the Q function into a value function and a quadratic advantage function. In some embodiments, the Q function can be learned by various Q-learning techniques, such as by minimizing the Bellman error. In some embodiments, the normalized advantage function of the Q-learning function can be used to train the trial strategy and adaptive trial strategy of the meta-learning model.

[0007] Additionally or alternatively, in some embodiments, the meta-learning model can be trained so that the trial strategy is separated from the adaptive trial strategy and optimized separately. In some of those embodiments, the trial strategy can be trained using imitation learning based on demonstrations of the agent performing a new task. The trial strategy can then be frozen and used to generate trial and error attempts to perform the new task. The adaptive trial strategy can then be trained to perform the new task using reinforcement learning and trial and error attempts.

[0008] In some embodiments, the demonstration of the robot performing the task may include a human-guided demonstration, in which a human demonstrates the task while controlling the robot. For example, the demonstration may be a human-guided demonstration via a virtual reality interface, in which a human demonstrates the robot task (through interaction in the virtual reality interface) without directly contacting the corresponding robot. As another example, the demonstration may be a human-guided demonstration as a kinesthetic demonstration, in which a human's body contacts and guides the robot to demonstrate the robot task. Additionally or alternatively, a human may provide a reward signal used when training a meta-learning model via reinforcement learning. For example, a human may provide a binary reward that indicates whether the robot successfully performs a task in a trial-and-error attempt. For example, a human may provide a reward (e.g., a binary reward) via a verbal user interface input (e.g., "correct" or "incorrect") or via a touch user interface input (e.g., selecting "thumbs up" or "thumbs down").

[0009] In some embodiments, a meta-learning model can be trained with a few human-guided demonstrations (e.g., less than 10 human-guided demonstrations, less than 100 human-guided demonstrations, less than 500 human-guided demonstrations, less than 1000 human-guided demonstrations, etc.). Several human-guided demonstrations can be used to autonomously generate additional training demonstrations without requiring an observer to indicate whether the additional training demonstrations successfully completed the task. For example, the meta-learning model can include an embedding network part and a control network part, wherein the embedding network and the control network are jointly trained using the same meta-strategy.

[0010] Demonstration pairs of the same task can be used to train the embedding network to generate a task embedding that embeds each demonstration into the task space (i.e., the latent space of the robot's tasks). When the task embedding is conditioned, the control network can be trained to perform the task. In some embodiments, the control network can be used to autonomously generate a test (also referred to as a rollout) of a robot performing a task based on the task embedding of a manually guided demonstration. The task embedding of the rollout can be compared with the task embedding of the corresponding manual demonstration to determine whether the robot performs the same task (e.g., completes the task) in the rollout. For example, a task space can be generated so that the distance between demonstrations of the same task is small, while the distance between demonstrations of different tasks is large. In other words, when the robot completes the same task, the distance between the first demonstration embedding and the manually guided demonstration embedding can be very small (conversely, when the robot does not complete the same task, the distance can be very large).

[0011] If the system determines that the artificially guided demonstration-first presentation pair is the same task based on the distance between the task embeddings, then the artificially guided demonstration-first presentation pair can be stored as additional training data. This additional training data can be used to train the meta-policy network, so the meta-policy network can be trained using autonomously generated training data without the need for an artificially provided reward signal. In other words, the meta-policy network can be conditioned on several artificially guided demonstration pairs. The meta-policy conditioned on several artificially guided demonstration pairs can be used to autonomously generate additional artificially guided demonstration-first presentation pairs. These additional artificially guided demonstration-first presentation pairs can then be used to train the meta-policy.

[0012] Additionally or alternatively, the rollouts may be stored in a rollout database. The task embeddings of the rollouts may be compared to determine whether any rollout pair corresponds to the same task. In other words, the distance between the task embeddings of the rollouts may determine whether the rollout pair corresponds to the same task. Rollout-rollout pairs corresponding to the same task may be used to experiment with meta-strategies.

[0013] Therefore, various embodiments provide techniques for training a meta-learning model to learn new tasks using meta-imitation learning and meta-reinforcement learning with a small sample size (or single sample learning). Traditional meta-learning techniques cannot utilize artificially guided demonstrations and trial-and-error attempts when learning to perform new tasks. In contrast, the techniques described herein can utilize demonstrations and trial-and-error attempts, thereby enabling the meta-learning model to be trained using a smaller amount of training data (e.g., using a single artificially guided demonstration to train the model, using less than 5 artificially guided demonstrations to train the model, etc.). Training the meta-learning model with only a few artificially guided demonstrations can save computing resources (e.g., processor cycles, memory, battery power, etc.) by reducing the amount of training data (i.e., artificially guided demonstrations) that needs to be captured and provided to the system. Additionally or alternatively, when training a meta-learning model, computing resources are saved by reducing the number of training episodes (episodes) required to train the model to perform a task.

[0014] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 An exemplary environment is illustrated in which embodiments described herein may be implemented.

[0016] Figure 2A An exemplary meta-learning model according to implementations described herein is illustrated.

[0017] Figure 2B Another exemplary meta-learning model according to implementations described herein is illustrated.

[0018] Figure 3 is a flowchart illustrating an exemplary process of training a meta-learning model according to embodiments disclosed herein.

[0019] Figure 4 is a flowchart illustrating an exemplary process of training a meta-learning model using imitation learning according to an embodiment disclosed herein.

[0020] Figure 5 is a flow chart illustrating an exemplary process for generating experiments for use in training a meta-learning model using reinforcement learning according to embodiments disclosed herein.

[0021] Figure 6 is a flow chart illustrating an exemplary process of training a trained meta-learning model for a new task according to embodiments disclosed herein.

[0022] Figure 7is a flowchart illustrating another exemplary process of training a meta-learning model according to embodiments disclosed herein.

[0023] Figure 8 is a flowchart illustrating an exemplary process of training a meta-learning model using imitation learning and reinforcement learning according to an embodiment disclosed herein.

[0024] Fig. 9 is a flow chart illustrating another exemplary process of training a trained meta-learning model for a new task according to embodiments disclosed herein.

[0025] Fig.10 An exemplary architecture of a robot is schematically depicted.

[0026] Fig.11 An exemplary architecture of a computer system is schematically depicted. DETAILED DESCRIPTION

[0027] Imitation learning can allow robotic agents to learn complex behaviors from demonstrations. However, for example, learning vision-based tasks may require an impractical number of demonstrations. Meta-imitation learning is an approach that aims to enable robotic agents to learn new tasks from one or a few demonstrations by leveraging previous data. However, without actually trying the task, a few demonstrations may not provide enough information to successfully infer a policy. The techniques described in this paper provide a method that can "learn to learn" from both demonstrations and trial-and-error experience with sparse reward feedback. In contrast to meta-imitation learning, the techniques described in this paper enable robotic agents to improve themselves autonomously beyond demonstration data. Compared to meta-reinforcement learning, the techniques described in this paper can be extended to a wider distribution of tasks because demonstrations reduce the burden of exploration.

[0028] Imitation learning enables robotic agents to learn complex behaviors from demonstrations that are simple and / or intuitive for users to provide. However, learning to express neural network policies from imitation requires a large number of demonstrations, especially when operating on high-dimensional inputs such as image pixels. Meta-imitation learning has emerged as a promising approach that allows robotic agents to leverage data from previous tasks in order to learn new tasks from only a few demonstrations. However, in many practical few-shot imitation settings, there is an identifiability problem: it may not be possible to accurately determine the policy from one or a few demonstrations without actually trying the task. For example, it may be difficult to discern from a demonstration where to grasp an object when it is in a new position and / or how much force to apply in order to slide the object without knocking it over. In essence, demonstrations often convey what the task requires, but do not always accurately explain how to accomplish it, especially in new situations. Furthermore, providing useful negative demonstrations can be challenging because it may not be obvious to the user where the agent's difficulty stems from. Alternatively, it is easy for users to provide feedback on success or failure, but exploratory interactions are useful for learning the task. Therefore, the techniques described in this paper are directed toward building a robotic agent that can first learn a policy from a single demonstration, then try that policy out as it receives binary user feedback, and finally use that feedback to refine its policy so that it can consistently solve a task. Not only does this enable the agent to eliminate its a posteriori estimate of the task from user feedback, but the continuous trial-and-error refinement process also enables continuous adaptation so that the robotic agent can get better and better at new tasks as it receives additional feedback.

[0029] This may require a certain amount of prior knowledge or experience, which can be acquired through meta-learning across a range of previous tasks. To this end, this paper describes a meta-learning process that combines elements of imitation learning with trial-and-error reinforcement learning. Compared to previous meta-imitation learning methods that learn a single-shot imitation learning process through imitation, the techniques described in this paper enable the agent to continuously improve on the test task through trial and error. In addition, this method is also of great appeal from the perspective of meta-reinforcement learning algorithms that aim to learn efficient reinforcement learning processes: as meta-reinforcement learning expands towards a wider distribution of tasks and increasingly general reinforcement learning processes, exploration and efficiency can become extremely difficult. However, demonstrations can significantly narrow the search space while also providing users with a practical means of communicating goals, enabling agents to achieve small-shot learning of behaviors.

[0030] In some embodiments, a meta-learning algorithm can be trained so that the meta-learning model can learn new behaviors with a single demonstration and a small number of trials. Specifically, after receiving a demonstration illustrating a new goal, the meta-trained agent can learn to achieve the goal through trial and error with only a few binary success or failure labels. This method can effectively learn tasks with new, retained objects using a single demonstration and a small number of trials, while significantly outperforming meta-imitation learning, meta-reinforcement learning, and previous methods that combine demonstration and reward feedback.

[0031] Learning to learn, or meta-learning, has a long history in the machine learning literature. In some embodiments, meta-learning methods build on and significantly improve upon meta-imitation learning and meta-reinforcement learning, and extend the model-independent meta-learning (MAML) algorithm. Unlike prior work in few-shot imitation learning, this approach enables the agent to additionally use trial-and-error experience to continuously improve by using an actor-evaluator model with an internal goal based on temporal difference errors. Compared to work in multi-task and meta-reinforcement learning, this approach learns to solve a meta-exploration problem using one demonstration. Additionally or alternatively, the technique may not require a large amount of on-policy data collection.

[0032] Some embodiments described herein are built on the idea of ​​meta-learning or learning to learn, with the goal of acquiring a policy for a new task using one demonstration and a short trial-and-error phase (e.g., a single trial-and-error phase, two trial-and-error phases, less than five trial-and-error phases, less than ten trial-and-error phases, etc.). A model-independent meta-learning algorithm (MAML) can enable an agent to combine complementary information from demonstrations and experiments to solve tasks with extremely high efficiency. The goal of a small sample meta-learning algorithm is to learn new tasks with only a few data points. Meta-learning methods achieve this goal by using data collected from many different tasks. These tasks may require different skills, such as grasping objects or sliding objects, manipulating different objects—such as grasping a cup or grasping a teapot, or recognizing and manipulating objects with different visual appearances and textures. Meta-learning assumes that these tasks have some shared structure (e.g., belonging to the class of desktop, object manipulation tasks) and come from some basic distribution At a high level, meta-learning algorithms aim to discover shared structure between meta-training tasks in a way that allows learning new meta-testing tasks more efficiently than learning from scratch. MAML aims to solve this problem by learning a set of initial parameters or features such that a few steps of gradient descent on a small amount of training data leads to effective performance on the task and generalization to new data points.

[0033] From the meta-training set Sampling a within-task training set for a single task and the in-task validation set The goal of MAML is to follow, with respect to the model parameters θ One or more steps of gradient descent on the same task loss, minimizing Task loss on

[0034] By differentiating (and thereby optimizing) the internal weight updates MAML can obtain the best gradient descent-based learner for a modest amount of data. Sampling This “internal” learning process is assessed.

[0035] In some embodiments, a task can be defined as a finite-scope Markov decision process with a state s, an action a, and a scalar reward r(s, a). In order to utilize off-policy data during meta-training, an actor-critic reinforcement learning technique can be established. In some embodiments, the normalized advantage function (NAF) technique can be used because it provides a simple way to represent and optimize the actor and the critiquer with a single computational graph and objective. Specifically, the normalized advantage function decomposes the Q-value function into a value function and a quadratic advantage function:

[0036] Q(s t , a t θ Q )=A(s t , a t θ A )+V(s t θ V )

[0037]

[0038] Among them, the parameter θ Q ={θ μ ,θ P ,θ V}θ Q ={θ μ ,θ P ,θ V}. This Q function can be learned by standard Q-learning techniques, for example by minimizing the Bellman error:

[0039]

[0040] In some implementations, it may be assumed that the task distribution, from which to extract meta-training tasks and the retained meta-test task During meta-training, expert demonstration trajectories can be used The form of supervision and can be used for each meta-training task The binary reward function r for the query i After meta-training with this supervision, the goal at meta-test time is to quickly learn the new meta-test task In some implementations, a single demonstration is provided to the robotic agent. The goal is to successfully complete the task within K trials, where for each trial the agent receives a binary reward.

[0041] The techniques disclosed herein can learn to learn from demonstrations and from its own trial-and-error episodes. In some embodiments, the techniques extend MAML: the agent is meta-trained so that given a demonstration of a new task and optionally some number of trial episodes, it can take a gradient step on an adaptive objective to successfully complete the task. This objective differs from the typical single-shot imitation learning problem in two ways: first, the agent can use both demonstrations and trials at the meta-test time, and second, it is crucial that the agent be able to infer a good policy with a variable number of trajectories because it will be learning incrementally at the meta-test time. Since the inner loop process is run at the meta-test time, the inner loop of meta-learning needs to be defined according to these two constraints.

[0042] In order to exploit both demonstration and trial episode information, the Q-value function Q(s, a; θ Q ), which is decomposed into a value and a normalized advantage function, as shown in Equation (2). Note that to combine both the expert's demonstrations and the agent's trials, it is not necessary to have an imitation learning objective, since reinforcement learning objectives can also be learned from successful trajectories. Therefore, in some embodiments, the Bellman error Can be used in the internal adaptation step. Adaptive Q-value function Can be compared with The corresponding batch of demonstrations and trial episodes {τ1, ..., τ K}Evaluated on Take the gradient step size to get:

[0043]

[0044] where the first episode τ1 is the demonstration, and where k∈{1, ..., K} are the trials made by the agent. In some embodiments, during meta-training, k can be varied between 1 and K (inclusive), so the robotic agent can infer good policies with different amounts of data.

[0045] When using the NAF representation of the Q-value function, with parameters Then we can get the strategy corresponding to the Q value function In some embodiments, Adaptive strategies can be used to learn the same task through direct imitation, behavioral cloning, and / or other imitation learning techniques. Another demonstration of τ * Perform meta-training:

[0046]

[0047] Combined, the overall meta-optimization goal is as follows:

[0048]

[0049]

[0050] Although it can be collected before meta-training time, it is represented as , but the trial episodes need to be chosen more carefully. Random episodes or episodes that are not close to solving the task will not be informative. Therefore, trial-and-error episodes should be collected from the agent while training. A distributed setting can be used when the training workers continuously sample data from the replay buffer. At the same time, a separate collection worker collects trial-and-error experience in the environment using the agent's latest saved weights and pushes these trial episodes into the replay buffer.

[0051] The embodiments described above with reference to equations (3)-(5) include a trial policy and an adaptive trial policy that share weights. One challenge with the trial policy and the adaptive trial policy sharing weights may be that updates to the adaptive trial policy will also affect the trial policy, thereby changing the distribution of trial trajectories that the adaptive trial policy should expect. This may lead to constantly recalling same-policy trial trajectories from the environment during meta-training, which is particularly difficult in real-world problem settings with a wide distribution of tasks.

[0052] In some embodiments, the test strategy and the adaptive test strategy can be represented and trained separately, decoupling their optimization. The test strategy and the adaptive test strategy can be parameterized by θ and φ, respectively. The parameterized test strategy can be represented as The parameterized adaptive test strategy can be expressed as In some embodiments, the training tracking strategy And freeze the weights of the experimental strategy, and for each meta-training task From the environment The collected test data can be used to train the adaptive test strategy without having to access the environment again.

[0053] The trial policy can be trained in a way that will provide useful exploration for the inference task. A simple and effective exploration strategy is posterior sampling or Thompson sampling, which greedily acts according to the policy's current belief about the task. To this end, the trial policy can be trained using a meta-imitation learning setting, where for any task Experiment with strategies in one or several training demonstrations {d i，k} is conditioned and trained to maximize another demonstration of the same task The possibility of the following actions (which can be obtained from Sampling without replacement). This leads to the goal:

[0054]

[0055] The adaptive trial strategy can be implemented in a similar way by using one or several trial trajectories {τ i，l} to train with additional conditions. In some embodiments, the test trajectory can be generated by executing the test strategy in the environment. In some embodiments, for any task A set of demo-test pairs can be generated In some embodiments, the adaptive trial objectives are:

[0056]

[0057] During meta-training, this can be accomplished by using mini-batches of tasks and the corresponding demonstration The trial policy is trained by minimizing Eq. (6). After training, θ can be frozen to have a fixed trial policy. A set of task demonstrations {d i，k}, and using the demonstration conditional experiment strategy One or several trial trajectories can be collected in the environment. The resulting demonstration-trial pairs ({{d i，k}, {τ i，l})} can be stored in the dataset In. This can be done by using small batch tasks and the corresponding demonstration test dataset Minimize Equation (7) to train the adaptive trial strategy At meta-test time, for any test task Experimental strategies can be used Generate test {τ j，l Then, you can use the adaptive experiment strategy Perform an adaptive trial episode.

[0058] The above embodiments of equations (1)-(7) use reward signals provided by human observers to generate additional training data. Additional or alternative embodiments may include autonomously generating additional training data. In some embodiments, meta-imitation can be used to learn from autonomous data collection while minimizing the amount of human supervision required for learning. A robotic agent can be guided from meta-imitation learning of, for example, human teleoperation episodes for several different tasks. This initial meta-imitation dataset can be used to (a) learn a meta-imitation policy that can perform various skills depending on the demonstrations it is conditioned on, and (b) learn a latent space of useful skills, which can later be used to characterize new skills discovered during autonomous operation. Using the learned meta-imitation policy, meaningful behavior episodes can be collected by randomly sampling from the space of useful skills (or tasks). These new episodes are then organized into a meta-imitation dataset using the learned latent space of skills. In some embodiments, the meta-strategy can be retrained using this expanded meta-imitation dataset.

[0059] In some embodiments, a meta-policy consisting of two networks can be trained: an embedding network and a control network. A dataset of paired demonstrations (i.e., the best trajectory pairs corresponding to the same task) can be used to train the meta-policy. In some embodiments, the meta-policy can be optimized so that the embedding network learns to embed each demonstration into a task space, where the task space is a latent space of useful skills. Additionally or alternatively, the control network can learn how to perform a task when conditioned on a specific skill in the form of a task embedding. In some embodiments, a meta-policy (also referred to as a behavioral policy) can be used to sample trajectories in a new environment, and these sampled trajectories (also referred to as debuts in this article) can be added to the dataset of debuts. In some embodiments, each collected debut can be embedded in the task space. The distance in the task space can be used to determine whether any two embedded debuts belong to the same task. If a pair of debuts is determined to belong to the same task, the pair can be added to the original dataset of the best demonstrations. In some embodiments, a new meta-policy can be learned from this expanded dataset.

[0060] The framework can allow the system to perform meta-imitation learning on large, autonomously collected datasets. In some embodiments, many policy first demonstrations can be collected and autonomously labeled as corresponding to the same task. The autonomously labeled first demonstrations can be used to improve the performance of the learned policy. In some embodiments, the goal is to learn a policy that can adapt to new, unseen tasks from a single demonstration for that task.

[0061] In some implementations, it may be assumed that the distribution of tasks A set of meta-training tasks can be derived from the task distribution In some embodiments, a set of paired demonstrations corresponding to a subset of the meta-training tasks may be used Each demo i Includes observations and action trajectories that represent the best behavior for the task d i =[(o1, a1), ..., (o H , a H )]. However, unlike the meta-imitation learning setting which only assumes access to a static set of demonstrations, some implementations can learn from the meta-training task This is similar to using meta-reinforcement learning. However, no task-specific reward function is required for any task in the meta-training set, which can allow new trajectories to be sampled autonomously without external supervision.

[0062] In some embodiments, a meta-strategy consisting of two neural networks can be used: an embedding network and a control network. The embedding network, denoted as ψ, can be used to process as input a video demonstration of a task to be performed, where the video demonstration of the task to be performed is denoted as d v In some embodiments, the embedding network can consist of a convolutional neural network followed by a 1-D temporal convolution. In some embodiments, the embedding network can be used to embed the demonstration video into a fixed-length vector, which is denoted as ψ(d v ) and are called demonstration embeddings. In some implementations, demonstration embeddings can have the following property: if demonstrations correspond to the same task, demonstration embeddings can be close to each other, while for demonstrations belonging to different tasks, demonstration embeddings can be farther away from each other. In other words, when and For the same task, the distance can be low, and can be high when they do not correspond to the same task. In some embodiments, a contrastive loss function can be used

[0063]

[0064] Among them, when two demonstrations and The variable Y = 0 when the two demonstrations correspond to the same task, and the variable Y = 1 when the two demonstrations correspond to different tasks.

[0065] In some implementations, the control network can process an image of the current scene (along with other parts of the robot state, such as the end effector pose) to generate robot actions. To learn to predict actions from this information, a maximum likelihood behavior cloning loss can be used. In some embodiments, the complete loss function can be expressed as In some of these simulations, the Adam optimizer can be used to minimize Additionally or alternatively, the embedding network and the control network can be trained jointly, sharing convolutional layers between them. The output of this process leads to a behavioral policy (also referred to as the meta-policy in this paper), where θ represents the learned parameters of the policy.

[0066] In some embodiments, the behavioral policy can be used to collect data in a new environment. Given a scenario, the learning policy can be trained on a dataset of demonstrations. The randomly sampled demonstrations in the dataset are conditioned on Initially including human-guided demonstrations. The policy can then be first demonstrated in new scenarios, and the corresponding first demonstrations can be added to the first demonstration dataset After one or more iterations, the first demonstration dataset can be filtered to remove spurious behaviors. For example, if the strategy knocks over an object on a table in a given scene, the first demonstration can be deleted from the dataset because the system does not want to replicate this behavior. The dot products (in the embedding space) are computed for all possible pairs of first appearances in , and pairs with high dot products (measured by some predefined threshold α) can be labeled as belonging to the same task and added to the dataset. In some implementations, an extended dataset may be used (e.g., the dataset contains self-labeled first-time pairs) to perform meta-imitation learning, optimizing the loss defined in Equation (8) To generate strategy π ψ , whose architecture is similar to π θ same.

[0067] Turning now to the accompanying drawings, Figure 1 An exemplary robot 100 is illustrated in FIG. The robot 100 is a "robot arm" having multiple degrees of freedom to enable a gripping end effector 102 to traverse any of multiple potential paths to position the gripping end effector 102 at a desired location. The robot 100 further controls two opposing "jaws" of its gripping end effector 102 to actuate the jaws between at least an open position and a closed position (and / or optionally multiple "partially closed" positions).

[0068] exist Figure 1 An exemplary visual component 106 is also illustrated in FIG. Figure 1In the embodiment of the present invention, the vision component 106 is mounted in a fixed posture relative to the base or other stationary reference point of the robot 100. The vision component 106 includes one or more sensors that can generate images and / or other visual data related to the shape, color, depth, and / or other characteristics of objects within the sensor's line of sight. The vision component 106 can be, for example, a monoscopic camera, a stereo camera, and / or a 3D laser scanner. The 3D laser scanner can be, for example, a time-of-flight 3D laser scanner or a triangulation-based 3D laser scanner and can include a position sensitive detector (PDS) or other optical position sensor.

[0069] Vision component 106 has a field of view of at least a portion of the workspace of robot 100, such as a portion of the workspace that includes exemplary object 104A. Figure 1 1, but those objects may be resting on a table, tray, and / or other surface. Objects 104 may include a spatula, a stapler, and a pencil. In other embodiments, more objects, fewer objects, additional objects, and / or alternative objects may be provided during all or part of a grasping attempt by the robot 100, as described herein.

[0070] Despite Figure 1 A particular robot 100 is illustrated in the drawings, but additional and / or alternative robots may be utilized, including additional robot arms similar to robot 100, robots having other robot arm forms, robots having humanoid forms, robots having animal forms, robots that move via one or more wheels (e.g., self-balancing robots), submersible robots, unmanned aerial vehicles (“UAVs”), etc. Furthermore, although Figure 1 A particular grasping end effector is illustrated in FIG, but additional and / or alternative end effectors may be utilized, such as alternative impact-type grasping end effectors (e.g., those with grasping "plates," end effectors with more or fewer "fingers" / "claws"), intrusive grasping end effectors, retracting grasping end effectors, continuous grasping end effectors, or non-grasping end effectors. Additionally, although in Figure 1 Specific mountings of the vision assembly 106 are illustrated in FIG, but additional and / or alternative mountings may be utilized. For example, in some embodiments, the vision assembly may be mounted directly to the robot, such as on a non-actuable component of the robot or on an actuable component of the robot (e.g., an end effector or a component proximate to an end effector). Further, for example, in some embodiments, the vision assembly may be mounted on a non-stationary structure separate from the robot with which it is associated and / or may be mounted in a non-stationary manner on a structure separate from the robot with which it is associated.

[0071] Data (e.g., state data) from the robot 100 can be utilized to train the meta-learning model 114 using the meta-learning model training engine 108. For example, the meta-learning model training engine 108 can use meta-learning to train the trial strategy and / or adaptive trial strategy of the meta-learning model 114. The meta-learning model training engine 108 can include an imitation learning training engine 110, a reinforcement learning training engine 112, and / or additional engines (not depicted). In some embodiments, the imitation learning training engine 110 can use human-guided demonstrations 116 to train the meta-learning model 114, where each demonstration captures the robot 100 performing a task. For example, the trial strategy of the meta-learning model 114 can be trained using imitation learning by the imitation learning engine 110. Additionally or alternatively, the reinforcement learning training engine 112 can train the meta-learning model 114 based on one or more trials 118 of the robot 100 performing a task. The trial generator engine 120 can generate one or more trials 118 using the trial strategy of the meta-learning model 114.

[0072] In some embodiments, the test strategy and the adaptive test strategy are integrated into a single model, wherein an update to one or more parts of the test strategy also changes one or more parts of the adaptive test strategy. Similarly, changes to one or more parts of the adaptive test policy will change one or more parts of the test policy. In some such embodiments, when the test strategy changes over time, the test generator engine 120 can continuously generate tests based on the test strategy. Then, the reinforcement learning training engine 112 can use these tests to train the adaptive strategy of the meta-learning model 114, which in turn will also update the test strategy of the meta-learning model 114. The updated test strategy can then be used by the test generator engine 120 to generate one or more additional tests. These additional tests can then be used by the reinforcement learning training engine 112 to update the adaptive test strategy, which in turn will update the test strategy. The cycle of updating the test strategy based on the training adaptive test strategy, generating additional tests using the updated test strategy, and updating the adaptive test strategy using the additional tests can be repeated until the training meta-learning model is completed.

[0073] In some other embodiments, the trial strategy is decoupled from the adaptive trial strategy and optimized separately. In some such embodiments, the imitation learning training engine 110 can use one or more human-guided demonstrations 116 of a robot performing a new task to train the trial strategy of the meta-learning model 114. The trial strategy can then be frozen. The trial generator engine 120 can use the frozen trial strategy to generate one or more trials 118 in which the robot attempts to perform the task. The reinforcement learning training engine 112 can use these trials 118 to train the adaptive strategy of the meta-learning model 114. Decoupling the trial strategy and the adaptive trial strategy can eliminate the need to continuously generate trials of the same strategy.

[0074] In addition, the reinforcement learning training engine 112 can utilize a reward indicating whether the robot successfully completes a new task in a trial 118 when training the adaptive trial strategy. In some embodiments, the reward can be a sparse reward signal. For example, a human can provide a binary reward indication of success or failure through one or more user interface inputs at a client device, indicating whether the robot successfully completes a task in a trial and error attempt.

[0075] Figure 2A An exemplary meta-learning model 200 according to embodiments disclosed herein is illustrated. Environment state data 202 captures information about the robot's current environment. In some embodiments, the environment state data may use a visual component such as Figure 1 . For example, the environment state 202 may include one or more images of the environment captured using an RGB camera and / or other visual data captured using a visual component. In addition, for example, the environment state 202 may additionally or alternatively include environmental features predicted based on visual data and / or other sensor data. For example, the environment state data 202 may include the pose and / or classification of objects in the environment determined based on the visual data. The environment state data 202 may be processed using a visual network 204 to generate an environment state embedding 206. In some embodiments, the environment state embedding 206 represents one or more visual features of the robot environment. Additionally or alternatively, in some embodiments, the robot may be a simulated robot, wherein the environment state data 202 is captured from the simulated robot. For example, the environment state data 202 may include simulated visual data (e.g., a rendered image) that captures a simulated environment of the simulated robot.

[0076] The environment state embedding can be combined with the robot state data 208. For example, the environment state embedding 206 can be cascaded with the robot state data 208. In some embodiments, the robot state data 208 can include a representation of a current end effector pose, a current end effector angle, a current end effector velocity, and / or additional information about the current position of the robot and / or one or more components of the robot. For example, the robot state data can include at least a representation of the pose of the end effector in the task space, such as a six-dimensional pose that can indicate the X, Y, Z position of the end effector, and the orientation of the end effector. In the illustrated example, the actor network 210 is a trained meta-learning model that can be used to generate one or more robot actions 212 based on the environment and the current state of the robot to perform the robot task.

[0077] Figure 2B An exemplary meta-learning model 200 according to embodiments disclosed herein is illustrated. As described above, environment state data 202 can be processed using a visual network 204 to generate an environment state embedding 206. Environment state embedding 206 can be combined with robot state data 208. In the example shown, environment state embedding 206 and robot state data 208 are additionally combined with context embedding 260. For example, environment state embedding 206 can be concatenated with robot state data 208 and context embedding 260.

[0078] The context embedding 260 may provide a representation of the new task. The demonstration embedding 252 may capture features of a human-guided demonstration of the task. For example, the demonstration data may include a video of a human-guided demonstration of a robot performing a task. The video may be sampled (e.g., a series of still images may be captured from the demonstration video), and the sample of the demonstration video may be processed using an additional vision network to generate a demonstration embedding.

[0079] Similarly, trial embedding 254 can capture the characteristics of a trial-and-error attempt to perform a task. Trial reward 256 can provide an indication of whether the trial captured in trial embedding 254 was a successful attempt to perform the task. Demonstration embedding 252, trial embedding 254, and trial reward 256 can be combined. For example, demonstration embedding 252 can be cascaded with trial embedding 254 and trial reward 256. The combination can be provided to context network 258 to generate context embedding 260. The combination of environment state embedding 206, robot state data 208, and context embedding 260 can be provided to actor network 210 to generate robot actions 212 to perform the robot task.

[0080] Figure 3300 is a flowchart illustrating an exemplary process 300 for training a meta-learning model using multiple training tasks and training an already trained meta-learning model for a new task according to an embodiment disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include one or more components of a robot and / or a computing device, such as a processor of the robot and / or computing device and / or a robot control system of robots 100, 1025 and / or other robots. In addition, although the operations of process 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0081] At block 302, the system trains a meta-learning model based on a plurality of training tasks, where each task includes (1) one or more human-guided demonstrations of a robot and (2) one or more trials of the robot performing the task. In some embodiments, one or more trials are generated at block 304. In some embodiments, based on Figure 4 The process of 400 training the meta-learning model.

[0082] At block 304, the system generates one or more trials of the robot performing the task based on (1) one or more human-guided demonstrations of the robot performing the task and (2) the trial strategy of the meta-learning model. For example, the generated trials may include a series of robot actions and corresponding robot states, where the robot performs actions to transition from a current state to a next state. In some embodiments, the state may include environmental state data (such as a state that captures the current environment of the robot) Figure 2A and 2B Environmental state data 202) and robot state data (such as capturing the current position and / or other characteristics of components of the robot) Figure 2A and Figure 2B In some embodiments, the system generates a trial strategy for the meta-learning model at block 302. In some embodiments, the system generates a trial strategy for the meta-learning model at block 302. Figure 5 The process 500 generates one or more trials.

[0083] At block 306, the system determines whether to generate any additional trials of the robot performing the task. If so, the system can proceed to block 304 and generate one or more additional trials based on the trial strategy of the meta-learning model. In some embodiments, the system can generate one or more additional trials based on the trial strategy and the same task as the previous iteration. In some other embodiments, the system can generate one or more additional trials based on the trial strategy and based on the additional task (e.g., using a guided demonstration for the additional task and trials for the additional task). If not, the system can proceed to block 310. For example, when the system has finished training the meta-learning model, the system can determine not to generate any additional trials.

[0084] At block 308, the system determines whether to perform any additional training of the meta-learning model. If so, the system can proceed to block 302 and continue to train the adaptive trial strategy of the meta-learning model using one or more generated trials. If not, the system can proceed to block 310. For example, the system can determine not to perform any additional training of the meta-learning model when the meta-learning model has been trained using a threshold number of training tasks, trained using all training tasks, trained for a threshold duration, and / or when additional or alternative conditions have been met.

[0085] At block 310, the system trains a previously trained meta-learning model for a new task that is different from the plurality of training tasks, and performs the training based on: (1) the trained meta-learning model, (2) one or more human-guided demonstrations of a robot performing the new task, and (3) one or more trials of the robot performing the new task. Figure 6 The process 600 trains a trained meta-learning model for a new task.

[0086] Figure 4 4 is a flowchart illustrating an exemplary process 400 for training a meta-learning model using imitation learning and reinforcement learning according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include one or more components of a robot and / or computing device, such as a processor of the robot and / or computing device and / or a robot control system of robots 100, 1025 and / or other robots. In addition, although the operations of process 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added. .

[0087] At block 402, the system initializes the trial strategy of the meta-learning model. In some embodiments, the trial strategy can be learned using a Q-learning function. In some of these embodiments, the Q-learning function can be a normalized advantage function.

[0088] At block 404 , the system selects a training task from a plurality of different robotic training tasks.

[0089] At block 406 , the system selects a human-led demonstration for the selected training task.

[0090] At block 408, the system selects one or more trials in which the robot performs the selected training task. Figure 5 The process 500 continuously generates one or more experiments based on the experiment strategy.

[0091] At block 410, the system generates an adaptive trial strategy based on (1) the selected human-guided demonstration and (2) the selected one or more trials. In some embodiments, the adaptive trial strategy is generated using reinforcement learning. In some embodiments, a reward is provided by a human observer indicating whether a task was successfully completed in a trial. The reward can be used when training the adaptive trial strategy using reinforcement learning.

[0092] At block 412, the system selects an additional human-led demonstration of the robot performing the selected training task. In some implementations, the human-led demonstration is a sampling of demonstration data (e.g., a video of the robot performing the task). In some of these implementations, the additional human-led demonstration is an additional sampling of the same demonstration data. Additionally or alternatively, the additional demonstration can be a sampling of demonstration data from a separate human-led demonstration of the captured task.

[0093] At block 414, the system uses imitation learning to update the trial strategy based on (1) additional human-guided demonstrations and (2) an adaptive trial strategy.

[0094] At block 416, the system determines whether to perform any additional training of the meta-learning model. If so, the system returns to block 404 and selects additional training tasks before proceeding to blocks 406, 408, 410, 412, and 414 with the additional training tasks. If the system determines not to perform additional training, the process ends. For example, the system may determine not to perform additional training when the meta-learning model has been trained using all training tasks, when a threshold number of training epochs has been completed, etc.

[0095] Figure 5 is a flow chart illustrating an exemplary process 500 for generating one or more trials for training a meta-learning model according to embodiments disclosed herein. For convenience, the operations of the flow chart are described with reference to a system performing the operations. The system may include one or more components of a robot and / or computing device, such as a processor of the robot and / or computing device and / or a robot control system of the robot 100, 1025 and / or other robots. In addition, although the operations of process 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0096] At block 502 , the system selects a training task from a plurality of robotic training tasks.

[0097] At block 504 , the system selects a human-led demonstration for the selected training task.

[0098] At block 506, the system stores the selected human-guided presentation in a buffer.

[0099] At block 508, the system uses reinforcement learning to generate an adaptive trial strategy based on (1) the current contents of the buffer and (2) the trial strategy of the meta-learning model. Figure 4 The process 400 generates an experimental strategy for a meta-learning model.

[0100] At block 510 , the system generates new trials of the robot performing the training task based on the adaptive trial strategy.

[0101] At block 512, the system stores the generated new trial as the next item in the buffer.

[0102] At block 514, the system determines whether to generate any additional trials. If so, the system returns to block 508 and generates an updated adaptive trial strategy based on the trial strategy and the contents of the buffer (including the generated new trials) before returning to blocks 510 and 512 based on the updated adaptive trial strategy. If at block 514, the system determines not to generate any additional trials, the system proceeds to block 516. For example, the system may determine whether the robot successfully completed the task in the trial, and may generate additional trials until the robot successfully completes the task. Additionally or alternatively, additional trials may be generated until a threshold number of trials are generated (e.g., 3 trials have been generated, 5 trials have been generated, 10 trials have been generated, etc.) and / or other conditions are met.

[0103] At block 516, the system determines whether to generate trials for any additional training tasks. If so, the system returns to block 502 and selects additional training tasks before proceeding to blocks 504, 506, 508, 510, 512, and 514 based on the additional training tasks. If the system determines not to generate trials for any additional training tasks, process 500 ends. For example, when training of the meta-learning model is complete, the system can determine not to generate trials for any additional training tasks.

[0104] Figure 6 600 for training an already trained meta-learning model for a new task according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include one or more components of a robot and / or computing device, such as a processor of the robot and / or computing device and / or a robot control system of the robot 100, 1025 and / or other robots. In addition, although the operations of process 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added. .

[0105] At block 602, the system receives a training meta-learning model trained using a plurality of training tasks. Figure 4Process 400 trains the already trained meta-learning model.

[0106] At block 604, the system selects a new task that is not included in the plurality of training tasks used to train the trained meta-learning model. At block 604, the system also selects a human-guided demonstration of a robot performing the selected new task.

[0107] At block 606, the system stores the selected presentation as the first item in the buffer.

[0108] At block 608 , the system generates an adaptive trial strategy based on (1) the current item in the buffer and (2) the trial strategy of the trained meta-learning model.

[0109] At block 610, the system generates a trial for the robot to perform a new task based on the adaptive trial parameter set. Figure 5 The process 500 generates a test.

[0110] At block 612, the system stores the generated trial as the next item in a buffer.

[0111] At block 614, the system determines whether the robot successfully performed the task in the generated trial. If yes, the system proceeds to block 616. If no, the system returns to block 608, generates and updates an adaptive trial strategy based on the trial strategy and the contents of the buffer (including the trial generated in the previous iteration), and proceeds to blocks 610 and 612 based on the updated adaptive trial strategy. In some implementations, a human observer provides an indication of whether the robot successfully performed the task in the generated trial.

[0112] Additionally or alternatively, even when there are no successful trials, in some embodiments the system may proceed to box 616 after a threshold number of trials (e.g., 1 trial, 2 trials, 3 trials, 5 trials, 10 trials, etc.) has been generated and / or other conditions are met.

[0113] At block 616 , the system causes the robot to perform the new task based on the adaptive trial strategy.

[0114] At block 618, the system determines whether to train a trained meta-learning model for any additional new tasks. If so, the system returns to block 604, selects an additional new task, and proceeds to blocks 606, 608, 610, 612, 614, and 616 based on the additional new task. If the system determines not to train a trained meta-learning model for any additional new tasks, the process ends.

[0115] Figure 6Process 600 is an illustrative example of few-shot learning of a new task, where a meta-learning model is trained to perform the new task using a single human-guided demonstration at block 604. Additionally or alternatively, the system can learn the new task with a few shots by performing multiple iterations of meta-learning model training based on the selected demonstration (i.e., performing iterations of blocks 606, 608, 610, 612, and 614 based on the selected demonstration). For example, the system can train the meta-learning model using a few demonstrations (e.g., two demonstrations, three demonstrations, less than 5 demonstrations, less than 10 demonstrations, etc.).

[0116] Figure 7 700 is a flowchart illustrating an exemplary process 700 for training a meta-learning model using multiple training tasks and training an already trained meta-learning model for a new task according to an embodiment disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include one or more components of a robot and / or a computing device, such as a processor of the robot and / or computing device and / or a robot control system of robots 100, 1025 and / or other robots. In addition, although the operations of process 700 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0117] At block 702, the system trains a meta-learning model based on a plurality of training tasks, wherein each training task includes (1) one or more human-guided demonstrations of a robot performing the training task and (2) one or more trials of the robot performing the training task. Figure 8 The process of 800 training the meta-learning model.

[0118] At block 704, the system trains the trained meta-learning model for a new task that is different from the plurality of training tasks, wherein the training is based on: (1) the trained meta-learning model; (2) one or more human-guided demonstrations of the robot performing the new task; and (3) one or more trials of the robot performing the new task. Fig. 9 Process 900 trains an already trained meta-learning model for a new task.

[0119] Figure 8800 is a flowchart illustrating an exemplary process 800 for training a meta-learning model using imitation learning and reinforcement learning according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include one or more components of a robot and / or computing device, such as a processor of the robot and / or computing device and / or a robot control system of the robot 100, 1025 and / or other robots. In addition, although the operations of process 800 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0120] At block 802, the system initializes a trial strategy and an adaptive trial strategy of a meta-learning model. In some implementations, the trial strategy and the adaptive trial strategy are decoupled and optimized separately.

[0121] At block 804 , the system selects a robot training task from a plurality of robot training tasks.

[0122] At block 806 , the system updates the trial policy using imitation learning based on one or more human-led demonstrations for the selected task.

[0123] At block 808, the system determines whether to use any additional robot training tasks to perform any additional imitation learning. If so, the system returns to block 804, selects an additional training task from a plurality of robot training tasks and returns to blocks 806 and 808 based on the additional training task. If not, the system proceeds to block 810.

[0124] At block 810, the system freezes the test strategy. In some embodiments, freezing the test strategy prevents the system from having to continuously generate tests. Figure 5 The process of 500 and Figure 6 This is in contrast to process 600, where the test strategy and the adaptive test strategy share weights. Thus, in process 500 and process 600, trials are continuously generated as the test strategy is updated based on the generated trials.

[0125] At block 812, the system selects a human-guided demonstration of a robot performing a task. In some implementations, the selected human-guided demonstration is used as one of the one or more demonstrations used to update the trial strategy at block 806. In some other implementations, the selected human-guided demonstration is not used to update the trial strategy at block 806.

[0126] At block 814 , the system generates a trial of the robot performing the task based on the trial strategy.

[0127] At block 816, the system stores the manually guided demonstration and the generated trial as the next demonstration-trial pair in a buffer.

[0128] At block 818, the system determines whether to generate any additional trials. If so, the system returns to block 812, selects another human-guided demonstration, and proceeds to blocks 814 and 816 based on the additional demonstration. If the system determines not to generate any additional trials, the system proceeds to block 820. For example, the system may generate additional trials until the system generates trials in which the robot successfully completes the task. Additionally or alternatively, the system may generate trials until a threshold number of trials (e.g., two trials, three trials, five trials, ten trials, etc.) has been generated and / or other conditions have been met.

[0129] At block 820 , the system updates the adaptive trial policy using reinforcement learning based on the demonstration-trial pairs stored in the buffer.

[0130] Fig. 9 900 is a flowchart illustrating an exemplary process 900 for training an already trained meta-learning model to perform a new task according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include one or more components of a robot and / or computing device, such as a processor of the robot and / or computing device and / or a robot control system of the robot 100, 1025 and / or other robots. In addition, although the operations of process 900 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0131] At block 902, the system receives a trained meta-learning model, wherein the meta-learning model is trained using a plurality of training tasks. Figure 8 Process 800 trains the already trained meta-learning model.

[0132] At block 904, the system initializes an adaptive trial strategy.

[0133] At block 906 , the system selects a new task, wherein the new task is a task that is not included in the plurality of training tasks used to train the already trained meta-learning model.

[0134] At block 908, the system selects a human-led demonstration of a robot performing a new task.

[0135] At block 910 , the system generates trials of the robot performing a new task using the trial policy of the trained meta-learning model.

[0136] At block 912, the system updates the adaptive trial strategy based on (1) the selected demonstration and (2) the generated trial.

[0137] At block 914, the system determines whether the robot successfully completed the task in the generated trial. If yes, the system proceeds to block 916. If no, the system returns to block 908, selects another human-guided demonstration, and proceeds to blocks 910, 912, and 914 using the other human-guided demonstration. In other words, the system generates another demonstration-trial pair. In some embodiments, the human observer provides an indication of whether the robot successfully completed the task in the trial.

[0138] At block 916, the system causes the robot to perform the task based on the adaptive policy network.

[0139] Fig.10 An exemplary architecture of a robot 1025 is schematically depicted. The robot 1025 includes a robot control system 1060, one or more operating components 1040a-1040n, and one or more sensors 1042a-1042m. The sensors 1042a-1042m may include, for example, vision components, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, etc. Although the sensors 1042a-m are depicted as being integrated with the robot 1025, this is not meant to be limiting. In some embodiments, the sensors 1042a-m may be located external to the robot 1025, such as as a stand-alone unit.

[0140] The operating components 1040a-1040n may include, for example, one or more end effectors and / or one or more servo or other actuators to achieve movement of one or more components of the robot. For example, the robot 1025 may have multiple degrees of freedom and each actuator may control the actuation of the robot 1025 in one or more degrees of freedom in response to a control command. As used herein, the term actuator covers mechanical or electrical devices (e.g., motors) that create motion, in addition to any driver that may be associated with an actuator and converts a received control command into one or more signals for driving a force actuator. Thus, providing a control command to an actuator may include providing a control command to a driver that converts the control command into an appropriate signal for driving an electrical or mechanical device to create the desired motion.

[0141] The robotic control system 1060 can be implemented in one or more processors, such as a CPU, GPU, and / or other controller of the robot 1025. In some embodiments, the robot 1025 can include a "brain box" that can include all or aspects of the control system 1060. For example, the brain box can provide real-time bursts of data to the operating components 1040a-n, where each real-time burst includes a set of one or more control commands that specify, among other things, motion parameters (if any) for each of the one or more operating components 1040a-n. In some embodiments, the robotic control system 1060 can perform one or more aspects of the processes 300, 400, 500, 600, 700, 800, 900 and / or other methods described herein.

[0142] As described herein, in some embodiments, all or aspects of the control commands generated by the control system 1060 when positioning the end effector to grasp an object can be based on end effector commands generated using a meta-learning model. For example, the vision components of the sensors 1042a-m can capture environmental state data. The environmental state data can be processed along with the robot state data using the policy network of the meta-learning model to generate one or more end effector control commands for controlling the movement and / or grasping of the robot's end effector. Although the control system 1060 is Fig.10 1025, but in some embodiments, all or aspects of the control system 1060 may be implemented in a component separate from the robot 1025 but in communication with the robot 1025. For example, all or aspects of the control system 1060 may be implemented on one or more computing devices, such as the computing device 1110, that are in wired and / or wireless communication with the robot 1025.

[0143] Fig.11 1 is a block diagram of an exemplary computing device 1110 that may optionally be used to perform one or more aspects of the techniques described herein. The computing device 1110 typically includes at least one processor 1114 that communicates with a number of peripheral devices via a bus subsystem 1112. These peripheral devices may include a storage subsystem 1124, including, for example, a memory subsystem 1125 and a file storage subsystem 1126; a user interface output device 1120; a user interface input device 1122, and a network interface subsystem 1116. The input and output devices allow a user to interact with the computing device 1110. The network interface subsystem 1116 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0144] User interface input devices 1122 may include a keyboard; a pointing device such as a mouse, trackball, touchpad, or graphics tablet; a scanner; a touch screen incorporated into a display; an audio input device such as a voice recognition system; a microphone; and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways of inputting information into computing device 1110 or over a communications network.

[0145] User interface output devices 1120 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visual image. The display subsystem may also provide a non-visual display such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from the computing device 1110 to a user or another machine or computing device.

[0146] Storage subsystem 1124 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 1124 may include a processor that executes the modules described herein. Figure 3 , 4 , the logic of selected aspects of the processes of 5, 6, 7, 8, 9 and / or other methods.

[0147] These software modules are typically executed by the processor 1114 alone or in conjunction with other processors. The memory 1125 used in the storage subsystem 1124 may include multiple memories, including a main random access memory (RAM) 1130 for storing instructions and data during program execution and a read-only memory (ROM) 1132 for storing fixed instructions. The file storage subsystem 1126 can provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored in the storage subsystem 1124 by the file storage subsystem 1126, or in other machines accessible to the processor 1114.

[0148] The bus subsystem 1112 provides a mechanism for the various components and subsystems of the computing device 1110 to communicate with each other as intended. Although the bus subsystem 1112 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0149] The computing device 1110 may be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Fig.11 The description of the computing device 1110 depicted in FIG. 1 is intended only as a specific example for the purpose of illustrating some embodiments. Many other configurations of the computing device 1110 may have more Fig.11 The computing devices depicted may have greater or fewer components.

[0150] In some embodiments, a method implemented by one or more processors is provided, including generating an adaptive policy network of a trained meta-learning model based on a human-guided demonstration of a robot performing a new task for use in controlling the robot to perform the new task, wherein multiple different tasks are used and the meta-learning model is not trained for the new task. Generating the adaptive policy network further includes: generating an initial adaptation of the policy network based on the human-guided demonstration and the experimental policy network of the meta-learning model. The method further includes using the initial adaptation of the policy network to generate an initial sequence of robot actions and corresponding states for the robot to perform the new task. The method further includes causing the robot to perform an initial sequence of robot actions and corresponding robot states. The method further includes determining whether the initial sequence of robot actions and corresponding robot states successfully completes the new task. The method further includes generating an adaptive policy network based on determining that the initial sequence of robot actions and corresponding robot states successfully completes the new task.

[0151] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0152] In some embodiments, determining whether the initial sequence of robot actions and corresponding robot states successfully completes the new task includes determining that the robot successfully completes the new task using the initial sequence of robot actions and corresponding robot states. In some of those embodiments, generating an adaptive policy network based on determining whether the initial sequence of robot actions and corresponding robot states successfully completes the new task includes generating an adaptive policy network based on an initial adaptation of the policy network.

[0153] In some embodiments, determining whether the initial sequence of robot actions and corresponding robot states successfully completes the new task includes determining, using the initial sequence of robot actions and corresponding robot states, that the robot did not successfully complete the new task. In some of these embodiments, generating an adaptive policy network based on determining whether the initial sequence of robot actions and corresponding robot states successfully completes the new task includes: generating additional adaptations of the adaptive policy network based on (1) the human-guided demonstration, (2) the initial sequence of robot actions and corresponding robot states, and (3) the experimental policy network of the meta-learning model. In some embodiments, the method further includes generating additional sequences of machine actions and corresponding robot states of the robot for the robot to perform the new task using the additional adaptations of the adaptive policy network. In some embodiments, the method further includes causing the robot to perform additional sequences of robot actions and corresponding robot states. In some embodiments, the method further includes determining whether the additional sequences of robot actions and corresponding robot states successfully complete the new task. In some embodiments, the method further updates the adaptive policy network based on determining whether additional sequences of robot actions and corresponding robot states successfully complete the new task.

[0154] In some embodiments, prior to generating an initial adaptation of the policy network based on the manually guided demonstrations for a set of experimental parameters of the meta-learning model, the method further includes storing the manually guided demonstrations as an initial item in a buffer. In some embodiments, in response to determining that the robot did not successfully complete the new task using the initial sequence of robot actions and corresponding robot states, the method further includes storing the initial sequence of robot actions and corresponding robot states as a next item in the buffer. In some embodiments, generating additional adaptations of the adaptive policy network based on (1) the manually guided demonstrations, (2) the initial sequence of robot actions and corresponding robot states, and (3) the experimental policy network of the meta-learning model includes generating additional adaptations of the adaptive policy network based on the contents of the buffer and the experimental parameter set of the meta-learning model.

[0155] In some embodiments, training the meta-learning model using multiple different tasks includes initializing an experimental strategy for the meta-learning model. In some embodiments, the method further includes selecting a training task from a plurality of training tasks. In some embodiments, the method further includes determining a manually guided training demonstration of the robot performing the training task. In some embodiments, the method further includes generating one or more training sequences of robot actions and corresponding robot states of the robot performing the training task based on the experimental strategy network and the manually guided training demonstration. In some embodiments, the method further includes determining additional manually guided training demonstrations of the robot performing the training task. In some embodiments, the method further includes using meta-imitation learning to update the experimental strategy network of the meta-learning model based on the additional manually guided demonstrations and the one or more training sequences.

[0156] In some embodiments, generating one or more training sequences of robot actions and corresponding robot states for a robot to perform a training task based on a trial policy network and a manually guided training demonstration includes generating an initial training adaptation of a policy network based on the manually guided training demonstration and the trial policy network. In some embodiments, the method further includes using the initial training adaptation of the policy network to generate an initial training sequence of robot actions and corresponding robot states for a robot to perform a training task. In some embodiments, the method further includes using meta-reinforcement learning to update a training adaptation policy network based on (1) the manually guided training demonstration, (2) the initial training sequence, and (3) the initial training adaptation of the policy network.

[0157] In some embodiments, generating one or more training sequences of robot actions and corresponding robot states for the robot to perform a training task based on the trial policy network and the manually guided training demonstration further includes using the trained adaptive policy network to generate other training sequences of robot actions and corresponding robot states for the robot to perform a training task. In some embodiments, the method further includes using reinforcement learning to update the trained adaptive policy network based on (1) the manually guided training demonstration, (2) the initial training sequence, (3) the other training sequences, and (4) the trained adaptive policy network.

[0158] In some embodiments, training a meta-learning model using a plurality of different tasks includes initializing a trial policy network and an adaptive trial policy network of the meta-learning model. In some embodiments, the method further includes selecting a training task from a plurality of training tasks. In some embodiments, the method further includes determining a manually guided training demonstration of a robot performing the training task. In some embodiments, the method further includes using imitation learning to update the trial policy network based on the manually guided demonstration. In some embodiments, the method further includes freezing the trial policy network of the meta-learning model. In some embodiments, the method further includes generating one or more training sequences of robot actions and corresponding robot states of the robot performing the training task based on the frozen trial policy network and the manually guided training demonstration. In some embodiments, the method further includes using reinforcement learning to update the adaptive trial policy network based on (1) one or more training sequences of robot actions and corresponding robot states and (2) manually guided training demonstrations.

[0159] In some embodiments, generating one or more training sequences of robot actions and corresponding robot states for the robot to perform a training task based on the frozen trial strategy network and the manually guided training demonstration includes generating an initial training sequence of robot actions and corresponding robot states for the robot to perform the task based on the trial strategy network and the manually guided training demonstration. In some embodiments, the method further includes storing the manually guided training demonstration and the generated initial training sequence as a first demonstration-trial pair in a training buffer. In some embodiments, the method further includes determining whether the initial training sequence of robot actions and corresponding robot states successfully completes the training task.

[0160] In some embodiments, generating one or more training sequences of robot actions and corresponding robot states for the robot to perform a training task based on the frozen trial strategy network and the manually guided training demonstration further includes, in response to determining that the initial training sequence of robot actions and corresponding robot states did not successfully complete the training task, generating other training sequences of robot actions and corresponding robot states for the robot to perform the task based on the trial strategy network and the manually guided training demonstration. In some embodiments, the method further includes storing the manually guided training demonstration and the generated other training sequences as the next demonstration-trial pair in a training buffer.

[0161] In some embodiments, updating the adaptive trial policy network using reinforcement learning based on (1) one or more training sequences of robot actions and corresponding robot states and (2) manually guided training demonstrations includes determining a reward signal for each demonstration-trial pair in the training buffer based on whether the robot successfully completes the training task in the corresponding demonstration-trial pair. In some embodiments, the method further includes updating the adaptive trial policy network using reinforcement learning based on the demonstration-trial pairs in the training buffer and the corresponding reward signals.

[0162] In some implementations, the new task is represented by a finite-horizon Markov decision process.

[0163] In some implementations, the trial policy network is learned via a Q-learning function.

[0164] In some implementations, the adaptive trial strategy network is learned via a Q-learning function.

[0165] In addition, some embodiments include one or more processors (e.g., central processing units (CPUs), graphics processing units (GPUs), and / or tensor processing units (TPUs)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause any of the methods described herein to be performed. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the methods described herein.

[0166] Although several embodiments have been described and illustrated herein, various other means and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein may be utilized, and each of these changes and / or improvements is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on the specific application or application for which the teachings are used. Those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein using experiments that do not exceed routine. Therefore, it should be understood that the aforementioned embodiments are presented only as examples, and within the scope of the attached claims and their equivalents, these embodiments may be implemented in a manner different from that specifically described and claimed. The embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, if these features, systems, articles, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method for training a robotic agent using meta-learning, the method being implemented by one or more processors and comprising: Based on a human-guided demonstration of a robot performing a new task, generating an adaptive policy network of a trained meta-learning model for use in controlling the robot to perform the new task, wherein the meta-learning model is trained using a plurality of different tasks and is not trained for the new task, and wherein generating the adaptive policy network comprises: generating an initial adaptation of the adaptive policy network based on the manually guided demonstration and the trial policy network of the trained meta-learning model; generating an initial sequence of robot actions and corresponding robot states for the robot to perform a new task using the initial adaptation of the adaptive policy network; causing said robot to perform said initial sequence of robot actions and corresponding robot states; determining whether the initial sequence of robot actions and corresponding robot states successfully completes the new task; and The adaptive policy network is generated based on determining whether the initial sequence of robot actions and corresponding robot states successfully completes the new task.

2. The method according to claim 1, wherein: Determining whether the initial sequence of robot actions and corresponding robot states successfully completes the new task comprises: determining that the robot successfully completed the new task using the initial sequence of robot actions and corresponding robot states, and Wherein, generating the adaptive policy network based on determining whether the initial sequence of robot actions and corresponding robot states successfully completes the new task comprises: The adaptive policy network is generated based on the initial adaptation of the adaptive policy network.

3. The method according to claim 1, wherein: Determining whether the initial sequence of robot actions and corresponding robot states successfully completes the new task comprises: using the initial sequence of robot actions and corresponding robot states, determining that the robot has not successfully completed the new task, and Wherein, generating the adaptive policy network based on determining whether the initial sequence of robot actions and corresponding robot states successfully completes the new task comprises: generating additional adaptations of the adaptive policy network based on (1) the human-guided demonstration, (2) the initial sequence of robot actions and corresponding robot states, and (3) the trial policy network of the meta-learning model; generating, using the other adaptations of the adaptive policy network, other sequences of robot actions and corresponding robot states for the robot to perform the new task; causing said robot to perform said other sequence of robot actions and corresponding robot states; determining whether the other sequence of robot actions and corresponding robot states successfully completes the new task; and The adaptive policy network is updated based on determining whether the other sequence of robot actions and corresponding robot states successfully completes the new task.

4. The method according to claim 3, further comprising: Before generating the initial adaptation of the adaptive policy network based on the artificially guided demonstration of a set of trial parameters of the meta-learning model, storing the artificially guided demonstration as an initial item in a buffer; In response to determining that the robot did not successfully complete the new task using the initial sequence of robot actions and corresponding robot states, storing the initial sequence of robot actions and corresponding robot states as a next item in the buffer; as well as Wherein, based on (1) the human-guided demonstration, (2) the initial sequence of robot actions and corresponding robot states, and (3) the trial policy network of the meta-learning model, generating the other adaptations of the adaptive policy network includes: The further adaptations of the adaptive policy network are generated based on the contents of the buffer and the set of trial parameters of the meta-learning model.

5. The method according to claim 3, wherein: Training the meta-learning model using the multiple different tasks includes: Initializing the trial strategy network of the meta-learning model; selecting a training task from the plurality of different tasks; Determining a human-guided training demonstration in which the robot performs the training task; generating one or more training sequences of robot actions and corresponding robot states for the robot to perform the training task based on the trial strategy network and the manually guided training demonstration; determining an additional human-guided training demonstration of the robot performing the training task; and The trial policy network of the meta-learning model is updated based on the additional human-guided training demonstrations and the one or more training sequences using meta-imitation learning.

6. The method according to claim 5, wherein: Generating one or more training sequences of robot actions and corresponding robot states for the robot to perform the training task based on the test strategy network and the manually guided training demonstration includes: generating an initial training adaptation of the adaptive policy network based on the manually guided training demonstration and the experimental policy network; generating an initial training sequence of robot actions and corresponding robot states for the robot to perform the training task using the initial training adaptation of the adaptive policy network; and Using meta-reinforcement learning, the trained adaptive policy network is updated based on (1) the manually guided training demonstrations, (2) the initial training sequences, and (3) the initial training adaptations of the adaptive policy network.

7. The method according to claim 6, wherein: Generating one or more training sequences of robot actions and corresponding robot states for the robot to perform the training task based on the trial strategy network and the manually guided training demonstration further includes: Using the trained adaptive policy network, generating other training sequences of robot actions and corresponding robot states for the robot to perform the training task; and Using reinforcement learning, the trained adaptive policy network is updated based on (1) the manually guided training demonstration, (2) the initial training sequence, (3) the other training sequences, and (4) the trained adaptive policy network.

8. The method according to claim 3, wherein: Training the meta-learning model using the multiple different tasks includes: Initializing the trial strategy network and the adaptive strategy network of the meta-learning model; selecting a training task from the plurality of different tasks; Determining a human-guided training demonstration in which the robot performs the training task; updating the trial policy network based on the human-guided demonstration using imitation learning; Freezing the trial strategy network of the meta-learning model; generating one or more training sequences of robot actions and corresponding robot states for the robot to perform the training task based on the frozen trial policy network and the manually guided training demonstration; and Using reinforcement learning, the adaptive policy network is updated based on (1) the one or more training sequences of robot actions and corresponding robot states and (2) the human-guided training demonstrations.

9. The method according to claim 8, wherein: Generating the one or more training sequences of robot actions and corresponding robot states of the robot performing the training task based on the frozen trial strategy network and the artificially guided training demonstration comprises: generating, based on the trial strategy network and the manually guided training demonstration, an initial training sequence of robot actions and corresponding robot states for the robot to perform the training task; storing the manually guided training demonstration and the generated initial training sequence as a first demonstration-trial pair in a training buffer; and A determination is made as to whether the initial training sequence of robot actions and corresponding robot states successfully completes the training task.

10. The method according to claim 9, wherein: Generating the one or more training sequences of robot actions and corresponding robot states of the robot performing the training task based on the frozen trial strategy network and the manually guided training demonstration further comprises: In response to determining that the initial training sequence of robot actions and corresponding robot states did not successfully complete the training task, generating, based on the trial strategy network and the manually guided training demonstration, other training sequences of robot actions and corresponding robot states for the robot to perform the training task; and The manually guided training demonstration and the generated other training sequences are stored in the training buffer as the next demonstration-trial pair.

11. The method according to claim 10, wherein: Using reinforcement learning to update the adaptive policy network based on (1) the one or more training sequences of robot actions and corresponding robot states and (2) the human-guided training demonstrations includes: For each demonstration-trial pair in the training buffer: determining a reward signal based on whether the robot successfully completes the training task in the corresponding demonstration-trial pair; and The adaptive policy network is updated based on the demonstration-trial pairs and corresponding reward signals in the training buffer using reinforcement learning.

12. The method according to any one of claims 1 to 11, wherein: The new task is represented by a finite-horizon Markov decision process.

13. The method according to any one of claims 1 to 11, wherein: The trial policy network is learned via a Q-learning function.

14. The method according to any one of claims 1 to 11, wherein: The adaptive policy network is learned via a Q-learning function.

15. A computer program comprising instructions which, when executed by one or more processors of a computing system, cause the computing system to perform the method of any one of claims 1 to 14.

16. A computing system configured to perform the method according to any one of claims 1 to 14.

17. A computer-readable storage medium storing instructions executable by one or more processors of a computing system to implement the method according to any one of claims 1 to 14.

Images