Methods, systems, and non-transitory computer storage media for robots

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributing robot demonstration learning skill templates. One of the methods includes receiving, by a skill template distribution system from a user device, a selection of an available skill template. The skill template distribution system provides a skill template, where the skill template includes information representing a state machine of one or more tasks, and where the skill template specifies which of the one or more tasks are demonstration subtasks that require local demonstration data. The skill template distribution system trains a machine learning model of the demonstration subtasks using the local demonstration data to generate learned parameter values.

Description

Technical Field

[0001] This specification relates to methods, systems, and one or more computer storage media for robots, and particularly to planning robot motion. Background Technology

[0002] Robot control refers to controlling the physical movements of a robot to perform a task. For example, an industrial robot used in car manufacturing can be programmed to first pick up a car part and then weld that part onto the car's frame. Each of these actions may involve dozens or hundreds of individual movements of the robot's motors and actuators.

[0003] Robot planning typically requires extensive manual programming to precisely define how robot components should move to accomplish specific tasks. Manual programming is tedious, time-consuming, and error-prone. Furthermore, a schedule manually generated for one work cell is often incompatible with other work cells. In this specification, a work cell is the physical environment in which the machine will operate. A work cell has specific physical properties, such as physical dimensions, that restrict how the robot moves within it. Therefore, a schedule manually programmed for one work cell may be incompatible with work cells that have different robots, different numbers of robots, or different physical dimensions.

[0004] Some research has moved towards using machine learning control algorithms, such as reinforcement learning, to control robots to perform specific tasks. However, robots have some drawbacks that often make traditional learning methods unsatisfactory.

[0005] First, robots naturally possess a very complex, high-dimensional, continuous action space. Therefore, generating and evaluating all possible candidate actions is computationally expensive. Second, robot control operates in an environment with extremely sparse rewards, as most possible actions do not lead to the completion of a specific task. A technique called reward shaping has been used to alleviate the sparse reward problem, but it is generally not scalable to hand-designed reward functions.

[0006] Another complicating issue is that traditional techniques for robot control using robot learning are extremely fragile. This means that even if a viable model is successfully trained, a very small change in the task, robot, or environment can render the entire model completely unusable.

[0007] All of these problems mean that traditional approaches using techniques such as reinforcement learning for robot control result in computationally expensive processing, are simply not effective, do not scale well, and cannot be generalized to other situations. Summary of the Invention

[0008] This specification describes a technique for distributing skill templates for demonstration-based robot learning. Specifically, it describes how a skill template distribution system can distribute skill templates that allow robots to be programmed to perform robotic tasks using custom control strategies learned from demonstration data.

[0009] In this specification, a task refers to a specific robot capability involving the performance of one or more sub-tasks. For example, a connector insertion task is the ability of a robot to insert a wired connector into an interface. This task typically includes two sub-tasks: 1) moving the robot's tool to the location of the interface, and 2) inserting the connector into the interface at the specific location.

[0010] In this specification, a subtask is an operation performed by the robot using a tool. For example, when the robot has only one tool, a subtask can be described as an operation that the robot as a whole will perform. Example subtasks include welding, dispensing, part positioning, and surface polishing. Subtasks are typically associated with the type of tool required to perform the subtask and the location within the coordinate system of the work cell where the subtask will be performed.

[0011] In this specification, a skill template (or simply "template") is a collection of data and software that allows a robot to be adapted to perform a specific task. Skill template data represents one or more subtasks required to perform a task, along with information describing which subtasks of the skill require local demonstration learning and which perception flows will be needed to determine success or failure. Therefore, a skill template can define demonstration subtasks that require local demonstration learning, non-demonstration subtasks that do not require local demonstration learning, or both.

[0012] These techniques are particularly advantageous for robotic tasks that are traditionally difficult to control using machine learning, such as reinforcement learning. These tasks include those involving physical contact with objects in the workspace, such as grinding, joining, and insertion tasks, as well as wiring. To name just a few examples.

[0013] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. Learning using the demonstration data described in this specification addresses the sparse rewards and lack of generalization problems of traditional reinforcement learning methods.

[0014] The system can perform tasks using visual, proprioceptive (joint) data, tactile data, and any other features, allowing it to rapidly adapt to specific robot models with high precision. The emphasis is on "sensor-rich robot manipulation," contrary to the classic view of minimal perception in robotics. Generally, this means that the same task can be completed using a cheaper robot in a shorter time.

[0015] The techniques described below allow machine learning techniques to be rapidly adapted to any suitable robot with an appropriately installed hardware abstraction. In a typical scenario, a single non-expert can train a robot to perform a skill template in less than a day of setup time. This is a significant improvement over traditional methods, which may require teams of experts to work for weeks on the problem to design reward functions and weeks of training time in very large data centers. This effectively allows machine learning robot control to be widely distributed to many types of robots, and even robots that the system has never seen before.

[0016] These technologies can effectively realize robot learning as a service, thereby enabling people to use the technology better. This, in turn, makes the entire robotics industry safer and more efficient.

[0017] The combination of reinforcement learning, perceptual data processing using machine learning, and enhanced impedance / admittance control will enable robotic skills to be performed with a very high success rate, as required by industrial applications, despite the complexity of the task.

[0018] Details of one or more embodiments of the subject matter in this specification are set forth in the accompanying drawings and description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

[0019] Brief description of the attached figures

[0020] Figure 1 This is a schematic diagram demonstrating an example learning system.

[0021] Figure 2A This is a schematic diagram of an example system that uses a custom control strategy based on local demo data to execute subtasks.

[0022] Figure 2B This is a schematic diagram of another example system that performs subtasks using local demo data.

[0023] Figure 2C This is a schematic diagram of another example system that uses residual reinforcement learning to perform subtasks.

[0024] Figure 3A This is a flowchart of an example process for combining sensor data from multiple different sensor streams.

[0025] Figure 3B This is a diagram of a camera wristband.

[0026] Figure 3C This is another example view of the camera wristband.

[0027] Figure 3D This is another example view of the camera wristband.

[0028] Figure 4 An example skill template is shown.

[0029] Figure 5 This is a flowchart of an example process for configuring a robot to perform skills using skill templates.

[0030] Figure 6A This is a flowchart illustrating an example of force-guided task processing using skill templates.

[0031] Figure 6B This is a flowchart illustrating an example of training skill templates using a cloud-based training system.

[0032] Figure 7 This is a schematic diagram of an example system for distributing skill templates.

[0033] Figure 8 This is a flowchart illustrating an example process of distributing skill templates through a skill template distribution system.

[0034] Similar reference numerals and model designations in the various figures indicate similar elements. Detailed Implementation

[0035] Figure 1 This is a schematic diagram illustrating an example demonstration learning system. System 100 is an example of a system capable of implementing the demonstration-based learning techniques described in this specification.

[0036] System 100 includes multiple functional components, including an online execution system 110, a training system 120, and a robot interface subsystem 160. Each of these components can be implemented as a computer program installed on one or more computers at one or more locations, and these components are coupled to each other via any suitable communication network, such as an intranet or the Internet, or a combination of networks.

[0037] System 100 operates in two basic modes to control robots 170a-n: demonstration mode and execution mode.

[0038] In demonstration mode, the user can control one or more robots 170a-n to perform a specific task or subtask. While doing so, the online execution system 110 collects state information 135 and online observations 145 to generate local demonstration data. The demonstration data collector 150 is a module that can generate local demonstration data 115 from the state information 135 and online observations 145, which the online execution system 110 can then provide to the training system 120. The training system can then generate a custom control strategy 125 tailored to the specific characteristics of both the task and the robot performing the task.

[0039] In this specification, a control strategy is a module or subsystem that generates one or more next actions for the robot to execute given observed inputs. The output of the control strategy can affect the movement of one or more robot components, such as motors or actuators, or can be a command directly output by the strategy, or a higher-level command used by multiple robot components through the mechanisms of the robot control stack. Therefore, a control strategy can include one or more machine learning models that translate environmental observations into one or more actions.

[0040] In this specification, local demonstration data is data collected when a user controls the robot to demonstrate how the robot can perform specific tasks by causing it to perform physical movements. Local demonstration data may include kinematic data, such as joint positions, orientations, and angles. Local demonstration data may also include sensor data, such as data collected from one or more sensors. Sensors may include force sensors; vision sensors, such as cameras, depth cameras, and lidar; electronic connection sensors; accelerometers; audio sensors; gyroscopes; contact sensors; radar sensors; and proximity sensors, such as infrared proximity sensors, capacitive proximity sensors, or inductive proximity sensors, to name just a few.

[0041] Typically, local demo data is obtained from one or more robots that are very close to the user controlling them in demo mode. However, close physical proximity between the user and the robot is not a necessary condition for obtaining local demo data. For example, a user can remotely obtain local demo data from a specific robot via a remote user interface.

[0042] Training system 120 is a computer system that can use machine learning techniques to generate custom control policies 125 from local demonstration data 115. Training system 120 typically has more computing resources than online execution system 110. For example, training system 120 may be a cloud-based computing system with hundreds or thousands of computing nodes.

[0043] To generate a custom control policy 125, the training system 120 can first acquire or pre-generate a basic control policy for the task. A basic control policy is a control policy for a specific task whose expected performance is sufficient to make any sufficiently similar robot relatively close to being able to perform that task. For the vast majority of tasks, a basic control policy alone is insufficient to complete the task with sufficient precision and achieve sufficiently reliable success. For example, connection and insertion tasks typically require sub-millimeter accuracy, which cannot be achieved without the specific details provided by local demonstration data for a particular robot.

[0044] Basic control policies for a specific task can be generated in various ways. For example, basic control policies can be manually programmed, trained using traditional reinforcement learning techniques, or used using demonstration-based learning techniques described in this specification. All of these techniques are suitable for pre-generating basic control policies before receiving local demonstration data for the task, as time is a less significant factor when generating basic control policies.

[0045] In some embodiments, the training system generates a basic control strategy from generalized training data 165. However, while the local demonstration data 115 collected by the online execution system 110 is typically specific to a particular robot or a particular robot model, the generalized training data 165 can be generated from one or more other robots that do not need to be the same model, located in the same location, or manufactured by the same manufacturer. For example, the generalized training data 165 can be generated offline from dozens, hundreds, or thousands of different robots with different characteristics and different models. Furthermore, the generalized training data 165 does not even need to be generated from a physical robot. For example, the generalized training data can include data generated from simulations of physical robots.

[0046] Therefore, local demo data 115 is local because it is specific to the particular robot that the user can access and manipulate. Thus, local demo data 115 represents robot-specific data, but it can also represent local variables, such as specific characteristics of a particular task and specific characteristics of a particular work environment.

[0047] A basic control strategy can also be defined using system demonstration data collected during the skill template development process. For example, the engineering team associated with the entity generating the skill template can perform the demonstration using one or more robots in a facility remote from and / or unrelated to System 100. The robot used to generate the system demonstration data also does not need to be the same robot or the same robot model as robots 170a-n in work cell 170. In this case, the system demonstration data can be used to guide the actions of the basic control strategy. The basic control strategy can then be adapted into a custom control strategy using more computationally expensive and complex learning methods.

[0048] Tuning the basic control policy using local demonstration data offers highly desirable results; it is relatively fast compared to generating the basic control policy, for example, by collecting system demonstration data or training using generalized training data 165. For instance, the size of generalized training data 165 for a specific task is often several orders of magnitude larger than local demonstration data 115, therefore, training the basic control policy is expected to take much longer than tuning it for a specific robot. For example, training the basic control policy may require substantial computational resources; in some instances, data centers with hundreds or thousands of machines have been working for days or weeks to train the basic control policy from generalized training data. Conversely, tuning the basic control policy using local demonstration data 115 may only take a few hours.

[0049] Similarly, collecting system demonstration data to define a basic control strategy may require more iterations than local demonstration data. For example, to define a basic control strategy, an engineering team might demonstrate 1,000 successful tasks and 1,000 unsuccessful tasks. Conversely, a well-tuned basic control strategy might only require 50 successful demonstrations and 50 unsuccessful demonstrations.

[0050] Therefore, the training system 120 can refine the basic control strategy using local demo data 115 to generate a custom control strategy 125 for the specific robot used to generate the demo data. The custom control strategy 125 adjusts the basic control strategy to take into account the characteristics of the specific robot and the local variables of the task. Training the custom control strategy 125 using local demo data takes significantly less time than training the basic control strategy. For example, while training the basic control strategy might take days or weeks, a user can generate local demo data 115 using a robot in just 1-2 hours, which can then be uploaded to the training system 120. The training system 120 can then generate the custom control strategy 125 in a much shorter time (e.g., perhaps only one or two hours) than training the basic control strategy.

[0051] In execution mode, execution engine 130 can automatically execute tasks using a custom control strategy 125 without any user intervention. Online execution system 110 can use the custom control strategy 125 to generate commands 155 that will be provided to robot interface subsystem 160, which drives one or more robots, such as robots 170a-n, within work cell 170. Online execution system 110 can consume state information 135 generated by robots 170a-n and online observations 145 performed by one or more observing sensors 171a-n within work cell 170. Figure 1As shown, each sensor 171 is coupled to its respective robot 170. However, sensors do not need to correspond one-to-one with a robot, nor do they need to be coupled to a robot. In fact, each robot can have multiple sensors, which can be mounted on fixed or movable surfaces within the work cell 170.

[0052] The execution engine 130 can use state information 135 and online observation 145 as inputs to a custom control strategy 125 received from the training system 120. Therefore, the robots 170a-n can react in real time, completing tasks based on their specific characteristics and the specific features of the task.

[0053] Therefore, adjusting the control strategy using local demo data results in a drastically different user experience. From the user's perspective, training the robot to perform tasks very precisely using a custom control strategy—including generating local demo data and waiting for the custom control strategy to be generated—is a very fast process, potentially requiring less than a day of setup time. This speed comes from utilizing a pre-computed basic control strategy.

[0054] This arrangement introduces a significant technological improvement compared to existing robot learning methods, which typically require weeks to test and generate hand-designed reward functions, weeks to generate suitable training data, and weeks to train, test, and refine the model to make it suitable for industrial production.

[0055] Furthermore, unlike traditional robot reinforcement learning, using local demo data is highly robust to minor perturbations in the characteristics of the robot, task, and environment. If a company purchases a new robot model, the user only needs to spend a day generating new local demo data for the new custom control strategy. This contrasts with existing reinforcement learning methods, where any change to the robot's physical characteristics, task, or environment can require an entire process that can take weeks to complete from scratch.

[0056] To initiate a demonstration-based learning process, the online execution system can receive a skill template 105 from the training system 120. As described above, the skill template 105 can specify a sequence of one or more subtasks required to perform the skill, which subtasks require local demonstration learning, which subtasks will require which perceptual flows, and specify transition conditions for when to transition from performing a subtask of the skill template to the next subtask.

[0057] As mentioned above, skill templates can define demonstration subtasks that require local demonstration learning, non-demonstration subtasks that do not require local demonstration learning, or both.

[0058] The demonstration subtasks implicitly or explicitly depend on a basic control policy, which, as mentioned above, can be pre-computed from generalized training data or system demonstration data. Therefore, a skill template can include a separate basic control policy for each demonstration subtask in the template, or an identifier for the basic control policy.

[0059] The skill template may also include software modules that, for each demonstration subtask, require tuning using local demonstration data. Each demonstration subtask may rely on different types of machine learning models and may use different techniques for tuning. For example, a motion demonstration subtask may rely heavily on camera images from the local work cell environment to find a specific task target. Therefore, the tuning procedure for a motion demonstration subtask may tune the machine learning model more to identify features in the camera images captured in the local demonstration data. Conversely, an insertion demonstration subtask may rely heavily on force feedback data that senses the edges of the connection interface and uses appropriate, gentle force to insert the connector into the interface. Therefore, the tuning procedure for an insertion demonstration subtask may tune the machine learning model that processes force perception and corresponding feedback more. In other words, even if the underlying models of the subtasks in the skill template are the same, each subtask can have its own tuning procedure that incorporates local demonstration data in different ways.

[0060] Non-demonstration subtasks may or may not be associated with a basic control strategy. For example, a non-demonstration subtask may simply specify moving to a specific coordinate position. Alternatively, a non-demonstration subtask may be associated with a basic control strategy, such as one calculated from another robot, specifying how the joint should move to a specific coordinate position using sensor data.

[0061] The purpose of skill templates is to provide a generalized framework for programming robots to possess specific task capabilities. In particular, skill templates can be used to tailor a robot to perform similar tasks with relatively little effort. Tailoring a skill template for a specific robot and a specific environment therefore involves performing a training process on each demonstration subtask within the skill template. For simplicity, this process can be referred to as training the skill template, even though it may involve multiple, separately trained models.

[0062] For example, a user can download a connector insertion skill template that specifies performing a first motion subtask followed by a connector insertion subtask. The connector insertion skill template can also specify that the first subtask relies on a visual perception stream (e.g., from a camera), while the second subtask relies on a force perception stream (e.g., from a force sensor). The connector insertion skill template can also specify that only the second subtask requires local demonstration learning. This is likely because moving the robot to a specific location is not typically highly dependent on the task at hand or the working environment. However, if the working environment has tight space requirements, the template can also specify that the first subtask requires local demonstration learning so that the robot can quickly learn to navigate through the tight space requirements of the working environment.

[0063] To equip the robot with connector insertion skills, users only need to guide the robot to perform sub-tasks requiring local demonstration data, as indicated by the skill template. The robot will automatically capture the local demonstration data, which the training system can use to refine the basic control strategy associated with the connector insertion sub-task. Once the custom control strategy is trained, the robot only needs to download the final trained custom control strategy to be equipped to perform the sub-task.

[0064] It's worth noting that the same skill template can be used for many different kinds of tasks. For example, the same connector insertion skill template can be used to equip a robot to perform HDMI cable insertion or USB cable insertion, or both. All the user needs to do is demonstrate these different insertion sub-tasks to refine the basic control strategy for the demonstration sub-task being learned. As mentioned above, this process typically requires far less computational power and time than developing or learning a complete control strategy from scratch.

[0065] Furthermore, the skill template approach can be hardware-agnostic. This means that skill templates can be used to equip a robot to perform tasks even if the training system has never trained a control policy for a particular robot model. Therefore, this technique addresses many problems associated with using reinforcement learning to control robots. In particular, it solves the fragility problem, where even very small hardware changes require relearning the control policy from scratch, which is costly and repetitive.

[0066] To support the collection of local demonstration data, system 100 may also include one or more UI devices 180 and one or more demonstration devices 190. UI devices 180 can help guide users to obtain the local demonstration data most beneficial for generating custom control strategies 125. UI devices 180 may include user interfaces that teach users which actions to perform or repeat, and augmented reality devices that allow users to control the robot without actually being near it.

[0067] Demonstration device 190 is a device that assists in the main operation of system 100. Generally, demonstration device 190 is a device that allows a user to demonstrate skills to the robot without introducing irrelevant force data into the local demonstration data. In other words, demonstration device 190 can reduce the likelihood that the user's demonstration actions will affect the actual readings of the sensor during execution.

[0068] In operation, the robot interface subsystem 160 and the online execution system 110 can operate according to different time constraints. In some embodiments, the robot interface subsystem 160 is a real-time software control system with hard real-time requirements. A real-time software control system is a software system that requires execution within strict time requirements to achieve normal operation. Time requirements are typically specified, and certain actions or outputs must be performed within a specific time window to prevent the system from entering a fault state. In a fault state, the system can abort execution or take other actions that interrupt normal operation.

[0069] The online execution system 110, on the other hand, typically offers greater operational flexibility. In other words, the online execution system 110 can, but is not required to, provide commands 155 within each real-time time window in which the robot interface subsystem 160 operates. However, to provide the ability to make sensor-based responses, the online execution system 110 can still operate under stringent time requirements. In a typical system, the real-time requirements of the robot interface subsystem 160 dictate that the robot provides commands every 5 milliseconds, while the online requirements of the online execution system 110 specify that the online execution system 110 should provide commands 155 to the robot interface subsystem 160 every 20 milliseconds. However, even if no such command is received within the online time window, the robot interface subsystem 160 does not necessarily have to enter a fault state.

[0070] Therefore, in this specification, the term "online" refers to the time and stiffness parameters of operation. The time window is larger than the time window of the real-time robot interface subsystem 160 and generally offers more flexibility when time constraints are not met. In some implementations, the robot interface subsystem 160 provides a cross-hardware interface to ensure compatibility of commands 155 issued by the field execution engine 150 with multiple different robot versions. During execution, the robot interface subsystem 160 can report status information 135 back to the online execution system 110 so that the field execution system 150 can make online adjustments to the robot's motion, for example, due to local faults or other unforeseen conditions. The robot can be a real-time robot, meaning that the robot is programmed to continuously execute its commands according to a highly constrained schedule. For example, each robot may be expected to receive commands from the robot interface subsystem 160 at a specific frequency, such as 100 Hz or 1 kHz. If the robot does not receive the expected command, it can enter a fault mode and stop operating.

[0071] Figure 2A This is a schematic diagram of an example system 200 for performing subtasks using a custom control strategy based on local demonstration data. Typically, data from multiple sensors 260 is fed through multiple separately trained neural networks and combined into a low-dimensional task state representation 205. This low-dimensional representation 205 is then used as input to adjust a control strategy 210, which is configured to generate robot commands 235 to be executed by a robot 270. Therefore, system 200 can implement a custom control strategy based on local demonstration data by modifying the basic control strategy through subsystem 280.

[0072] Sensor 260 may include a perception sensor that generates a perception data stream representing visual features of a target in the robot or robotic work cell. For example, to achieve better visual capabilities, the robotic tool may be equipped with multiple cameras, such as visible light cameras, infrared cameras, and depth cameras, to name just a few.

[0073] Different sensing data streams 202 can be processed independently by their respective convolutional neural networks 220a-n. Each sensing data stream 202 can correspond to a different sensing sensor, such as a different camera or a camera of a different type. Data from each camera can be processed by a different convolutional neural network.

[0074] Sensor 260 also includes one or more robot state sensors that generate a robot state data stream 204 representing the physical characteristics of the robot or robot components. For example, the robot state data stream 204 may represent forces, torques, angles, positions, velocities, and accelerations of the robot or its components, to name just a few. Each of the robot state data streams 204 may be processed by its own deep neural network 230a-m.

[0075] The modified subsystem 280 can have any number of neural network subsystems that process sensor data in parallel. In some implementations, the system includes only a perception stream and a robot state data stream.

[0076] The outputs of the neural network subsystem are respective portions of the task state representation 205, which cumulatively represents the state of the subtasks performed by the robot 270. In some implementations, the task state representation 205 is a low-dimensional representation with fewer than 100 features, for example, 10, 30, or 50 features. Having a low-dimensional task state representation means fewer model parameters need to be learned, which further improves the speed at which local demonstration data can be used to tune specific subtasks.

[0077] Then, the task state representation 205 is used as input to the adjustment control strategy 210. During execution, the adjustment control strategy 210 generates robot instructions 235 from the input task state representation 205, which are then executed by the robot 270.

[0078] During training, the training engine 240 generates parameter corrections 255 using the representation of the local demonstration action 275 and the suggested commands 245 generated by the adjustment control policy 210. The training engine can then use the parameter corrections 255 to refine the adjustment control policy 210 so that the commands generated by the adjustment control policy 210 in future iterations will be closer to the local demonstration action 275.

[0079] During training, the regulation control policy 210 can be initialized using a basic control policy associated with the demonstration subtask being trained. The regulation control policy 210 can be iteratively updated using the local demonstration action 275. The training engine 240 can use any suitable machine learning technique to adjust the regulation control policy 210, such as supervised learning, regression, or reinforcement learning. When the regulation control policy 210 is implemented using a neural network, parameter correction 235 can be backpropagated through the network so that in future iterations, the output suggested command 245 is closer to the local demonstration action 275.

[0080] As described above, each subtask of the skill template can have a different training priority, even if their underlying model architectures are the same or similar. Therefore, in some implementations, the training engine 240 may optionally take subtask hyperparameters 275, which specify how to update the adjustment control policy 210, as input. For example, the subtask hyperparameters may indicate that visual sensing is critical. Therefore, the training engine 240 can more aggressively adjust the adjustment control policy 210 to align with the camera data captured using the local demonstration action 275. In some implementations, the subtask hyperparameters 275 identify separate training modules for each different subtask.

[0081] Figure 2B This is a schematic diagram of another example system that performs subtasks using local demonstration data. In this example, the system does not have only a single adjustment control strategy, but includes multiple independent control strategies 210a-n. Each control strategy 210a-n can use the task state representation 205 to generate its own robot sub-instructions 234a-n. The system can then combine these sub-instructions to generate a single robot command 235 executed by the robot 270.

[0082] Having multiple individually adjustable control strategies is advantageous in sensor-rich environments, for example, by using data from multiple sensors with different update rates. Different control strategies 210a-210n can be executed at different update rates, allowing the system to incorporate both simple and more complex control algorithms within the same system. For instance, one control strategy can focus on robot commands using current force data, which updates at a faster rate than image data. Simultaneously, another control strategy can focus on robot commands using current image data, which may require more complex image recognition algorithms with potentially nondeterministic runtimes. The result is a system that can rapidly adjust to force data while adapting to image data without slowing down its adjustment to force data. During training, subtask hyperparameters can identify separate training procedures for each individually adjustable strategy 210-an.

[0083] Figure 2C This is a schematic diagram of another example system that uses residual reinforcement learning to perform subtasks. In this example, instead of a single regulatory control policy that generates robot commands, the system uses a residual reinforcement learning subsystem 212 to generate corrective actions 225 that modify the basic actions 215 generated by the basic control policy 250.

[0084] In this example, the basic control strategy 250 takes sensor data 245 from one or more sensors 260 as input and generates basic actions 215. As described above, the output of the basic control strategy 250 can be one or more commands, which are consumed by the various components of the robot 270.

[0085] During execution, the reinforcement learning subsystem 212 generates a corrective action 225 from the input task state representation 205 to combine with the basic action 215. The corrective action 225 is corrective because it modifies the basic action 215 from the basic control policy 250. The generated robot command 235 can then be executed by the robot 270.

[0086] Traditional reinforcement learning processes use two phases: (1) the action phase, where the system generates new candidate actions, and (2) the training phase, where the model's weights are adjusted to maximize the cumulative reward for each candidate action. As mentioned in the background section above, traditional methods of reinforcement learning for robots suffer from a severe sparse reward problem, which means that randomly generated actions in the action phase are extremely unlikely to obtain any type of reward through the task's reward function.

[0087] Unlike traditional reinforcement learning, using local demo data provides all the information about which action to choose during the action phase. In other words, local demo data provides a set of actions, so actions do not need to be randomly generated. This technique significantly limits the problem space and makes the model converge faster.

[0088] During training, local demonstration data is used to drive robot 270. In other words, robot commands 235 generated from correction actions 225 and basic actions 215 need to be used to drive robot 270. At each time step, reinforcement learning subsystem 210 receives representations of demonstration actions for physically moving robot 270. Reinforcement learning subsystem 210 also receives basic actions 215 generated by basic control policy 250.

[0089] Then, the reinforcement learning subsystem 210 can generate a reconstructed corrective action by comparing the demonstration action with the basic action 215. The reinforcement learning subsystem 210 can also use a reward function to generate the actual reward value for the reconstructed corrective action.

[0090] The reinforcement learning subsystem 210 can also generate a predicted correction action based on the current state of the reinforcement learning model, and a predicted reward value that will be generated using the predicted correction action. The predicted correction action is the correction action that the reinforcement learning subsystem 210 will generate based on the current task state representation 205.

[0091] Then, the reinforcement learning subsystem 210 can use the predicted correction action, the predicted reward value, the reconstructed correction action, and the actual reward value to compute weight updates for the reinforcement model. During iterations of the training data, the weight updates are used to adjust the predicted correction action to a reconstructed correction action reflected by the demonstration action. The reinforcement learning subsystem 210 can compute weight updates according to any appropriate reward maximization procedure.

[0092] Figure 2A The architecture shown in -C provides the capability to combine multiple different models for sensor streams with varying update rates. Some real-time robots have very demanding control loop requirements, so they can be equipped with force and torque sensors that generate high-frequency updates, such as 100, 1000, or 10,000 Hz. In contrast, very few cameras or depth cameras operate at frequencies exceeding 60 Hz.

[0093] Figure 2A The architecture shown in -C features multiple parallel and independent sensor streams, and optionally multiple different control strategies, allowing for the combination of these different data rates.

[0094] Figure 3A This is a flowchart of an example process for combining sensor data from multiple different sensor streams. This process can be performed by a computer system having one or more computers at one or more locations, for example... Figure 1 System 100. The process is described as being executed by a system of one or more computers.

[0095] The system selects a basic update rate (302). The basic update rate will define the rate at which the learning subsystem, for example, adjusts the control strategy 210 to generate commands that drive the robot. In some implementations, the system selects the basic update rate based on the robot's minimum real-time update rate. Alternatively, the system may select the basic update rate based on the sensors that generate data at the fastest speed.

[0096] The system generates the various parts of the task state representation at their respective update rates (304). Since the neural network subsystem can operate independently and in parallel, it can repeatedly generate the various parts of the task state representation at the rate specified by the rate of its respective sensor.

[0097] To enhance system independence and parallelism, some implementations maintain multiple independent storage devices or partitions, where different portions of the task state representation are written. This prevents different neural network subsystems from competing for memory accesses while generating their outputs at high frequencies.

[0098] The system repeatedly generates a task state representation at a basic update rate (306). During each time period defined by the basic update rate, the system can generate a new version of the task state representation by reading the most recently updated sensor data output by multiple neural network subsystems. For example, the system can read from multiple independent storage devices or storage partitions to generate a complete task state representation. It is worth noting that this means that some neural network subsystems generate data at a rate different from their generation rate. For example, for sensors with a slower update rate, data can be consumed at a rate much faster than its generation rate.

[0099] The system repeatedly uses the task state representation to generate robot commands at a basic update rate (308). By using independent and parallel neural network subsystems, the system can ensure that commands are generated at a sufficiently fast update rate to power the robot with hard real-time constraints.

[0100] This arrangement also means that the system can simultaneously feed multiple independent control algorithms with different update frequencies. For example, as mentioned above... Figure 2B The system can include multiple independent control strategies, each generating sub-commands, rather than the system generating a single command. The system can then generate a final command by combining these sub-commands into a final, hybrid robot command, which represents the output of multiple different control algorithms.

[0101] For example, vision control algorithms can enable a robot to move towards a recognized object more quickly. Meanwhile, force control algorithms can enable a robot to track along surfaces it contacts. Even though vision control algorithms are typically updated much more slowly than force control algorithms, the system can still function effectively. Figure 2A The architecture described in -C powers both at a basic update rate.

[0102] Figure 2A The architecture shown in -C provides numerous opportunities to expand the system's capabilities without requiring major redesign. Multiple parallel and independent data streams allow for the implementation of machine learning functions that are beneficial for local demonstration learning.

[0103] For example, integrating sensors that take into account local environmental data can be highly advantageous in order to more thoroughly tune a robot’s performance in a particular environment.

[0104] One example of using local environment data is considering functions related to electrical connections. Electrical connections can serve as a reward factor for a variety of challenging robotic tasks involving establishing current between two components. These tasks include plugging cables into jacks, inserting power plugs into electrical outlets, and screwing in light bulbs, to name just a few.

[0105] To integrate the electrical connection into the modified subsystem 280, an electrical sensor, such as one of the sensors 260, can be configured in the working unit to detect when a current is established. The output of the electrical sensor can then be processed by a separate neural network subsystem, and the result can be added to the task state representation 205. Alternatively, the output of the electrical sensor can be provided directly as input to the system implementing the regulation control strategy or the reinforcement learning subsystem.

[0106] Another example of using local environment data is a function that considers certain types of audio data. For instance, many connector insertion tasks have very distinctive sounds when successfully completed. Therefore, the system can use a microphone to capture audio, and the output of the audio processing neural network can be added to the task state representation. The system can then use a function that considers the specific acoustic characteristics of the connector insertion sound, forcing the learning subsystem to learn what the sound of a successful connector insertion sounds like.

[0107] Figure 3B This is a schematic diagram of a camera wrist strap. Camera wrist straps are examples of a wide variety of instruments that can be used for high-precision demonstration and learning using the architecture described above. Figure 3B It is a perspective view of the tool at the end of the robot arm that is closest to the viewer.

[0108] In this example, the camera wrist strap is mounted on the robotic arm 335, just before the tool 345 located at the very end of the robotic arm 335. The camera wrist strap is mounted on the robotic arm 335 using a clamp 345 and has four radially mounted cameras 310a-310d.

[0109] The hoop 345 can have any suitable protruding shape, allowing it to be securely mounted at the end of the robotic arm. The hoop 345 can be designed to be added to robots manufactured by third-party manufacturers. For example, a system distributing skill templates could also distribute camera wristbands to help non-professional users quickly converge the model. Alternatively or additionally, the hoop 345 can be integrated into the robotic arm by the manufacturer during the manufacturing process.

[0110] The hoop 345 can be elliptical in shape, such as circular or oval, or rectangular. The hoop 345 can be formed from a single solid volume and secured to the end of the robotic arm before the tool 345 is secured. Alternatively, the hoop 345 can be opened and securely closed by a fastening mechanism, such as a latch or latch. The hoop 345 can be made of any suitable material to provide a secure connection to the robotic arm, such as hard plastic; fiberglass; fabric; or metal, such as aluminum or steel.

[0111] Each camera 310a-d has a respective base 325a-d that secures the sensor, other electronics, and respective lenses 315a-d to a clamp 345. The clamp 345 may also include one or more lights 355a-b that illuminate the volume captured by the camera 310a-d. Generally, the cameras 310a-d are arranged to capture different, respective views of the working volume, either from the tool 345 or outside the tool 345.

[0112] The example camera wristband has four radially mounted cameras, but any suitable number of cameras can be used, such as 2, 5, or 10. As described above, modifying the architecture of subsystem 280 allows any number of sensor streams to be included in the task state representation. For example, the computer system associated with the robot can implement different, respective convolutional neural networks to process the sensor data generated by each camera 310a-d in parallel. The processed camera outputs can then be combined to generate a task state representation, which, as described above, can be used to power multiple control algorithms running at different frequencies. As described above, the processed camera outputs can be combined with the outputs of other networks that independently process the outputs of force sensors, torque sensors, position sensors, velocity sensors, or tactile sensors, or any suitable combination of these sensors.

[0113] Generally, using a camera wristband during demonstration learning leads to faster model convergence because the system is able to recognize reward conditions for more locations and orientations. Therefore, using a camera wristband can effectively further reduce the training time required to fine-tune the basic control policy using local demonstration data.

[0114] Figure 3C This is another example view of the camera wristband. Figure 3C Further instrumentation for implementing a camera wristband is shown, including cables 385a-d for supplying the camera's output to respective convolutional neural networks. Figure 3C The diagram also illustrates how an additional depth camera 375 can be mounted on the hoop 345. As described above, the system's architecture allows for the integration of any other sensors into the sensing system; therefore, for example, a separately trained convolutional neural network can process the output of the depth camera 375 to generate another part of the task state representation.

[0115] Figure 3D This is another example view of the camera wristband. Figure 3D This is a perspective view of the camera wristband, which has a metal band and four radially mounted cameras 317a-d.

[0116] With these basic mechanisms for refining control strategies using local demo data, users can orchestrate tasks to build skill templates across hardware, which can be downloaded and used to quickly deploy tasks on many different types of robots and in many different environments.

[0117] Figure 4 Example skill template 400 is shown. Generally, a skill template defines a state machine for multiple subtasks that need to be performed. It is worth noting that skill templates are hierarchically composable, meaning that each subtask can be an independent task or another skill template.

[0118] Each subtask of a skill template has a subtask ID and includes subtask metadata, including whether the subtask is a demonstration subtask or a non-demonstration subtask, or whether the subtask refers to another skill template that should be trained separately. The subtask metadata can also indicate which sensor streams will be used to perform the subtask. Subtasks serving as demonstration subtasks will additionally include a base policy ID, which identifies the base policy that will be combined with corrective actions learned from local demonstration data. Each demonstration subtask will also be explicitly or implicitly associated with one or more software modules that control the training process of the subtask.

[0119] Each subtask in a skill template also has one or more transition conditions, specifying the conditions under which a transition should occur to another task within the skill template. Transition conditions can also be referred to as subtask objectives of the subtask.

[0120] Figure 4 The example illustrates a skill template for performing a task notoriously difficult to achieve in traditional robot learning techniques. This task is a grasping and connecting insertion task, requiring the robot to locate a wire within a work cell and insert a connector at one end of the wire into a socket also within the work cell. This problem is difficult to generalize using traditional reinforcement learning techniques because wires have many different textures, diameters, and colors. Furthermore, if the grasping subtask of this skill fails, traditional reinforcement learning techniques cannot tell the robot what to do next or how to make progress.

[0121] Skill Template 400 includes four sub-tasks, in Figure 4 In this context, nodes are represented as nodes in the graph that define the state machine. In practice, Figure 4 All information within can be represented in any suitable format, such as a plain text configuration file or a record in a relational database. Alternatively or additionally, the user interface device can generate a graphical skill template editor, allowing users to define skill templates through a graphical user interface.

[0122] The first subtask in skill template 400 is the movement subtask 410. Movement subtask 410 is designed to locate a wire within a work cell, requiring the robot to move from an initial position to a desired location on the wire, for example, as placed by a previous robot in an assembly line. Moving from one location to the next typically does not heavily rely on the robot's local characteristics; therefore, the metadata for movement subtask 410 specifies that this subtask is a non-demonstration subtask. The metadata for movement subtask 410 also specifies that a camera stream is required for locating the wire.

[0123] The moving subtask 410 also specifies the transition condition 405 for "gaining wire vision," which indicates when the robot should transition to the next subtask in the skill template.

[0124] The next subtask in skill template 400 is the grasping subtask 420. Grasping subtask 420 is designed to grasp a wire in a work cell. This subtask is highly dependent on the characteristics of the wire and the robot, especially the tool used to grasp the wire. Therefore, grasping subtask 420 is designated as a demonstration subtask that needs to be refined with local demonstration data. Grasping subtask 420 is also associated with a basic policy ID, which identifies the previously generated basic control policy typically used for grasping wires.

[0125] Subtask 420 also specifies that both the camera stream and the force sensor stream need to execute this subtask.

[0126] The grasping subtask 420 also includes three transition conditions. The first transition condition, the "loss of wire vision" transition condition 415, is triggered when the robot loses visual contact with the wire. This can happen, for example, when the wire is accidentally moved within the work cell, for instance, by a human or another robot. In this case, the robot transitions back to the moving subtask 410.

[0127] The second transition condition for grasping subtask 420, namely the "grabbing failure" transition condition 425, is triggered when the robot attempts to grasp the wire but fails. In this scenario, the robot can simply backtrack and try grasping subtask 420 again.

[0128] The third transition condition for the grabbing subtask 420, namely the "grab successful" transition condition 435, is triggered when the robot attempts to grab the wire and succeeds.

[0129] The demonstration subtask can also indicate which transition conditions require local demonstration data. For example, a specific subtask could indicate that all three transition conditions require local demonstration data. Therefore, users can demonstrate how a grasp succeeds, how a grasp fails, and the robot's vision when it loses its wire.

[0130] The next subtask in skill template 400 is the second movement subtask 430. Movement subtask 430 is designed to move the grasped wire to a location within the work unit near the outlet. In many cases where users expect the robot to perform connections and insertions, the outlet is in a height-restricted space, such as inside a dishwasher, television, or microwave being assembled. Because movement in that height-restricted space is highly dependent on the subtask and the work unit, the second movement subtask 430 is designated as a demonstration subtask, even though it only involves moving from one location within the work unit to another. Therefore, a movement subtask can be either a demonstration or a non-demonstration subtask, depending on the skill requirements.

[0131] Although the second motion subtask 430 is designated as a demonstration subtask, it does not specify any base policy ID. This is because some subtasks heavily rely on local working cells that include base policies that would only hinder model convergence. For example, if the second motion task 430 requires moving the robot in a very specific orientation within an appliance, then a generalized base policy for movement would be unhelpful. Therefore, the user can perform refinement to generate local demonstration data showing how the robot should move through working cells to reach a specific orientation within the device.

[0132] The second motion subtask 430 includes two transition conditions 445 and 485. The first "acquire socket vision" transition condition 445 is triggered when the camera stream comes into contact with the socket vision.

[0133] If the robot happens to drop a wire while moving towards the socket, a second "dropped wire" transition condition 485 is triggered. In this case, skill template 400 specifies that the robot will need to return to motion subtask 1 in order to restart the skill. These transition conditions in the skill template provide the robot with a built-in robustness and dynamic response level that traditional reinforcement learning techniques cannot offer.

[0134] The last subtask in skill template 400 is insertion subtask 440. Insertion subtask 440 is designed to insert a connector of a captured wire into a socket. Insertion subtask 440 is highly dependent on the type of wire and the type of socket; therefore, skill template 400 indicates that insertion subtask 440 is a demonstration subtask typically associated with the basic strategy ID related to insertion subtasks. Insertion subtask 440 also indicates that this subtask requires both camera flow and force sensor flow.

[0135] Insertion subtask 440 includes three transition conditions. The first, "Insertation Failure" transition condition 465, is triggered when insertion fails for any reason and specifies that the insertion should be attempted again. The second, "Loss of Socket Visibility" transition condition 455, is triggered when the socket happens to move out of the camera's line of sight and again specifies that the wire should be moved to the socket's location within a height-restricted space. Finally, the "Dropped Wire" transition condition 475 is triggered when the wire is dropped while performing the insertion task. In this case, skill template 400 specifies that the process should return to the first movement subtask 410.

[0136] Figure 4 One of the main advantages of the skill templates shown is their composability for developers. This means that new skill templates can be composed of already developed subtasks. This functionality also includes hierarchical composability, meaning that each subtask within a particular skill template can reference another skill template.

[0137] For example, in an alternative implementation, insertion subtask 440 may actually reference a state machine insertion skill template that defines multiple finely controlled motions. For example, the insertion skill template may include a first motion subtask whose objective is to align the connector and receptacle as precisely as possible, a second motion subtask whose objective is to achieve contact between the sides of the connector and receptacle, and a third motion subtask whose objective is to achieve a complete connection by using the sides of the receptacle as force guides.

[0138] Furthermore, skill templates can be layered from skill template 400. For example, skill template 400 could be a small piece of a more complex set of subtasks required to assemble electronic appliances. The entire skill template could have multiple connectors inserted into subtasks, each referencing the skill template to implement that subtask, such as skill template 400.

[0139] Figure 5 This is a flowchart illustrating an example process for configuring a robot to perform skills using a skill template. This process can be performed by a computer system with one or more computers at one or more locations, for example... Figure 1 System 100. This process will be described as being performed by a system of one or more computers.

[0140] The system receives a skill template (510). As described above, the skill template defines a state machine with multiple subtasks, and defines the transition conditions when the robot should transition from performing one task to the next. Furthermore, the skill template can define which tasks are demonstration subtasks that need to be refined using local demonstration data.

[0141] The system obtains the basic control strategy (520) for the demonstration subtask of the skill template. This basic control strategy can be a generalized control strategy generated from multiple different robot models.

[0142] The system receives local demonstration data (530) for the demonstration subtask. The user can use an input device or user interface to enable the robot to execute the demonstration subtask in multiple iterations. During this process, the system automatically generates local demonstration data for the executed subtask.

[0143] The system trains a machine learning model for the demonstration subtask (540). As described above, the machine learning model can be configured to generate commands to be executed by the robot for one or more input sensor streams, and the machine learning model can be tuned using local demonstration data. In some implementations, the machine learning model is a residual reinforcement learning model that generates corrective actions combined with basic actions generated by the basic control policy.

[0144] The system executes the skill template (550) on the robot. After training all the demonstration subtasks, the system can use the skill template to enable the robot to fully perform the task. During this process, the robot will use refined demonstration subtasks that are specifically tailored for the robot's hardware and operating environment using local demonstration data.

[0145] Figure 6A This is a flowchart illustrating an example of force-guided task processing using skill templates. The skill template arrangement described above provides a relatively easy way to generate very complex tasks consisting of multiple highly complex subtasks. The connector inserts an example of such a task, which uses force data as guidance. This allows the robot to achieve a higher level of accuracy than other methods can. This processing can be performed by a computer system with one or more computers in one or more locations, for example, Figure 1 System 100. This process will be described as being performed by a system of one or more computers.

[0146] The system receives a skill template (602) with transition conditions requiring the establishment of physical contact forces between an object held by the robot and a surface in the robot's environment. As described above, the skill template can define a state machine with multiple tasks. The transition conditions can define the transition between the first and second subtasks of the state machine.

[0147] For example, the first subtask could be a movement subtask, and the second subtask could be an insertion subtask. Transition conditions can be specified, such that the connector held by the robot and to be inserted into the socket needs to make physical contact with the edge of the socket.

[0148] The system receives local demonstration data (604) for the transition. In other words, the system can ask the user to demonstrate the transition between the first and second subtasks. The system can also ask the user to demonstrate a failure scenario. One such failure scenario is the loss of physical contact with the edge of the socket. If this occurs, the skill template can specify a return to the first motion subtask of the template so that the robot can re-establish physical contact as specified in the transition conditions.

[0149] The system trains a machine learning model using local demo data (606). As described above, through training, the system learns to avoid actions that result in loss of physical contact force and learns to select actions that may maintain physical contact force throughout the second task.

[0150] The system executes a trained skill template (608) on the robot. This allows the robot to automatically perform subtasks and transitions defined by the skill template. For connection and insertion tasks, for example, local demo data can enable the robot to be highly adapted to inserting a specific connector.

[0151] Figure 6B This is a flowchart illustrating an example process of training a skill template using a cloud-based training system. Generally, the system can generate all the demo data locally and then upload it to the cloud-based training system to train all the demo subtasks of the skill template. This process can be performed by a computer system that has one or more computers in one or more locations, for example... Figure 1 System 100. This process will be described as being performed by a system of one or more computers.

[0152] The system receives a skill template (610). For example, the online execution system may download the skill template from a cloud-based training system that trains demonstration subtasks of the skill template, or download it from another computer system.

[0153] The system identifies one or more demonstration subtasks defined in the skill template (620). As mentioned above, each subtask defined in the skill template can be associated with metadata indicating whether the subtask is a demonstration subtask or a non-demonstration subtask.

[0154] The system generates a separate local demonstration dataset (630) for each of one or more demonstration subtasks. As described above, the system can instantiate and deploy individual task systems, each generating local demonstration data as the user manipulates the robot to perform subtasks in a local work cell. The task state representation can be generated at the basic rate of the subtasks, regardless of the update rate of the sensors contributing data to the task state representation. This provides a convenient way to store and organize local demonstration data, rather than generating many different sensor datasets that must all be coordinated in some way later.

[0155] The system uploads a local demonstration dataset to a cloud-based training system (640). Most robotic facilities employing real-world tasks lack on-site data centers suitable for training complex machine learning models. Therefore, while local demonstration data can be collected on-site by a system co-located with the robot performing the task, the actual model parameters can be generated by a cloud-based training system accessible only via the internet or other computer networks.

[0156] As mentioned above, the size of the local demo data is expected to be several orders of magnitude smaller than the size of the data used to train the basic control policy. Therefore, although the local demo data may be large, the upload burden is manageable within a reasonable timeframe, for example, a few minutes to an hour.

[0157] In the cloud-based training system, individual training model parameters (650) are generated for each set of local demonstration data. As described above, the training system can train the learning system to generate robot commands, which can be, for example, a combination of corrective actions from basic actions generated by a basic control policy. As part of this process, the training system can obtain individual basic control policies for each demonstration subtask, which can be local or from another computer system, such as a third-party computer system that publishes the task or skill template.

[0158] Cloud-based training systems typically possess significantly more computational power than online execution systems. Therefore, while training each demo subtask involves a substantial computational burden, these operations can be parallelized on a cloud-based training system at a massive scale. Consequently, in typical scenarios, training a skill template from local demo data on a cloud-based training system takes no more than a few hours.

[0159] The system receives trained model parameters (660) generated by a cloud-based training system. The size of the trained model parameters is typically much smaller than the size of the local demo data for a specific subtask, so the time required to download the trained parameters after model training is negligible.

[0160] The system uses trained model parameters generated by a cloud-based training system to execute a skill template (670). As part of this process, the system can also download the basic control policy for the demonstration subtask, for example, from the training system, from its original source, or from another source. The trained model parameters can then be used to generate commands for the robot to execute. In a reinforcement learning system for the demonstration subtask, the parameters can be used to generate corrective actions that modify the basic actions generated by the basic control policy. The online execution system can then repeatedly issue the generated robot commands to drive the robot to perform specific tasks.

[0161] Figure 6B The processing described herein can be completed by a team of one person in a single day, enabling robots to perform highly precise skills in a way that is tailored to their environment. This is a significant improvement over traditional manual programming methods or even traditional reinforcement learning methods, which require many teams of engineers to work for weeks or months to design, test, and train models that do not generalize well to other scenarios.

[0162] Figure 7 This is a schematic diagram of an example system 700 for distributing skill templates. As mentioned above, skill templates provide the ability to quickly adapt robot tasks to a wide range of robot models. This means that the system can popularize the process of creating skill templates and distribute them to multiple potential robot installations maintained by affiliated or unaffiliated entities.

[0163] The system includes a skill template distribution system 710, which is a computer system that maintains a skill template set 780 and a basic control strategy set 790 for demonstration subtasks defined in a skill template set 760.

[0164] For training skill templates, basic control strategies, or both, the skill template distribution system 710 communicates with the training system 720. The skill template distribution system 710 may also optionally communicate with a robot execution system 730, which has one or more robots, for developing and testing new skill templates to be added to the skill template set 760.

[0165] In operation, in step 1, one or more skill template developers 740 may provide skill template 705 to skill template distribution system 710. Each skill template developer 740 may be an entity, such as an individual, team, or organization that designs skill templates for completing specific robotic skills. Each skill template developer 740 may be affiliated with or employed by the entity 710 that operates the skill template distribution system. Alternatively or additionally, skill template developers 740 may be third-party entities that only provide skill template definitions 705 to the skill template distribution system 710.

[0166] Skill template developer 740 may or may not access the physical robot to generate basic control strategies. Therefore, skill template developer 740 may or may not provide corresponding basic control strategies for skill template definition 705.

[0167] A key advantage of the skill templates described in this specification is that they are composable, meaning that arbitrarily complex behaviors can be represented from individual subtasks. For example, one use case for composable skill templates is furniture assembly. Prefabricated furniture is typically assembled using nearly identical fastening hardware. Therefore, skill templates can be assembled by developers associated with or employed by the entity that manufactures the furniture. The skill template can be reused multiple times for demonstration subtasks related to manipulating the fastening hardware. Thus, very complex skill templates can be developed by skill developers and used to enable robots to automatically assemble entire pieces of furniture.

[0168] Furthermore, the composability of skill templates simplifies the process of modifying or pushing new updates for specific tasks. For example, a skill developer related to furniture manufacturing can push a new skill template each time a new version of furniture is released. Individual stores can then download the skill template and adjust it to their specific robot models to quickly adapt their robots for automated furniture assembly.

[0169] After receiving the skill template, in step 2, the skill template distribution system 710 can generate or obtain a basic control strategy 715 for each demonstration subtask that needs to perform the task. (See above for reference.) Figure 1 The basic control policy can be obtained from demonstration data or learned from generalized training data. Therefore, the skill template distribution system 710 can use a training system 720, which can be a local or remote cloud-based computer system, to generate a basic control policy 715 for a specific skill. The skill template distribution system can then add a skill template 705 to a skill template set 780 and add the basic control policy 715 to a basic control policy set 790. As part of the process, the skill template distribution system 710 associates each demonstration subtask in the skill template with the corresponding basic control policy in the set of basic control policies 790.

[0170] The skill template distribution system 710 then makes the collection of skill templates available for installation on any suitable robot. Many industrial robotics facilities worldwide lack the engineering personnel, resources, and computing power to attempt to implement reinforcement learning. However, as described above, this capability becomes possible by simply downloading and training skill templates from the skill template distribution system 710. In some implementations, the robotics facility team first installs low-level control software to make the robot's hardware compatible with the actions of the skill templates. The low-level control software can be organized as a multi-level, cross-hardware software stack, increasing hardware specificity in one direction and software abstraction in another. Therefore, it is possible that only the lowest one or two levels of the software stack are specifically relevant to a particular robot model, and the skill template distribution system 710 can also make this level of implementation available for many different types of robots. The architecture of the cross-hardware software control stack will be described in more detail below.

[0171] exist Figure 7 In the example shown, robotic facility 780 downloads and trains skill templates to deploy skills on one or more local robots. The described operations should be understood as being performed by one or more computers installed on robotic facility 780. Operations involving user input should be understood as being performed using appropriate user devices, such as mobile phones, tablets, laptops, or desktop computers.

[0172] Therefore, in step 3, a user of the robotic facility 780 can submit a request 725 to the skill template distribution system 710 for available skill templates. In response, in step 4, the skill template distribution system 710 can provide the skill marketplace interface 735 back to the industrial robotic facility. In some implementations, the skill marketplace interface 735 can be filtered manually or automatically based on a specific robot type. For example, a user can specify which robot model is installed in the robotic facility 780, and the skill template distribution system 710 can filter skill templates so that only skill templates compatible with a specific robot type are provided in the interface 735. In this context, the skill marketplace interface 735 refers to a software mechanism, application, or tool provided by the skill template distribution system 710 that allows a user to select skill templates to be trained for deployment on one or more robots. Therefore, the skill marketplace interface 735 can be implemented using any suitable user interface front-end. For example, the skill marketplace interface 735 can be a command-line interface or a graphical user interface. A skill marketplace does not necessarily imply financial transactions, although some skill marketplaces may require payment for users to download and / or train skill templates.

[0173] In step 5, the user of the robotic facility 780 can provide a skill template selection 745, and in response, in step 6, the skill template distribution system 710 can provide a skill template 755. As described above, the skill template 755 defines the state machine of the task and indicates which are the demonstration subtasks that require local demonstration data. The skill template 755 may also include software modules that allow the basic control strategy to be adjusted based on the local characteristics of the robotic facility 780 and the robot.

[0174] In step 7, users of the robotics facility 780 upload local demo data 765 collected from the demo task to the field robot located at the robotics facility 780. Typically, engineers at the industrial robotics facility 780 are not expected to be machine learning experts, or even experts in acquiring local demo data. Therefore, the skill template distribution system 710 can provide various software and hardware tools to facilitate the collection of the local demo data needed to most effectively train skill templates. Examples of hardware and software tools that help non-expert users collect local demo data will be described in detail below.

[0175] In step 8, the skill template distribution system 710 provides local demonstration data 765 to the training system 720, which trains a model for each demonstration subtask, which can refine the basic control strategy of the demonstration subtask.

[0176] In step 9, the training system 720 provides the trained model parameters 775 back to the skill template distribution system 710.

[0177] In step 10, the skill template distribution system 710 provides the industrial robot facility 780 with the basic control strategy and training model parameters 785 for the demonstration subtask of the selected skill template 755.

[0178] At this point, the robotic facility 780 can execute the trained skill template, enabling the robot to automatically perform skill tasks based on the state machine defined by the skill template. Because the skill template has been customized based on local demonstration data, the robot is expected to perform skills with high accuracy in the environment of the robotic facility 780. Moreover, as mentioned above, for a single non-specialist in the industrial robotic facility 780, the entire process from step 3 to step 10 typically requires less than a day of setup time. This contrasts sharply with traditional methods using robot reinforcement learning, which require weeks of design, training, and testing by a large team of highly specialized expert engineers.

[0179] Another use case for a skills template distribution system is an organization with multiple factories producing different products, but without identical robot models. Therefore, a skills template distribution system can generate skills templates for assembling products using one or more appropriate basic control strategies. The system can then distribute these templates to all factories, each of which can quickly adjust the skills templates to suit its specific robot model using local demo data collected at the factory. Thus, all factories can be up and running in less than a day of setup time.

[0180] Another use case for skill template distribution systems is for organizations with multiple retail outlets selling products that require consumer assembly. Organizations can offer consumers the option of automated robotic assembly of products on-site by distributing skill templates to all retail outlets. Each retail outlet can then tailor the skill template to its specific robot model and the environment of the retail location. Therefore, when a skill template for a new product is distributed, each retail outlet can very quickly provide automated robotic assembly of the new product.

[0181] However, another use case for the skill template distribution system is for manufacturers to distribute skill templates for home robots. Consumers of the product, for example, can access the website of the skill template distribution system to download skill templates that will be performed on a specific model of home robot. Consumers can then provide local demonstration data, allowing the templates for a specific robot model to be tailored to their specific environment, which can be important given the varying lighting conditions and space constraints in a consumer's home. Within a short timeframe, a consumer's robot can be configured with the trained skill templates generated in the cloud by the training system 720, allowing the robot to automatically assemble the product.

[0182] Figure 8 This is a flowchart illustrating an example process of distributing skill templates through a skill template distribution system. This process can be performed by a computer system having one or more computers at one or more locations, for example... Figure 7 The skill template distribution system 710. This process will be described as being performed by a system of one or more computers.

[0183] The system receives a selection of skill templates from the user device (810). As described above, this selection can be made through a user interface that presents available skill templates and selectively selects only skill templates compatible with a specific robot type or robot model associated with the selection.

[0184] The system provides the selected skill template to the user device (820). The skill template indicates which subtasks are demonstration subtasks that require local demonstration data.

[0185] The system receives local demonstration data (830) for demonstration sub-tasks used in the skill template. In a typical scenario, local demonstration data is generated within a facility housing a robot that will perform the final training on the skill template. Therefore, local demonstration data is typically collected by demonstrating to the robot how to perform the demonstration sub-task within the facility.

[0186] The system uses local demo data to train a machine learning model for the demo subtask (840). As mentioned above, the facility housing the bot may not be affiliated with the training system that will ultimately train the machine learning model. In other words, the entity that owns the bot and the entity that distributes the skill templates are different entities that distribute the skill templates on a fair exchange. Furthermore, this also generally means that the user who generates the local demo data can be employed by a different entity, rather than the user who sets up and maintains the training system for training the machine learning model.

[0187] The system provides learning parameter values ​​for the machine learning model and a basic control policy (850) for demonstrating subtasks. By deploying the basic control policy and customizing it using the learned parameter values, the skill template can be executed by the robot with high precision, taking into account both the robot's own characteristics and the characteristics of the robot's personalized environment.

[0188] In this specification, a robot is a machine having a basic position, one or more movable parts, and a kinematic model that can be used to map a desired position, pose, or both in a coordinate system, such as Cartesian coordinates, to commands for physically moving one or more movable parts to the desired position or pose. In this specification, a tool is a device that is part of the kinematic chain of one or more movable parts of the robot and connected to its end. Example tools include grippers, welding equipment, and grinding equipment.

[0189] In this specification, a task is an operation performed by a tool. For the sake of brevity, when the robot has only one tool, a task can be described as an operation performed by the robot as a whole. Examples of tasks include welding, dispensing, part positioning, and surface polishing, to name just a few. Tasks are typically associated with the tool required to perform the task and the type of location within the work cell where the task is performed.

[0190] In this specification, a motion plan is a data structure that provides information for performing an action, which can be a task, a cluster of tasks, or a transition. A motion plan can be fully constrained, meaning that all values ​​for all controllable degrees of freedom of the robot are explicitly or implicitly represented; or underconstrained, meaning that some values ​​for controllable degrees of freedom are unspecified. In some implementations, for the motion plan to actually perform the action corresponding to it, the motion plan must be fully constrained to include all necessary values ​​for all controllable degrees of freedom of the robot. Therefore, some motion plans may be underconstrained during the planning process described in this specification, but may be fully constrained when the motion plan is actually executed on the robot. In some implementations, a motion plan represents an edge of a task graph between two configuration states of a single robot. Therefore, typically each robot has one task graph.

[0191] In this specification, motion scan volume refers to the area of ​​space occupied by at least a portion of the robot or tool throughout the execution of the motion plan. Motion scan volume can be generated by the collision geometry associated with the robot-tool system.

[0192] In this specification, a transition is a motion plan describing the movement to be performed between a start point and an end point. The start and end points can be represented by pose, position in a coordinate system, or the task to be performed. Transitions can be under-constrained due to the lack of one or more values ​​for one or more of the robot's respective degrees of freedom (DOFs). Some transitions represent free motion. In this specification, free motion is a transition in which no degree of freedom is constrained. For example, a robot movement that simply moves from pose A to pose B without any restrictions on how it moves between the two poses is free motion. During the process, the DOF variables for free motion are eventually assigned values, and the path planner can use any appropriate value for the motion that does not conflict with the physical constraints of the work cell.

[0193] The robot functions described in this specification can be implemented using a software stack across hardware, or, for brevity, only at least partially using a software stack across hardware. In other words, the software stack can accept commands generated by the planning process described above as input, without requiring the commands to be specifically related to a particular robot model or a particular robot component. For example, the software stack can be at least partially implemented using… Figure 1 The field execution engine 150 and robot interface subsystem 160 are implemented.

[0194] A software stack can comprise multiple layers, increasing hardware specialization in one direction and software abstraction in another. At the bottom of the software stack are the robot components, including devices that perform low-level actions and sensors that report low-level states. For example, a robot may include various low-level components, including motors, encoders, cameras, actuators, grippers, application-specific sensors, linear or rotary position sensors, and other peripherals. As an example, a motor can receive commands indicating the amount of torque to be applied. In response to these commands, the motor can report the current position of the robot joints, for example, using encoders, to higher levels of the software stack.

[0195] Each sub-layer in the software stack can implement interfaces that support multiple different underlying implementations. Generally, each interface between layers provides state information from lower to higher layers and commands from higher to lower layers.

[0196] Typically, commands and status information are generated cyclically within each control cycle; for example, each control cycle contains one status message and one command. Lower layers of the software stack usually have stricter real-time requirements than higher layers. At the lowest layer of the software stack, such as the control cycle, there may be actual real-time requirements. In this specification, real-time means that within a specific control cycle, commands received at a level of the software stack must be executed, and optionally, status information must be fed back to higher layers of the software stack. If this real-time requirement is not met, the robot can be configured to enter a fault state, for example, by freezing all operations.

[0197] At the next higher level, the software stack can include software abstractions of specific components, which will be referred to as motor feedback controllers. A motor feedback controller can be a software abstraction of any suitable low-level component, not just a literal motor. Therefore, the motor feedback controller receives state through an interface, accesses the lower-level hardware component, and sends commands to the lower-level hardware component through the interface based on commands received from higher levels in the stack. The motor feedback controller can have any suitable control rules that determine how upper-level commands should be interpreted and translated into lower-level commands. For example, a motor feedback controller can use anything from simple logic rules to more advanced machine learning techniques to translate upper-level commands into lower-level commands. Similarly, a motor feedback controller can use any suitable fault rules to determine when a fault state is reached. For example, if the motor feedback controller receives an upper-level command but does not receive a lower-level state within a specific part of the control cycle, the motor feedback controller can cause the robot to enter a fault state that stops all operations.

[0198] At the next higher level, the software stack may include actuator feedback controllers. Actuator feedback controllers may include control logic for controlling multiple robot components via their respective motor feedback controllers. For example, some robot components, such as articulated arms, may actually be controlled by multiple motors. Therefore, the actuator feedback controller can provide a software abstraction of the articulated arm by sending commands to the motor feedback controllers of multiple motors using its control logic.

[0199] At the next higher level, the software stack can include joint feedback controllers. A joint feedback controller can represent a joint mapped to the robot's logical degrees of freedom. Thus, for example, while a robot's wrist might be controlled by a complex network of actuators, a joint feedback controller can abstract away this complexity and expose that degree of freedom as a single joint. Therefore, each joint feedback controller can control arbitrarily complex networks of actuator feedback controllers. For example, a six-DOF robot can be controlled by six different joint feedback controllers, each controlling an independent network of actual feedback controllers.

[0200] Each level of the software stack can also enforce level-specific constraints. For example, if a specific torque value received by the actuator feedback controller exceeds an acceptable range, the actuator feedback controller can modify it to be within the acceptable range or enter a fault state.

[0201] To drive the inputs to the joint feedback controller, the software stack can use command vectors that include command parameters for each component in the lower layers, such as the position, torque, and speed of each motor in the system. To expose the state from the joint feedback controller, the software stack can use state vectors that include state information for each component in the lower layers, such as the position, speed, and torque of each motor in the system. In some implementations, the command vectors also include constraint information about the constraints enforced by the lower-level controller.

[0202] At the next higher level, the software stack can include joint collection controllers. Joint collection controllers can handle the issuance of commands and state vectors as a set of partially abstracted public entities. Each entity can include a kinematic model, for example, for performing inverse kinematics calculations, limiting information, and joint state vectors and joint command vectors. For example, a single joint collection controller can be used to apply different sets of policies to different subsystems at lower levels. Joint collection controllers can effectively decouple the physical representation of motors and control policies from the relationships between these entities. Thus, for example, if a robot arm has a movable base, the joint collection controller can be used to enforce one set of restrictive policies on the movement of the arm and a different set of restrictive policies on the movement of the movable base.

[0203] At the next higher level, the software stack can include a joint selection controller. The joint selection controller is responsible for dynamically selecting commands issued from different sources. In other words, the joint selection controller can receive multiple commands within a single control cycle and select one of them to execute. This ability to dynamically select from multiple commands within a real-time control cycle significantly increases the control flexibility of traditional robot control systems.

[0204] At the next higher level, the software stack can include joint position controllers. Joint position controllers can receive target parameters and dynamically calculate the commands needed to achieve those parameters. For example, a joint position controller can receive a position target and calculate the setpoint to achieve that target.

[0205] At the next lower level, the software stack can include a Cartesian position controller and a Cartesian selection controller. The Cartesian position controller receives the target in Cartesian space as input and uses an inverse kinematics solver to compute the output in joint position space. Then, before passing the computed results in joint position space to the next lower level joint position controller in the stack, the Cartesian selection controller can implement constraint policies on the results computed by the Cartesian position controller. For example, the Cartesian position controller can be assigned three independent target states in Cartesian coordinates x, y, and z. For some levels, the target state might be position, while for others, it might be the desired velocity.

[0206] Therefore, the capabilities provided by the software stack offer extensive flexibility for control commands, allowing them to be easily expressed as target states and naturally integrated with the higher-level planning techniques described above. In other words, when the planning process uses process definition diagrams to generate specific actions to be taken, these actions do not need to be specified as low-level commands for individual robot components. Instead, they can be expressed as high-level goals, which are accepted by the software stack and translated through different levels until they finally become low-level commands. Furthermore, actions generated through the planning process can be specified in Cartesian space, making them understandable to human operators, which makes debugging and analyzing plans easier, faster, and more intuitive. Moreover, actions generated through the planning process do not need to be tightly coupled to any specific robot model or low-level command format. Instead, the same actions generated during the planning process can actually be executed by different robot models, as long as they support the same degrees of freedom and have the appropriate control hierarchy implemented in the software stack.

[0207] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, in tangible computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination thereof. Alternatively or additionally, program instructions may be encoded onto artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiving device for execution by the data processing device.

[0208] The term "data processing apparatus" refers to data processing hardware, including various types of devices, apparatuses, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The apparatus may also be, or include, special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0209] A computer program (also referred to as or described as a program, software, software application, application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program can, but does not need to, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, for example, in one or more scripts stored in a markup language file, in a single file dedicated to the program, or in multiple coordinating files, for example, in a file storing one or more modules, subroutines, or portions of code. A computer program can be deployed to execute on one computer, or on multiple computers located in one location, or distributed across multiple locations and interconnected via a data communication network.

[0210] For a system consisting of one or more computers configured to perform a specific operation or action, it means that the system has software, firmware, hardware, or a combination thereof installed on it, which, in operation, causes the system to perform that operation or action. For one or more computer programs configured to perform a specific operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0211] As used in this specification, "engine" or "software engine" refers to an input / output system implemented in software that provides outputs different from the inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet, laptop, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device that includes one or more processors and computer-readable media. Furthermore, two or more engines can be implemented on the same computing device or on different computing devices.

[0212] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry, such as FPGAs or ASICs, or by a combination of special-purpose logic circuitry and one or more programmed computers.

[0213] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated into, special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to that device, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device, such as a Universal Serial Bus (USB) flash drive, to name a few.

[0214] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as semiconductor storage devices like EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM discs.

[0215] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer with a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse, trackball, or presence-sensitive display or other surface through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and any form of input from the user can be received, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving files from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from the web browser. Additionally, the computer can interact with the user by sending text messages or other forms of information to a personal device, such as a smartphone, running a messaging application, and receiving responses from the user.

[0216] Embodiments of the subject matter described in this specification may be implemented in a computing system including backend components, such as data servers; or in a computing system including middleware components, such as application servers; or in a computing system including frontend components, such as client computers with graphical user interfaces, web browsers, or applications through which users can interact with embodiments of the subject matter described in this specification, or with any combination of one or more such backend, middleware, or frontend components. Components of the system may be interconnected through any form of digital data communication medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0217] A computing system may include clients and servers. Clients and servers are typically geographically isolated and usually interact via a communication network. The client-server relationship arises from computer programs running on their respective computers, and a client-server relationship exists between them. In some embodiments, the server transmits data, such as HTML pages, to a user device, for example, to display data to a user interacting with the device as a client and to receive user input. Data generated on the user device, such as the results of user interactions, can be received from the device on the server.

[0218] In addition to the embodiments described above, the following embodiments are also innovative:

[0219] Example 1 is a method comprising:

[0220] Receive selections of available skill templates from user devices through a skill template distribution system;

[0221] Skill templates are provided by a skill template distribution system, wherein the skill templates include information representing a state machine for one or more tasks, and wherein the skill templates specify which of the one or more tasks are demonstration subtasks that require local demonstration data;

[0222] The skill template distribution system receives local demo data for the demo subtasks of skill templates.

[0223] Train a machine learning model for the demo subtask using local demo data to generate learning parameter values; and

[0224] Provide user equipment with learning parameter values ​​and demonstration subtask basic control strategies.

[0225] Example 2 is the method as described in claim 1, further comprising:

[0226] The system receives requests for training using available skill templates from user devices through a skill template distribution system, enabling the robot to perform skills with multiple tasks; and

[0227] In response to the request, information describing one or more available skill templates is provided to the user device.

[0228] Example 3 is the method as described in Example 2, wherein the request specifies a particular robot type or model, and wherein providing information describing one or more available skill templates includes providing one or more skill templates compatible with the particular robot type or model.

[0229] Example 4 is the method as described in any one of Examples 1-3, further comprising:

[0230] Receive new skill templates generated by the skill template developer; and

[0231] Add new skill templates to the set of available skills;

[0232] The selection of available skill templates includes the option to choose new skill templates generated by skill template developers.

[0233] Example 5 is the method as described in Example 4, wherein the skill template developer is an entity that is not affiliated with the skill template distribution system or an organization that owns user devices.

[0234] Example 6 is a method as described in any one of Examples 1-5, wherein the user equipment is located in a facility with a robot, and wherein the local demonstration data is data captured by demonstrating to the robot how to perform a demonstration subtask of a selected skill template.

[0235] Example 7 is the method as described in Example 6, wherein the training skill template is executed in a training system not attached to a facility with robots.

[0236] Example 8 is the method as described in Example 7, wherein the basic control strategy is generated by a training system that is not attached to a facility with a robot.

[0237] Example 9 is a system comprising: one or more computers and one or more storage devices storing operable instructions that, when executed by the one or more computers, cause the one or more computers to perform the method described in any one of Examples 1 to 8.

[0238] Example 10 is a computer storage medium encoded with a computer program including operable instructions that, when executed by a data processing device, cause the data processing device to perform the method described in any one of Examples 1 to 8.

[0239] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the content of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the content of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although the foregoing features may be described as functioning in certain combinations, or even originally claimed in this way, in some cases one or more features may be removed from the claimed combination, and the claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0240] Similarly, although the operations in the accompanying drawings are described in a specific order, this should not be construed as requiring that these operations be performed in the specific order shown or sequentially, or that all of the shown operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments; it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0241] Specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the preceding claims. For example, the actions described in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific order shown, or a sequential order, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for robots, the method performed by one or more computers, comprising: receiving, by a skill template distribution system from a user device, a request for available skill templates to be trained to cause a robot to perform a skill having a plurality of tasks, wherein each skill template defines one or more subtasks, which of the one or more subtasks are demonstration subtasks to be adjusted with local demonstration data, and which perception streams are needed to determine success or failure, wherein the request specifies a particular robot type or model; in response to the request, providing, by the skill template distribution system to the user device, information describing one or more available skill templates stored by the skill template distribution system that are compatible with the particular robot type or model; receiving, by the skill template distribution system from the user device, a selection of an available skill template from the one or more available skill templates that are compatible with the robot; providing, by the skill template distribution system, a skill template corresponding to the selection received from the user device, wherein the skill template includes information representing a state machine of one or more tasks; receiving, by the skill template distribution system, local demonstration data for a demonstration subtask of the skill template; training, by a cloud-based training system, a machine learning model for the demonstration subtask using local demonstration data received from an online execution system that monitors a robot to generate learned parameter values; and providing the learned parameter values and a base control policy for the demonstration subtask to the user device.

2. The method of claim 1, further comprising: receiving, from a skill template developer, a new skill template generated by the skill template developer; and adding the new skill template to a set of available skills, wherein the selection of the available skill template includes a selection of the new skill template generated by the skill template developer.

3. The method of claim 2, wherein the skill template developer is an entity that is not affiliated with the organization that owns the skill template distribution system or the user device.

4. The method of claim 1, wherein the user device is located in a facility that has a robot, and wherein the local demonstration data is data captured by demonstrating to the robot how to perform a demonstration subtask of the selected skill template.

5. The method of claim 4, wherein training the skill template is performed in a training system that is not affiliated with the facility that has the robot.

6. The method of claim 5, wherein the base control policy is generated by the training system that is not affiliated with the facility that has the robot.

7. A system for robots, comprising: one or more computers and one or more storage devices storing operational instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: ​ receiving, by a skill template distribution system from a user device, a request for available skill templates to be trained to cause a robot to perform a skill having a plurality of tasks, wherein each skill template defines one or more subtasks, which of the one or more subtasks are demonstration subtasks to be adjusted with local demonstration data, and which perception streams are needed to determine success or failure, wherein the request specifies a particular robot type or model; in response to the request, providing, by the skill template distribution system to the user device, information describing one or more available skill templates stored by the skill template distribution system that are compatible with the particular robot type or model; receiving, by the skill template distribution system from a user device, a selection of an available skill template from the one or more available skill templates that are compatible with the robot; providing, by the skill template distribution system, a skill template corresponding to the selection received from the user device, wherein the skill template includes information representing a state machine of one or more tasks; receiving, by the skill template distribution system, local demonstration data for a demonstration subtask of the skill template; training, by a cloud-based training system, a machine learning model for the demonstration subtask using the local demonstration data received from an online execution system that monitors the robot to generate learned parameter values; and providing the learned parameter values and a base control policy for the demonstration subtask to the user device.

8. The system of claim 7, wherein the operations further comprise: receiving, from a skill template developer, a new skill template generated by the skill template developer; and adding the new skill template to a set of available skills, wherein the selection of the available skill template includes a selection of the new skill template generated by the skill template developer.

9. The system of claim 8, wherein the skill template developer is an entity that is not affiliated with the organization that owns the skill template distribution system or the user device.

10. The system of claim 7, wherein the user device is located in a facility having a robot, and wherein the local demonstration data is data captured by demonstrating to the robot how to perform a demonstration subtask of the selected skill template.

11. The system of claim 10, wherein training the skill template is performed in a training system that is not affiliated with the facility having the robot.

12. The system of claim 11, wherein the base control policy is generated by the training system that is not affiliated with the facility having the robot.

13. One or more non-transitory computer storage media encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any of claims 1-6.

Images