Efficient robotic control based on input from remote client device

Abstract

The invention relates to efficient robot control based on input from a remote client device. User interface input from a remote client device is utilized when controlling a robot in an environment. Implementations involve generating training instances based on object manipulation parameters defined by instances input by a user interface, and training a machine learning model to predict the object manipulation parameters. Those implementations may then utilize a trained machine learning model to reduce the number of instances requesting input from the remote client device when performing a given group of robotic operations, and / or to reduce the extent of input from the remote client device when performing a given group of robotic operations. Implementations additionally or alternatively relate to reducing idle time of a robot by utilizing visual data captured by an object to be manipulated by the robot before the object is transmitted to a robot workspace in which the robot can reach and manipulate the object.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 202080053232.7, with a filing date of July 24, 2020. Background Art

[0003] In industrial or commercial settings, robots are often preprogrammed to repeatedly perform specific tasks. For example, a robot may be preprogrammed to repeatedly apply fasteners to a specific assembly part in an assembly line. Also, for example, a robot may be preprogrammed to repeatedly grasp a specific assembly part and move it from a fixed first position to a fixed second position. When grasping an object, the robot may use a grasping end effector, such as an "impact" end effector (e.g., using a "claw" or other finger to apply force to an area of ​​the object), an "intrusion" end effector (e.g., using a nail, needle, etc. to physically penetrate the object), a "retraction" end effector (e.g., using suction or vacuum to pick up the object) and / or one or more "contact" end effectors (e.g., using surface tension, freezing, or adhesives to pick up the object).

[0004] Such methods may work well in environments where constrained actions are repeatedly performed on constrained groups of components. However, such methods may fail in environments where robots are tasked with performing a wide variety of actions and / or performing actions on different groups of components, optionally including new components for which the robot has not been preprogrammed. Furthermore, such methods require significant engineering effort (and associated use of computing resources) when preprogramming the robot. Furthermore, such methods may require significant reconfiguration of an industrial or commercial environment in order to accommodate the robot in the environment.

[0005] Additionally, some have proposed some methods in a loop approach, where a human repeatedly provides the same type of guidance to assist the robot in performing a task. However, such methods may suffer from various disadvantages. For example, some methods may cause the robot to be idle while requesting and / or waiting for human guidance, which results in inefficient operation of the robot. Furthermore, for example, some methods always request human guidance and / or the same type of guidance. This limits the ability of the robot to operate more efficiently. Summary of the invention

[0006] Implementations disclosed herein relate to utilizing user interface input from a remote client device when controlling a robot in an environment. An instance of the user interface input provided at the remote client device (directly or indirectly) indicates one or more object manipulation parameters to be used by the robot when manipulating at least one object. For example, the object manipulation parameters indicated by the instance of the user interface input may include one or more of the following: a grasping posture; a placement posture; a sequence of waypoints to be encountered when traversing to the grasping posture; a sequence of waypoints to be encountered when traversing toward the placement posture (after grasping the object); a complete path or trajectory (i.e., a path with velocity, acceleration, jerk, and / or other parameters) when traversing to and / or from a manipulation posture (e.g., a grasping posture or other manipulation posture); and / or other object manipulation parameters, such as but not limited to those described in further detail herein.

[0007] The user interface input of the instance is provided with reference to a visual representation of an object representation including at least one object. The visual representation may also optionally include an environmental representation of other environmental objects (e.g., a work surface, a container in which at least one object is to be placed) and / or a robot representation of all or part of the robot. The visual representation may be rendered, for example, on a separate display screen controlled by a remote client device or on a virtual reality (VR) head mounted device controlled by a remote client device. For example, user interface input may be provided via a mouse, a touch screen, a VR hand controller, and / or a VR glove. Additional descriptions of example visual representations and how they are rendered are provided herein, including a description of implementations of generating visual representations in a manner that reduces network traffic and / or reduces latency when rendering the visual representations.

[0008] Some implementations disclosed herein involve generating training instances based on object manipulation parameters defined by instances input by a user interface. These implementations also involve training a machine learning model based on the training instances for use by the trained machine learning model in predicting object manipulation parameters. In some of these implementations, in response to determining that a measure of success of a manipulation attempted based on the corresponding object manipulation parameters satisfies a threshold, the training instance may be generated and / or marked as a positive training instance. The measure of success may be generated based on sensor data from one or more sensors and may be generated in a manner that depends on the manipulation being performed. As an example, if the manipulation is a grasping with an impact end effector, the measure of success may indicate whether the grasping is successful. The measure of success may be based on, for example, one or more of: sensor data from a sensor of the impact end effector (e.g., using the position of a finger determined based on data from a position sensor and / or a torque indicated by a torque sensor to determine whether the impact end effector is grasping an object); visual data from a visual sensor of the robot (e.g., determining whether the impact end effector is grasping an object and / or whether the object has moved from its previous position), a weight sensor in the environment (e.g., determining whether an object is lifted from one position and / or placed in another position), etc. As another example, if the manipulation includes grabbing an object and then placing the object in a container, the success measure of the placement manipulation can indicate whether the object was successfully placed in the container and / or the degree to which the placement in the container conforms to the desired placement. As yet another example, if the manipulation includes joining of two objects, the success measure of the placement manipulation can indicate whether the objects were successfully joined together and / or the accuracy of their joining.

[0009] Implementations of training a machine learning model based on the generated training instances also involve subsequently utilizing the trained machine learning model. Utilizing the trained machine learning model reduces the number of instances in which input from a remote client device is requested when performing a given set of robotic manipulations (thereby reducing network traffic) and / or reduces the extent of input from a remote client device when performing a given set of robotic manipulations (thereby providing efficient resource utilization at the remote client device). These implementations can enable robots in an environment to operate more efficiently by reducing instances and / or durations in which robots are idle while waiting for user interface input. These implementations can further facilitate an operator of a remote client device to perform technical tasks of controlling, supervising, and / or monitoring the operation of one or more robots, and can enable an operator to provide input for a greater number of manipulations and / or for a greater number of robots within a given time period.

[0010] As a specific example, assume that one or more robots are newly deployed in a given environment to perform manipulations, each of which includes grabbing a corresponding object from a conveyor belt and placing the object in an appropriate one of N available containers (e.g., a shipping box). First, a user interface input may be requested for each manipulation to determine object manipulation parameters including any one, any combination, or all of the following: a sequence of waypoints to be encountered when traversing to a grasping posture for grasping the object; a grasping posture; a sequence of waypoints to be encountered when traversing toward an appropriate one of the N available containers; and a placement posture for placing the object in the container. Those determined manipulation parameters may be used to control the robot when performing the manipulation.

[0011] Over time, training instances can be generated for each of one or more machine learning models based on corresponding visual data (and / or other sensor data), one or more of the object manipulation parameters, and optionally based on success measurements. Each of the machine learning models can be trained to process visual data and / or other sensor data when predicting one or more corresponding manipulation parameters. In addition, the machine learning models can be trained based on training instances. For example, assume that a machine learning model is trained to process visual data to generate corresponding probabilities (e.g., of successful object manipulations such as grasping) for each of N grasping postures. For a manipulation that includes a successful grasp (as determined based on a success measurement for the grasp), a positive training instance can be generated based on the corresponding visual data and the corresponding grasping posture defined by the user interface input.

[0012] The trained machine learning model can then be at least selectively used to predict one or more corresponding object manipulation parameters, which are then at least selectively used to control the robot. For example, the predicted object manipulation parameters can be used automatically (without prompting for a confirmatory user interface input) and / or can be used after presenting an indication of the predicted object manipulation parameters (e.g., as part of a visual representation) and receiving a confirmatory user interface input in response. In these and other ways, object manipulation parameters can be determined and used without requiring a user interface input (e.g., when it is used automatically) and / or with a reduced number of user interface inputs (e.g., when a confirmatory user interface input is provided instead of a more time-consuming full input for defining the object manipulation parameters). This can reduce the duration required to determine the object manipulation parameters, enabling the robot and / or remote operator to operate more efficiently.

[0013] In some implementations, after determining that one or more conditions are satisfied, only the trained machine learning model is utilized when predicting object manipulation parameters that are at least selectively utilized. The one or more conditions may include, for example, at least a threshold amount of training and / or validation of the trained machine learning model. Validation of the trained machine learning model may include comparing predictions generated using the machine learning model with ground truth object manipulation parameters based on user interface input (optionally for instances of visual data (and / or other sensor data) on which the machine learning model has not been trained). In various implementations, and as described herein, the trained machine learning model may continue to be trained even after it is actively used to predict object manipulation parameters that are at least selectively utilized in the operation of the robot. For example, additional training instances may be generated based on the predicted and utilized object manipulation parameters and marked as positive or negative based on the determined measure of success. In addition, for example, additional training instances may be generated based on the predicted object manipulation parameters and marked as negative if the user interface input rejects the predicted object manipulation parameters.

[0014] As a specific example, assume again that a machine learning model is trained to process visual data to generate a corresponding probability for each of N grasping postures. When the visual data is processed using the trained machine learning model, it results in a probability of a corresponding grasping posture exceeding a first threshold (e.g., 85% or other threshold), and the grasping posture can be automatically utilized without prompting for a confirmatory user interface input. If no grasping posture exceeds the first threshold but the probability of the grasping posture does exceed a second threshold (e.g., 50% or other threshold), an indication of one or more of those grasping postures and an object representation in the visual representation can be presented, and the grasping posture is only utilized when the confirmatory input points to the grasping posture. If no grasping posture exceeds the first threshold or the second threshold, a user interface input can be requested for determining the grasping posture without providing any indication of the predicted grasping posture. The grasping posture determined based on the user interface input can then be used to generate a training instance, optionally also taking into account a measure of success of the grasp. The training instance can then be used to further train the model. Note that such training examples are “hard negative” training examples, which can be particularly beneficial in efficiently updating the parameters of a machine learning model to increase the accuracy and / or robustness of the model.

[0015] Thus, for a given deployment of a robot in an environment, an instance of a user interface input may be initially used to determine object manipulation parameters that are used to control the robot when performing a manipulation. In addition, training instances may be generated based on object manipulation parameters determined using instances of user interface input and based on corresponding visual data and / or other sensor data and optionally based on success measurements determined by the sensor data. The training instances may be used to train a machine learning model for predicting object manipulation parameters. In response to satisfying one or more conditions, the trained machine learning model may then be "online" and used to generate predicted object manipulation parameters. The predicted object manipulation parameters are at least selectively automatically used to control the robot, and / or a corresponding indication of the predicted object manipulation parameters is rendered on a remote client device and utilized if a confirmatory user interface input is received in response. In addition, even after being online, the trained machine learning model may continue to be trained, thereby increasing its accuracy and efficiency, thereby increasing the number of instances in which predictions can be automatically used to control the robot and / or rendered as suggestions for confirmatory approval.

[0016] In these and other ways, robots can be deployed in new environments and / or for new tasks and used immediately without requiring extensive use of engineering and / or computing resources prior to deployment. For example, object manipulation parameters initially utilized at deployment may be based largely (or even exclusively) on user interface inputs from a remote device. However, over time, user interface inputs from the remote device may be used to train a machine learning model that is brought online to reduce the number and / or extent of user interface inputs required when operating the robot in the environment. This enables the robot to operate more efficiently in the environment and reduces the amount of network traffic to the remote device for a given number or manipulations of the robot. In addition, this enables an operator of a remote client device to assist in controlling a greater number of robot manipulations.

[0017] Some implementations disclosed herein additionally or alternatively relate to specific techniques for determining object manipulation parameters for manipulating a given object based on user interface input from a remote operator. Some of these implementations relate to techniques for mitigating (e.g., reducing or eliminating) idle time of a robot while waiting to provide user interface input. Reducing the idle time of a robot increases the overall efficiency of the robot's operation.

[0018] Some implementations seek to slow down the idle time of the robot by utilizing visual data that captures an object to be manipulated by the robot before it is transferred to a robot workspace where the robot can reach and manipulate the object. For example, a visual component (e.g., a single-view and / or stereoscopic vision camera, a lidar component, and / or other visual component) can have a field of view of a first area of ​​an environment that is different from the robot workspace. When the object is in the first area before being transferred to the robot workspace, the visual data from the visual component can capture features of the object. For example, the first area can be the part where the object is transferred from a portion of a conveyor system to the robot workspace. The visual data capturing the object in the first area can be used to generate a visual representation including at least an object representation of the object, which is generated based on object features of the object captured in the visual data.

[0019] The visual representation may be transmitted to the remote client device before the object is transferred to the robot workspace (e.g., while the object is being conveyed by the conveyor system, but before it reaches the robot workspace). Additionally, data may be received from the remote client device before the object is transferred to the robot workspace, wherein the data is generated based on a user interface input directed to the visual representation when rendered at the remote client device.

[0020] The received data directly or indirectly indicates one or more object manipulation parameters for manipulating an object in a robot workspace. Therefore, the object manipulation parameters can be determined based on the data, and the object manipulation parameters can be optionally determined before the object is transferred to the robot workspace. Then, the determined object manipulation parameters can be used to control the robot so that the robot manipulates the object when the object is in the robot workspace after the object is transferred to the robot workspace. Since at least the visual representation is transmitted and the response data is received before the object is transferred to the robot workspace, once the object is in the robot workspace, the robot can quickly manipulate the object based on the manipulation parameters determined according to the data. For example, the robot can determine when the object is in the robot workspace based on visual data from its own visual components, and operate according to the object manipulation parameters in response to such a determination. The robot can optionally wait for the object to be in a posture corresponding to the posture defined for the object manipulation parameters, or can transform the object manipulation parameters into a newly detected posture of the object in the robot workspace (for example, when it is different from the posture defined for the object manipulation parameters). If the robot workspace itself includes a conveyor portion for conveying objects, the conveyor portion can optionally be temporarily stopped when the robot manipulates the object. In other implementations, objects may be transferred to the robotic workspace using a conveyor or other transport device (eg, air tube, separate transport robot, by a person), and the robotic workspace itself may not include a conveyor portion.

[0021] Optionally, where a trained machine learning model has been brought online for predicting object manipulation parameters, visual data from the first region can be used to predict the object manipulation parameters. This enables prediction of the object manipulation parameters before completing the transfer of the object to the robot workspace. The predicted object manipulation parameters can be automatically used as some of the object manipulation parameters, and / or an indication of the predicted object manipulation parameters can be provided with the visual representation, and if the received data indicates confirmation of the predicted object manipulation parameters, one or more of the predicted object manipulation parameters are utilized.

[0022] In some implementations, the pose of the vision component in the first area and the pose of the robot vision component are known, enabling determination of a transformation between a reference coordinate system of the vision component in the first area and a robot coordinate system of the robot vision component. Use of this transformation enables input at the remote client device to be defined directly in the robot coordinate system, or initially defined in the first coordinate system and then transformed to the robot coordinate system.

[0023] In some implementations, the visual representation transmitted to the remote client device includes an object representation of the object and optionally includes one or more object representations of other nearby dynamic objects (dynamic in the first area), but omits other static parts of the first area. In some of these implementations, only the representation of the object and optionally the nearby dynamic objects are rendered at the remote client device. In some other implementations, all or part of the robot and / or the robot workspace is also rendered at the remote client device (although not captured in the visual data capturing the first area). For example, the remote client device can execute a robot simulator, or communicate with an additional device that executes a robot simulator. The robot simulator can simulate all or part of the robot and / or all or part of the robot workspace, and can render a simulation of the object together with the robot simulation and / or the robot workspace simulation. The above transformation can be used to determine the posture of the object relative to the robot simulation and / or the robot workspace simulation. This can enable a human operator to provide a user interface input that manipulates the robot simulation to define object manipulation parameters. For example, in order to define a grasping posture, a human operator can provide a user interface input that adjusts the robot simulation until it is in a desired posture, and then provide a further user interface input to define the desired posture as a grasping posture.

[0024] Implementations of simulated robots and / or robot workspaces enable visual representations of smaller data sizes to be transmitted from the environment to a remote client device. This may be the result of transmissions that only define dynamic objects, static features that do not define the robot workspace, and / or features that do not define the robot. In addition to saving network resources, this can slow down delays when rendering visual representations at the remote device because smaller data sizes can be transmitted to the remote client device faster and / or rendered faster at the remote client device. This reduction in delay can also slow down the robot's idle time. In addition, note that even in implementations where object representations are generated based on robot visual data (instead of visual data from different areas), simulated robots and / or robot workspaces can still enable visual representations of smaller data sizes to be transmitted - and slow down the robot's idle time.

[0025] Some implementations additionally or alternatively seek to slow down the robot's idle time by generating an object representation of a visual representation to be rendered at a client device, which renders the object with a lower accuracy than the full representation, but has a smaller data size than the full representation. For example, an object may be represented by one or more bounding boxes and / or other bounding shapes that approximate the surface of the object. For example, an object may be defined by a plurality of connected bounding boxes, each of which may be defined by a center point, a height dimension, and a width dimension—which includes significantly less data than a representation that defines the color, texture, and / or depth of each pixel or voxel corresponding to the surface of the object. In addition to saving network resources, a less accurate object representation may slow down the latency of rendering a visual representation at a remote device, because a smaller data size may be transmitted faster at a remote client device and / or rendered faster at a remote client device. In addition, a less accurate object representation may blur or remove potentially sensitive data from an object, or blur the object itself, thereby preventing an operator of the remote device from ascertaining the data and / or the object.

[0026] Although some examples are described herein with reference to manipulations including grasping and / or placing an object, it should be understood that the techniques described herein can be used for various robotic manipulations of an object. For example, the techniques can be used for manipulations including pushing and / or pulling an object to move it to a different location and / or pairing it with another object. In addition, for example, the techniques can be used for manipulations including grasping a first object, grasping a second object, coupling the first and second objects together, and placing the coupled objects in a particular location. As yet another example, the techniques can be used for manipulations including operating an object using an end effector that includes an etching tool, a screwdriver tool, a cutting tool, and / or other tools.

[0027] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of these implementations and other implementations are described in more detail below.

[0028] Other implementations may include a transitory or non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more methods described above and / or elsewhere herein. Other implementations may also include a system of one or more computers and / or one or more robots including one or more processors operable to execute stored instructions to perform methods such as one or more methods described above and / or elsewhere herein.

[0029] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1A An example environment is shown in which the implementations described herein may be implemented.

[0031] Figure 1B The various implementations described herein are shown Figure 1A An example of how the components can interact.

[0032] Figure 2A , 2B , 2C, 2D, 2E and 2F each illustrate examples of rendering at a remote client device a visual representation of an object representation including an object to be manipulated by a robot and examples of user interface inputs that may be provided to define and / or confirm object manipulation parameters for manipulating the object by the robot.

[0033] Figure 3 is a flow chart illustrating an example method of causing a robot to manipulate an object according to object manipulation parameters determined based on data generated at a remote client device in response to a visual representation including an object representation of the object.

[0034] Figure 4 is a flow chart illustrating an example method for generating training instances based on robotic object manipulation attempts and using the training instances in training a predictive model.

[0035] Figure 5 is a flow chart illustrating an example method for selectively utilizing a trained predictive model in determining object manipulation parameters used by a robot in manipulating an object.

[0036] Figure 6 is a flow chart illustrating an example method of training a predictive model, validating the predictive model, deploying the predictive model, and optionally further training the deployed predictive model.

[0037] Figure 7 An example architecture of a robot is schematically depicted.

[0038] Figure 8 An example architecture of a computer system is schematically depicted. DETAILED DESCRIPTION

[0039] Figure 1A An example environment is shown in which the implementations described herein may be implemented. Figure 1AIncludes a first robot 170A and an associated robot vision component 174A, a second robot 170B and an associated robot vision component 174B, and an additional vision component 194. The additional vision component 194 can be, for example, a monocular vision camera (e.g., generating a 2D RGB image), a stereo vision camera (e.g., generating a 2.5D RGB image), a laser scanner (e.g., generating a 2.5D "point cloud"), and can be operably connected to one or more systems disclosed herein (e.g., system 110). Optionally, multiple additional vision components can be provided, and visual data from each vision component utilized as described herein. The robot vision components 174A and 174B can be, for example, a monocular vision camera, a stereo vision camera, a laser scanner, and / or other vision components, and visual data from them can be provided to and utilized by the respective robots 170A and 170B described herein. Although in Figure 1A 170B, but in other implementations, the robot vision components 174A and 174B may instead be coupled directly to the robots 170A and 170B (eg, attached proximate to the end effectors 172A and 172B).

[0040] The robots 170A and 170B, the robot vision components 174A and 174B, and the additional vision component 194 are all deployed in an environment such as a manufacturing facility, a packaging facility, or other environment. Although the environment may include additional robots and / or additional vision components, for simplicity, only the robots 170A and 170B and the additional vision component 194 are shown in FIG.

[0041] Each of the robots 170A and 170B is a "robotic arm" having multiple degrees of freedom to enable the respective gripping end effector 172A, 172B to traverse any of a plurality of potential paths to position the gripping end effector at a desired location. Each of the robots 170A and 170B further controls two opposing "claws" of its respective gripping end effector 172A, 172B to actuate the claws between at least an open position and a closed position (and / or optionally a plurality of "partially closed" positions). Although in Figure 1A Specific robots 170A and 170B are shown in the drawings, but additional robots and / or alternative robots may be utilized, including additional robotic arms similar to robots 170A and 170B, robots having other robotic arm forms, robots having humanoid forms, robots having animal forms, robots that move via one or more wheels, unmanned aerial vehicles (“UAVs”), etc. Furthermore, although Figure 1ASpecific gripping end-effectors 172A and 172B are shown, but additional end-effectors and / or alternative end-effectors (or even no end-effectors) may be utilized, such as alternative impact gripping end-effectors (e.g., those with gripping "plates," those with more or fewer "fingers" / "claws"), "dip" gripping end-effectors, "retract" gripping end-effectors, or "contact" gripping end-effectors, or non-grasping end-effectors (e.g., welding tools, cutting tools, etc.). For example, a retracting end-effector with multiple suction cups may be used to pick up and / or place multiple objects (e.g., four objects may be picked up and placed simultaneously by using multiple suction cups).

[0042] Robot 170A can access robot workspace 101A. Figure 1A , the workspace includes sunglasses 192A on a conveyor portion 103A of a conveyor system, and also includes a container 193A. The robot 170A can utilize object manipulation parameters determined as described herein to grasp the sunglasses 192A and appropriately place them in the container 193A. Other objects can be on the conveyor portion 103A in the robot workspace 101A at different times, and different containers can also be on the conveyor portion at different times (e.g., containers can be placed by separate systems or on separate conveyor systems). For example, as the conveyor system moves, other objects will be transferred into the robot workspace 101A and can be manipulated by the robot 170A while in the robot workspace 170A. The robot 170A can likewise utilize corresponding object manipulation parameters to pick up and place such objects, and / or perform other manipulations on such objects.

[0043] Robot 170B can access robot workspace 101B. Figure 1A In the embodiment of the present invention, the workspace includes a stapler 192B on the conveyor portion 103B of the conveyor system, and also includes a container 193B. The robot 170B can use the object manipulation parameters determined as described herein to grab the stapler 192B and place it appropriately in the container 193B. Other objects can be on the conveyor portion 103B in the robot workspace 101B at different times, and different containers can also be on the conveyor portion at different times. The robot 170B can also use corresponding object manipulation parameters to pick up and place such objects, and / or perform other manipulations on such objects.

[0044] The additional vision component 194 has a field of view of an area 101C of the environment that is different from the robot workspace 101A and different from the robot workspace 101B. Figure 1A, the area includes the conveyor portion 103C of the conveyor system and also includes the spatula 192C. Area 101C may be "upstream" of the robot workspace 101A and / or the robot workspace 101B because the objects to be manipulated first pass through area 101 before being transferred to the robot workspace 101A or the robot workspace 101B. For example, the conveyor system may first pass the objects through area 101C and then route the objects to the robot workspace 101A or the robot workspace 101B via the conveyor system. For example, in Figure 1A , spatula 192C is in area 101C, but has not yet been transferred to robot workspace 101A or robot workspace 101B.

[0045] As described in detail herein, in various implementations, the additional vision component 194 can capture visual data capable of capturing features of the spatula 192C. In addition, the system 110 (described below) can utilize the visual data to determine object manipulation parameters to enable the robot 170A or the robot 170B to manipulate (e.g., pick up and place) the spatula 192C. For example, the system 110 can determine the object manipulation parameters based at least in part on user interface input from the remote client device 130 for a visual representation generated at least in part based on the visual data captured by the additional vision component 194 (e.g., object features based at least in part on the visual data capturing features of the spatula 192C). By utilizing the additional vision component 194 located "upstream" of the robotic workspaces 101A and 101B, visual representations may be provided to the remote client device 130, user interface inputs may be provided at the remote client device 130, and / or object manipulation parameters may be determined based on data corresponding to the user interface inputs before the spatula 192C is in the robotic workspace 101A or the robotic workspace 101B (i.e., before the spatula is transferred to either of the robotic workspaces 101A, 101B). In these and other ways, the robots 170A and 170B may operate more efficiently because object manipulation parameters for manipulating an object may be quickly determined, optionally even before the object reaches the robotic workspaces 101A and 101B.

[0046] Figure 1A The example environment also includes system 110, remote client device 130, training data engine 143, training data database 152, training engine 145, and one or more machine learning models 165 (also referred to herein as "prediction models").

[0047] The system 110 may be implemented by one or more computing devices. The one or more computing devices may be located in an environment with robots 170A and 170B, and / or may be located in a remote server farm. The system 110 includes one or more prediction engines 112, visual representation engines 114, and manipulation parameter engines 116. The system 110 may execute Figure 3 Method 300 and / or Figure 5 One or more (eg, all) operations of method 500 , both of which are described in detail below.

[0048] The remote client device 130 may optionally be within the environment, but in various implementations, is located in a different structure that may be several miles away from the environment. The remote client device 130 includes a display engine 132, an input engine 134, and an input device 136. Note that in various implementations, multiple remote client devices 130 may be accessible to the system 110 at any given time. In these implementations, a given remote client device 130 may be selected at a given time based on various considerations, such as whether the given remote client device 130 has any pending requests in its queue, the number of pending requests in its queue, and / or the expected duration for addressing pending requests in its queue.

[0049] The prediction engines 112 of the system 110 may receive visual data from the visual components 194, 174A, and / or 174B, and optionally receive other sensor data. The prediction engines 112 may each process the visual data and / or other sensor data using a corresponding one of the machine learning models 165 to generate one or more predicted object manipulation parameters for manipulating the object captured by the visual data. For example, one of the prediction engines 112 may process the visual data from the additional visual component 194 and use a corresponding one of the machine learning models 165 to generate a predicted grasping posture for grasping the spatula 192C. In addition, for example, one of the prediction engines 112 may additionally or alternatively process the visual data from the additional visual component 194 and use a corresponding one of the machine learning models 165 to generate a predicted placement posture for placing the spatula 192C. In addition, for example, one of the prediction engines 112 may additionally or alternatively process the visual data from the additional visual component 194 and use a corresponding one of the machine learning models 165 to generate a predicted waypoint to be encountered when traversing to the grasping posture for the spatula. As described herein, the prediction engine 112 and corresponding machine learning model 165 (if any) that is online and used by the system 110 may change over time and may depend on sufficient training and / or validation of the machine learning model (e.g., by the training engine 145).

[0050] The predicted object manipulation parameters (if any) generated by the prediction engine 112 for a given object manipulation may be automatically used as manipulation parameters by the manipulation parameter engine 116, may be first presented for confirmation by the visual representation engine 114 before being utilized, or may be discarded and not utilized. For example, one of the prediction engines 112 may generate the predicted object manipulation parameters and a confidence measure for the predicted object manipulation parameters. If the confidence measure meets a first threshold, the prediction engine may specify that the predicted object manipulation parameters will be used by the manipulation parameter engine 116 without prompting for confirmation. If the confidence measure fails to meet the first threshold but meets the second threshold, the prediction engine may specify that an indication of the predicted object manipulation parameters is to be included in the visual representation by the visual representation engine 114 and is only utilized when a deterministic user interface input for the indication is received. If the confidence measure fails to meet the first threshold and the second threshold, the prediction engine may specify that the predicted object manipulation parameters are not utilized and the visual representation engine 114 prompts to define the corresponding object manipulation parameters.

[0051] The visual representation engine 114 receives visual data from the visual components 194, 174A, and / or 174B and generates a visual representation for transmission to the remote client device 130 for rendering by the display engine 132 of the remote client device 130. The transmission to the remote client device 130 may be via one or more networks (not shown), such as the Internet or other wide area network (WAN).

[0052] The visual representation generated by the visual representation engine 114 includes an object representation of at least one object captured by the visual data. For example, the visual representation may include an object representation of the spatula 192 captured in the visual data from the additional visual component 194. For example, the visual representation may include an object representation that is a two-dimensional (2D) image of the spatula 192. Examples of 2D images of the spatula 192 are described in more detail below. Figure 2D and 2E . Moreover, for example, the visual representation may include an object representation that is a three-dimensional (3D) representation of the spatula 192. For example, the 3D representation of the spatula 192 may define the location (e.g., x, y, z location) of one or more points on the surface of the spatula and may optionally include one or more color values ​​for each location. Examples of 3D representations of the spatula 192 are described in more detail below. Figure 2A , 2B 2C. The visual representation may also optionally include an indication of the predicted object manipulation parameters (if any) from the prediction engine 112. Figure 2EAn example of predicted object manipulation parameters from the prediction engine 112 is shown in . The visual representation may also optionally include an environment representation of other environment objects (e.g., a work surface, a container in which at least one object is to be placed) and / or a robot representation of all or part of the robot.

[0053] In some implementations, the visual representation generated by the visual representation engine 114 and transmitted to the remote client device 130 includes an object representation of the object and optionally one or more object representations of other nearby dynamic objects, but omits static other parts. In some of these implementations, only the object and optionally nearby dynamic objects are rendered at the remote client device 130. In some implementations, all or part of the robot and / or the robot workspace is also rendered at the remote client device 130, although not captured in the visual data transmitted to the remote client device 130. For example, the display engine 132 of the remote client device may include a robot simulator. The robot simulator can simulate all or part of the robot and / or all or part of the robot workspace, and can render a simulation of the object together with the robot simulation and / or the robot workspace simulation. The robot simulator can be used to simulate an environment including corresponding objects, simulate all or part of the robot operating in the simulated environment (e.g., at least the end effector of the robot), and optionally simulate the interaction between the simulated robot and the simulated environment objects in response to the simulated robot action. Various simulators can be used, such as physics engines that simulate collision detection, soft and rigid body dynamics, etc. A non-limiting example of such a simulator is the BULLET physics engine.

[0054] As a specific example, the display engine 132 of the client device may receive a visual representation that includes only a 3D object representation of an object to be manipulated. The display engine 132 may place the 3D object representation in a simulated robot workspace and / or relative to the simulated robot. For example, the robot simulator of the display engine 132 may have a visual representation of a robot workspace and / or a preloaded visual representation of a robot, and may place the 3D object representation relative to those objects. When the object representation is based on visual data from the additional visual component 194, the pose of the object relative to the robot simulation and / or the robot workspace simulation may optionally be determined using a transformation between the pose of the additional visual component 194 and the pose of the corresponding one of the robot visual components 174A, 174B. The simulated robot may be set to a default state (e.g., a start state), or the current state of the robot (e.g., the current position of the joint) may optionally be provided with a visual representation for rendering the simulated robot in the current state. The implementation of the simulated robot and / or the robot workspace enables a visual representation of a smaller data size to be transmitted from the system 110 to the remote client device 130.

[0055] In some implementations, the visual representation engine 114 generates an object representation of the visual representation to be rendered at the client device that renders the object with a lower degree of accuracy than the full representation, but has a smaller data size than the full representation. For example, the visual representation engine 114 can generate an object representation that includes one or more bounding boxes and / or other bounding shapes that approximate the surface of the object. For example, the visual representation engine 114 can generate an object representation that is composed of a plurality of connected bounding boxes, each of which can be defined by a center point, a height dimension, and a width dimension. As described in more detail below, Figure 2F A non-limiting example is shown in . A less detailed object representation can be more concise data, thereby saving network resources. In addition, a less detailed object representation can reduce the delay of rendering the visual representation at the remote device and / or can blur or remove potentially sensitive data from the object, or blur the object itself.

[0056] An operator of the remote client device 130 interacts with the visual representation provided by the display engine 132 using one or more input devices 136 of the remote client device 130. The input device 136 may include, for example, a mouse, a touch screen, a VR hand controller, and / or a VR glove. The input device 136 may form an integral part of the remote client device (e.g., a touch screen), or may be a peripheral device coupled to the remote device 130 using a wired and / or wireless protocol.

[0057] The input engine 134 of the remote client device 130 processes the user interface input provided via the input device 136 to generate (directly or indirectly) data indicating one or more object manipulation parameters to be used in the object manipulation. For example, the object manipulation parameters indicated by the data generated by the input engine 134 of the instance of the user interface input may include: a grab gesture; a placement gesture; a sequence of waypoints to be encountered when traversing to the grab gesture; a sequence of waypoints to be encountered when traversing toward the placement gesture (after grabbing the object); a complete path or trajectory (i.e., a path with speed, acceleration, jerk, and / or other parameters) when traversing to and / or from a manipulation gesture (e.g., a grab gesture or other manipulation gesture); and / or other object manipulation parameters. The user interface input of the instance is provided by an operator of the remote client device 130 with reference to a visual representation rendered by the display engine 132. For example, the instance of the user interface input may indicate a complete trajectory utilized during the assembly of a portion utilizing multiple component parts.

[0058] The manipulation parameter engine 116 determines the manipulation parameters based on the data provided by the input engine 134. In some implementations, the data directly defines the object manipulation parameters, and the manipulation parameter engine 116 determines the object manipulation parameters by utilizing the object manipulation parameters defined by the data. In other implementations, the manipulation parameter engine 116 transforms and / or otherwise processes the data when determining the object manipulation parameters.

[0059] The manipulation parameter engine 116 transmits the determined object manipulation parameters and / or commands generated based on the object manipulation parameters to the robot 170A or 170B. In some implementations, the manipulation parameter engine 116 transmits the object manipulation parameters and / or high-level commands based on the object manipulation parameters. In those implementations, the control system of the corresponding robot transforms the object manipulation parameters and / or high-level commands into corresponding low-level actions, such as control commands issued to the actuators of the robot. In other implementations, the object manipulation parameters may define low-level actions themselves (e.g., when a complete trace is defined via a user interface input) and / or low-level actions may be generated based on the object manipulation parameters, and the manipulation parameter engine 116 transmits the low-level actions to the corresponding robot for control based on the low-level actions.

[0060] The training data engine 143 generates training instances and stores the training instances in the training data database 152. Each of the training instances is generated for a corresponding one of the machine learning models 165 and is generated based on corresponding manipulation parameters of the instance, visual data and / or other data for the instance, and optionally a measure of success for the instance (also referred to herein as a "success measure").

[0061] As an example, the training data engine 143 may receive from the manipulation parameter engine 116 the manipulation parameters used in controlling one of the robots 170A, 170B when performing the manipulation. The manipulation parameters may be manipulation parameters generated based on user interface input from the remote client device 130, predicted by one of the prediction engines 112 and confirmed based on user interface input from the remote client device, or predicted by one of the prediction engines 112 and automatically used. The training data engine 143 may also receive visual data of the instance, such as visual data capturing an object being manipulated in the manipulation. The visual data may come from the additional vision component 194, or from one of the robot vision components 174A or 174B. Note that in some implementations, the visual data used by the training data engine 143 when generating the training instance may be different from the visual data used when generating the object manipulation parameters. For example, the object manipulation parameters may be defined based on the user interface input, which points to an object representation generated based on the visual data from the additional vision component 194, but the visual data (capturing the object) from the robot vision component 174A may be used to generate the training instance.

[0062] The training data engine 143 may optionally further determine the success measure of the manipulation (as a whole, and / or for part of the object manipulation parameters) based on the visual data and / or data from other sensors 104. Other sensors 104 may include, for example, weight sensors in the environment, non-visual sensors of the robot (e.g., torque sensors, position sensors), and / or other sensors. The training data engine 143 may then generate a training instance based on the visual data, the object manipulation parameters, and the optional success measure. For example, a training instance may include visual data and object manipulation parameters (e.g., representations thereof) as training instance inputs, and include success measures as training instance outputs. As another example, a training instance may include visual data as a training instance input, object manipulation parameters as a training instance output, and a training instance that may be marked as positive or negative based on success measures. As yet another example, a training instance may include visual data as a training instance input, and include values ​​corresponding to object manipulation parameters and determined based on success measures as training instance outputs.

[0063] The training engine 145 trains the machine learning model 165 using corresponding training instances of the training data database 152. The trained machine learning model can then be at least selectively used by one of the prediction engines 112 to predict one or more corresponding object manipulation parameters, which are then at least selectively used to control the robot. In some implementations, after the training engine 145 determines that one or more conditions are met, the trained machine learning model is only utilized when predicting the object manipulation parameters that are at least selectively utilized. The one or more conditions may include, for example, at least a threshold amount of training and / or validation of the trained machine learning model as described herein. In some implementations, the training data engine 143 and the training engine 145 may implement the methods described in detail herein. Figure 4 One or more aspects of method 400.

[0064] Now turn to Figure 1B ,Should Figure 1B The various implementations described herein are shown Figure 1A An example of how the components of can interact with each other. Figure 1B In the example of FIG. 1 , visual data from the additional visual component 194 is provided to the prediction engine 112 and the visual representation engine 114. For example, the visual data may capture Figure 1A . The prediction engine 112 can generate predicted object manipulation parameters 113 based on processing the visual data using one or more machine learning models 165. The visual representation engine 114 generates a visual representation 115 that includes at least an object representation of the object, wherein the object representation is based on object features of the visual data. In some implementations, the visual representation 115 may also include an indication of the predicted object manipulation parameters 113 (e.g., when the corresponding confidence measurement indicates that confirmation is required). Additionally or alternatively, and as indicated by the dashed arrow, the predicted object manipulation parameters 113 can be directly provided to the manipulation parameter engine 116 without including an indication thereof in the visual representation 115 or requesting confirmation (e.g., when the corresponding confidence measurement indicates that confirmation is not required).

[0065] The visual representation 115 is transmitted to a display engine 132, which renders the visual representation, optionally along with other simulated representations (e.g., simulated robots and / or simulated workspaces). Input data 135 is generated by an input engine 134 in response to one or more user interface inputs provided at one or more input devices 136 and directed to the visual representation. The input data 135 directly or indirectly indicates one or more additional object manipulation parameters and / or confirmation of any predicted object manipulation parameters indicated in the visual representation 115.

[0066] The manipulation parameter engine 116 utilizes the input data and optionally any directly provided predicted object manipulation parameters 113 to generate object manipulation parameters 117 that are provided to the robot 170A for implementation. For example, the robot 170A may generate control commands based on the object manipulation parameters 117 and may implement them in response to determining that an object has entered the robot workspace of the robot 170A and / or is in a particular posture within the robot workspace. For example, the robot 170A may make such a determination based on robot vision data from the robot vision component 174A.

[0067] The training data engine 143 can generate training instances 144 based on the implemented manipulation parameters 117. Each training instance 144 can include a training instance input based on visual data from the additional vision component 194 and / or from the robot vision component 174. Each training instance 144 can also be based on a corresponding one of the manipulation parameters 117 (e.g., a training instance input or output can be based on a manipulation parameter). Each training instance 144 can also be based on a corresponding success measurement determined by the training data engine, based on visual data from the vision components 174A and / or 194, and / or based on data from other sensors 104. The training instances 144 are stored in the training data database 152 for use by the training engine 145 (FIG. 1) in training one or more machine learning models 165.

[0068] Now turn to Figure 2A , 2B , 2C, 2D, 2E and 2F, each of which shows that the remote client device 130 ( Figure 1A ) or other remote client device. Each visual representation includes an object representation of an object to be manipulated by the robot and illustrates an example of a user interface input that can be provided to define and / or confirm object manipulation parameters for manipulating the object by the robot.

[0069] Figure 2A Shown includes Figure 1A A visual representation of a simulation environment of a robot simulation 270A of one of the robots of FIG. 1 is also shown in the simulation environment. Figure 1A As described herein, the pose of the object representation 292A can be determined based on the visual data that captured the spatula 192C and used to generate the object representation 292A, optionally taking into account the transformation of the robot reference coordinate system. Figure 2A The visual representation of can be rendered, for example, via a VR head-mounted device.

[0070] An operator (e.g., via a VR controller) has provided user interface input to define a path 289A1 of the robot's end effector from a starting pose (not shown) to the illustrated grasping pose. The operator may, for example, actuate a first virtual button (e.g., virtual button 282A1) or hardware button to begin defining the path 289A1, and actuate a second virtual or hardware button to define the end of the path 289A1, which also constitutes the grasping pose. Although not shown, it is noted that the simulated robot 270A may "move" during the definition of the trajectory 289A1 to provide the operator with visual feedback of the path 289A1 as it will be implemented by the robot 270A.

[0071] Figure 2A Also shown is a virtual button 282A2 that can be selected by the operator to use a predefined path that was "saved" by the operator after being defined for a previous instance of user interface input. Selecting the virtual button 282A2 can paste the predefined path into the virtual environment along with the user's option to modify the predefined path to adapt it to a specific object. Figure 2A Also shown is a virtual button 282A3 that can be selected by an operator to define path 289A1 as a path that can be selected as a "predefined path" at a later time. Enabling an operator to save and reuse certain paths can reduce the number of user interface inputs required to redefine the path or a path that is a slight variation of a predefined path. Additionally, this can enable a path for the current instance to be defined more quickly, which can reduce idle time of the robot while waiting for definition of object manipulation parameters and / or can increase operator productivity.

[0072] Figure 2B Shown includes Figure 1A A visual representation of a simulated environment of a robot simulation 270B of one of the robots of FIG. Figure 1A An object representation 292B of a spatula 192C is shown. An operator (e.g., via a VR controller) has provided user interface input to define waypoints 289B1 and 289B2 (instead of a full path) and a grasping posture 289B3, where the waypoints are encountered when traversing to the grasping posture 289B3, and the grasping posture 289B3 is utilized when grasping the spatula 192C. The operator may, for example, actuate a first hardware button (e.g., of a VR controller) in a first manner for defining the waypoints 289B1 and 289B2, and may actuate the first hardware button (or actuate a second hardware button) in a second manner for defining the grasping posture 289B3. Although not shown, note that the simulated robot 270B may "move" during definition of the waypoints 289B1, 289B2, and / or the grasping posture 289B3 to provide visual feedback to the operator. Although in Figure 2BNot shown, but virtual buttons may also be provided for saving waypoints 289B1 and 289B2, and / or reusing (and possibly adapting) predefined waypoints.

[0073] Figure 2C Shown includes Figure 1A A visual representation of a simulation environment of a robot simulation 270C of one of the robots of FIG. 1 is also shown in the simulation environment. Figure 1A The object representation 292C of the spatula 192C. The operator (e.g., via the VR controller) has provided user interface input to define only the grasping posture 289C1. The operator may, for example, actuate a first hardware button (e.g., of the VR controller) to define the grasping posture 289C1. Although not shown, it is noted that the simulated robot 270C may "move" during the definition of the grasping posture 289C1 to provide visual feedback to the operator. In some implementations, a user interface input similar to Figure 2A and / or a visual representation of 2B until a machine learning model is trained that enables prediction of paths or waypoints that can be at least selectively implemented automatically (without confirmation), and can then provide a method similar to Figure 2C A visual representation of for defining only a grasping posture via a user interface input. Optionally, Figure 2C The predicted path and / or predicted waypoints may also be indicated visually, and prompts may be provided to confirm the predicted waypoints or path, or to redefine the predicted waypoints or path if not confirmed.

[0074] Figure 2D Shown include Figure 1A The object representation 292D of the spatula 192C is a visual representation, i.e., a 2D image (e.g., an RGB image) of the spatula. The visual representation can be rendered, for example, on a touch screen of a remote client device. An operator of the client device is prompted by indication 282D to swipe on the touch screen to define an antipodal grasp. In response, the operator touches the touch screen at 289D1 and swipes to 289D2, at which point the operator releases his / her touch. As a result, an antipodal grasp is defined with a first point at 289D1 and a second point at 289D2. Points 289D1 and 289D2 can be transformed from 2D points to 3D points using, for example, a mapping between a 2D image and corresponding 2.5D or 3D visual data.

[0075] Figure 2E Shown include Figure 1AThe object representation 292E of the spatula 192C is a visual representation, i.e., a 2D image (e.g., an RGB image) of the spatula. The visual representation also includes an indication 288E of the predicted antipodal grasp. The visual representation can be rendered, for example, on the screen of the remote client device. The operator of the client device is prompted by indication 282E1 to confirm the predicted antipodal grasp of indication 288E, or to define a replacement grasp (via indication 282E2). If the operator agrees with the predicted antipodal grasp of indication 288E, he / she can simply click / tap on indication 282E1. If the operator disagrees with the predicted antipodal grasp of indication 288E, he / she can click / tap on indication 282E2 and change indication 288E (e.g., drag it up / down, change the width, etc.) or define a new antipodal grasp from scratch.

[0076] Figure 2F Shown include Figure 1A 192C includes three connected bounding boxes (dashed lines) that approximate the surface of the spatula 192A. As described herein, the object representation 292F may be compared to Figure 2D and 2E The representation of the remote client device may be more data efficient and / or may prevent potentially sensitive data from being viewed by an operator of the client device. The visual representation may be rendered, for example, on a touch screen of the remote client device. The operator of the client device is prompted by indication 282DF to swipe on the touch screen to define an antiplantar grab. In response, the operator touches the touch screen at 289F1 and swipes to 289FD2, at which point the operator releases his / her touch. As a result, an antiplantar grab is defined with a first point at 289F1 and a second point at 289F2.

[0077] exist Figures 2A to 2F Various examples of visual representations and interactions with visual representations are shown in . However, it is to be understood that additional visual representations and / or interactions and / or replacement visual representations and / or interactions may be utilized in various implementations disclosed herein.

[0078] Now turn to Figure 3 , Figure 3 An example method 300 is shown for causing a robot to manipulate an object according to object manipulation parameters determined based on data generated at a remote client device in response to a visual representation of an object including an object representation. For convenience, some operations of the method 300 are described with reference to a system that performs the operations. The system may include various computer systems and / or various components of a robot, such as Figure 1A and 1B In addition, although the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0079] At box 352, the system receives visual data that captures object features of one or more objects from one or more vision components. In some implementations or iterations of method 300, the vision component is a robot vision component that observes the robot workspace of the corresponding robot, and the visual data captures the object features when the object is in the robot workspace. In some other implementations or iterations, the vision component is in a first area of ​​the environment that is different from the robot workspace of the environment, and the visual data captures the object features when the object is in the first area and before completing the transfer of the object to the robot workspace. In some of these implementations, one or more of boxes 354, 356, 358, 360, 362 and / or 364 can be completed before completing the transfer of the object to the robot workspace.

[0080] At optional box 354, the system generates one or more predicted object manipulation parameters based on the visual data and the prediction model. For example, the system can use the corresponding prediction model that has been trained and online to process the visual data and / or other sensor data to generate the predicted grasping posture, and optionally generate a predicted probability (e.g., successful object manipulation) for the predicted grasping posture. As another example, the system can additionally or alternatively use the corresponding prediction model that has been trained and online to process the visual data and / or other sensor data to generate a predicted classification of the object to be grasped, and optionally generate a predicted probability of the predicted classification (e.g., the probability that the predicted classification is correct). The predicted classification can be used to determine the predicted placement location of the object (e.g., in a specific container corresponding to the predicted classification in multiple available containers). The predicted probability of the predicted classification can optionally be used as the probability of the corresponding predicted placement location (e.g., a location with a defined relationship to the predicted classification).

[0081] At optional box 356, the system determines whether: (a) multiple object manipulation parameters other than the predicted manipulation parameters of box 354 are required in order to manipulate the object; and / or (b) one or more of the predicted object manipulation parameters need to be confirmed by a remote user interface input (e.g., because the corresponding predicted probability fails to meet a threshold).

[0082] If the determination at block 356 is “no,” the system proceeds directly to block 360 and causes the robot to manipulate the object according to the object manipulation parameters, which in this case correspond to the predicted object manipulation parameters of block 354 .

[0083] If the determination at block 356 is “yes,” the system proceeds to optional block 358 or block 360 .

[0084] Blocks 354 and 356 are shown as optional (as indicated by dashed lines) because they may not be used in method 300 in various implementations and / or because they may only be utilized in some iterations in other implementations. For example, in some other implementations, block 354 may be implemented only when at least one prediction model has been trained and brought online, which may be contingent on the satisfaction of one or more conditions as described herein.

[0085] At optional box 358, the system selects a remote client device from a plurality of client devices. The system can select a remote client device based on various considerations. For example, the system can select a remote client device in response to determining that the remote client device currently has no requests for object manipulation parameters in its queue. In addition, for example, the system can select a remote client device additionally or alternatively in response to determining that the number of pending requests for the remote client device and / or the expected duration of the pending requests are less than the number of pending requests and / or the expected duration of the pending requests of other candidate remote client devices (e.g., those that can be used in the environment of the robot utilized in the deployment method 300). As another example, the system can select a remote client device based on the proficiency measurement of the operator of the remote client device. The proficiency measurement can be based on past success measurements for manipulation, which are based on object manipulation parameters determined based on user interface input from the operator, and the proficiency measurement can be a global proficiency measurement or specific to one or more specific manipulations (e.g., a first proficiency measurement for grabbing and placing manipulations, a second proficiency measurement for grabbing two objects and linking them together, etc.).

[0086] At box 360, the system transmits a visual representation based on the visual data of box 352 to a remote client device (e.g., the one selected at box 358). The visual representation includes at least an object representation of the object features based on the visual data of box 352. In some implementations, the object representation includes less data than the object features of the visual data of box 352. For example, the object representation can define the bounding shapes of the corresponding areas of the respective approximate objects without defining colors and / or other values ​​for the individual pixels or voxels contained in the bounding shapes in the visual data. For example, the values ​​of 64 pixels or voxels in the visual data can be replaced by 7 values: 3, which define the x, y, z coordinates of the center of the bounding box; 2, which collectively define the orientation of the bounding box; and two, which define the width and height of the bounding box.

[0087] In some implementations, the visual representation transmitted at box 360 lacks any representation of the robot and / or lacks any representation of one or more static objects and / or other objects in the robot's robot workspace. In some of these implementations, the client device renders the transmitted visual representation together with a simulation of the robot and / or a simulation of all or part of the robot workspace. For example, the remote client device can execute a robot simulator that simulates the robot and the robot workspace, and can render object representations within the robot simulator and together with the simulated robot and the robot workspace. Note that this can save network resources by eliminating the need to transmit a representation of the robot and / or the robot workspace with each transmission of the visual representation to the remote client device. It should also be noted that even when the visual data of box 352 is captured in a first area different from the robot workspace, the simulated robot and / or the simulated robot workspace can be rendered and the object representations can be appropriately rendered using it.

[0088] Optionally, box 360 includes sub-box 360A, in which the system generates a visual representation based on the visual data and based on the predicted manipulation parameters of box 354 (if any). For example, if a predicted grasping pose is generated at box 354, an indication of the predicted grasping pose may optionally be included in the visual representation. For example, the indication of the predicted grasping pose may be a representation of the robot end effector rendered in the predicted grasping pose along with the object representation. An operator of the remote client device may confirm the predicted grasping pose, or suggest an alternative grasping pose (e.g., by adjusting the representation of the robot end effector). As another example, if a predicted series of waypoints are generated at box 354, indications of the waypoints may optionally be included in the visual representation. For example, the indication of the waypoints may be circles or other markings of the waypoints rendered with the object representation and / or robot simulation.

[0089] At block 362, the system receives from the remote client device data generated based on user interface inputs directed to the visual representation transmitted at block 360. The user interface inputs may include those that (directly or indirectly) define object manipulation parameters and / or those that confirm predicted object manipulation parameters.

[0090] At block 364, the system determines object manipulation parameters for manipulating the object by the robot based on the data received at block 362. The object manipulation parameters may include those based on the predicted object manipulation parameters indicated in the visual representation (if any), if the data indicates confirmation of those predicted object manipulation parameters. The object manipulation parameters may additionally or alternatively include those defined based on user interface inputs independently of any predicted object manipulation parameters.

[0091] In some implementations, the data generated at the remote client device directly defines the object manipulation parameters and can be used directly as the object manipulation parameters. In some other implementations, the data indirectly defines the object manipulation parameters and is further processed when determining the object manipulation parameters. As a non-limiting example, box 364 can optionally include a sub-box 364A in which the system transforms the pose and / or point to the robot coordinate system of the robot. For example, the pose, point (e.g., waypoint) and / or other features defined by the data received at box 362 can be defined relative to a given coordinate system different from the robot coordinate system and then transformed to the robot coordinate system. For example, the given coordinate system can be a first coordinate system of the visual component of box 352 that is different from the robot visual component of the robot.

[0092] In box 360, the system causes the robot to manipulate the object according to the object manipulation parameters. The object manipulation parameters may include those based on the predicted object manipulation parameters and / or those defined based on the user interface input independently of any predicted object manipulation parameters. In some implementations, the system provides the robot with the object manipulation parameters and / or high-level commands based on the object manipulation parameters. In those implementations, the control system of the robot transforms the object manipulation parameters and / or high-level commands into corresponding low-level actions, such as control commands issued to the actuators of the robot. For example, the robot may include a controller that converts the high-level commands into more specific control commands to be provided to one or more actuators of the robot. The control commands may include one or more speed control commands issued to the actuators of the robot at the corresponding instances to control the movement of the robot. For example, when controlling the movement of the robot, a speed control command may be issued to each actuator that controls the movement of the end effector of the robot. In other implementations, the object manipulation parameters may define the low-level actions themselves (e.g., when defining a complete trajectory via a user interface input), and / or the low-level actions may be generated based on the object manipulation parameters, and the low-level actions are provided to the robot for control based on the low-level actions.

[0093] In implementations where the vision component is in a first area of ​​the environment that is different from a robot workspace of the environment, block 360 may include causing the robot to manipulate the object in further response to determining that the object is within the robot workspace. In some of those implementations, the robot may determine that the object is within the robot workspace based on robot vision data from the vision component of the robot. In some additional or alternative implementations, the object may be determined to be within the workspace based on data from a conveying device of the object indicating that the object is within the workspace. For example, when the conveying device includes a conveyor system, the arrival time of the object within the robot workspace may be determined based on operational data of the conveyor system.

[0094] After block 360, the system then returns to block 352. Note that in various implementations, multiple iterations of method 300 can be run in parallel for a given environment, thereby enabling visual representations of new objects to be generated, transmitted, corresponding data to be received, and / or corresponding object manipulation parameters to be determined before method 300 is completed for previous objects (e.g., at least before block 360 is completed). For example, multiple iterations of method 300 can be run in parallel, with each iteration being for a different robot of the environment. Further, for example, multiple iterations of method 300 can be run in parallel for a given robot, thereby enabling object manipulation parameters to be determined for each of multiple different objects before they arrive in the robot workspace of the given robot and are manipulated by the given robot.

[0095] Now turn to Figure 4 , Figure 4 An example method 400 for generating training examples based on robot object manipulation attempts and using the training examples in training a prediction model is shown. For convenience, some operations of the method 400 are described with reference to a system that performs these operations. The system may include various computer systems and / or various components of a robot, such as Figure 1A and 1B In addition, although the operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0096] At block 2, the system identifies: (1) object manipulation parameters utilized by the robot in an object manipulation attempt; and (2) visual data associated with the object manipulation attempt. For example, the object manipulation parameters may include a grasping pose and a placing pose defined based on a user interface input directed to a visual representation generated based on visual data from a first area, and the visual data may be robot visual data from a robot workspace different from the first area.

[0097] At optional box 4, the system generates a success measure for the object manipulation attempt based on sensor data from the sensor. In some implementations, the system generates a single success measure for the entire object manipulation attempt. For example, for a pick and place manipulation, the system can determine a single success measure based on whether the object is successfully placed and / or the accuracy of the placement. In some other implementations, the system generates multiple success measures for the object manipulation attempt, wherein each success measure corresponds to a corresponding subset of the object manipulation parameters. For example, for a pick and place manipulation, the system can determine a first success measure for the pick manipulation (e.g., based on whether the object is successfully grasped) and a second success measure for the placement manipulation (e.g., based on whether the object is successfully placed and / or the accuracy of the placement). The sensors that the success measure can be based on can include, for example, position sensors of the robot, torque sensors of the robot, robot vision data from a vision component of the robot, weight sensors in the environment, and / or other robot and / or environmental sensors.

[0098] At block 6, the system generates training instances based on the object manipulation parameters, visual data, and optionally success measurements. As indicated by the arrow from block 6 to block 2, the system may continue to perform iterations of blocks 2, 4, and 6 to generate additional training instances based on additional object manipulation attempts.

[0099] As an example of box 6, assume that a pick and place manipulation has a grasping posture and a placing posture manipulation parameter. A first training instance can be generated based on visual data and grasping postures and based on a success measure (e.g., a success measure for grasping or an overall success measure for picking and placing). For example, the first training instance can be used for a grasp prediction model that approximates the value function, and is used to process the visual data and the grasping posture, and use the grasping posture and in view of the visual data to predict the success probability of grasping an object. In such an example, the training instance input will include the visual data and the grasping posture (e.g., a representation of the x, y, and z position and orientation), and the training instance output will include a success measure (e.g., "0" if the success measure indicates a failed grasp, and "1" if the success measure indicates a successful grasp). In addition, for example, the first training instance can be used instead to process the visual data (also without processing the grasping posture) and generate a prediction model for the corresponding probability for each of the N grasping postures. In such an instance, the training instance input would include visual data, and the training instance output would include an output value corresponding to the grasping posture that is "1" if the success measure indicates a successful grasp, and optionally all other values ​​are "0". A second training instance can be generated based on the visual data and the placement posture and based on the success measure (e.g., a success measure for the grasp or an overall success measure for the pick and place). For example, the second training instance can be used for a placement prediction model that approximates the value function and is used to process the visual data and the placement posture and predict the probability of successful placement of the object when the grasping posture is used given the visual data. In such an instance, the training instance input would include visual data and the placement posture (e.g., a representation of the x, y, and z position and orientation), and the training instance output would include a success measure (e.g., "0" if the success measure indicates an unsuccessful placement, "1" if the success measure indicates a successful placement, "0.7" if the success measure indicates a successful but not completely accurate placement, etc.).

[0100] As another example of block 456, assume that a maneuver has maneuver parameters that include a waypoint sequence defined based on a user interface input. A training instance may be generated based on the visual data and the waypoint sequence. For example, the training instance may be for a waypoint prediction model that approximates a value function and is used to process the visual data and the waypoint sequence and predict the probability of the waypoint sequence given the visual data (e.g., the probability that the waypoint sequence is appropriate). In such an instance, the training instance input would include a representation of the visual data and the waypoint sequence (e.g., an embedding of the sequence generated using a recurrent neural network model or a transformer network), and the training instance output would include "1" (or other "positive" value) based on the sequence that has been defined based on the user interface input.

[0101] In block 458, the system uses the generated training instances to update the parameters of the prediction model. In the case where different training instances for different prediction models are generated in block 456, appropriate training instances for the corresponding prediction models may be utilized in each iteration of block 458. For example, some iterations of block 458 may use training instances of a first type to train a first prediction model, other iterations may use training instances of a second type to train a second prediction model, and so on. Furthermore, multiple iterations of blocks 458, 460, and 462 may optionally be operated in parallel, with each iteration dedicated to training a corresponding prediction model.

[0102] At block 460, the system determines whether more training is needed. In some implementations, this can be based on whether a threshold amount of training has occurred, whether a threshold duration of training has occurred, and / or whether one or more performance characteristics of the prediction model have been observed (e.g., high probability predictions and / or successful maneuvers in at least X% of maneuvers when the prediction model is used). In some implementations, training of the prediction model can continue indefinitely, at least on a periodic basis.

[0103] If the determination of block 460 is "yes", the system waits for the availability of another training instance at block 462 and returns to block 458 based on the available training instance. If the determination of block 460 is "yes", the system proceeds to block 464 and ends the training of the prediction model (although the training of other prediction models can continue). The trained prediction model can be utilized in method 300 or method 500, and can optionally continue to be trained during utilization.

[0104] Now turn to Figure 5 , Figure 5 An example method 500 is shown for selectively utilizing a trained prediction model to determine object manipulation parameters for use by a robot in manipulating an object. Method 500 illustrates some implementations of method 300. For convenience, some operations of method 500 are described with reference to a system that performs these operations. The system may include various computer systems and / or various components of a robot, such as Figure 1A and 1B In addition, although the operations of method 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0105] In block 552, the system receives visual data capturing object features of one or more objects from one or more vision components. In some implementations or iterations of method 500, the vision component is a robot vision component that observes a robot workspace of the corresponding robot, and the visual data captures the object features when the object is in the robot workspace. In some other implementations or iterations, the vision component is in a first area of ​​the environment that is different from the robot workspace of the environment, and the visual data captures the object features when the object is in the first area and before completing the transfer of the object to the robot workspace. In some of these implementations, one or more blocks prior to block 572 may be completed before completing the transfer of the object to the robot workspace.

[0106] At block 554 , the system selects one or more object manipulation parameters of a plurality of object manipulation parameters that need to be solved for manipulation of the object by the robot.

[0107] In box 556, the system determines whether the trained model for the object manipulation parameters has been online as described herein for the selected object manipulation parameters. If not, the system proceeds to box 558, and prompts the user interface input at the remote client device to specify the object manipulation parameters. For example, the system can generate a visual representation based on the visual data of box 552, transmit the visual representation to the client device, and render prompts at the client device based on box 558, for defining the object manipulation parameters via the user interface input pointing to the visual representation. Then, in box 570, any object manipulation parameters defined by the user interface input received in response to the prompt at box 558 can be used as the selected object manipulation parameters.

[0108] If, at block 556 , the system determines that a trained model for the object manipulation parameter is already online for the selected object manipulation parameter, the system proceeds to block 560 .

[0109] At block 560, the system generates predicted object manipulation parameters and corresponding confidence measures based on the visual data and the prediction model of block 552. For example, the system can select a prediction model corresponding to the object manipulation parameters and use the prediction model to process the visual data and / or other data to generate the predicted object manipulation parameters and corresponding confidence measures.

[0110] The system then proceeds to box 562 and determines whether the confidence measurement for the predicted object manipulation parameters meets one or more thresholds (e.g., 90% or other thresholds). If not, the system proceeds to box 564 and prompts to confirm the predicted object manipulation parameters at the remote client device and / or prompts to specify the corresponding object manipulation parameters by the user interface input at the remote client device. For example, the system can generate a visual representation including an indication of one or more of the predicted object manipulation parameters, transmit the visual representation to the client device, and based on box 564, render the prompt at the client device. The prompt can request the client device operator to confirm the predicted object manipulation parameters via user interface input, or define corresponding alternative object manipulation parameters via user interface input. In addition, for example, the system can additionally or alternatively prompt one or more of the object manipulation parameters to be specified by the user interface input at the remote client device without presenting the option of confirming the predicted object manipulation parameters. In some implementations, if the confidence measure for a given predicted object manipulation parameter does not satisfy the threshold value but satisfies an additional lower threshold value (e.g., 65% or other threshold value) in box 562, the system can cause a prompt for confirmation of the given predicted object manipulation parameter. In those implementations, if the confidence measure for a given predicted object manipulation parameter does not satisfy the additional lower threshold value, the system can prompt for definition of the corresponding object manipulation parameter, optionally without providing any indication of the given predicted object manipulation parameter. Then, in box 570, any object manipulation parameter defined by the user interface input received in response to box 564 can be used as all or part of the selected object manipulation parameter.

[0111] If the system determines at block 562 that the confidence measure satisfies the threshold, the system proceeds to block 566 and uses the predicted object manipulation parameters without prompting for confirmation of the predicted object manipulation parameters.

[0112] At box 568, the system then determines whether there are multiple object manipulation parameters that need to be solved for manipulating the object by the robot. If yes, the system returns to box 554 and selects additional object manipulation parameters. If no, the system proceeds to box 572. It should be noted that in the instance of method 500 where the determination of box 556 or box 562 is "no" for more than one iteration of box 556 or box 562, the prompt at the client device can be a single prompt for all object manipulation parameter definitions and / or confirmations for the object manipulation parameters for which a "no" determination was made at box 556 or box 562. In other words, for each of the N iterations, there will not necessarily be N separate prompts. Instead, there can optionally be a single prompt containing the request for each of the N iterations.

[0113] At block 572, the system causes the robot to manipulate the object according to the object manipulation parameters. The object manipulation parameters may include those object manipulation parameters from one or more iterations of block 566 and / or from one or more iterations of block 570. For example, the object manipulation parameters may include those based on predicted object manipulation parameters (with or without confirmation) and / or those defined based on user interface inputs independently of any predicted object manipulation parameters.

[0114] The system then returns to block 552. Note that in various implementations, multiple iterations of method 500 can be run in parallel for a given environment, thereby enabling visual representations of new objects to be generated, transmitted, corresponding data to be received, and / or corresponding object manipulation parameters to be determined before method 500 is completed for previous objects (e.g., at least before completing block 572).

[0115] Now turn to Figure 6 , Figure 6 An example method 600 for training a prediction model, validating a prediction model, deploying a prediction model, and optionally further training the deployed prediction model is shown. For convenience, some operations of method 600 are described with reference to a system that performs these operations. The system may include various components of various computer systems and / or robots, such as Figure 1A and 1B In addition, although the operations of method 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0116] At block 652 , the system trains a predictive model based on data from an operator-directed object manipulation attempt. For example, the system may train a predictive model based on training examples generated in blocks 452 , 454 , and 456 of method 400 .

[0117] At block 654, the system determines whether one or more conditions have been met. If not, the system returns to block 652. If yes, the system proceeds to block 656. The conditions considered at block 654 may include, for example, a threshold number of training instances that have been used in training at block 652 and / or a threshold duration of training at block 652.

[0118] At block 656, the system attempts to validate the prediction model based on comparing the predictions generated using the prediction model with the operator-guided ground truth. For example, the system may compare the predicted object manipulation parameters made using the model with the corresponding object manipulation parameters defined based on the user interface input (i.e., the operator-guided ground truth). The system may determine an error measure for the prediction based on the comparison. The operator-guided ground truth may optionally be validated based on the determined success measure. In other words, the operator-guided ground truth may be considered to be ground truth only if the corresponding success measure indicates the overall success of the corresponding manipulation and / or the success of the portion of the manipulation corresponding to the defined object manipulation parameters.

[0119] At block 658, the system determines whether the validation was successful. If not, the system returns to block 652 and optionally adjusts the conditions of block 654 (e.g., to require a greater degree of training). In determining whether the validation was successful, various metrics may be utilized. For example, the system may determine, based on the comparison of block 656, that a successful validation is when at least a threshold percentage of the predictions are less than a threshold error measure.

[0120] If the determination at block 658 is that the validation is successful, the system proceeds to block 660. At block 660, the system deploys the prediction model for use in the generated suggested and / or automatically implemented predictions. For example, the prediction model can be deployed for use in method 300 and / or method 500.

[0121] At optional block 662 , the system also trains the predictive model based on operator feedback on recommendations during deployment and / or based on sensor-based success measurements during deployment.

[0122] Figure 7 An example architecture of a robot 725 is schematically depicted. The robot 725 includes a robot control system 760, one or more operating components 740a-740n, and one or more sensors 742a-742m. The sensors 742a-742m may include, for example, visual components, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, and barometers, etc. Although the sensors 742a-742m are depicted as being integrated with the robot 725, this is not meant to be limiting. In some implementations, the sensors 742a-742m may be located external to the robot 725, for example, as a separate unit.

[0123] The operating components 740a-740n may include, for example, one or more end effectors and / or one or more servo motors or other actuators to achieve movement of one or more components of the robot. For example, the robot 725 may have multiple degrees of freedom, and each actuator may control the actuation of the robot 725 in one or more degrees of freedom in response to a control command. As used herein, the term actuator also includes a mechanical or electrical device (e.g., a motor) that establishes motion, in addition to any driver that may be associated with the actuator and converts a received control command into one or more signals for driving the actuator. Therefore, providing a control command to an actuator may include providing a control command to a driver that converts the control command into an appropriate signal to drive the electrical or mechanical device to establish the desired motion.

[0124] The robotic control system 760 may be implemented in one or more processors such as a CPU, GPU, and / or other controller of the robot 725. In some implementations, the robot 725 may include a "brain box," which may include all or aspects of the control system 760. For example, the brain box may provide real-time bursts of data to the operating components 740a-740n, where each real-time burst includes a set of one or more control commands that, among other things, indicate motion parameters (if any) for each of the one or more operating components 740a-740n. In some implementations, the robotic control system 760 may perform one or more aspects of one or more methods described herein.

[0125] As described herein, in some implementations, all or aspects of the control commands generated by control system 760 can be generated based on object operating parameters generated according to the techniques described herein. Figure 7 760 is shown as an integral part of the robot 725, but in some implementations, all or aspects of the control system 760 may be implemented in a component separate from but in communication with the robot 725. For example, all or aspects of the control system 760 may be implemented on one or more computing devices (e.g., computing device 810) that communicate with the robot 725 in wired and / or wireless communication.

[0126] Figure 8is a block diagram of an example computing device 810 that may optionally be used to perform one or more aspects of the techniques described herein. For example, in some implementations, the computing device 810 may be used to execute the simulator 120, the simulated difference engine 130, the real fragment system 110, the simulated training data system 140, and / or the training engine 145. The computing device 810 generally includes at least one processor 814 that communicates with a plurality of peripheral devices via a bus subsystem 812. These peripheral devices may include: a storage subsystem 824, including, for example, a memory subsystem 825 and a file storage subsystem 826; a user interface output device 820; a user interface input device 822; and a network interface subsystem 816. The input and output devices allow a user to interact with the computing device 810. The network interface subsystem 816 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0127] User interface input devices 822 may include a keyboard; a pointing device such as a mouse, trackball, touchpad, or graphics tablet; a scanner; a touch screen incorporated into a display; an audio input device such as a voice recognition system; a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of entering information into the computing device 810 or a communication network.

[0128] The user interface output device 820 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for generating a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from the computing device 810 to a user or another machine or computing device.

[0129] The storage subsystem 824 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 824 may include logic to perform selected aspects of one or more methods described herein.

[0130] These software modules are typically executed by the processor 814 alone or in combination with other processors. The memory 825 used in the storage subsystem 824 may include multiple memories, including a main random access memory (RAM) 830 for storing instructions and data during program execution and a read-only memory (ROM) 832 in which fixed instructions are stored. The file storage subsystem 826 can provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored in the storage subsystem 824 by the file storage subsystem 826, or in other machines accessible to the processor 814.

[0131] The bus subsystem 812 provides a mechanism for the various components and subsystems of the computing device 810 to communicate with each other as intended. Although the bus subsystem 812 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0132] The computing device 810 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 8 The description of the computing device 810 depicted in FIG. 8 is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of the computing device 810 may have more Figure 8 The computing device depicted may have more or fewer components.

[0133] In some implementations, a method is provided that includes receiving visual data from one or more visual components in a first area of ​​an environment that captures features of the first area at a first time. The captured features include object features of an object located in the first area at the first time. The method also includes: transmitting a visual representation generated based on the visual data to a remote client device via one or more networks before completing the transfer of the object from the first area to a different robot workspace of the environment that is not captured by the visual data; and receiving data generated based on one or more user interface inputs from the remote client device via the one or more networks. The visual representation includes an object representation generated based on the object features. The user interface input is at the remote client device and points to the visual representation when rendered at the remote client device. The method also includes determining one or more object manipulation parameters for manipulating the object by a robot operating in the robot workspace based on the data. The method also includes causing the robot to manipulate the object according to the one or more object manipulation parameters in response to detecting that the object is within the robot workspace. At a second time after the object is transferred from the first area to the robot workspace and after the first time, the object is within the robot workspace.

[0134] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0135] In some implementations, determining the one or more object manipulation parameters also occurs prior to completing the transfer of the object from the first area to the robotic workspace.

[0136] In some implementations, the one or more object manipulation parameters include a grasping pose for grasping the object. In those implementations, causing the robot to manipulate the object according to the one or more object manipulation parameters in response to detecting the object within the robot workspace after the transmitting includes causing an end effector of the robot to traverse to the grasping pose and attempting to grasp the object after traversing to the grasping pose.

[0137] In some implementations, the data defines one or more poses and / or one or more points relative to a first reference coordinate system. In some of those implementations, generating the one or more object manipulation parameters includes transforming the one or more poses and / or the one or more points to a robot coordinate system different from the reference coordinate system; and using the transformed poses and / or points in generating the object manipulation parameters.

[0138] In some implementations, the method further includes: after causing the robot to manipulate the object, determining a measure of success for the manipulation based on additional sensor data from one or more additional sensors; generating a positive training instance based on the success measure satisfying a threshold; and training a machine learning model based on the positive training instance. In some versions of those implementations, the one or more additional sensors include: a robot vision component, a torque sensor of the robot, and a weight sensor in the environment. In some additional or alternative versions of those implementations, generating the positive training instance includes: generating a training instance input for the positive training instance based on the visual data or based on robot vision data from one or more robot vision components of the robot; and / or generating a training instance output for the positive training instance based on the object manipulation parameters. In some of those additional or alternative versions, the method further includes: after training the machine learning model based on the positive training instance, using the machine learning model to process additional visual data capturing an additional object; generating one or more predicted object manipulation parameters for the additional object based on the processing; and causing the robot to manipulate the additional object according to the one or more predicted object manipulation parameters. Additionally, the method may also optionally include: transmitting a visual indication of the predicted object manipulation parameters to the remote client device or an additional remote client device; receiving an indication from the remote client device or the additional remote client device that an affirmative user interface input has been received in response to the visual indication of the predicted object manipulation parameters. Causing the robot to manipulate the additional object according to the one or more predicted object manipulation parameters may be in response to receiving an indication that the affirmative user interface input has been received. Optionally, the method also includes generating a confidence measure for the one or more predicted object manipulation parameters based on the processing. Transmitting the visual indication of the predicted object manipulation parameters may be in response to the confidence measure failing to meet a threshold confidence measure. Additionally or alternatively, the method may also optionally include: after training the machine learning model based on the positive training instances, using the machine learning model to process additional visual data capturing an additional object; generating one or more predicted object manipulation parameters for the additional object based on the processing; transmitting a visual indication of the predicted object manipulation parameters to the remote client device or an additional remote client device; receiving from the remote client device or the additional remote client device an indication of alternative object manipulation parameters defined via user interface input received in response to presentation of the visual indication of the predicted object manipulation parameters; and in response to receiving the alternative object manipulation parameters, causing the robot to manipulate the additional object according to the one or more alternative object manipulation parameters.The method may also optionally include further training the machine learning model using training examples having labeled outputs based on the surrogate object manipulation parameters.

[0139] In some implementations, the method further includes: receiving, before the robot manipulates the object, visual data capturing features of the first area from one or more visual components in the first area at a third time, the third time being after the first time but before the second time, wherein the visual data includes new object features of a new object located in the first area at the third time; transmitting to the remote client device a new visual representation generated based on the new visual data, the new visual representation including a new object representation generated based on the new object features; receiving from the remote client device new data generated at the remote client device based on one or more new user interface inputs, the one or more new user interface inputs pointing to the new visual representation when rendered at the remote client device; and determining, based on the data, one or more new object manipulation parameters for manipulating the new object by a robot operating in the robot workspace. In some of those implementations, the method further includes: after the robot manipulates the object, in response to the robot detecting, via the one or more robot visual components, that the new object is within the robot workspace, causing the robot to manipulate the new object according to the one or more new object manipulation parameters. At a fourth time after transmitting the new object and after the second time, the new object is within the robot workspace.

[0140] In some implementations, transferring the objects from the first area to the robotic workspace is via one or more conveyors.

[0141] In some implementations, the method further comprises, for each of a plurality of remote client devices, accessing corresponding queue data, the corresponding queue data defining a number and / or duration of outstanding robotic manipulation assistant requests. In some of those implementations, the method further comprises selecting a remote client device from the plurality of remote client devices based on corresponding query data for the remote client device. Transmitting the visual representation to the remote client device may be responsive to selecting the remote client device.

[0142] In some implementations, the object representation is a rendering of the object, wherein the rendering is generated based on the object features and omits one or more features of the object that are visible in the visual data.

[0143] In some implementations, the object is detected by the robot in the robot workspace based on robot vision data from one or more robot vision components of the robot.

[0144] In some implementations, a method is provided that includes receiving visual data capturing features of an environment from one or more visual components in an environment, the features of the environment including object features of an object located in the environment. The method also includes generating, based on processing the visual data using a machine learning model: predicted object manipulation parameters for the object, and a confidence measure for the predicted object manipulation parameters. The method also includes determining whether the confidence measure for the predicted object manipulation parameters satisfies a threshold confidence measure. The method also includes: in response to determining that the confidence measure fails to satisfy the threshold confidence measure, transmitting to a remote client device via one or more networks: (1) an object representation of the object generated based on the object features and (2) a visual indication of the predicted object manipulation parameters; and receiving data generated based on one or more user interface inputs from the remote client device via the one or more networks. The user interface inputs are at the remote client device and in response to rendering the object representation and the visual indication at the remote client device. The method also includes determining, based on the data, to utilize the object manipulation parameters or substitute object manipulation parameters. The method also includes causing a robot to manipulate the object according to the determined object manipulation parameters or the substitute object manipulation parameters. The method also includes, in response to determining that the confidence measure satisfies the threshold confidence measure, causing the robot to manipulate the object according to the object manipulation parameters without transmitting the visual indication to any remote client device for confirmation prior to manipulating the object according to the object manipulation parameters.

[0145] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0146] In some implementations, the vision component is in a first area of ​​the environment, and determining whether a confidence measure for the predicted object manipulation parameter satisfies the threshold confidence measure occurs prior to transferring the object to a different robot workspace of the robot.

Claims

1. A method comprising: receiving, from one or more visual components in a first area of ​​an environment, visual data capturing characteristics of the first area at a first time, the characteristics of the first area including object characteristics of objects located in the first area at the first time; Prior to completing the transfer of the object from the first area to a different robotic workspace of the environment not captured by the visual data: transmitting, via one or more networks, to a remote client device a visual representation generated based on the visual data capturing the features of the first area at the first time, the visual representation comprising an object representation generated based on the object features, Wherein, the object representation includes a boundary shape defined by: two or more coordinates defining the center of the bounding shape, one or more orientation values ​​defining the orientation of the bounding shape, and at least one dimension value defining a dimension of the bounding shape; wherein the visual representation omits one or more other objects visible in the visual data of the first area; and receiving, from the remote client device via the one or more networks, data generated based on one or more user interface inputs at the remote client device, the one or more user interface inputs directed to the visual representation as the visual representation is rendered at the remote client device, generated based on the visual data capturing the feature of the first area at the first time; determining one or more object manipulation parameters for manipulating the object by a robot operating in the robot workspace based on the data; and causing the robot to manipulate the object according to the one or more object manipulation parameters, wherein, in response to detecting, using one or more robot sensors, that the object is within the robot workspace, the robot manipulates the object according to the one or more object manipulation parameters, wherein the one or more robotic sensors are in addition to the vision component in the first area of ​​the environment, and Wherein, after transferring the object from the first area to the robot workspace and at a second time after the first time, the object is within the robot workspace.

2. The method according to claim 1, wherein: Determining the one or more object manipulation parameters also occurs prior to completing the transfer of the object from the first area to the robot workspace.

3. The method of claim 1 , wherein the one or more object manipulation parameters include a grasping pose for grasping the object, and wherein causing the robot to manipulate the object according to the one or more object manipulation parameters comprises: The end effector of the robot is caused to traverse to the grasping pose, and after traversing to the grasping pose, an attempt is made to grasp the object.

4. The method according to claim 1, wherein: The data defines one or more poses and / or one or more points relative to a first reference coordinate system, and wherein generating the one or more object manipulation parameters comprises: transforming the one or more poses and / or the one or more points to a robot coordinate system different from the reference coordinate system; and The transformed poses and / or points are used in generating the object manipulation parameters.

5. The method according to claim 1, further comprising: After causing the robot to manipulate the object: determining a measure of success of the maneuver based on additional sensor data from one or more additional sensors; generating a positive training instance based on the success measure satisfying a threshold; as well as A machine learning model is trained based on the positive training examples.

6. The method according to claim 5, wherein: The one or more additional sensors include one or more of: the one or more sensors and a weight sensor in the environment.

7. The method according to claim 5, wherein: Generating the positive training instance includes generating a training instance input for the positive training instance based on the vision data or based on robot vision data from one or more robot vision components of the robot.

8. The method according to claim 7, wherein: Generating the positive training instance includes generating a training instance output for the positive training instance based on the object manipulation parameters.

9. The method according to claim 8, further comprising: After training the machine learning model based on the positive training instance, the method further includes: using the machine learning model to process additional visual data capturing additional objects; generating one or more predicted object manipulation parameters for the additional object based on the processing; and The robot is caused to manipulate the additional object according to the one or more predicted object manipulation parameters.

10. The method according to claim 9, further comprising: transmitting a visual indication of the predicted object manipulation parameter to the remote client device or an additional remote client device; receiving, from the remote client device or the additional remote client device, an indication of receiving an affirmative user interface input in response to presentation of the visual indication of the predicted object manipulation parameter; Wherein causing the robot to manipulate the additional object according to the one or more predicted object manipulation parameters is in response to receiving an indication that an affirmative user interface input was received.

11. The method according to claim 10, further comprising: generating a confidence measure for the one or more predicted object manipulation parameters based on the processing; Wherein transmitting the visual indication of the predicted object manipulation parameter is in response to the confidence measure failing to satisfy a threshold confidence measure.

12. The method according to claim 8, further comprising: After training the machine learning model based on the positive training instance, the method further includes: using the machine learning model to process additional visual data capturing additional objects; generating one or more predicted object manipulation parameters for the additional object based on the processing; transmitting a visual indication of the predicted object manipulation parameter to the remote client device or an additional remote client device; receiving, from the remote client device or the additional remote client device, an indication of an alternate object manipulation parameter defined via a user interface input received in response to presentation of the visual indication of the predicted object manipulation parameter; and In response to receiving the substitute object manipulation parameters, the robot is caused to manipulate the additional object according to the one or more substitute object manipulation parameters.

13. A method comprising: receiving visual data capturing characteristics of the environment from one or more visual components in the environment, the characteristics of the environment including object characteristics of objects located in the environment; Based on processing the visual data using a machine learning model to generate: predicted object manipulation parameters for the object, the predicted object manipulation parameters being predicted grasp poses for grasping the object or waypoints to be encountered when traversing the robot to interact with the object, and a confidence measure for the predicted object manipulation parameters; determining whether the confidence measure for the predicted object manipulation parameter satisfies a threshold confidence measure; In response to determining that the confidence measure fails to satisfy the threshold confidence measure: transmitting, via one or more networks to a remote client device, an object representation of the object generated based on the object features and a visual indication of the predicted object manipulation parameters, wherein transmitting the visual indication of the predicted object manipulation parameter to the remote client device is further responsive to determining that the confidence measure satisfies a lower threshold confidence measure, the lower threshold confidence measure being a lesser indication of confidence than the threshold confidence measure; receiving, from the remote client device via the one or more networks, data generated at the remote client device based on one or more user interface inputs in response to rendering the object representation and the visual indication at the remote client device, determining to utilize the object manipulation parameters or to substitute object manipulation parameters based on the data, and causing the robot to manipulate the object according to the determined object manipulation parameters or the substitute object manipulation parameters; and In response to determining that the confidence measure satisfies the threshold confidence measure: The robot is caused to manipulate the object according to the object manipulation parameters without transmitting the visual indication to any remote client device for confirmation before manipulating the object according to the object manipulation parameters.

14. The method according to claim 13, wherein: The vision component is in a first area of ​​the environment, and wherein determining whether the confidence measure for the predicted object manipulation parameter satisfies the threshold confidence measure occurs prior to transferring the object to a different robot workspace of the robot.

15. The method according to claim 13, further comprising: In response to determining that the confidence measure fails to satisfy the lower threshold confidence measure: The object representation of the object is transmitted to the remote client device without transmitting any visual indication of the predicted object manipulation parameters.

16. A method comprising: receiving visual data capturing characteristics of the environment from one or more visual components in the environment, the characteristics of the environment including object characteristics of objects located in the environment; Based on processing the visual data using a machine learning model to generate: predicted object manipulation parameters for the object, and a confidence measure for the predicted object manipulation parameters; determining whether the confidence measure for the predicted object manipulation parameter satisfies a threshold confidence measure; Generating a visual representation for transmission to a remote client device via one or more networks, generating the visual representation comprises: including in the visual representation an object representation of the object generated based on the object features; and determining whether to include a visual indication of the predicted object manipulation parameter in the visual representation based on whether the confidence measure satisfies a lower threshold confidence measure; transmitting the visual representation to the remote client device; receiving from the remote client device via the one or more networks data generated at the remote client device based on one or more user interface inputs in response to rendering the visual representation at the remote client device; and The robot is caused to manipulate the object based on the received data.

17. A method comprising: receiving visual data capturing characteristics of the environment from one or more visual components in the environment, the characteristics of the environment including object characteristics of objects located in the environment; Based on processing the visual data using a machine learning model to generate: the predicted parameters for controlling the robot in the environment, and confidence measures for the predicted parameters; determining whether the confidence measure for the predicted parameter satisfies a threshold confidence measure; In response to determining that the confidence measure fails to satisfy the threshold confidence measure: transmitting, via one or more networks to a remote client device, an object representation of the object generated based on the object features and a visual indication of the predicted parameters, wherein transmitting the visual indication of the predicted parameter to the remote client device is further responsive to determining that the confidence measure satisfies a lower threshold confidence measure, the lower threshold confidence measure being a lesser indication of confidence than the threshold confidence measure; receiving, from the remote client device via the one or more networks, data generated at the remote client device based on one or more user interface inputs in response to rendering the object representation and the visual indication at the remote client device; and causing the robot to be controlled in the environment based on the received data; and In response to determining that the confidence measure satisfies the threshold confidence measure: The robot is caused to be controlled in the environment according to the predicted parameters without transmitting the visual indication to any remote client device for confirmation before the robot is caused to be controlled according to the predicted parameters.

18. The method according to claim 17, further comprising: In response to determining that the confidence measure fails to satisfy the lower threshold confidence measure: The object representation of the object is transmitted to the remote client device without transmitting any visual indication of the predicted parameter.

19. The method according to claim 17, wherein: The received data indicates confirmation of the predicted parameters, and wherein causing the robot to be controlled in the environment according to the received data includes causing the robot to be controlled in the environment according to the predicted parameters in response to the received data indicating confirmation of the predicted parameters.

20. The method according to claim 19, wherein: A confirmation user interface element is rendered at the remote client device along with rendering the object representation and the visual indication at the remote client device, and wherein, in response to a user interface input being directed to the confirmation user interface element at the remote client device, the received data indicates confirmation of the predicted parameter.

21. The method according to claim 17, wherein: The received data indicates alternative parameters, and wherein causing the robot to be controlled in the environment in accordance with the received data comprises causing the robot to be controlled in the environment in accordance with the alternative parameters in response to the received data indicating the alternative parameters.

22. The method according to claim 21, wherein: An alternative parameter user interface element is rendered at the remote client device along with rendering the object representation and the visual indication at the remote client device, and wherein, in response to a user interface input directed to the alternative parameter user interface element at the remote client device, the received data indicates the alternative parameter.

23. The method according to claim 21, further comprising: generating a positive training instance based on the alternative parameter indicated by the received data; as well as The machine learning model is trained based on the positive training examples.

24. The method of claim 17, wherein: Both a confirmation user interface element and an alternative parameter user interface element are rendered at the remote client device along with rendering the object representation and the visual indication at the remote client device.

25. A method comprising: receiving visual data capturing characteristics of the environment from one or more visual components in the environment, the characteristics of the environment including object characteristics of objects located in the environment; Based on processing the visual data using a machine learning model to generate: the predicted parameters for controlling the robot in the environment, and confidence measures for the predicted parameters; determining whether the confidence measure for the predicted parameter satisfies a threshold confidence measure; Generating a visual representation for transmission to a remote client device via one or more networks, generating the visual representation comprises: including in the visual representation an object representation of the object generated based on the object features; and determining whether to include a visual indication of the predicted parameter in the visual representation based on whether the confidence measure satisfies a lower threshold confidence measure; transmitting the visual representation to the remote client device; receiving, from the remote client device via the one or more networks, data generated at the remote client device based on one or more user interface inputs in response to rendering the visual representation at the remote client device; and The robot is controlled in the environment according to the received data.

26. The method according to claim 25, wherein: Based on determining that the confidence measure fails to satisfy the lower threshold confidence measure, the visual representation lacks any visual indication of the predicted parameter.

27. The method according to claim 25, wherein: The received data indicates confirmation of the predicted parameters, and wherein causing the robot to be controlled in the environment according to the received data includes causing the robot to be controlled in the environment according to the predicted parameters in response to the received data indicating confirmation of the predicted parameters.

28. The method according to claim 27, wherein: A confirmation user interface element is rendered at the remote client device along with rendering the object representation and the visual indication at the remote client device, and wherein, in response to a user interface input being directed to the confirmation user interface element at the remote client device, the received data indicates confirmation of the predicted parameter.

29. The method according to claim 25, wherein: The received data indicates alternative parameters, and wherein causing the robot to be controlled in the environment in accordance with the received data comprises causing the robot to be controlled in the environment in accordance with the alternative parameters in response to the received data indicating the alternative parameters.

30. The method of claim 29, wherein: An alternative parameter user interface element is rendered at the remote client device along with rendering the object representation and the visual indication at the remote client device, and wherein, in response to a user interface input directed to the alternative parameter user interface element at the remote client device, the received data indicates the alternative parameter.

31. The method of claim 29, further comprising: generating a positive training instance based on the alternative parameter indicated by the received data; as well as The machine learning model is trained based on the positive training examples.

32. The method of claim 25, wherein: Both a confirmation user interface element and an alternative parameter user interface element are rendered at the remote client device along with rendering the object representation and the visual indication at the remote client device.

33. A system comprising: one or more visual components of the environment; a memory for storing instructions; one or more processors operable to execute the instructions to: receiving, from the one or more visual components, visual data capturing characteristics of the environment, the characteristics of the environment including object characteristics of objects located in the environment; Based on processing the visual data using a machine learning model to generate: the predicted parameters for controlling the robot in the environment, and confidence measures for the predicted parameters; determining whether the confidence measure for the predicted parameter satisfies a threshold confidence measure; In response to determining that the confidence measure fails to satisfy the threshold confidence measure: transmitting, via one or more networks to a remote client device, an object representation of the object generated based on the object features and a visual indication of the predicted parameters, wherein transmitting the visual indication of the predicted parameter to the remote client device is further responsive to determining that the confidence measure satisfies a lower threshold confidence measure, the lower threshold confidence measure being a lesser indication of confidence than the threshold confidence measure; receiving, from the remote client device via the one or more networks, data generated at the remote client device based on one or more user interface inputs in response to rendering the object representation and the visual indication at the remote client device; causing the robot to be controlled in the environment based on the received data; and In response to determining that the confidence measure satisfies the threshold confidence measure: The robot is caused to be controlled in the environment according to the predicted parameters without transmitting the visual indication to any remote client device for confirmation before the robot is caused to be controlled according to the predicted parameters.

34. The system of claim 33, wherein: One or more of the processors are further operable to execute the instructions to: In response to determining that the confidence measure fails to satisfy the lower threshold confidence measure: The object representation of the object is transmitted to the remote client device without transmitting any visual indication of the predicted parameter.

35. A system comprising a memory storing instructions and one or more processors operable to execute the instructions so as to cause performance of a method according to any preceding claim.

Images