Simulation-driven robot control of real robots

Abstract

Active utilization of a robot simulator in controlling one or more real-world robots. The simulated environment of the robot simulator can be configured to reflect a real-world environment in which the real robot is currently located or will be located in the future. The robot simulator can then be used to determine a set of robot actions to be used by the real-world robot in performing at least a portion of a robot task. The set of robot actions can be applied to the simulated robot of the robot simulator to generate a set of predicted simulated state data instances. The real robot can be controlled to implement the set of robot actions. The implementation of one or more of the robot actions can be contingent on the real state data instances having at least a threshold similarity to a corresponding one of the predicted simulated state data instances.

Description

[Technical field]

[0001] Simulation-driven robot control of real robots. [Background technology]

[0002] The use of robot simulators has been proposed to generate simulated training data that is utilized to train machine learning models. Once the machine learning models are trained based on the simulated training data, they can be utilized during the perception, planning, and / or action stages of robotic control of a real robot (i.e., a physical, real-world robot that is not simulated). However, the utilization of the machine learning models by the real robot in the robotic control does not actively involve the robot simulator. Rather, the machine learning models are simply trained based on simulated training data generated using the robot simulator. Additionally, the use of the robot simulator has been proposed to test / validate a robot control stack prior to implementation of the control stack on the real robot. However, once the robot control stack is implemented on the real robot, the robot simulator is no longer actively utilized. Summary of the Invention [Means for solving the problem]

[0003] Implementations disclosed herein are directed to the active use of a robot simulator in controlling one or more real-world robots. A robot simulator used to control a real-world robot (also referred to herein as a "real robot") can be implemented using hardware (e.g., processor, RAM, and ROM) local to the real-world robot and / or can be implemented using hardware remote from, but in network communication with, the real-world robot.

[0004] In various implementations, the simulated environment of the robot simulator can be configured to reflect a real-world environment (also referred to herein as a "real environment") in which a real robot is currently located or will be located in the future. The simulated environment can be configured based on real environment state data based on sensor-based observations of the environment. The real environment state data can include raw data from the observations that can identify identifiers of objects in the real environment and their poses in the real environment and / or can be processed to identify objects and their poses. The simulated environment can be configured based on identifiers of objects and their poses. For example, identifiers of objects can be utilized to identify matching three-dimensional models of the objects and the objects to be positioned in the simulated environment according to their corresponding poses.

[0005] The sensor-based observations may include robot sensor-based observations from a vision component and / or other sensors integrated with the real robot being controlled. In some of those implementations, the real environment state data includes real environment state data generated by the real robot using a perception system and provided by the real robot to the robot simulator. The real environment state data may be generated based on the robot sensor-based observations and / or the environmental sensor-based observations (described below). Such perception data may include, for example, a corresponding pose and a corresponding object identifier for each of a plurality of objects in the real environment. The corresponding object identifier may be a higher level classification, such as a coffee mug, a cup, or a bowl, or may be a more granular identifier, such as an identifier for a particular coffee mug or a particular cup. Optionally, the more granular identifier of the object may be determined based on a QR code or other coded identifier printed on or attached to the object. The sensor-based observations may additionally or alternatively include environmental sensor-based observations from a vision component and / or other sensors present in the real environment but not integrated with the real robot being controlled. For example, environmental sensor-based observations may include observations from a stand-alone stereo camera or other vision components.

[0006] Once the simulated environment is configured to reflect the real environment, the robotics simulator can be used to determine a sequence of robotic actions to be used by the real-world robot in performing at least a portion of a robotic task. The robotic task may be specified by a higher level planning component of the robotics simulator, by the real-world robot, or based on user interface input. As a non-limiting example, a robotic task may include grasping an object and placing the object in a container, where part of the task may include grasping the object. As another non-limiting example, a robotic task may include loading a dishwasher, where part of the robotic task may include repositioning an object, grasping an object to be placed in the dishwasher, or placing an already grasped object in the dishwasher.

[0007] In determining a set of robot actions, the robot simulator may utilize one or more machine learning models and / or rules accessible to the robot simulator to determine the set of robot actions from the perspective of a simulated environment of the robot simulator. For example, simulated state data from the simulated environment may be applied to the machine learning models and / or rules to determine that the set of robot actions is a candidate set of robot actions. For example, the applied simulated state data may include simulated images of the simulated environment, such as simulated images rendered from the perspective of a simulated vision component of a simulated robot (that simulates a real-world robot) of the robot simulator.

[0008] In some implementations, the applied simulated state data may include simulated state data that simulates real state data that cannot be captured by any of the robot sensors and / or environmental sensors in the real world at all. For example, the simulated state data may include simulated / synthetic visual data captured from a particular viewpoint (e.g., "overhead"), but in the absence of manual human intervention, there may not be any environmental or robotic visual components capable of capturing the real visual data from a corresponding viewpoint in the real world. It should be noted that even if there is no real-world visual component capable of capturing the real visual data from a corresponding viewpoint, the simulated visual data will still reflect the real world as seen from a simulated environment that is based on the real world and configured using a three-dimensional object model. As yet another example, the simulated state data may include simulated LIDAR data, but there may not be any LIDAR visual components in the real world. In these and other aspects, a robot simulator can be utilized to generate simulated state data that lacks any real-world counterpart and utilize such simulated state data in determining a course of robotic actions. Utilizing such simulated state data may result in more robust and / or more accurate determination of a course of robotic actions in various scenarios. As one example, it may enable the utilization of additional and / or alternative machine learning models, such as those tailored to such simulated state data. As another example, it may enable the application of machine learning models and / or rules to multiple heterogeneous simulated state data instances to collectively determine a course of action based on the multiple heterogeneous simulated state data instances.

[0009] In some additional or alternative implementations, the applied simulated state data may include simulated state data that may be captured by one or more of the robot sensors and / or environmental sensors in the real world, but only if power is consumed when traversing the pose of those real world sensors and / or repositioning environmental objects. For example, the applied simulated state data may include a simulated three-dimensional point cloud of an object. A vision component of a real robot may be able to capture data to generate a corresponding real three-dimensional point cloud, but only if the real robot consumes a significant amount of power when traversing the entire perimeter of the object and / or when picking up and rotating the object. On the other hand, a robot simulator may be able to generate a simulated three-dimensional point cloud of an object with less power consumption. As another example, the applied simulated state data instance may include multiple simulated images, each from a different viewpoint. A vision component of a real robot may be able to capture corresponding real images from corresponding different viewpoints, but only if the real robot consumes a significant amount of power when traversing the vision component to the different viewpoints. On the other hand, the robot simulator is capable of rendering simulated images from different viewpoints with less power consumption.

[0010] In some implementations, candidate sequences of robot actions may be applied to a simulated robot in a robot simulator and in a simulated environment, while further simulated state data during and / or after this application can be used to determine whether these candidate sequences of robot actions should be determined as the sequence of robot actions provided for use by the real-world robot. Optionally, multiple candidate sequences of robot actions can be considered, each applied in the robot simulator and corresponding further simulated data each used in determining which are the determined sequence of robot actions provided for use by the real-world robot.

[0011] As an example, one or more features of the additional simulated state data can be analyzed to determine whether a candidate sequence of robotic actions should be determined as a sequence of robotic actions provided for use by a real-world robot. For example, the feature may include a task success feature indicating whether a portion of a task was successful (e.g., if a portion of the task is grasping an object, does the simulated state data indicate that a simulated representation of the object was grasped by applying the sequence of robotic actions). In such a case, the candidate sequence of robotic actions may be determined as a sequence of robotic actions provided for use only if the task success feature indicates success of that portion of the task. Also, for example, the feature may include an efficiency feature, such as an efficiency feature indicating the amount of robotic actions in the sequence of robotic actions and / or the distance traversed by components of the robot in performing the sequence of robotic actions (e.g., in joint space, the cumulative distance traversed by the joints of the robot and / or in task space, the distance traversed by the end effector of the robot). In such cases, a candidate set of robotic actions can be determined as a set of robotic actions offered for use only if the efficiency features satisfy absolute and / or relative thresholds, such as relative thresholds based on the efficiency features of other candidate sets of actions being considered. It should be noted that for some sets of robotic actions, the efficiency features cannot be known without applying them in a simulation. For example, a set of robotic actions can be determined using a machine learning model, but determining downstream actions in the set of robotic actions can depend on processing simulated state data that is affected by the execution of upstream actions in the set of robotic actions.

[0012] Various technical effects can be achieved by analyzing additional simulated state data during and / or after applying a sequence of robotic actions in a simulation, and simply providing the sequence of robotic actions for implementation by a real robot if the analysis satisfies certain conditions. As one example, when task success characteristics are considered, the probability that the real robot will successfully perform that portion of the task is increased. This can avoid wasting energy when controlling a real-world robot to perform actions that do not result in successfully performing that portion of the task, and / or can avoid unnecessary wear and tear on the real-world robot that would otherwise be a consequence of performing actions that do not result in successfully performing that portion of the task. As another example, when efficiency characteristics are considered, a real robot can perform that portion of the task in a shorter duration and / or with fewer movements of components. As yet another example, a robot simulator can apply a sequence of robotic actions in a simulation in a shorter actual elapsed time than would be possible for a real robot. This allows a sequence of unsuccessful or otherwise suboptimal robot actions to be determined more quickly and for a shorter duration than if a real robot were instead controlled to implement the suboptimal sequence of robot actions, resulting in real-world state data being analyzed to determine that the sequence of robot actions is suboptimal.

[0013] It should be noted that these and other technical effects may not be achievable in various implementations without the use of the robot simulator described herein. As an example, assume that a utilized machine learning model indicates a first series of robotic actions as most likely to successfully accomplish a portion of a task and a second series of robotic actions as second most likely to successfully accomplish that portion of the task. Further assume that by applying the first and second series of robotic actions in a simulation and analyzing corresponding further simulated state data, it is determined that the second series of robotic actions resulted in the portion of the task being successfully performed in the simulation, whereas the first series of robotic actions did not. As a result, the second series of robotic actions can be provided for implementation by a real robot. Without the use of a robot simulator, the first series of robotic actions that were not successful may have been selected and implemented as a result of the machine learning model indicating that it was more likely to be successful.

[0014] After a sequence of robot actions is determined using the robot simulator, a real robot can be controlled to implement the sequence of robot actions. For example, each robot action can directly or indirectly indicate a corresponding control command to be implemented at a corresponding time step, which in turn is implemented at the corresponding time step. Implementing a control command can include sending the control command to a corresponding actuator of the real robot.

[0015] As described above, in some implementations, a series of robot actions are applied in a robot simulator prior to implementation of the series of robot actions by a real robot. In some of those implementations, a series of simulated state data instances are generated based on simulated data from the robot simulator while the series of robot actions are applied. Each of the simulated state data instances defines, for a corresponding point in the series of robot actions, corresponding predicted environmental state data for the corresponding point and / or corresponding predicted robot state data for the corresponding point. In those implementations, each of the series of simulated state data instances can be compared with a corresponding real state data instance while the series of robot actions are implemented. Each of the real state data instances defines, for a corresponding point in the series of robot actions, a real environmental state (e.g., robot sensor-based and / or environmental sensor-based) based on one or more updated sensor-based observations of the real environment and / or real robot state data based on one or more sensor-based observations of the real robot. It should be noted that the simulated state data instances can each be captured from the perspective of a real-world sensor (robot and / or environment) and a corresponding simulated sensor (e.g., a simulated vision component) to enable the comparisons described herein.

[0016] Each of the simulated state data instances can be compared to a corresponding real state data instance to determine whether a threshold similarity exists between them. If a threshold similarity exists, this may indicate that the implementation of the robot action on the real robot is affecting the real environment in a manner consistent with its effect on the simulated environment when implemented in the simulation. This may additionally or alternatively indicate that the real environment is in a state corresponding to the simulated environment state. For example, it may indicate that no unpredicted objects have been introduced into the environment and / or that no predicted objects have been removed from the environment. Still further, this may additionally or alternatively indicate that the implementation of the robot action on the real robot is affecting the real robot in a manner consistent with its effect on the simulated robot when implemented in the simulation.

[0017] The simulated state data instance for a point can be compared with a corresponding real state data instance for that point prior to implementation of the sequence of robotic actions following that point on the real robot. Implementation of the robotic actions following that point can be contingent on the comparison indicating a threshold similarity. If the comparison indicates that the threshold similarity does not exist, control of the real robot based on the sequence of robotic actions can be stopped. This prevents the real robot from continuing to implement the sequence of robotic actions in a situation where environmental conditions have changed. Thus, energy is not wasted in continuing to control the real robot and / or damage to the real robot and / or environmental objects is prevented. Optionally, when control of the real robot based on the sequence of robotic actions is stopped, the updated real environment state data is used to reconstruct a simulated environment of the robot simulator, and the robot simulator is used to determine an updated sequence of robotic actions to implement based on the reconstructed simulated environment. The updated sequence of robotic actions can then be used in controlling the real robot. Additionally, an updated set of simulated state data instances may be provided along with an updated set of robot actions to again enable comparison with corresponding real state data instances to ensure a threshold similarity between the two before fully implementing the updated set of robot actions.

[0018] As one particular example, each of the simulated state data instances may include corresponding predicted environmental state data, where the corresponding predicted environmental state data is based on a corresponding rendered image and is rendered from the perspective of a simulated vision component of the simulated robot (corresponding to a real vision component of the real robot). For example, the environmental state data for a simulated state data instance may include a segmentation mask of the corresponding rendered image and / or an embedding of the corresponding rendered image (e.g., an embedding generated based on processing the rendered image with an encoder). Thus, the environmental state data may provide an indication of a predicted environmental state at a corresponding point in a sequence of robot actions. Additionally, each of the real state data instances may include corresponding real environmental state data, where the real environmental state data is based on a corresponding real image from the real vision component of the robot. For example, the real environmental state data for a real state data instance may include a segmentation mask of the corresponding real image and / or an embedding of the corresponding real image (e.g., an embedding generated based on processing the real image with an encoder). Continuing with this example, the predicted instances of environmental state data can be compared to corresponding instances of real environmental state data prior to implementing the sequence of robotic actions following the points corresponding to those instances. For example, the predicted segmentation mask can be compared to the real segmentation mask (e.g., by determining the difference in size and / or location of the segmentation mask) to determine whether a threshold similarity exists between the two. Also, for example, the predicted embedding can be compared to the real embedding (e.g., by determining the distance in embedding space between the two) to determine whether a threshold similarity exists between the two.Utilizing segmentation masks, embeddings, and / or non-pixel level abstractions of the visual data may be more robust to real-world gaps that may exist between rendered visual data from a simulation and real-world visual data, e.g., utilization of such abstractions may be more robust to real-world gaps than directly comparing pixels of the visual data.

[0019] The comparison between the simulated state data instances and the real state data instances can be performed by the robot simulator (e.g., based on the real state data instances provided by the real robot) and / or by the real robot (e.g., based on the simulated state data instances provided by the robot simulator).

[0020] In various implementations, the sequence of robot actions that has been determined by the robot simulator and / or implemented by the real robot at a given time is for only a portion of the task implemented by the real robot. In those implementations, the robot simulator can be utilized to determine the next sequence of robot actions to be implemented to perform another portion of the task, which is implemented by the real robot following the implementation of the sequence of robot actions. This process can continue until all sequences of robot actions required to complete the task have been determined and implemented. Thus, the robot simulator can effectively be in the control loop for the real robot, continually providing an updated sequence of robot actions for use by the real robot.

[0021] In some of these implementations, the next sequence of robot actions may be determined by the robot simulator prior to the sequence of robot actions being fully implemented by the real robot. In various circumstances, the next sequence of robot actions may be determined based on the simulated environment as it progresses in the robot simulator. For example, the simulated environment may be utilized as it progresses when a comparison between the simulated state data instances and the real state data instances satisfies a threshold similarity to previously implemented robot actions. In various other circumstances, the next sequence of robot actions may be determined based on a reconstructed simulated environment that has been reconstructed based on updated real environment state data. For example, the simulated environment may be reconstructed in response to a determination that the threshold similarity is not satisfied, as described above.

[0022] Various technical effects have been mentioned above that may be achieved by the techniques described herein. Additionally or alternatively, the techniques described herein may enable more robust planning of robotic actions and / or more power-efficient use of real robots. For example, in an implementation where a robot simulator is implemented by hardware remote from the real robot, the planning for the robotic actions implemented on the real robot may be executed by the robot simulator. This allows for more robust planning on the real robot with limited hardware capabilities. For example, the robot simulator may utilize machine learning models in the planning that require GPU and / or CPU that may be available to the robot simulator but not present on the real robot. Executing the planning on a remote robot simulator may additionally or alternatively conserve limited battery life in a battery-powered real robot because the real robot does not need to utilize processor cycles and / or other power-intensive resources for planning.

[0023] The above description is provided as merely an overview of some implementations disclosed herein. These and other implementations are described in more detail herein, including in the detailed description, claims, and accompanying documents.

[0024] It should be understood that all combinations of the above concepts, and additional concepts described in more detail herein, are contemplated as part of the subject matter disclosed herein, for example, all combinations of claimed subject matter listed at the end of this disclosure are contemplated as part of the subject matter disclosed herein. [Brief description of the drawings]

[0025] [Figure 1]1 is an illustration of an example environment in which implementations disclosed herein can be implemented. [Diagram 2] 1 is a flowchart illustrating an exemplary method for simulation-driven robotic control of a real robot, according to various implementations disclosed herein. [Diagram 3] 2 is an illustration of examples of the various components of FIG. 1 and interactions that may occur between the various components in a simulation-driven robotic control of a real robot. [Figure 4A] 1 illustrates an example of comparing real and simulated state data instances at a first frequency when determining whether to continue control of a real robot based on a sequence of robot actions determined using a robot simulator. [Figure 4B] 13 illustrates an example of comparing real and simulated state data instances at a second frequency when determining whether to continue control of a real robot based on a sequence of robot actions determined using a robot simulator. [Diagram 5] FIG. 1 is a schematic diagram of an exemplary architecture of a robot. [Figure 6] FIG. 1 is a schematic diagram of an exemplary architecture of a computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] FIG. 1 illustrates an example environment in which implementations disclosed herein may be implemented. The example environment includes a robot 110 and a dishwasher 195. The example environment also includes a bowl 192, a cup 193, and a coffee mug 194 on a table 191. The example environment of FIG. 1 may be a real-world environment in which the robot 110 can utilize the techniques described herein in performing various tasks, such as manipulating one or more of the objects 191-195. For example, the tasks may include pushing, grasping, or otherwise manipulating one or more of the objects 191-195. As another example, the tasks may include more complex tasks, such as loading each of the objects 191-194 into the dishwasher 195.

[0027] The robot 110 illustrated in Figure 1 is a particular mobile robot. However, using the techniques disclosed herein, additional and / or alternative robots may be utilized, such as additional robots that differ in one or more respects from the robot 110 illustrated in Figure 1. For example, a mobile forklift robot, a robotic arm, an unmanned aerial vehicle ("UAV"), and / or a humanoid robot may be utilized in place of or in addition to the robot 110 in the techniques described herein.

[0028] The robot 110 includes a base 113 provided with wheels on opposite sides thereof for movement of the robot 110. The base 113 may include one or more motors for driving the wheels of the robot 110 to achieve, for example, a desired direction, speed, and / or acceleration of movement of the robot 110.

[0029] The robot 110 also includes a vision component 111 capable of generating images relating to the shape, color, depth, and / or other characteristics of objects present in the line of sight of the vision component 111A. The vision component 111 may be, for example, a monographic camera (e.g., generating a two-dimensional RGB image), a stereographic camera (e.g., generating a 2.5-dimensional RGB-D image), and / or a laser scanner (e.g., a LIDAR generating a 2.5-dimensional depth (D) image or point cloud). The robot 110 also includes a robot arm 114 with an end effector 115 having the form of a gripper with two opposing "fingers" or "digits."

[0030] The robot 110 also includes one or more processors that, for example, implement robotic actions (e.g., by providing corresponding control commands to its actuators and / or other motion components), determine object poses and / or identifiers based on the vision component data, and optionally implement a robot simulator. For example, the one or more processors of the robot 110 may implement all or some aspects of the method 200 of FIG. 2. Additional description of some example structures and functions of various robots is provided herein.

[0031] 1 also includes one or more environmental vision components 170 that are not coupled to or otherwise integrated with the robot 110. In addition, the example environment includes a real robot system 120, a robot simulator 140, an object model database 152, and one or more machine learning (ML) models 154.

[0032] The real robot system 120 is implemented by one or more processors of the robot 110. In Figure 1, the real robot system 120 is illustrated as having a real state data engine 122, a control engine 124, and an optional comparison engine 126. In various implementations, additional engines may be included in the real robot system 120.

[0033] The real state data engine 122 processes the sensor-based observations to generate the real environment state data described herein and / or to generate the real state data instances described herein. In generating the real environment state data and / or the real state data instances, the real state data engine 122 may utilize one or more locally stored machine learning models (not shown) to process the sensor-based observations.

[0034] As one example, the real state data engine 122 can process the visual data (from the vision component 111 and / or from the vision component 170) to generate real environment state data including poses and / or classifications of objects in the environment of the robot 110 using machine learning models. As another example, the real state data engine 122 can process the visual data to generate poses for objects in the environment using machine learning models, and can separately process the visual data to determine fine-grained identifiers for the objects, such as identifiers reflected in labels attached to the objects and captured in the visual data. The real environment state data can be provided to the robot simulator 140 by the real robot system 120 and used by the robot simulator's configuration engine 142 in configuring (i.e., initially configuring or reconfiguring) the simulated environment of the robot simulator 140.

[0035] As another example, the real state data engine 122 can process visual data (from the vision component 111 and / or from the vision component 170) using a machine learning model to generate real state data instances, each including a corresponding segmentation mask and / or visual data embedding. The real state data instances can be used by the comparison engine 126 of the real robot system 120 or the comparison engine 150 of the robot simulator 140 in comparing the real state data instances with corresponding simulated state data instances to determine whether a threshold similarity exists between the real state data instances and the simulated state data instances. The real state data engine 122, in various implementations, can include a perception system and / or implement block 252A of the method 200 of FIG. 2.

[0036] The control engine 124 implements the robot actions determined by the simulated action engine 144 of the robot simulator 140. Implementing the robot actions may include sending corresponding control commands to corresponding actuators of the robot 110, where the control commands are directly or indirectly indicated by the robot actions.

[0037] The comparison engine 126 can compare real state data instances (from the real state data engine 122) and simulated state data instances (from the simulated state data engine 148 of the robot simulator 140) for corresponding points during implementation to determine whether a threshold similarity exists between the compared real state data instances and simulated state data instances during implementation of a series of robot actions by the control engine 124. The comparison engine 126 can be in communication with the control engine 124 and can cause the control engine 124 to continue implementing the robot actions provided by the robot simulator as long as the comparison indicates a threshold similarity. The comparison engine 126 can additionally or alternatively cause the control engine 124 to stop implementing the robot actions provided by the robot simulator when one or more comparisons do not indicate a threshold similarity. For example, the comparison engine 126 can cause the control engine 124 to stop the implementation when a single comparison does not indicate the threshold similarity, or when two or more comparisons, such as N sequential comparisons or N comparisons of the most recent N+M comparisons, where N is an integer greater than 1 and M is an integer, do not indicate the threshold similarity. The comparison engine 126 can implement block 262 and / or block 264 of the method 200 of FIG. 2 in various implementations.

[0038] The robot simulator 140 may be implemented by one or more hardware components (e.g., processors) of the robot 110 or by one or more computing devices that are remote from but in network communication with the robot 110. For example, the computing devices may be in network communication with the robot 110 via a local area network and / or a wide area network. The robot simulator 140 is used to simulate an environment including corresponding environmental objects, simulate a robot operating in the simulated environment (e.g., simulating the robot 110 and / or other robots), simulate responses of the simulated robot in response to virtual implementations of various simulated robot actions, and simulate interactions between the simulated robot and simulated environmental objects in response to the simulated robot actions. Various simulators may be utilized, such as physics engines that simulate collision detection, soft and rigid body dynamics, etc. A non-limiting example of such a simulator is the BULLET physics engine.

[0039] The robot simulator 140 includes a configuration engine 121 , a simulated action engine 144 with an optional evaluation module 145 , a simulated application engine 146 , a simulated state data engine 148 , and a comparison engine 150 .

[0040] The configuration engine 121 configures various parameters for the simulated environment of the robot simulator 140 and / or configures various parameters for the simulated robot of the robot simulator 140. In configuring the simulated environment, the configuration engine 121 utilizes real environment state data. In some implementations, the real environment state data may include poses and / or object identifiers provided by the real state data engine 122 of the real robot system 120 and / or by real state data engines of other robots (not shown) in the environment. In some implementations, the real environment state data additionally or alternatively includes raw sensor-based observations, such as visual data from the vision component 111 of the robot 110 and / or visual data from the environmental vision component 170. In those implementations, the configuration engine 121 may include a perception system, e.g., to generate poses and / or object identifiers of objects based on the sensor-based observations.

[0041] In configuring a simulated environment, configuration engine 142 can access object model database 152 to identify three-dimensional (3D) models for use in the simulated environment. For example, object model database 152 can include accurate three-dimensional models for some or all of the objects in the environment, and configuration engine 142 can match the three-dimensional models to corresponding objects, for example, using object identifiers. For example, bowl 192 can include an identifier thereon (e.g., a QR code or other identifier), while the real environment state data can reflect that identifier and a pose for that identifier. Configuration engine 142 can use the identifier to retrieve a three-dimensional model for bowl 192 from object model database 152 (e.g., it can be indexed based on the identifier), and can place the three-dimensional model in the simulated environment based on the pose.

[0042] In some implementations, configuring the simulated environment may also include configuring the simulated robot to reflect the robot 110 and / or its current state in the real environment. In such implementations, the real state data engine 122 may provide real environment state data including an identifier of the robot 110 and / or a current state of the robot 110 in the real environment, such as a current pose for each of the robot's joints. In configuring the simulated robot, the configuration engine 142 may utilize a robot model. The robot model may define the physical configuration of the simulated robot and the constraints of the simulated robot, a dynamic model of the simulated robot, and / or other parameters of the simulated robot. The configuration engine 121 may implement all or some aspects of block 252 and / or block 272 of the method 200 of FIG. 2 in various implementations.

[0043] The simulated action engine 144 can determine a sequence of robotic actions to be used by the robot 110 (and / or other robots not shown) in performing at least a portion of the robotic task based on the simulated environment configured by the configuration engine 142. In determining the sequence of robotic actions, the simulated action engine 144 can utilize one or more machine learning models 154 and / or rules accessible to the robot simulator to determine the sequence of robotic actions from the perspective of the simulated environment. As an example, the machine learning model 154 can include a policy model trained using reinforcement or imitation learning, and the simulated action engine 144 can determine the sequence of robotic actions by applying simulated data to the policy model to generate a probability distribution over the robotic actions and selecting a sequence of robotic actions based on the probability distribution. It should be noted that in selecting each next robotic action, the simulated data can reflect a new state of the simulated environment and / or the simulated robot, where the new state reflects a rollout (e.g., by the simulated application engine 146) from a prior robotic action in the simulation. As another example, the machine learning model 154 may include a machine learning model that may be used to process the current simulated visual data and corresponding candidate robotic actions at each time step and generate a value that reflects the likelihood that implementing the candidate robotic action will result in at least a portion of the task being successfully performed. Multiple candidate robotic actions may be considered by the simulated action engine 144 at each time step, and one is selected for inclusion in the set of robotic actions based on the generated value.It should be noted that in selecting each next robot action, the simulated data may reflect a new state of the simulated environment and / or the simulated robot, where the new state reflects the rollout of a preceding robot action in the simulation.

[0044] In some implementations, the simulated action engine 144 generates a set of candidate robot actions and determines which one is determined as the set of robot actions to be provided for use by the robot 110. In some versions of those implementations, an evaluation module 145 of the simulated action engine 144 is utilized to evaluate the set of candidate robot actions and, based on the evaluation, determine only one of the set of candidate robot actions to be provided for use by the robot 110. In some versions of those implementations, a simulated application engine 146 can apply each of the candidate robot actions to a simulated robot in a robot simulator and in a simulated environment. Additionally, the simulated state data engine 148 can generate simulated state data during and / or after application, and the evaluation module 145 can utilize the simulated state data in evaluating one or more sets of candidate robot actions.

[0045] As an example, the evaluation module 145 can utilize the simulated state data for a series of robotic actions to generate a task success characteristic that indicates whether a portion of a task intended to be implemented by the series of robotic actions was successful. For example, if the portion of the task is to grasp an object, the evaluation module 145 can generate the task success characteristic based on whether the simulated state data indicates that a simulated representation of the object was grasped by applying the series of robotic actions. In such an example, the evaluation module 145 can determine that the series of robotic actions is provided for use by the robot 110 only if the task success characteristic indicates success of the portion of the task.

[0046] As an example, the evaluation module 145 can utilize the simulated state data for a series of robotic actions to additionally or alternatively generate an efficiency feature indicative of the amount of robotic actions in the series and / or the distance traversed by a component of the robot in performing the series of robotic actions. The distance can be, for example, a cumulative distance in joint space traversed by joints of the simulated robot and / or a distance in task space traversed by an end effector of the simulated robot. In such an example, the evaluation module 145 can determine that a series of robotic actions is to be provided for use by the robot 110 only if the efficiency feature satisfies absolute and / or relative thresholds. The simulated action engine 144 can implement all or some aspects of block 256 of the method 200 of FIG. 2 in various implementations.

[0047] The simulated adaptation engine 146 can apply a set of robot actions to the simulated robot and in the simulated environment, such that the simulated robot is moved according to the robot actions and the simulated environment is affected by at least some of the robot actions. As described above, the simulated adaptation engine 146 can optionally apply a set of different candidate robot actions, for example to generate simulated state data that is utilized by the evaluation module 145 in evaluating the set of different candidate robot actions. In such a situation, the simulated adaptation engine 146 can apply each of the different candidate sets of robot actions with the same starting environment and robot state, but roll out the simulation separately for each and according to the corresponding applied robot actions. The simulated adaptation engine 146 can implement all or some aspects of the block 256 of the method 200 of FIG. 2 in various implementations.

[0048] The simulated state data engine 148 may generate simulated state data for application of robotic actions by the simulated application engine 146. The generated simulated state data may include data utilization by the evaluation module 145 and / or may include simulated state data instances for utilization by the comparison engine 126 and / or the comparison engine 150. For example, for a series of robotic actions applied by the simulated application engine 146, the simulated state data engine 148 may generate a series of simulated state data instances based on simulated data from the robot simulator during application of the series of robotic actions. For example, the simulated state data engine 148 may generate a first simulated state data instance prior to any application of the series of robotic actions, but may generate simulated state data instances after application of each robotic action, after application of every other robotic action, after application of every third robotic action, or at some other frequency. The simulated state data instances each define, for a corresponding point in the series of robotic actions, corresponding predicted environmental state data for that corresponding point and / or corresponding predicted robot state data for that corresponding point. The simulated state data engine 148 may implement all or some aspects of block 256A and / or block 258 of the method 200 of FIG. 2 in various implementations.

[0049] The comparison engine 150 may share one or more aspects with the comparison engine 126 of the real robot system 120 described above. In various implementations, only one of the comparison engine 126 and the comparison engine 150 is provided, but not the other. In implementations in which the comparison engine 150 is included, it can obtain real state data instances from the real state data engine 122 and compare them with the simulated state data instances generated by the simulated state data engine 148. In implementations in which the comparison engine 126 is included, it can obtain simulated state data instances from the simulated state data engine 148 and compare them with the real state data instances generated by the real state data engine 122. The comparison engine 150, in various implementations, can implement all or some aspects of block 262 and / or block 264 of the method 200 of FIG. 2.

[0050] Additional description of the various components of FIG. 1 and interactions that may occur between them are described below with respect to FIG. 3 and FIG. 4. First, however, method 200 of FIG. 2 is described, which illustrates an example method of simulation-driven robotic control of a real robot according to various implementations disclosed herein. For convenience, the operations of the flowcharts are described with reference to a system that performs those operations. The system may include one or more components (e.g., a processor) of the robot and / or of a computing device in network communication with the robot. Furthermore, while the operations of method 200 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0051] In block 252, the system identifies real environment state data. The real environment state data is based on sensor-based observations of the real environment. The sensor-based observations are from real sensors in the real environment, such as sensors of a real robot in the real environment and / or sensors in the real environment but not part of the real robot. The real environment state data may include, for example, a pose of a real object in the real environment, an object identifier of the real object, and / or the sensor-based observations themselves.

[0052] Block 252 may optionally include a sub-block 252A in which the system generates real environment state data based on sensor-based observations of the real environment. For example, the system may generate a pose of a real object in the real environment and / or an object identifier for the real object by processing the sensor-based observations. For example, in generating the object identifier and / or the pose, the system may employ a perception system to process the sensor-based observations utilizing one or more machine learning models.

[0053] In block 254, the system configures a simulated environment for the robot simulator based on the real environment condition data identified in block 252. The system configures the simulated environment based on the real environment condition data to reflect the real environment as closely as possible.

[0054] In block 256, the system uses a robot simulator to determine a set of robot actions for a robot task. The robot task can be one that is performed immediately by a real robot currently present in the real environment, or one that is performed at a future time (e.g., 5 minutes from now) when a real robot is present in the real environment. The real environment reflected in the simulated environment can be, for example, a portion of a room, an entire room, multiple rooms, an entire building, and / or other defined space in the world.

[0055] Block 256 may optionally include sub-block 256A, in which the system generates a plurality of different sequences of robot actions, separately applies each of the sequences of robot actions to a robot simulator in its current configuration (the initial configuration in the first iteration of sub-block 256A), and determines a sequence of robot actions based on one or more features from the application. The features for a sequence of robot actions may be determined based on simulated data from the application of the sequence of robot actions and may include, for example, task success features and / or efficiency features.

[0056] In block 258, the system generates simulated state data instances for the sequence of robotic actions determined in block 256. Each simulated state data instance defines corresponding predicted environmental state data and / or robot state data for a corresponding point in the sequence of robotic actions. The system generates the simulated state data instance for a point in the sequence of robotic actions based on the simulated state data from that point. For example, given a simulated state data instance following implementation of a second robotic action in a sequence, it may be based on a simulated environment state and / or a simulated robot state following implementation of the second robotic action. In some implementations, when block 256 includes sub-block 256A, block 258 may include generating a simulated state data instance based on application of block 256A to the determined sequence of robotic actions.

[0057] In block 260, the system controls the real robot to implement one or more next robot actions in the sequence of robot actions determined in block 256. In a first iteration through the sequence of robot actions, this may include implementing a first robot action and optionally one or more subsequent robot actions. Although not illustrated in FIG. 2, it should be noted that in some implementations, prior to the first iteration through the sequence of robot actions, the system may compare a current real state data instance with a simulated state data instance that reflects a predicted environment and / or robot state prior to implementation of the sequence of robot actions. This may ensure that the real robot state and / or real environment state have not changed significantly compared to the initial simulated robot state and / or initial simulated environment state based on which the sequence of robot actions are determined.

[0058] In block 262, the system compares the real state data instances with corresponding simulated state data instances generated in block 258, each generated after or at a corresponding point during the implementation of one of the robot actions in the most recent iteration of block 260.

[0059] In block 264 , the system determines whether the comparison of block 262 indicates a threshold degree of similarity between the real and simulated state data instances being compared in block 262 .

[0060] If, at block 264, the system determines that a threshold similarity is indicated, the system proceeds to block 266. In block 266, the system determines whether there are one or more remaining robot actions in the sequence of robot actions. If so, the system returns to block 260 to control the real robot to implement the next robot action in the sequence of robot actions. If, at block 266, the system determines that there are no remaining robot actions in the sequence of robot actions, the system proceeds to block 268.

[0061] In block 268, the system determines whether there are any further robot actions for the completion of the task being performed by the real robot. If not, the system proceeds to block 270 and the method 200 ends for that task (but may be restarted for other tasks). In that case, the system returns to block 256 and uses the robot simulator to determine the next set of robot actions for that robot task. It should be noted that in some implementations, the next iteration of block 256 may be performed while the iteration of block 260 is still being performed, based in whole or in part on the set of robot actions determined during the previous iteration of block 256. It should also be noted that in some implementations where block 256 is executed again without encountering block 272 before execution, the robot simulator may be in a state that matches its state after the application of the most recent set of robot actions in the robot simulator. In some other implementations where block 256 is executed again without encountering block 272 prior to execution, the robot simulator may be in a state where it is updated based on updated real environment state data from during or after the application of the most recent series of robot actions by the real robot (e.g., the pose of the simulated object may be updated based in part on the pose from the updated real environment state data).

[0062] If in block 264 the system determines that the threshold similarity is not indicated, the system proceeds to block 272. In block 272, the system reconstructs the simulated environment based on the updated real environment state data. The updated real environment state data may be based on sensor-based observations of the real environment following the most recent iteration of block 260. Block 272 may share one or more aspects with block 254, but may be based on the updated real environment state data. After executing block 272, the system may return to block 256 and determine a new set of robot actions using the robot simulator and the reconstructed simulated environment.

[0063] 3, the various components of FIG. 1 are illustrated along with example interactions that may occur between the various components in a simulation-driven robotic control of a real robot. In FIG. 3, the robot vision component 111 generates visual data 112A and provides the visual data 112A to the real state data engine 122 and / or the configuration engine 142. Additionally, the environment vision component 170 generates visual data 171A and provides the visual data 171A to the configuration engine 142. The real state data engine 122 can process the visual data 112A to generate pose data and / or object identifier data 112 that reflects poses and / or object identifiers for objects in the real environment of the real robot 110 (FIG. 1). The pose data and / or object identifier data 112 is provided to the configuration engine 142.

[0064] The configuration engine 142 utilizes the visual data 112A, the visual data 171A, and / or the pose data and / or object identifier data 112 in generating the configured simulated environment 143. The configuration engine 142 may also utilize the object model database 152 in generating the configured simulated environment 143. For example, the pose data and / or object identifier data 112 determined based on the visual data 112A or 117A may be utilized to retrieve corresponding three-dimensional models of objects from the object model database 152. Those three-dimensional models may be included in the configured simulated environment and may be included in the corresponding poses from the pose data and / or object identifier data 112 determined based on the visual data 112A or 117A.

[0065] The simulated action engine 144 utilizes the configured simulated environment 143 to determine one or more candidate sequences of robot actions 144A-N. The simulated action engine 144 can determine the sequence of candidate robot actions 144A-N based on processing simulated data using a machine learning model 154. Additionally, the simulated action engine 144 can determine the sequence of candidate robot actions 144A-N in terms of a task 101 to be performed by a real robot. The task 101 can be based, for example, on user interface input from a user and / or on a higher level planning system.

[0066] For each of the candidate sets of robot actions 144A-N, the simulated application engine 146 can apply a set of robot actions 147 in the robot simulator. It should be noted that in some implementations, one or more of the downstream robot actions of one or more sets of robot actions 144A-N can be generated by the simulated action engine 144 using updated simulation data from the robot simulator after the simulated application engine 146 applies the set of upstream robot actions.

[0067] The simulated state data engine 148 generates simulated state data for each of the candidate sets of robotic actions 144A-N and based on the application of the set of robotic actions 147 by the simulated application engine. The simulated state data may be utilized to generate corresponding features 112A-N for each of the candidate sets of robotic actions 144A-N. The evaluation module 145 may utilize those features 112A-N in selecting (e.g., from among the multiple candidate sets of robotic actions 144A-N) the determined set of robotic actions 144A to provide to the control engine 124 for implementation. The evaluation module 145 may also provide feedback to the simulated action engine 144 regarding the evaluation of the set of robotic actions already evaluated. The simulated action engine 144 may generate one or more of the sets of candidate robotic actions 144A-N based on that feedback. For example, the simulated action engine 144 may initially generate only a single candidate set of robot actions, but may generate additional sets of robot actions only if feedback indicates that the initially generated set of robot actions does not meet the conditions.

[0068] The simulated state data engine 148 also generates a simulated state data instance 149A for the determined sequence of robot actions 144A and provides the simulated state data instance 149A to the comparison engine 126 or 150.

[0069] The control engine 124 implements the determined set of robot actions 144A in the real robot 110. The comparison engine 126 or 150 compares the simulated state data instances with the real state data instances during implementation. The real state data instances may be based on updated visual data from the robot vision component 111, updated visual data from the environment vision component 170, and / or updated real state data from the real state data engine 122. If the comparison indicates a threshold similarity, the implementation of the set of robot actions 144A may continue. On the other hand, if one or more comparisons do not indicate a threshold similarity, the comparison engine 126 or 150 may cause the control engine 124 to stop implementing the determined set of robot actions 144A. The comparison engine 126 may further send a reconfiguration signal 127 to the reconstruction engine 142 to cause the simulated environment to be reconfigured as described herein. FIG. 3 is described with respect to a single set of robot actions for simplicity. However, it should be noted that in various implementations, additional sets of robot actions may be determined using a robot simulator and provided for implementation by a real robot in performing a task. These additional sets of robot actions may be determined and implemented in a manner similar to that described in FIG. 3 and as described elsewhere herein.

[0070] 4A and 4B, two different examples of comparisons between real and simulated state data instances are illustrated. In FIG. 4A, three robot actions 144A1, 144A2, and 144A3 are illustrated, which may be all or part of a series of robot actions. Furthermore, a simulated state data instance 149A0 is provided, which corresponds to a predicted state before any of the series of robot actions are implemented. A simulated state data instance 149A1 is provided, which corresponds to a predicted state after the robot action 144A1 is implemented, a simulated state data instance 149A2 is provided, which corresponds to a predicted state after the robot action 144A2 is implemented, and a simulated state data instance 149A3 is provided, which corresponds to a predicted state after the robot action 144A3 is implemented.

[0071] 4A, the comparison engine 126 or 150 may compare the real state data instance 170A0 at the time to the simulated state data instance 149A0 before any of the series of robot actions are implemented, and may simply implement the robot action 144A1 if the comparison indicates a threshold similarity. After the implementation of the robot action 144A1, the comparison engine 126 or 150 may compare the real state data instance 170A1 from a point after the implementation of the robot action 144A1 to the simulated state data instance 149A1. The implementation of the robot action 144A2 may be contingent on that comparison indicating a threshold similarity, or the robot action 144A2 may be implemented in part (or in whole) during the comparison, but if the comparison does not indicate a threshold similarity, control of the real robot based on the series of robot actions is stopped. After implementation of the robot action 144A2, the comparison engine 126 or 150 can compare the real state data instance 170A2 from a point after implementation of the robot action 144A2 with the simulated state data instance 149A2. Implementation of the robot action 144A3 can be contingent on the comparison indicating a threshold similarity, or the robot action 144A3 can be partially (or entirely) implemented during the comparison, but if the comparison does not indicate a threshold similarity, control of the real robot based on the set of robot actions is stopped. Additionally, after implementation of the robot action 144A3, the comparison engine 126 or 150 can compare the real state data instance 170A3 from a point after implementation of the robot action 144A3 with the simulated state data instance 149A3.Implementation of further robot actions in the series of robot actions (if any), or the next robot action in the series (if robot action 144A3 is the terminal robot action in the series), may be contingent on the comparison indicating a threshold similarity, or robot action 144A3 may be implemented in part (or in whole) during the comparison, but if the comparison does not indicate a threshold similarity, control of the real robot based on the series of robot actions is stopped.

[0072] FIG. 4B is similar to FIG. 4A, with like reference numbers referring to like components. However, in FIG. 4A, simulated state data instances are not provided after each action. Rather, only the initial simulated state data instance 149A0 and the simulated state data instance 149A3 are provided. Thus, the comparison engine 126 or 150 performs an initial comparison between the real state data instance 170A0 and the simulated state data instance 149A0, as well as a comparison between the real state data instance 170A3 and the simulated state data instance 149A3, but does not perform a comparison based on the real state data after the robot actions 144A1 and 144A2. Thus, FIG. 4A and FIG. 4B illustrate two different non-limiting example frequencies at which comparisons can be performed by the comparison engine 126 or 150.

[0073] In some implementations, the frequency at which the comparison is performed can be fixed throughout the duration of the attempted execution of a robotic task, and optionally for each of multiple different robotic tasks. In some alternative implementations, the frequency at which the comparison is performed can be adapted throughout the duration of the attempted execution of a robotic task and / or among multiple different robotic tasks. In some of these implementations, the frequency at which the comparison is performed at a given time can be determined depending on the real-world environment, the part of the robotic task being performed, and / or the actual or predicted entropy of the robotic task being performed. For example, the frequency can be increased if the approaching simulated state data instance indicates that the real-world environment is likely to change (e.g., due to interaction between the real robot and real-world objects). On the other hand, the frequency can be decreased or a lower frequency can be maintained if the approaching simulated state data instance indicates that the real-world environment is likely to remain static. As another example, an initial frequency can be utilized for a robotic task, or a part of a robotic task, i.e., navigating an empty hallway. On the other hand, a second frequency higher than the first frequency may be utilized for a robotic task, or part of a robotic task, i.e., when attempting to grasp an object. In these and other aspects, the frequency of comparison may be dynamically adapted depending on current or predicted robotic actions and / or environmental conditions. This may include increasing the frequency for complex robotic actions (e.g., those involving interaction with environmental objects) and / or high entropy environments, while decreasing the frequency for less complex robotic actions (e.g., moving forward or moving components without interaction with environmental objects) and / or low entropy environments.Thus, the frequency of the comparison can be selectively reduced, which can result in lower utilization of the computational and / or power resources of the real-world robot. Furthermore, in some implementations, the frequency of generating the real-state data instances can even be adjusted to reflect the frequency of the comparison. For example, in some of those implementations, the real-state data instances will only be generated at a frequency that matches the frequency of the comparison. This can include only processing the sensor data instances at that frequency and / or even adapting the sensing frequency of the corresponding sensor components based on that frequency. Adapting the sensing frequency of the corresponding sensor components can include adapting them to sense at that frequency (if possible) or at a corresponding given frequency as close to that frequency as possible, and optionally more frequently than that frequency. As a result of adapting the sensing frequency of the corresponding sensor components, the utilization of the computational and / or power resources of the real-world robot can be reduced.

[0074] 5 illustrates, in schematic form, an example architecture of a robot 520. The robot 520 includes a robot control system 560, one or more motion components 540a-540n, and one or more sensors 542a-542m. The sensors 542a-542m may include, for example, visual sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and the like. Although the sensors 542a-m are shown as integral to the robot 520, this is not intended to be limiting. In some implementations, the sensors 542a-m may be located external to the robot 520, for example, as stand-alone units.

[0075] The motion components 540a-540n may include, for example, one or more end effectors and / or one or more servo motors or other actuators for effecting movement of one or more components of the robot. For example, the robot 520 may have multiple degrees of freedom, and each of the actuators may control the actuation of the robot 520 in one or more of the degrees of freedom in response to a control command. As used herein, the term actuator encompasses a mechanical or electrical device (e.g., a motor) that produces movement, as well as any driver that may be associated with the actuator that converts a received control command into one or more signals for driving the actuator. Thus, providing a control command to an actuator may include providing the control command to a driver that converts the control command into an appropriate signal that drives an electrical or mechanical device to produce the desired movement.

[0076] The robot control system 560 may be implemented in one or more processors, such as the CPU, GPU, and / or other controller of the robot 520. In some implementations, the robot 520 may include a "brain box" that may include all or some aspects of the control system 560. For example, the brain box may provide real-time bursts of data to the motion components 540a-n, each including a set of one or more control commands that, among other things, dictate movement parameters (if any) for each one or more of the motion components 540a-n. The control commands may be based on robot actions determined utilizing a robot simulator described herein. In some implementations, the robot control system 560 may execute certain blocks of the method 200 of FIG. 2.

[0077] 6 is a block diagram of an exemplary computing device 610 that may be optionally utilized to perform one or more aspects of the techniques described herein. For example, a robot simulator may be implemented on the computing device 610 or in a cluster of multiple computing devices 610. The computing device 610 typically includes at least one processor 614 that communicates with several peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624, including, for example, a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices enable a user to interact with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0078] The user interface input devices 622 may include a keyboard, a pointing device, e.g., a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 610 or onto a communications network.

[0079] The user interface output devices 620 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a flat panel device, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 610 to a user or to another machine or computing device.

[0080] Storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, storage subsystem 624 may include logic for performing certain aspects of the method of FIG.

[0081] These software modules are generally executed by the processor 614, either alone or in combination with other processors. The memory 625 used in the storage subsystem 624 may include several memories, including a main random access memory (RAM) 630 for storage of instructions and data during program execution, and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of a particular implementation may be stored by the file storage subsystem 626 in the storage subsystem 624, or in other machines accessible by the processor 614.

[0082] The bus subsystem 612 provides a mechanism for allowing the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is shown generally as a single bus, alternative implementations of the bus subsystem could use multiple buses.

[0083] The computing device 610 may be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Because the nature of computers and networks is ever-changing, the description of the computing device 610 shown in Figure 6 is intended only as a specific example for the purposes of illustrating some implementations. Many other configurations of the computing device 610 are possible, having more or fewer components than the computing device shown in Figure 6.

[0084] In some implementations, a method implemented by one or more processors is provided, the method including configuring a simulated environment of a robot simulator based on real environment state data. The real environment state data is based on one or more sensor-based observations of the real environment. The method further includes determining, with the robot simulator, a set of robot actions for controlling the real robot in performing at least a portion of a robot task in the real environment. The method further includes applying the set of robot actions to the simulated robot of the robot simulator to generate a set of simulated state data instances. Each of the simulated state data instances defines, for a corresponding point in the set of robot actions, one or both of corresponding predicted environmental state data for the corresponding point and corresponding predicted robot state data for the corresponding point. The method further includes controlling the real robot to implement a first subset of the set of robot actions, and comparing the real state data instances with a corresponding one of the simulated state data instances for the corresponding point in the set of robot actions. The real state data instance defines, for the corresponding point, one or both of updated real environment state data based on one or more updated sensor-based observations of the real environment and real robot state data based on one or more sensor-based observations of the real robot. The method further includes controlling the real robot to implement a second subset of the set of robot actions in response to the comparing step indicating a threshold similarity between the updated real state data and a corresponding one of the predicted state data instances for the corresponding point.

[0085] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0086] In some implementations, the method further includes ceasing control of the real robot based on the set of robot actions in response to the comparing step not indicating a threshold similarity. In some versions of those implementations, the method further includes, in response to the comparing step not indicating a threshold similarity, reconstructing the simulated environment based on the updated real environment state data; determining, using a robot simulator, an updated set of robot actions for controlling the real robot in performing the robotic task; applying the updated set of robot actions to the simulated robot of the robot simulator to generate an updated set of simulated state data instances; controlling the real robot to implement a first updated subset of the updated set of robot actions; comparing the updated real state data instances with a corresponding updated one of the updated simulated state data instances for a corresponding updated point in the updated set of robot actions; and, in response to the comparing step indicating a threshold similarity, controlling the real robot to implement an updated second subset of the updated set of robot actions. Each updated simulated state data instance defines, for a corresponding updated point in the updated sequence of robot actions, one or both of corresponding updated predicted environmental state data and real robot state data based on one or more sensor-based observations of the real robot.

[0087] In some implementations, the method includes controlling the real robot to implement a second subset of the set of robot actions and to implement any additional subset of the set of robot actions. In some versions of those implementations, the method further includes, prior to full implementation of the set of robot actions, determining, with a robot simulator, a next set of robot actions to use in controlling the real robot in performing another portion of the robot task, and applying the next set of robot actions to a simulated robot in the robot simulator to generate a next set of simulated state data instances with a simulated robot and an environment in the robot simulator that corresponds to one of the terminations of the simulated state data instances. In some versions of those implementations, the method further includes controlling the real robot to implement the next set of robot actions after full implementation of the set of robot actions. The step of controlling the real robot to implement the next set of robot actions may include the steps of controlling the real robot to implement a first next subset of the next set of robot actions, comparing the next real state data instance with a corresponding one of the next simulated state data instances, and, in response to the comparing step indicating a threshold similarity, controlling the real robot to implement a second next subset of the next set of robot actions.

[0088] In some implementations, the one or more sensor-based observations include robot sensor-based observations based on one or more real sensors of a real robot.

[0089] In some implementations, the one or more sensor-based observations include environmental sensor-based observations based on one or more real sensors that are not coupled to the real robot but are present in the real environment.

[0090] In some implementations, determining a set of robot actions to use for controlling the real robot in performing at least a portion of the robot task includes applying the other set of robot actions to the robot simulator, comparing one or more other features from applying the other set of robot actions to the robot simulator with the one or more features from applying the set of robot actions to the robot simulator, and determining to utilize the set of robot actions instead of the other set of robot actions for use in controlling the real robot in performing at least a portion of the robot task based on comparing the one or more other features to the one or more features. In some versions of these implementations, the one or more features include one or both of a task success feature and an efficiency feature, and the one or more other features include one or both of another task success feature and another efficiency feature. In some versions of these implementations, the one or more features include a task success feature, and the method further includes generating a task success feature based on a terminal state of the robot simulator after applying the set of robot actions. Further, in some of those versions, the one or more features additionally or alternatively include an efficiency feature, and the method further includes generating the efficiency feature based on one or both of an amount of robotic actions in the series of robotic actions and a distance traversed by one or more simulated components of the simulated robot in performing the series of robotic actions.

[0091] In some implementations, the step of comparing the real state data instance with a corresponding one of the simulated state data instances for the corresponding point is performed by one or more processors of the real robot.

[0092] In some implementations, the robot simulator is implemented on one or more remote processors that are remote from the real robot but in network communication with the real robot.

[0093] In some implementations, a method is provided that is implemented by one or more processors of a real robot, the method including receiving a sequence of robot actions and a sequence of simulated state data instances. Each of the simulated state data instances defines, for a corresponding point in the sequence of robot actions, one or both of corresponding predicted environmental state data for the corresponding point and corresponding predicted robot state data for the corresponding point. The method further includes controlling the real robot to implement a first subset of the sequence of robot actions, and comparing the real state data instances with a corresponding one of the simulated state data instances for the corresponding point in the sequence of robot actions. The real state data instances define, for the corresponding point, one or both of updated real environment state data based on one or more updated sensor-based observations of a real environment and real robot state data based on one or more sensor-based observations of the real robot. The method further includes controlling the real robot to implement a second subset of the sequence of robot actions in response to the comparing step indicating a threshold similarity between the updated real state data and the corresponding one of the predicted state data instances for the corresponding point.

[0094] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0095] In some implementations, the method further includes ceasing control of the real robot based on the set of robot actions in response to the comparing step not indicating a threshold similarity.

[0096] In some implementations, the method further includes, in response to the comparing step not indicating a threshold similarity, sending updated real environment state data to the robot simulator component; receiving from the robot simulator component an updated set of robot actions generated based on the updated real environment state data and the updated set of simulated state data instances; controlling the real robot to implement a first updated subset of the updated set of robot actions; comparing the updated real state data instance with a corresponding updated one of the updated simulated state data instances for a corresponding updated point in the updated set of robot actions; and, in response to the comparing step indicating a threshold similarity, controlling the real robot to implement an updated second subset of the updated set of robot actions.

[0097] In some implementations, the robot simulator is implemented on one or more remote processors that are remote from the real robot but in network communication with the real robot.

[0098] In some implementations, a method implemented by one or more processors is provided, the method including configuring a simulated environment of a robot simulator based on real environment state data based on one or more sensor-based observations of a real environment. The method further includes determining, with the robot simulator, a first candidate set of robot actions and a second candidate set of robot actions for use in controlling a real robot in performing at least a portion of a robot task in the real environment. The method further includes applying the first candidate set of robot actions to a simulated robot of the robot simulator using the simulated environment after the configuring step to generate first simulated state data, and applying the second candidate set of robot actions to a simulated robot of the robot simulator using the simulated environment after the configuring step to generate second simulated state data. The method further includes determining, based on the first simulated state data and the second simulated state data, to utilize the first candidate set of robot actions instead of the second candidate set of robot actions for use in controlling a real robot in performing at least a portion of a robot task. The method further includes, in response to determining to utilize the first candidate set of robot actions, causing the real robot to implement the first candidate set of robot actions.

[0099] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0100] In some implementations, determining to utilize the first candidate set of robot actions instead of the second candidate set of robot actions based on the first simulated state data and the second simulated state data includes generating one or more first features based on the first simulated state data, generating one or more second features based on the second simulated state data, and determining to utilize the first candidate set of robot actions based on a comparison of the first features to the second features. In some versions of those implementations, the one or more first features include one or both of a first task success feature and a first efficiency feature, and the one or more second features include one or both of a second task success feature and a second efficiency feature.

[0101] In some implementations, a method implemented by one or more processors is provided, the method including configuring a simulated environment for a robot simulator based on real environment state data based on one or more sensor-based observations of a real environment. The method further includes determining, with the robot simulator, a first candidate set of robot actions and a second candidate set of robot actions for controlling the real robot in the real environment in performing at least a portion of a robot task. Determining the first candidate set of robot actions includes utilizing simulated state data that simulates real state data that cannot be generated at all using any sensors in the real environment. The method further includes causing the real robot to implement the first candidate set of robot actions.

[0102] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0103] In some implementations, the simulated state data is generated from a simulated viewpoint to simulate real state data that cannot be generated at all using any sensors in the real environment, and no sensors in the real environment are capable of capturing real visual data from the corresponding real viewpoint in the real environment without human intervention.

[0104] In some implementations, the simulated state data is generated from simulated sensors of a particular type to simulate real state data that cannot be generated at all using any sensors in the real environment, where no sensors in the real environment are of that particular type.

[0105] Other implementations may include a non-transitory, computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described herein. Still other implementations may include one or more computers and / or one or more robotic systems including one or more processors operable to execute the stored instructions to perform methods such as one or more of the methods described herein. [Explanation of symbols]

[0106] 101 Task 110 Robot 111 Robot Vision Components 111A Visual Components 112 Pose Data and / or Object Identifier Data 112A Visual Data 112A~N Features 113 Base 114 Robot Arm 115 End Effector 117A Visual Data 120 Real Robot System 121 Configuration Engine 122 Reality State Data Engine 124 Control Engine 126 Comparison Engine 127 Reconfiguration signal 140 Robot Simulator 142 Configuration Engine 143 configured simulated environments 144 Simulated Action Engine 144A Determined sequence of robot actions 144A~N A series of candidate robot actions 144A1 Robot Action 1 144A2 Robot Action 2 144A3 Robot Action 3 145 Evaluation Module 146 Simulated Application Engines 147 Robot Actions 148 Simulated State Data Engine 149A Simulated State Data Instances 149A0 Simulated State Data Instances 149A1 Simulated State Data Instances 149A2 Simulated State Data Instances 149A3 Simulated State Data Instances 150 Comparison Engine 152 Object Model Database 154 Machine Learning Models 170 Ambient Vision Components 170A0 Reality State Data Instance 170A1 Reality State Data Instance 170A2 Reality State Data Instance 170A3 Reality State Data Instance 171A Visual Data 191 Table 192 Bowl 193 cups 194 Coffee Mug 195 Dishwasher 200 ways 520 Robot 540a~n Operating components 542a~m Sensor 560 Robot Control System 610 Computing Devices 612 Bus Subsystem 614 processor 616 Network Interface Subsystem 620 User Interface Output Device 622 User Interface Input Devices 624 Storage Subsystem 625 Memory Subsystem 626 File Storage Subsystem 630 RAM 632 ROM

Claims

1. 1. A method implemented by one or more processors, comprising: configuring a simulated environment of the robot simulator based on real environment state data that is based on one or more sensor-based observations of the real environment; determining, using the robot simulator, a set of robot actions to be used to control a real robot in the real environment when performing at least a portion of a robot task; applying the sequence of robot actions to a simulated robot of the robot simulator to generate a sequence of simulated state data instances, each of the simulated state data instances comprising, for a corresponding point in the sequence of robot actions: corresponding predicted environmental condition data for the corresponding points; corresponding predicted robot state data for the corresponding points; and controlling the real robot to perform a first portion of the sequence of robotic actions; During or after the step of controlling the real robot to perform a first portion of the sequence of robot actions, comparing a real state data instance with a corresponding one of the simulated state data instances for the corresponding point in the sequence of robot actions, the real state data instance being, for the corresponding point, updated real environment state data based on one or more updated sensor-based observations of the real environment; and real robot state data based on one or more sensor-based observations of the real robot; and and in response to the comparing step indicating a threshold similarity between the real state data instance and the corresponding one of the simulated state data instances for the corresponding point, controlling the real robot to perform a second portion of the sequence of robotic actions; The method includes:

2. In response to the comparing step not indicating the threshold similarity, The method of claim 1 , further comprising the step of ceasing control of the real robot based on the set of robot actions.

3. In response to the comparing step not indicating the threshold similarity, reconstructing the simulated environment based on updated real environment condition data; determining, using the robot simulator, an updated set of robot actions to be used to control the real robot in performing the robot task; applying the updated set of robot actions to the simulated robot of the robot simulator to generate an updated set of simulated state data instances, each of the updated simulated state data instances being updated for a corresponding updated point in the updated set of robot actions by: corresponding updated predicted environmental state data for the corresponding updated points; and real robot state data based on one or more sensor-based observations of the real robot; and and controlling the real robot to perform a first updated portion of the updated set of robot actions; comparing an updated real state data instance with a corresponding updated one of the updated simulated state data instances for the corresponding updated point in the updated sequence of robot actions; in response to the step of comparing the updated real state data instance with the corresponding updated one of the updated simulated state data instances indicating the threshold similarity, controlling the real robot to perform an updated second portion of the updated set of robot actions; 3. The method of claim 2, further comprising:

4. The method of claim 1 , comprising controlling the real robot to perform the second portion of the sequence of robot actions and to perform any additional portions of the sequence of robot actions.

5. Prior to the complete execution of said sequence of robotic actions, determining, using the robot simulator, a next sequence of robot actions to be used to control the real robot in performing another portion of the robot task; applying the next set of robot actions to a simulated robot of the robot simulator to generate a next set of simulated state data instances using the simulated robot of the robot simulator and the environment corresponding to a last state data instance of the simulated state data instances; and after a complete execution of said sequence of robotic actions, The method of claim 4 , further comprising controlling the real robot to perform the next sequence of robotic actions.

6. controlling the real robot to perform the next set of robotic actions, controlling the real robot to perform a first portion of the next sequence of robot actions; comparing a next real state data instance with a corresponding one of the next simulated state data instances; controlling the real robot to perform a second portion of the next sequence of robot actions in response to the comparing step indicating the threshold similarity; 6. The method of claim 5, comprising:

7. The method of claim 1 , wherein the one or more sensor-based observations include robot sensor-based observations based on one or more real sensors of the real robot.

8. 8. The method of claim 1, wherein the one or more sensor-based observations include environmental sensor-based observations based on one or more real sensors not coupled to the real robot but present in the real environment.

9. determining the set of robot actions to be used to control the real robot in performing at least a portion of the robot task, comprising: applying another set of robot actions to the robot simulator; comparing one or more further features from the step of applying the set of robot actions to the robot simulator with one or more features from the step of applying the set of robot actions to the robot simulator; determining, based on the comparing step of the one or more further features to the one or more further features, to utilize the set of robot actions instead of the another set of robot actions for use in controlling the real robot in performing at least a portion of the robotic task; 9. The method of any one of claims 1 to 8, comprising:

10. 10. The method of claim 9, wherein the one or more features include one or both of a task success feature and an efficiency feature, and the one or more other features include one or both of another task success feature and another efficiency feature.

11. 11. The method of claim 10, wherein the one or more features include the task success feature, the method further comprising generating the task success feature based on a final state of the robot simulator after applying the set of robot actions.

12. The one or more features include the efficiency feature, and the method further comprises: an amount of robot actions in the sequence of robot actions; and a distance traversed by one or more simulated components of the simulated robot when performing the set of robotic actions; and The method of claim 10 , further comprising generating the efficiency feature based on one or both of:

13. 13. The method of claim 1, wherein the step of comparing the real state data instance with the corresponding one of the simulated state data instances for the corresponding point is performed by one or more processors of the real robot.

14. 14. The method of claim 1, wherein the robot simulator is implemented on one or more remote processors that are remote from the real robot but in network communication with the real robot.

15. 1. A method implemented by one or more processors of a reality robot, comprising: receiving a sequence of robot actions and a sequence of simulated state data instances, each of the simulated state data instances representing, for a corresponding point in the sequence of robot actions: corresponding predicted environmental condition data for the corresponding points; corresponding predicted robot state data for the corresponding points; and controlling the real robot to perform a first portion of the sequence of robotic actions; During or after the step of controlling the real robot to perform a first portion of the sequence of robot actions, comparing a real state data instance with a corresponding one of the simulated state data instances for the corresponding point in the sequence of robot actions, the real state data instance being, for the corresponding point, updated real-world environment state data based on one or more updated sensor-based observations of the real-world environment; and real robot state data based on one or more sensor-based observations of the real robot; and and in response to the comparing step indicating a threshold similarity between the real state data instance and the corresponding one of the simulated state data instances for the corresponding point, controlling the real robot to perform a second portion of the sequence of robotic actions; The method includes:

16. In response to the comparing step not indicating the threshold similarity, The method of claim 15 , further comprising ceasing control of the real robot based on the set of robot actions.

17. In response to the comparing step not indicating the threshold similarity, sending the updated real environment state data to a robot simulator component; receiving, from the robot simulator component, an updated set of robot actions generated based on the updated real-environment state data and an updated set of simulated state data instances, each of the updated simulated state data instances being updated for a corresponding updated point in the updated set of robot actions by: corresponding updated predicted environmental state data for the corresponding updated points; and real robot state data based on one or more sensor-based observations of the real robot; and and controlling the real robot to perform a first updated portion of the updated set of robot actions; comparing an updated real state data instance with a corresponding updated one of the updated simulated state data instances for the corresponding updated point in the updated sequence of robot actions; in response to the step of comparing the updated real state data instance with the corresponding updated one of the updated simulated state data instances indicating the threshold similarity, controlling the real robot to perform an updated second portion of the updated set of robot actions; 17. The method of claim 16, further comprising:

18. The method of claim 17 , wherein the robot simulator component is implemented on one or more remote processors that are remote from the real robot but in network communication with the real robot.

19. 1. A method implemented by one or more processors, comprising: configuring a simulated environment for a robot simulator based on real environment state data based on one or more sensor-based observations of a real environment, wherein a first candidate set of robot actions and a second candidate set of robot actions are selected as inputs to the robot simulator for use in controlling a real robot in performing at least a portion of a robotic task in the real environment using a machine learning model, the first candidate set of robot actions and the second candidate set of robot actions having different probabilities of successfully completing at least a portion of the robotic task; applying the first candidate set of robot actions to a simulated robot of the robot simulator using the simulated environment after the configuring step to generate first simulated state data; applying the second candidate set of robot actions to the simulated robot in the robot simulator to generate second simulated state data using the simulated environment after the configuring step; determining, based on the first simulated state data and the second simulated state data, to utilize the first candidate set of robotic actions instead of the second candidate set of robotic actions for controlling the real robot in performing at least a portion of the robotic task; in response to a determination to utilize the first candidate set of robotic actions; causing the real robot to perform the first candidate set of robot actions; The method includes:

20. determining, based on the first simulated state data and the second simulated state data, to utilize the first candidate set of robotic actions instead of the second candidate set of robotic actions, generating one or more first characteristics based on the first simulated condition data; generating one or more second features based on the second simulated condition data; determining to utilize the first candidate set of robotic actions based on a comparison of the first feature and the second feature; 20. The method of claim 19, comprising:

21. 21. The method of claim 20, wherein the one or more first features include a first task success feature and / or a first efficiency feature, and the one or more second features include a second task success feature and / or a second efficiency feature.

22. 1. A method implemented by one or more processors, comprising: configuring a simulated environment of the robot simulator based on real environment state data that is based on one or more sensor-based observations of the real environment; determining, using a robot simulator, a first candidate set of robot actions and a second candidate set of robot actions for controlling a real robot in the real environment when performing at least a portion of a robotic task, wherein determining the first candidate set of robot actions includes utilizing simulated state data that simulates real state data that cannot be generated at all using any sensors in the real environment; causing the real robot to perform the first candidate set of robot actions; A method comprising:

23. 23. The method of claim 22, wherein the simulated state data is generated from a simulated viewpoint to simulate the real state data that cannot be generated at all using any sensors in the real environment, and no sensors in the real environment are capable of capturing real visual data from a corresponding real viewpoint in the real environment.

24. 24. A method according to claim 22 or claim 23, wherein the simulated state data is generated from simulated sensors of a particular type, thereby simulating the real state data that cannot be generated at all using any sensors in the real environment, and no sensors in the real environment are of the particular type.

25. 25. A system comprising one or more processors configured to execute instructions to perform the method of any one of claims 1 to 24.

26. 26. The system of claim 25, further comprising one or more environmental sensors that capture sensor-based observations of the real-world environment.

27. 27. The system of claim 25 or claim 26, further comprising a robotic actuator.

28. A computer program comprising instructions which, when executed by one or more processors of a computing system, cause the computing system to perform the method of any one of claims 1 to 24.

Images