Simulated drive robot control of real robot

Abstract

Robot simulators are actively utilized in the control of one or more real-world robots. The simulation environment of the robot simulator may be configured to reflect a real-world environment in which a real robot is currently arranged or will be arranged. The robot simulator may then be used to determine a sequence of robot actions used by the real-world robot (s) in performing at least a portion of the robot task. A sequence of robot actions may be applied to a simulation robot of a robot simulator to generate a sequence of expected simulation state data instances. A real robot may be controlled to implement a sequence of robot actions. Implementation of one or more of the robotic actions may depend on real state data instances having at least a threshold similarity to a corresponding one of the expected simulated state data instances.

Description

[0001] This application is a divisional application of the invention patent application with application date of December 28, 2021 and application number 202180088815.8. Background Art

[0002] It has been proposed to use a robot simulator to generate simulated training data for training a machine learning model (or models). Once trained based on the simulated training data, the machine learning model (or models) can be utilized during the perception, planning, and / or action phase (or phases) of robotic control of a real robot (or robots) (i.e., a non-simulated physical real-world robot (or robots)). However, utilizing the machine learning model (or models) of a real robot (or robots) in robotic control does not actively involve the robot simulator (or robots). Instead, the machine learning model (or models) is trained only based on simulated training data generated using the robot simulator (or robots). In addition, it has been proposed to use a robot simulator (or robots) to test / validate a robot control stack before implementing the control stack on a real robot. However, once the robot control stack is implemented on a real robot, the robot simulator is no longer actively utilized. Summary of the invention

[0003] Implementations disclosed herein involve actively utilizing a robot simulator in the control of one or more real-world robots. The robot simulator utilized in controlling a real-world robot (also referred to herein as a "real robot") may be implemented using hardware (e.g., a processor(s), RAM, and ROM) local to the real robot and / or may be implemented using hardware remote from the real-world robot but in network communication with the real-world robot.

[0004] In various implementations, the simulated environment of the robot simulator can be configured to reflect the real world environment (also referred to herein as the "real environment") in which the real robot is currently deployed or will be deployed. The simulated environment can be configured based on real environment state data, which is based on sensor-based observations (or multiple) of the environment. The real environment state data can specify identifiers of objects in the real environment and their postures in the real environment, and / or can include raw data from the observations (or multiples), which can be processed to identify the objects and their postures. The simulated environment can be configured based on identifiers of objects and their postures. For example, the identifiers of objects can be used to identify matching three-dimensional models of objects, and to locate objects in the simulated environment according to their corresponding postures.

[0005] Sensor-based observations (or multiple) may include robot sensor-based observations (or multiple), which come from vision components (or multiple) and / or other sensors (or multiple) integrated with the real robot (or multiple) to be controlled. In some of those implementations, the real environment state data includes real environment state data generated by the real robot (or multiple) using a perception system and provided to the robot simulator by the real robot (or multiple). The real environment state data can be generated based on robot sensor-based observations (or multiple) and / or based on environmental sensor observations (or multiple) (described below). Such perception data may include, for example, corresponding postures and corresponding object identifiers for each of a plurality of objects in the real environment. The corresponding object identifier may be a higher-level category, such as a coffee cup, a cup, or a bowl, or may be a more fine-grained identifier, such as an identifier for a specific coffee cup or a specific cup. Alternatively, a more fine-grained identifier of an object may be determined based on a QR code or other coded identifier printed on or attached to the object. The sensor-based observation(s) may additionally or alternatively include environmental sensor-based observation(s) from the vision component(s) and / or other sensor(s) in the real environment but not integrated with the real robot(s) to be controlled. For example, the environmental sensor-based observation(s) may include observation(s) from an independent stereo camera or other vision component.

[0006] Once the simulated environment is configured to reflect the real environment, the robot simulator can be used to determine a sequence of robot actions used by the real-world robot in performing at least a portion of a robot task. The robot task can be a robot task specified by a higher-level planning component of the robot simulator or by the real-world robot (or multiple), or can be a robot task specified based on a user interface input. As a non-limiting example, the robot task can include grasping an object and placing the object in a container, and a portion of the task can include grasping the object. As another non-limiting example, the robot task can include loading a dishwasher, and a portion of the robot task can include repositioning an object, grasping an object to be placed in the dishwasher, or placing an already grasped object in the dishwasher.

[0007] In determining a sequence of robot actions, the robot simulator may determine the sequence of robot actions in view of a simulated environment of the robot simulator and using one or more machine learning models and / or rules accessible to the robot simulator. For example, simulated state data from the simulated environment may be applied to the machine learning model(s) and / or rule(s) to determine that the sequence of robot actions is a candidate sequence of robot actions. For example, the applied simulated state data may include simulated images(s) of the simulated environment, such as simulated images(s) rendered from the viewpoint of a simulated visual component of a simulated robot of the robot simulator (which simulates a real-world robot).

[0008] In some implementations, the simulated state data of the application may include simulated state data that simulates real state data, which real state data cannot be captured by any robot sensor (or multiple) and / or environmental sensor (or multiple) in the real world. For example, the simulated state data may include simulated / synthetic visual data captured from a specific viewpoint (e.g., "head"), and in the absence of manual human intervention, there may not be any environmental visual component or robot visual component that can capture real visual data from the corresponding viewpoint in the real world. It should be noted that even if there is no real-world visual component that can capture real visual data from the corresponding viewpoint, in view of the simulated environment based on the real world and configured with a three-dimensional object model, the simulated visual data will still reflect the real world. As another example, the simulated state data may include simulated LIDAR data, and there may not be any LIDAR visual component in the real world. In these and other ways, a robot simulator can be used to generate simulated state data that lacks any real-world counterparts, and utilize such simulated state data in determining a robot action sequence. Utilizing such simulated state data can lead to more robust and / or more accurate determination of robot action sequences in various scenarios. As one example, it may enable the use of additional and / or alternative machine learning models (or models), such as those customized for such simulation state data. As another example, it may enable the application of multiple different simulation state data instances to machine learning models (or models) and / or rules (or models) to enable the determination of action sequences based collectively on multiple different simulation state data instances.

[0009] In some additional or alternative implementations, the simulated state data of the application may include simulated state data that can be captured by one or more robot sensors (or multiples) and / or one or more environmental sensors (or multiples) in the real world, but can only be captured when consuming power while traversing the posture (or multiples) of those real-world sensors (or multiples) and / or repositioning environmental objects (or multiples). For example, the simulated state data of the application may include a simulated 3D point cloud of an object. The visual component (or multiples) of the real robot may be able to capture data to generate a corresponding real 3D point cloud, but only when the real robot consumes a lot of power when traversing and / or lifting and rotating the object completely around the object. On the other hand, the robot simulator can generate a simulated 3D point cloud of the object with less power consumption. As another example, the simulated state data instance of the application may include multiple simulated images, each simulated image from a different viewpoint. The visual component (or multiples) of the real robot may be able to capture corresponding real images from corresponding different viewpoints, but only when the real robot consumes a lot of power when traversing the visual component (or multiples) to different viewpoints. On the other hand, the robot simulator is able to render simulation images from different viewpoints with less power consumption.

[0010] In some implementations, the candidate sequence of robot actions may be applied to a simulated robot and a simulated environment of a robot simulator, and further simulation state data from during and / or after the application may be utilized to determine whether the candidate sequence of robot actions should be determined as the sequence of robot actions to be provided for use by the real-world robot(s). Alternatively, multiple candidate sequences of robot actions may be considered, each candidate sequence being applied to the robot simulator, and the corresponding further simulation data for each candidate sequence being used to determine which should be the determined sequence of robot actions to be provided for use by the real-world robot(s).

[0011] As an example, one or more features of the further simulated state data may be analyzed to determine whether the candidate sequence of robot actions should be determined as a robot action sequence to be provided for use by the real-world robot (or multiple). For example, the feature (or multiple) may include a task success feature indicating whether a portion of the task is successful (e.g., if a portion of the task is to grasp an object, whether the simulated state data indicates that the simulated representation of the object is grasped by applying the robot action sequence). In this case, the candidate sequence of robot actions may be determined as a robot action sequence to be provided for use only when the task success feature indicates that a portion of the task is successful. In addition, for example, the feature (or multiple) may include an efficiency feature (or multiple), such as an efficiency feature indicating the number of robot actions in the sequence and / or the distance traversed by the component (or multiple) of the robot when executing the robot action sequence (e.g., the cumulative distance in the joint space traversed by the joints of the robot and / or the distance in the task space traversed by the end effector of the robot). In this case, the candidate sequence of robot actions may be determined as a robot action sequence to be provided for use only when the efficiency feature (or multiple) satisfies an absolute and / or relative threshold (or multiple) such as a relative threshold based on the efficiency features of the alternative candidate action sequence being considered. Note that for some robot action sequences, the efficiency characteristics cannot be known without applying the efficiency characteristics in the simulation. For example, a machine learning model can be used to determine a robot action sequence, and determining the downstream action(s) in the sequence can depend on processing simulation state data that is affected by the execution of the upstream action(s) in the sequence.

[0012] By analyzing state data from further simulations during and / or after applying a robot action sequence in a simulation, and only providing a robot action sequence (or sequences) for implementation by a real robot (or sequences) if the analysis satisfies certain conditions (or conditions), various technical benefits can be achieved. As an example, when the task success feature (or conditions) is considered, the likelihood that the real robot will successfully perform the portion of the task increases. This can prevent energy from being wasted when controlling the real world robot (or sequences) to perform actions that do not result in the successful execution of the portion of the task and / or can prevent unnecessary wear and tear on the real world robot (or sequences), which would otherwise be caused by performing actions that do not result in the successful execution of the portion of the task. As another example, when the efficiency feature (or conditions) is considered, the real robot can perform a portion of the task in a shorter duration and / or with fewer component moves. As yet another example, a robot simulator can apply a robot action sequence in a simulation with less wall clock time than a real robot. This enables an unsuccessful or otherwise suboptimal sequence of robot actions to be determined more quickly, in a shorter duration than a real robot is controlled to achieve a suboptimal sequence of robot actions, and the resulting real-world state data is analyzed to determine that the sequence of robot actions is suboptimal.

[0013] It should be noted that these and other technical benefits cannot be achieved in various implementations without the use of a robot simulator as described herein. As an example, assume that the machine learning model utilized indicates a first sequence of robot actions as being most likely to successfully perform a portion of a task, and indicates a second sequence of robot actions as being second most likely to successfully perform a portion of the task. Further assume that by applying the first sequence and the second sequence in a simulation, and analyzing corresponding further simulation state data, it is determined that the second sequence results in successful execution of a portion of the task in the simulation, while the first sequence does not. As a result, the second sequence of robot actions can be provided for implementation by a real robot. Without the use of a robot simulator, the unsuccessful first sequence of robot actions would be selected and implemented because the machine learning model indicated the first sequence of robot actions as being more likely to succeed.

[0014] After determining the sequence of robot actions using the robot simulator, the real robot can be controlled to implement the sequence of robot actions. For example, each robot action can directly or indirectly indicate a corresponding control command (or multiple) to be implemented at a corresponding time step, and the corresponding control command to be implemented in sequence and at the corresponding time step. Implementing the control command can include sending the control command to the corresponding actuator (or multiple) of the real robot.

[0015] As described above, in some implementations, the robot action sequence is applied in a robot simulator before the sequence is implemented by a real robot. In some of those implementations, during the application of the robot action sequence, a sequence of simulated state data instances is generated based on simulated data from the robot simulator. For corresponding points in the robot action sequence, each simulated state data instance defines corresponding expected environmental state data for the corresponding point and / or corresponding expected robot state data for the corresponding point. In those implementations, during the implementation of the robot action sequence, each of the sequence of simulated state data instances can be compared with a corresponding real state data instance. Each real state data instance defines, for a corresponding point in the robot action sequence: a real environmental state, based on one or more updated sensor-based observations of the real environment (e.g., based on robot sensors and / or based on environmental sensors); and / or real robot state data, based on one or more sensor-based observations of the real robot. Note that the simulated state data instances can each be captured from the perspective of a simulated sensor (or sensors) (e.g., a simulated vision component) corresponding to a real world sensor (or sensors) (robot and / or environment) to achieve the comparison described herein.

[0016] Each simulated state data instance can be compared to a corresponding real state data instance to determine whether a threshold similarity exists between them. If a threshold similarity exists, this can indicate that implementation of the robot action on the real robot affects the real environment in a manner consistent with its effect on the simulated environment when implemented in the simulation. This can additionally or alternatively indicate that the real environment is in a state corresponding to the state of the simulated environment. For example, this can indicate that no unexpected objects have been introduced into the environment and / or expected objects have not been removed from the environment. Furthermore, this can additionally or alternatively indicate that implementation of the robot action on the real robot affects the real robot in a manner consistent with its effect on the simulated robot when implemented in the simulation.

[0017] The simulated state data instance of a point can be compared with the corresponding real state data instance of the point, and compared before the robot action (or multiple) of the sequence after the point is implemented on the real robot. The implementation of the robot action after the point can depend on the comparison indicating the threshold similarity. If the comparison indicates that there is no threshold similarity, the control of the real robot based on the robot action sequence can be stopped. This can prevent the real robot from continuing to implement the robot action of the sequence when the environmental conditions have changed. Therefore, energy is not wasted to continue to control the real robot and / or avoid damage to the real robot and / or environmental objects. Optionally, when the control of the real robot based on the robot action sequence is stopped, the simulated environment of the robot simulator is reconfigured using the updated real environment state data, and the robot simulator is used to determine the updated robot action sequence to be implemented based on the reconfigured simulated environment. Then the updated robot action sequence can be used to control the real robot. In addition, the updated sequence of the simulated state data instance can be provided together with the updated sequence of the robot action to enable the comparison with the corresponding real state data instance again, so as to ensure the threshold similarity between the two before the updated sequence is fully implemented.

[0018] As a specific example, each simulated state data instance may include corresponding expected environmental state data based on corresponding apparent images rendered from the viewpoint of the simulated visual component of the simulated robot (corresponding to the real visual component of the real robot). For example, the environmental state data of the simulated state data instance may include a segmentation mask of the corresponding rendered image and / or an embedding of the corresponding rendered image (e.g., an embedding generated based on processing the rendered image using an encoder). Therefore, the environmental state data can provide an indication of the expected environmental state at the corresponding point in the robot action sequence. In addition, each real state data instance may include corresponding real environmental state data based on the corresponding real image from the real visual component of the robot. For example, the real environmental state data of the real state data instance may include a segmentation mask of the corresponding real image and / or an embedding of the corresponding real image (e.g., an embedding generated based on processing the real image using an encoder). Continuing with this example, before implementing the robot action (or multiple) of the sequence corresponding to the points of those instances, the instance of the expected environmental state data can be compared with the corresponding instance of the real environmental state data. For example, an expected segmentation mask can be compared to a true segmentation mask (e.g., by determining a difference(s) between the size and / or position(s) of the segmentation masks) to determine if a threshold similarity exists between the two. Additionally, for example, an expected embedding can be compared to a true embedding (e.g., by determining a distance in embedding space between the two) to determine if a threshold similarity exists between the two. Utilizing segmentation masks, embeddings, and / or other non-pixel-level abstractions of visual data can be more robust to realistic gaps that may exist between rendered visual data from a simulation and real visual data. For example, utilizing such abstractions can be more robust to realistic gaps than directly comparing pixels of the visual data.

[0019] The comparison between the simulated state data instances and the real state data instances may be performed by the robot simulator (e.g., based on the real state data instances provided by the real robot) and / or by the real robot (e.g., based on the simulated state data instances provided by the robot simulator).

[0020] In various implementations, the sequence of robot actions determined by the robot simulator and / or implemented by the real robot at a given time is used for only a portion of the task implemented by the real robot. In those implementations, the robot simulator can be used to determine the next sequence of robot actions to be implemented to perform another portion of the task, as well as the next sequence of robot actions to be implemented by the real robot after implementing the sequence of robot actions. This process can continue until all sequences of robot actions required to complete the task have been determined and implemented. Thus, the robot simulator can effectively be in the control loop of the real robot, continuously providing updated robot action sequences for use by the real robot.

[0021] In some of those implementations, the robot simulator may determine the next robot action sequence before the real robot fully implements the robot action sequence. In various cases, the next robot action sequence may be determined based on a simulation environment performed in the robot simulator. For example, for robot actions implemented to date, when a comparison between a simulated state data instance and a real state data instance meets a threshold similarity, the simulation environment in which it has been performed may be utilized. In various other cases, the next robot action sequence may be determined based on a reconfigured simulation environment that has been reconfigured based on updated real environment state data. For example, as described above, in response to determining that a threshold similarity is not met, the simulation environment may be reconfigured.

[0022] Various technical benefits that can be achieved according to the techniques described herein have been mentioned above. In addition to or in lieu of such benefits, the techniques described herein can achieve more robust planning of robot actions and / or more energy-efficient utilization of real robots. For example, in an implementation where a robot simulator is implemented by hardware away from the real robot, planning can be performed by the robot simulator to implement robot actions on the real robot. This can enable more robust planning on real robots with limited hardware capabilities. For example, a robot simulator can utilize a machine learning model (or models) in planning, which requires a GPU (or models) and / or CPU (or models) that are available to the robot simulator but not present on the real robot. Executing planning on a remote robot simulator can additionally or alternatively preserve limited battery life on a battery-powered real robot because the real robot does not need to utilize processor cycles and / or other power-intensive resources for planning.

[0023] The above description is provided only as an overview of some implementations disclosed herein. These and other implementations are described in more detail herein, including in the detailed description, claims, and accompanying papers.

[0024] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 An example environment is shown in which implementations disclosed herein may be implemented.

[0026] Figure 2is a flow chart illustrating an example method of simulation-driven robotic control of a real robot according to various implementations disclosed herein.

[0027] Figure 3 Show Figure 1 Examples of various components of and the interactions that can occur between the various components in simulated driven robot control of a real robot.

[0028] Figure 4A An example is shown of comparing real state data instances with simulated state data instances at a first frequency when determining whether to continue controlling a real robot based on a robot motion sequence determined using a robot simulator.

[0029] Figure 4B An example is shown of comparing real state data instances with simulated state data instances at a second frequency when determining whether to continue controlling the real robot based on a robot motion sequence determined using a robot simulator.

[0030] Figure 5 An example architecture of a robot is schematically depicted.

[0031] Figure 6 An example architecture of a computer system is schematically depicted. DETAILED DESCRIPTION

[0032] Figure 1 An example environment in which the implementations disclosed herein may be implemented is shown. The example environment includes a robot 110 and a dishwasher 195. The example environment also includes a bowl 192, a cup 193, and a coffee cup 194 placed on a table 191. Figure 1 An example environment of can be a real-world environment in which the robot 110 can utilize the techniques described herein to perform various tasks, such as those involving manipulation of one or more of the objects 191-195. For example, the tasks can include pushing, grabbing, or otherwise manipulating one or more of the objects 191-195. As another example, the tasks can include more complex tasks, such as loading each of the objects 191-194 into the dishwasher 195.

[0033] Figure 1 The robot 110 shown is a specific mobile robot. However, additional and / or alternative robots may be used with the technology disclosed herein, such as in one or more aspects of the present invention. Figure 1 The robot 110 shown is different from additional robots. For example, a mobile forklift robot, a robotic arm, an unmanned aerial vehicle ("UAV"), and / or a humanoid robot may be utilized in place of or in addition to the robot 110 in the techniques described herein.

[0034] The robot 110 includes a base 113 having wheels provided on opposite sides thereof for movement of the robot 110. The base 113 may include, for example, one or more motors for driving the wheels of the robot 110 to achieve a desired direction, speed and / or acceleration of movement of the robot 110.

[0035] The robot 110 also includes a vision component 111 that can generate images related to the shape, color, depth, and / or other features of objects in the line of sight of the vision component 111A. The vision component 111 can be, for example, a thematic camera (e.g., generating a 2D RGB image), a stereo camera (e.g., generating a 2.5D RGB-D image), and / or a laser scanner (e.g., a LIDAR that generates a 2.5D depth (D) image or point cloud). The robot 110 also includes a robot arm 114 having an end effector 115, which takes the form of a gripper with two opposing "fingers" or "toes".

[0036] The robot 110 also includes one or more processors that, for example, implement robot actions (e.g., by providing corresponding control commands to actuators and / or other operating components thereof); determine the pose and / or identifier of an object based on the visual component data; and optionally implement a robot simulator. For example, the one or more processors of the robot 110 may implement Figure 2 All or aspects of method 200. Additional descriptions of some examples of the structure and functionality of various robots are provided herein.

[0037] Figure 1 The example environment also includes one or more environmental vision components 170 that are not coupled to or otherwise integrated with the robot 110. In addition, the example environment includes a real robot system 120, a robot simulator 140, an object model (or multiple) database 152, and one or more machine learning (ML) models 154.

[0038] The real robot system 120 is implemented by one or more processors of the robot 110. Figure 1 , a real robotic system 120 is shown with a real state data engine 122, a control engine 124, and an optional comparison engine 126. In various implementations, additional engines may be included in the real robotic system 120.

[0039] The real state data engine 122 processes sensor-based observations to generate the real environment state data described herein and / or to generate the real state data instances described herein. When generating the real environment state data and / or the real state data instances, the real state data engine 122 may utilize one or more locally stored machine learning models (not shown) to process sensor-based observations.

[0040] As one example, the real state data engine 122 may process visual data (from the vision component 111 and / or from the vision component(s) 170) using machine learning models(s) to generate real environment state data that includes pose(s) and / or classification(s) of the object(s) in the environment of the robot 110. As another example, the real state data engine 122 may process visual data using machine learning models(s) to generate pose(s) of the object(s) in the environment, and may process the visual data separately to determine granular identifier(s) of the object(s), such as an identifier(s) reflected in a tag attached to the object(s) and captured in the visual data. The real environment state data may be provided by the real robot system 120 to the robot simulator 140 and used by the configuration engine 142 of the robot simulator to configure (i.e., initially configure or reconfigure) the simulated environment of the robot simulator 140.

[0041] As another example, the real state data engine 122 may use a machine learning model(s) to process visual data (from the vision component 111 and / or from the vision component(s) 170) to generate real state data instances, each of which includes a corresponding segmentation mask and / or visual data embedding. The comparison engine 126 of the real robotic system 120 or the comparison engine 150 of the robotic simulator 140 may use the real state data instances to compare the real state data instances with corresponding simulated state data instances to determine whether a threshold similarity exists between the real state data instances and the simulated state data instances. In various implementations, the real state data engine 122 may include Figure 2 The perception system and / or implementation of method 200 is block 252A.

[0042] The control engine 124 implements the robot action determined by the simulated action engine 144 of the robot simulator 140. Implementing the robot action may include sending a corresponding control command(s) to a corresponding actuator(s) of the robot 110, wherein the control command(s) is directly or indirectly indicated by the robot action.

[0043] During implementation of a sequence of robotic actions by the control engine 124, the comparison engine 126 may compare real state data instances (from the real state data engine 122) with simulated state data instances (from the simulated state data engine 148 of the robot simulator 140) at corresponding points during the implementation to determine whether a threshold similarity exists between the compared real state data instances and the simulated state data instances. The comparison engine 126 may communicate with the control engine 124 and may cause the control engine 124 to continue implementing the robotic actions provided by the robot simulator as long as the comparison indicates a threshold similarity. When one or more comparisons do not indicate a threshold similarity, the comparison engine 126 may additionally or alternatively cause the control engine 124 to stop implementing the robotic actions provided by the robot simulator. For example, when a single comparison does not indicate a threshold similarity, or when two or more comparisons do not indicate a threshold similarity, such as N of N sequential comparisons or the last N+M comparisons, where N is an integer greater than 1 and M is an integer, the comparison engine 126 may cause the control engine 124 to stop implementing. In various implementations, the comparison engine 126 may implement Figure 2 262 and / or 264 of method 200 .

[0044] The robot simulator 140 is implemented by one or more hardware components (e.g., processor(s)) of the robot 110, or by one or more computing devices that are separate from the robot 110 but in network communication with the robot 110. For example, the computing device(s) may be in network communication with the robot 110 via a local area network and / or a wide area network. The robot simulator 140 is used to simulate an environment including corresponding environmental objects(s), simulate robots (e.g., simulated robot 110 and / or other robots) operating in the simulated environment, simulate the response of the simulated robot in response to virtual implementations of various simulated robot actions, and simulate the interaction between the simulated robot and the simulated environment objects in response to the simulated robot actions. Various simulators may be utilized, such as physics engines that simulate collision detection, soft and rigid body dynamics, etc. A non-limiting example of such a simulator is the BULLET physics engine.

[0045] The robot simulator 140 includes a configuration engine 142 , a simulation action engine 144 with an optional evaluation module 145 , a simulation application engine 146 , a simulation state data engine 148 , and a comparison engine 150 .

[0046] The configuration engine 121 configures various parameters of the simulation environment for the robot simulator 140 and / or configures various parameters about the simulated robot of the robot simulator 140. In configuring the simulation environment, the configuration engine 121 utilizes real environment state data. In some implementations, the real environment state data may include poses (or multiples) and / or object identifiers (or multiples) provided by the real state data engine 122 of the real robot system 120 and / or by the real state data engines of other robots (or multiples) (not shown) in the environment. In some implementations, the real environment state data additionally or alternatively includes raw sensor-based observations (or multiples), such as visual data from the vision component 111 of the robot 110 and / or visual data from the environment vision component (or multiples) 170. In those implementations, the configuration engine 121 may include a perception system and may generate, for example, poses (or multiples) and / or object identifiers (or multiples) of objects based on sensor-based observations.

[0047] When configuring the simulation environment, the configuration engine 142 can access the object model (or multiple) database 152 to identify the three-dimensional (3D) model (or multiple) for use in the simulation environment. For example, the object model (or multiple) database 152 may include accurate 3D models for some or all objects in the environment, and the configuration engine 142 may use, for example, object identifiers (or multiple) to match the 3D model with the corresponding object. For example, the bowl 192 may include an identifier (e.g., a QR code or other identifier) ​​thereon, and the real environment state data may reflect the identifier and the posture of the identifier. The configuration engine 142 can use the identifier to retrieve the 3D model of the bowl 192 from the object model (or multiple) database 152 (e.g., it can be indexed based on the identifier), and the 3D model can be placed in the simulation environment based on the posture.

[0048] In some implementations, configuring the simulation environment may also include configuring the simulated robot to reflect the robot 110 and / or reflect its current state in the real environment. In such an implementation, the real state data engine 122 may provide real environment state data, which includes an identifier of the robot 110 and / or the current state of the robot 110 in the real environment, such as the current posture of each joint of the robot. When configuring the simulated robot, the configuration engine 142 may utilize a robot model. The robot model may define the physical structure of the simulated robot, as well as the constraints of the simulated robot, the dynamic model of the simulated robot, and / or other parameters (or multiple) of the simulated robot. In various implementations, the configuration engine 121 may implement Figure 2 All or aspects of blocks 252 and / or 272 of method 200.

[0049] The simulation action engine 144 can determine a sequence of robot actions used by the robot 110 (and / or other robots (or multiple) not shown) in performing at least a portion of a robot task based on the simulation environment configured by the configuration engine 142. In determining the sequence of robot actions, the simulation action engine 144 can determine the sequence of robot actions in view of the simulation environment and using one or more machine learning models 154 and / or rules accessible to the robot simulator. As an example, the machine learning model (or multiple) 154 may include a policy model trained using reinforcement or imitation learning, and the simulation action engine 144 can determine the sequence of robot actions by applying simulation data to the policy model to generate a probability distribution of robot actions and selecting a sequence of robot actions based on the probability distribution. Note that when each next robot action is selected, the simulation data can reflect a new state of the simulation environment and / or the simulated robot, where the new state reflects the roll out of the previous robot action in the simulation (e.g., rolled out by the simulation application engine 146). As another example, the machine learning model(s) 154 may include a machine learning model that may be used at each time step to process current simulated visual data and corresponding candidate robot actions to generate a value reflecting the likelihood that implementing the candidate robot action will result in successful performance of at least a portion of the task. Multiple candidate robot actions may be considered by the simulated action engine 144 at each time step, and one candidate robot action may be selected for inclusion in the sequence of robot actions based on the generated values. Note that as each next robot action is selected, the simulation data may reflect a new state of the simulation environment and / or the simulated robot, wherein the new state reflects the rollout of the previous robot action in the simulation.

[0050] In some implementations, the simulation action engine 144 generates multiple candidate sequences of robot actions and determines which should be the determined robot action sequence to be provided for use by the robot 110. In some versions of these implementations, the evaluation module 145 of the simulation action engine 144 is used to evaluate the candidate sequences of robot actions and determine, based on the evaluation, that only one of the candidate sequences is provided for use by the robot 110. In some versions of those implementations, the simulation application engine 146 can apply each candidate robot action to a simulated robot of a robot simulator in a simulation environment. In addition, the simulation state data engine 148 can generate simulation state data during and / or after application, and the evaluation module 145 can utilize the simulation state data when evaluating one or more candidate sequences.

[0051] As an example, the evaluation module 145 can utilize the simulation state data of the sequence to generate a task success feature that indicates whether a portion of the task intended to be achieved by the sequence was successful. For example, if a portion of the task is to grasp an object, the evaluation module 145 can generate the task success feature based on whether the simulation state data indicates that a simulated representation of the object was grasped by applying the robot action sequence. In such an example, the evaluation module 145 can determine that the sequence is a sequence to be provided for use by the robot 110 only if the task success feature indicates that a portion of the task was successful.

[0052] As an example, the evaluation module 145 may utilize the simulated state data of the sequence to additionally or alternatively generate an efficiency characteristic(s) indicating the number of robot actions in the sequence and / or the distance traversed by the component(s) of the robot when executing the sequence. The distance may be, for example, the cumulative distance in joint space traversed by the joint(s) of the simulated robot and / or the distance in task space traversed by the end effector of the simulated robot. In such an example, the evaluation module 145 may determine that the sequence is to be provided for use by the robot 110 only if the efficiency characteristic(s) satisfies the absolute and / or relative threshold(s). In various implementations, the simulated motion engine 144 may implement Figure 2 All or aspects of block 256 of method 200 may be implemented.

[0053] The simulation application engine 146 can apply a series of robot actions to the simulated robot in the simulated environment. This causes the simulated robot to move according to the robot actions, and the simulated environment to be affected by at least some of the robot actions. As described above, the simulation application engine 146 can optionally apply multiple different candidate sequences of robot actions to, for example, generate simulation state data for use by the evaluation module 145 in evaluating the multiple different candidate sequences. In those cases, the simulation application engine 146 can apply each of the different candidate sequences with the same starting environment and robot state, but for each corresponding applied robot action and roll out the simulation separately according to it. In various implementations, the simulation application engine 146 can implement Figure 2 All or aspects of block 256 of method 200 may be implemented.

[0054] The simulation state data engine 148 can generate simulation state data for application of robot actions by the simulation application engine 146. The generated simulation state data can include data utilization of the evaluation module 145 and / or can include simulation state data instances for utilization by the comparison engine 126 and / or by the comparison engine 150. For example, for a sequence of robot actions applied by the simulation application engine 146, the simulation state data engine 148 can generate a sequence of simulation state data instances based on simulation data from a robot simulator during application of the robot action sequence. For example, the simulation state data engine 148 can generate an initial simulation state data instance before applying any robot action of the sequence, and can generate simulation state data instances after applying each robot action, after applying every other robot action, after applying every third robot action, or at other frequencies. For corresponding points in the robot action sequence, each simulation state data instance defines corresponding expected environmental state data for the corresponding point and / or corresponding expected robot state data for the corresponding point. In various implementations, the simulation state data engine 148 can implement Figure 2 All or aspects of block 256A and / or block 258 of method 200.

[0055] The comparison engine 150 may share one or more aspects with the comparison engine 126 of the real robot system 120 described above. In various implementations, only one of the comparison engine 126 and the comparison engine 150 is provided, and the other is omitted. In an implementation including the comparison engine 150, real state data instances may be obtained from the real state data engine 122 and compared with the simulated state data instances generated by the simulated state data engine 148. In an implementation including the comparison engine 126, simulated state data instances may be obtained from the simulated state data engine 148 and compared with the real state data instances generated by the real state data engine 122. In various implementations, the comparison engine 150 may implement Figure 2 All or aspects of block 262 and / or block 264 of method 200.

[0056] Below about Figure 3 and Figure 4 describes Figure 1 The following sections describe the various components of the and the interactions that can occur between them. However, first, a description of Figure 2Method 200 of the present invention illustrates an example method of simulated driven robot control of a real robot according to various implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include one or more components (e.g., processor(s)) of a robot and / or a computing device(s) in communication with a robot network. In addition, while the operations of method 200 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0057] At block 252, the system identifies real environment state data. The real environment state data is based on sensor-based observation(s) of the real environment. The sensor-based observation(s) are from real sensors(s) in the real environment, such as sensors(s) of a real robot(s) in the real environment and / or sensors(s) in the real environment but not part of the real robot(s). The real environment state data may include, for example, pose(s) of a real object(s) in the real environment, object identifier(s) of the real object(s), and / or the sensor-based observation(s) themselves.

[0058] Block 252 may optionally include subblock 252A, in which the system generates real environment state data based on sensor-based observations (or multiple) of the real environment. For example, the system may generate poses (or multiples) of real objects (or multiples) in the real environment and / or object identifiers (or multiples) of the real object (or multiples) by processing the sensor-based observations (or multiples). For example, when generating object identifiers and / or poses, the system may utilize one or more machine learning models to process the sensor-based observations using a perception system.

[0059] At block 254, the system configures the simulated environment of the robot simulator based on the real environment state data identified at block 252. The system configures the simulated environment to reflect the real environment as closely as possible based on the real environment state data.

[0060] At block 256, the system determines a sequence of robot actions for a robot task using the robot simulator. The robot task may be a task to be performed by a real robot (or multiple) currently in the real environment, or may be a task to be performed at a future time (e.g., within 5 minutes) when the real robot (or multiple) is in the real environment. The real environment reflected in the simulated environment may be, for example, a portion of a room, an entire room, multiple rooms, an entire building, and / or other defined spaces within the world.

[0061] Block 256 may optionally include subblock 256A, in which the system generates a plurality of different robot motion sequences, applies each of the plurality of sequences individually to the robot simulator in its current configuration (the initial configuration in the initial iteration of subblock 256A), and determines the sequence based on one or more features from the application. The feature(s) of the sequence may be determined based on simulation data from applying the sequence, and may include, for example, a task success feature and / or an efficiency feature(s).

[0062] At block 258, the system generates a simulation state data instance for the sequence of robot actions determined at block 256. Each simulation state data instance defines corresponding expected environment state data and / or robot state data for a corresponding point in the sequence. The system generates the simulation state data instance for the point based on the simulation state data from the point in the sequence. For example, a given simulation state data instance after implementing a second robot action of the sequence can be based on a state of the simulated environment and / or a state of the simulated robot after implementing the second robot action. In some implementations, when block 256 includes sub-block 256A, block 258 can include generating the simulation state data instance based on the application of block 256A to the determined sequence.

[0063] At block 260, the system controls the real robot to implement one or more subsequent robot actions in the robot action sequence determined at block 256. In an initial iteration of the sequence, this may include implementing an initial robot action and optionally one or more subsequent robot actions. Figure 2 , but it should be noted that in some implementations, prior to the initial iteration of the sequence, the system may compare the current real state data instance with a simulated state data instance that reflects the expected environment and / or robot state prior to implementing the sequence. This can ensure that the real robot state and / or real environment state has not changed significantly relative to the initial simulated robot state and / or initial simulated environment state on which the sequence was based.

[0064] At block 262, the system compares the real state data instance(s) to the corresponding simulated state data instance(s) generated at block 258. Each of the real state data instance(s) was generated at a corresponding point after or during performance of one of the robot actions(s) in the most recent iteration of block 260.

[0065] At block 264 , the system determines whether the comparison of block 262 indicates a threshold similarity between the actual state data instance(s) and the simulated state data instance(s) compared at block 262 .

[0066] At block 264, if the system determines that a threshold similarity is indicated, the system proceeds to block 266. At block 266, the system determines whether there are one or more remaining robot actions in the sequence. If so, the system returns to block 260 to control the real robot to perform the next robot action in the sequence. If at block 266, the system determines that there are no remaining robot actions in the sequence, the system proceeds to block 268.

[0067] At box 268, the system determines whether there are additional robot actions to complete the task (or multiple) performed by the real robot. If not, the system proceeds to box 270, and the method 200 ends for the task (but can be started again for another task). If right, the system returns to box 256 and determines the next robot action sequence for the robot task using the robot simulator. Note that in some implementations, the next iteration of box 256 can be performed in whole or in part while still performing iterations of box 260 based on the robot actions of the sequence determined in the immediately previous iteration of box 256. It should also be noted that in some implementations, when box 256 is executed again before execution without encountering box 272, the robot simulator can be in a state consistent with its state after applying the most recent robot action sequence in the robot simulator. In some other implementations, when box 256 is executed again before execution without encountering box 272, the robot simulator can be in a state that is updated based on updated real environment state data from the real robot application during or after the most recent robot action sequence (e.g., the pose (or poses) of the simulated object can be slightly updated based on the pose (or poses) from the updated real environment state data).

[0068] At block 264, if the system determines that a threshold similarity is not indicated, the system proceeds to block 272. At block 272, the system reconfigures the simulated environment based on the updated real environment state data. The updated real environment state data may be based on sensor-based observations of the real environment after the most recent iteration of block 260. Block 272 may share one or more aspects with block 254, but is based on updated real environment state data. After executing block 272, the system may return to block 256 and determine a new sequence of robot actions using the robot simulator and the reconfigured simulated environment.

[0069] Now go to Figure 3 , showing Figure 1 The various components of the system and example interactions that can occur between the various components in a simulated driven robotic control of a real robot. Figure 3In the embodiment, the robot vision component (or multiple) 111 generates visual data 112A and provides the visual data 112A to the real state data engine 122 and / or the configuration engine 142. In addition, the environmental vision component (or multiple) 170 generates visual data 171A and provides the visual data 171A to the configuration engine 142. The real state data engine 122 can process the visual data 112A to generate a real robot 110 ( Figure 1 )'s real environment. The posture data and / or object identifier data 112 are provided to the configuration engine 142.

[0070] Configuration engine 142 utilizes visual data 112A, visual data 171A, and / or gesture data and / or object identifier data 112 in the generated configured simulation environment 143. Configuration engine 142 may also utilize object model database(s) 152 in generating the configured simulation environment 143. For example, object identifiers from gesture data and / or object identifier data 112 and / or determined based on visual data 112A or 117A may be used to retrieve corresponding 3D models of objects from object model database 152. These 3D models may be included in the configured simulation environment and may be included at corresponding gestures from gesture data and / or object identifier data 112 and / or determined based on visual data 112A or 117A.

[0071] The simulated action engine 144 utilizes the configured simulation environment 143 to determine one or more candidate sequences of robot actions 144A-N. The simulated action engine 144 can determine the candidate sequence(s) of robot actions 144A-N based on processing the simulation data using the machine learning model(s) 154. In addition, the simulated action engine 144 can determine the candidate sequence(s) of robot actions 144A-N in view of the task 101 to be performed by the real robot. The task 101 can be based on, for example, user interface input from a user and / or based on a higher level planning system.

[0072] For each of the candidate sequences of robot actions 144A-N, the simulation application engine 146 may apply the sequence of robot actions 147 in the robot simulator. Note that in some implementations, one or more of the downstream robot actions in the one or more sequences 144A-N may be generated by the simulation action engine 144 using updated simulation data from the robot simulator after the simulation application engine 146 has applied the upstream robot action(s) of the sequence.

[0073] The simulation state data engine 148 generates simulation state data for each of the candidate sequences 144A-N of robot actions and based on the sequence 147 of robot actions applied by the simulation application engine. The simulation state data can be used to generate corresponding features (or multiples) 112A-N for each of the candidate sequences of robot actions 144A-N. The evaluation module 145 can select (e.g., from multiple candidate sequences 144A-N of robot actions) to provide to the control engine 124 for utilizing the features (or multiples) 112A-N in implementation. The evaluation module 145 can also provide feedback to the simulation action engine 144 regarding the evaluation of the sequence (or multiples) that has been evaluated. The simulation action engine 144 can generate one or more of the candidate sequences (or multiples) of robot actions 144A-N based on the feedback. For example, the simulation action engine 144 can initially generate only a single candidate sequence, and then generate additional sequences only when the feedback indicates that the initially generated sequence cannot meet the conditions.

[0074] The simulation state data engine 148 also generates a simulation state data instance 149A for the determined robot motion sequence 144A and provides the simulation state data instance 149A to the comparison engine 126 or 150 .

[0075] The control engine 124 implements the determined robot action sequence 144A at the real robot 110. During implementation, the comparison engine 126 or 150 compares the simulated state data instance with the real state data instance. The real state data instance can be based on updated visual data from the robot vision component (or multiple) 111, updated visual data from the environment vision component (or multiple) 170, and / or based on updated real state data from the real state data engine 122. If the comparison indicates a threshold similarity, the implementation of the robot action sequence 144A can continue. On the other hand, if one or more comparisons do not indicate a threshold similarity, the comparison engine 126 or 150 can cause the control engine 124 to stop implementing the determined robot action sequence 144A. The comparison engine 126 can also send a reconfiguration signal 127 to the reconfiguration engine 142 to cause the simulated environment to be reconfigured, as described herein. For the sake of brevity, the description of a single robot action sequence is described. Figure 3 However, it should be noted that in various implementations, additional sequences may be determined using the robot simulator and provided for implementation by the real robot when performing the task. Those additional sequences may be in the same Figure 3 Determined and implemented in a similar manner as described in and as described elsewhere in this document.

[0076] Now go to Figure 4A and Figure 4B, shows two different examples of comparing real state data instances and simulated state data instances. Figure 4A , three robot actions 144A1, 144A2, and 144A3 are shown, which may be all or part of the robot actions of the sequence. In addition, a simulation state data instance 149A0 is provided and corresponds to the expected state before implementing any robot action of the sequence. A simulation state data instance 149A1 is provided and corresponds to the expected state after implementing the robot action 144A1, a simulation state data instance 149A2 is provided and corresponds to the expected state after implementing the robot action 144A2, and a simulation state data instance 149A3 is provided and corresponds to the expected state after implementing the robot action 144A3.

[0077] exist Figure 4A , the comparison engine 126 or 150 may compare the real state data instance 170A0 at that time with the simulated state data instance 149A0 before implementing any robot action of the sequence, and implement the robot action 144A1 only when the comparison indicates a threshold similarity. After implementing the robot action 144A1, the comparison engine 126 or 150 may compare the real state data instance 170A1 from a point after the robot action 144A1 was implemented with the simulated state data instance 149A1. The implementation of the robot action 144A2 may be dependent on the comparison indicating the threshold similarity, or alternatively, the robot action 144A2 may be partially (or completely) implemented during the comparison, but if the comparison does not indicate the threshold similarity, then the control of the real robot based on the sequence is stopped. After implementing the robot action 144A2, the comparison engine 126 or 150 may compare the real state data instance 170A2 from a point after the robot action 144A2 was implemented with the simulated state data instance 149A2. The implementation of the robot action 144A3 may be dependent on the comparison indicating a threshold similarity, or alternatively, the robot action 144A3 may be partially (or completely) implemented during the comparison, but if the comparison does not indicate a threshold similarity, the control of the real robot based on the sequence is stopped. In addition, after implementing the robot action 144A3, the comparison engine 126 or 150 may compare the real state data instance 170A3 from the point after the implementation of the robot action 144A3 with the simulation state data instance 149A3. The implementation of further robot actions of the sequence (if any) or the next robot action sequence (if the robot action 144A3 is the terminal robot action of the sequence) may be dependent on the comparison indicating a threshold similarity, or alternatively, the robot action 144A3 may be partially (or completely) implemented during the comparison, but if the comparison does not indicate a threshold similarity, the control of the real robot based on the sequence is stopped.

[0078] Figure 4B Similar to Figure 4A , and the same reference numerals denote the same components. However, no Figure 4A Instead, only the initial simulated state data instance 149A0 and the simulated state data instance 149A3 are provided. Therefore, the comparison engine 126 or 150 will perform an initial comparison between the real state data instance 170A0 and the simulated state data instance 149A0, and between the real state data instance 170A3 and the simulated state data instance 149A3, but will not perform a comparison based on the real state data after the robot actions 144A1 and 144A2. Therefore, Figure 4A and Figure 4B Two different non-limiting example frequencies at which comparisons may be performed by comparison engine 126 or 150 are shown.

[0079] In some implementations, the frequency at which the comparison is performed can be fixed throughout the duration of the attempted execution of the robot task, and optionally, can be fixed for each of a plurality of different robot tasks. In some alternative implementations, the frequency at which the comparison is performed can be adjusted throughout the duration of the attempted execution of the robot task and / or between a plurality of different robot tasks. In some of those implementations, the frequency at which the comparison is performed at a given time can be determined based on the actual or expected entropy of the real-world environment, the portion of the robot task being executed, and / or the robot task being executed. For example, if the upcoming simulation state data instance (or instances) indicates that the real-world environment may change (e.g., due to the interaction (or multiples) of the real robot with the real-world object (or multiples)), the frequency can be increased. On the other hand, if the upcoming simulation state data instance (or instances) indicates that the real-world environment may remain static, the frequency can be reduced or maintained at a lower frequency. As another example, for a robot task or a portion of a robot task that navigates along an empty corridor, a first frequency can be utilized. On the other hand, for a robot task or a portion of a robot task that attempts to grasp an object, a second frequency greater than the first frequency can be utilized. In these and other ways, the frequency of comparison can be dynamically adjusted according to the current or expected robot action (or multiple) and / or environmental state. This can include increasing the frequency of complex robot actions (or multiple) (e.g., robot actions involving interaction with environmental objects (or multiple)) and / or high entropy environments, while reducing the frequency of less complex robot actions (or multiple) (e.g., navigating or moving components (or multiple) without interacting with environmental objects (or multiple)) and / or lower entropy environments. Therefore, the frequency of comparison can be selectively reduced, resulting in less computational and / or power resources of the real world robot being utilized. In addition, in some implementations, the frequency of generating real state data instances can even be adjusted to reflect the frequency of comparison. For example, in some of these implementations, real state data instances will be generated only at a frequency that matches the frequency of comparison. This can include processing only sensor data instances at this frequency and / or even adapting the sensing frequency of the corresponding sensor component (or multiple) based on this frequency. Adjusting the sensing frequency of the corresponding sensor component can include adjusting them to sense at this frequency (if possible) or at a corresponding given frequency that is as close to this frequency as possible and optionally greater than this frequency. Adapting the sensing frequency of the corresponding sensor assembly(ies) may result in utilizing fewer computational and / or power resources of the real-world robot.

[0080] Figure 5An example architecture of a robot 520 is schematically depicted. The robot 520 includes a robot control system 560, one or more operating components 540a-540n, and one or more sensors 542a-542m. The sensors 542a-542m may include, for example, vision sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, etc. Although the sensors 542a-m are depicted as being integrated with the robot 520, this is not meant to be limiting. In some implementations, the sensors 542a-m may be located external to the robot 520, such as as a standalone unit.

[0081] The operating components 540a-540n may include, for example, one or more end effectors and / or one or more servomotors or other actuators to achieve movement of one or more components of the robot. For example, the robot 520 may have multiple degrees of freedom, and each actuator may control actuation of the robot 520 within one or more degrees of freedom in response to a control command. As used herein, the term actuator includes a mechanical or electrical device (e.g., a motor) that produces motion, in addition to any driver (or drivers) that may be associated with the actuator and convert a received control command into one or more signals for driving the actuator. Thus, providing a control command to an actuator may include providing a control command to a driver that converts the control command into an appropriate signal for driving an electrical or mechanical device to produce the desired motion.

[0082] The robot control system 560 may be implemented in one or more processors, such as a CPU, GPU, and / or other controller(s) of the robot 520. In some implementations, the robot 520 may include a "brain box" that may include all or aspects of the control system 560. For example, the brain box may provide real-time data bursts to the operating components 540a-n, where each real-time burst includes a group of one or more control commands that indicate, among other things, motion parameters (if any) for each of the one or more operating components 540a-n. The control commands may be based on robot motions determined using a robot simulator as described herein. In some implementations, the robot control system 560 may execute Figure 2 Certain blocks of method 200.

[0083] Figure 66 is a block diagram of an example computing device 610 that may optionally be used to perform one or more aspects of the techniques described herein. For example, a robot simulator may be implemented on the computing device 610 or in a cluster of multiple computing devices 610. The computing device 610 typically includes at least one processor 614 that communicates with multiple peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624 (including, for example, a memory subsystem 625 and a file storage subsystem 626), a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow a user to interact with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0084] The user interface input device 622 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 610 or over a communication network.

[0085] User interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanisms for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Typically, the use of the term "output device" is intended to include devices and methods of all possible types of information output from computing device 610 to a user or another machine or computing device.

[0086] The storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include a Figure 2 The logic of some aspects of the method.

[0087] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 may include multiple memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 for storing fixed instructions. The file storage subsystem 626 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored in the storage subsystem 624 by the file storage subsystem 626, or in other machines accessible to the processor (or multiple) 614.

[0088] The bus subsystem 612 provides a mechanism for the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0089] The computing device 610 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 6 The description of computing device 610 depicted in FIG. 6 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device 610 may have more Figure 6 The computing devices depicted may have greater or fewer components.

[0090] In some implementations, a method implemented by one or more processors is provided, and the method includes configuring a simulated environment of a robot simulator based on real environment state data. The real environment state data is based on one or more sensor-based observations of the real environment. The method also includes determining a robot action sequence for controlling a real robot in the real environment to perform at least a portion of a robot task using the robot simulator. The method also includes applying the robot action sequence to a simulated robot of the robot simulator to generate a sequence of simulated state data instances. Each simulated state data instance defines one or both of the following for a corresponding point in the robot action sequence: corresponding expected environment state data for the corresponding point, and corresponding expected robot state data for the corresponding point. The method also includes controlling the real robot to implement a first subset of the robot action sequence, and comparing the real state data instance with a corresponding one of the simulated state data instances about the corresponding point in the robot action sequence. The real state data instance defines one or both of the following for the corresponding point: updated real environment state data based on one or more updated sensor-based observations of the real environment; and real robot state data based on one or more sensor-based observations of the real robot. The method further includes, in response to comparing the threshold similarity between the real state data indicating the update and a corresponding one of the expected state data instances for the corresponding point: controlling the real robot to perform a second subset of the robot action sequence.

[0091] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0092] In some implementations, the method further includes, in response to the comparison not indicating a threshold similarity, stopping control of the real robot based on the robot action sequence. In some versions of these implementations, the method further includes, in response to the comparison not indicating a threshold similarity: reconfiguring the simulated environment based on the updated real environment state data; determining an updated robot action sequence using a robot simulator for controlling the real robot when performing a robot task; applying the updated robot action sequence to a simulated robot of the robot simulator to generate a sequence of updated simulation state data instances; controlling the real robot to implement a first updated subset of the updated robot action sequence; comparing the updated real state data instance with a corresponding updated simulation state data instance in the updated simulation state data instance for a corresponding update point in the updated robot action sequence; and in response to comparing the updated real state data instance with a corresponding updated simulation state data instance in the updated simulation state data instance indicating a threshold similarity: controlling the real robot to implement an updated second subset of the updated robot action sequence. Each updated simulation state data instance defines one or both of the following for a corresponding update point in the updated robot action sequence: corresponding updated expected environment state data for the corresponding update point, and real robot state data based on one or more sensor-based observations of the real robot.

[0093] In some implementations, the method includes controlling the real robot to implement a second subset of the robot action sequence and implement any additional subsets of the robot action sequence. In some versions of those implementations, the method also includes, before fully implementing the robot action sequence: determining, using the robot simulator, a next robot action sequence for controlling the real robot in performing another portion of the robot task; and applying the next robot action sequence to the simulated robot of the robot simulator using the simulated robot and the environment of the robot simulator corresponding to the terminal simulated state data instance in the simulated state data instance to generate a next sequence of simulated state data instances. In some of these versions, the method also includes, after fully implementing the robot action sequence, controlling the real robot to implement the next sequence or robot action. Controlling the real robot to implement the next robot action sequence may include: controlling the real robot to implement a first next subset of the next robot action sequence; comparing the next real state data instance with a corresponding one of the next simulated state data instances; and in response to the comparison indicating a threshold similarity, controlling the real robot to implement a second next subset of the next robot action sequence.

[0094] In some implementations, the one or more sensor-based observations include robot sensor-based observations based on one or more real sensors of the real robot.

[0095] In some implementations, the one or more sensor-based observations include environmental sensor-based observations based on one or more real sensors that are not coupled to the real robot but are in the real environment.

[0096] In some implementations, determining a sequence of robot actions for controlling a real robot in performing at least a portion of a robot task includes: applying an alternative sequence of robot actions to a robot simulator; comparing one or more alternative features from applying the alternative sequence of robot actions to the robot simulator with one or more features from applying the sequence of robot actions to the robot simulator; and determining, based on comparing the one or more alternative features with the one or more features, to replace an alternative sequence of robot actions with a sequence of robot actions for controlling a real robot in performing at least a portion of a robot task. In some versions of these implementations, the one or more features include one or both of a task success feature and an efficiency feature, and the one or more alternative features include one or both of an alternative task success feature and an alternative efficiency feature. In some of those versions, the one or more features include a task success feature, and the method further includes generating a task success feature based on a terminal state of the robot simulator after applying the sequence of robot actions. In addition, in some of those versions, the one or more features additionally or alternatively include an efficiency feature, and the method further includes generating an efficiency feature based on one or both of the following: the number of robot actions in the sequence of robot actions, and the distance traversed by one or more simulated components of the simulated robot when performing the sequence of robot actions.

[0097] In some implementations, comparing the real state data instance to a corresponding one of the simulated state data instances for the corresponding point is performed by one or more processors of the real robot.

[0098] In some implementations, the robot simulator is implemented on one or more remote processors that are remote from the real robot but in network communication with the real robot.

[0099] In some implementations, a method implemented by one or more processors of a real robot is provided, and the method includes receiving a sequence of robot action sequences and simulated state data instances. Each simulated state data instance defines one or both of the following for a corresponding point in the robot action sequence: corresponding expected environment state data for the corresponding point, and corresponding expected robot state data for the corresponding point. The method also includes controlling the real robot to implement a first subset of the robot action sequence, and comparing the real state data instance with a corresponding one of the simulated state data instances for the corresponding point in the robot action sequence. The real state data instance defines one or both of the following for the corresponding point: updated real environment state data based on one or more updated sensor-based observations of the real environment; and real robot state data based on one or more sensor-based observations of the real robot. The method also includes, in response to a comparison indicating a threshold similarity between the updated real state data and the corresponding one of the expected state data instances for the corresponding point: controlling the real robot to implement a second subset of the robot action sequence.

[0100] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0101] In some implementations, the method further includes, in response to the comparing not indicating a threshold similarity, ceasing control of the real robot based on the robot motion sequence.

[0102] In some implementations, the method also includes, in response to the comparison not indicating a threshold similarity: sending updated real environment state data to the robot simulator component; receiving an updated robot action sequence from the robot simulator component, wherein the updated robot action sequence is generated based on a sequence of updated real environment state data and updated simulation state data instances; controlling the real robot to implement a first updated subset of the updated robot action sequence; comparing the updated real state data instance with a corresponding updated simulation state data instance in an updated simulation state data instance of a corresponding update point in the updated robot action sequence; and in response to comparing the updated real state data instance with a corresponding updated simulation state data instance in an updated simulation state data instance indicating a threshold similarity: controlling the real robot to implement a second updated subset of the updated robot action sequence.

[0103] In some implementations, the robot simulator is implemented on one or more remote processors that are remote from the real robot but in network communication with the real robot.

[0104] In some implementations, a method implemented by one or more processors is provided, and the method includes configuring a simulated environment of a robot simulator based on real environment state data, the real environment state data being based on one or more sensor-based observations of the real environment. The method also includes using the robot simulator to determine a first candidate sequence of robot actions and a second candidate sequence of robot actions for controlling a real robot in the real environment to perform at least a portion of a robot task. The method also includes applying the first candidate sequence of robot actions to a simulated robot of the robot simulator using the configured simulated environment to generate first simulated state data, and applying the second candidate sequence of robot actions to a simulated robot of the robot simulator using the configured simulated environment to generate second simulated state data. The method also includes determining to replace the second candidate sequence of robot actions with the first candidate sequence of robot actions based on the first simulated state data and the second simulated state data for controlling the real robot when performing at least a portion of the robot task. The method also includes causing the real robot to implement the first candidate sequence of robot actions in response to determining to utilize the first candidate sequence of robot actions.

[0105] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0106] In some implementations, determining to replace a second candidate sequence of robot actions with a first candidate sequence of robot actions based on the first simulation state data and the second simulation state data includes: generating one or more first features based on the first simulation data; generating one or more second features based on the second simulation data; and determining to utilize the first candidate sequence of robot actions based on comparing the first features with the second features. In some versions of those implementations, the one or more first features include one or both of a first mission success feature and a first efficiency feature, and wherein the one or more second features include one or both of a second mission success feature and a second efficiency feature.

[0107] In some implementations, a method implemented by one or more processors is provided, and the method includes configuring a simulated environment of a robot simulator based on real environment state data, the real environment state data being based on one or more sensor-based observations of the real environment. The method also includes determining, using the robot simulator, a first candidate sequence of robot actions and a second candidate sequence of robot actions for controlling a real robot in the real environment to perform at least a portion of a robot task. Determining the first candidate sequence includes utilizing simulated state data that simulates real state data that cannot be generated at all using any sensor in the real environment. The method also includes causing the real robot to implement the first candidate sequence of robot actions.

[0108] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0109] In some implementations, the simulated state data simulates real state data, which is generated from a simulated viewpoint and is completely unable to be generated using any sensor in the real environment, and any sensor in the real environment cannot capture real visual data from a corresponding real viewpoint in the real environment without human intervention.

[0110] In some implementations, the simulated state data is generated from a simulated sensor of a specific type to simulate real state data that cannot be generated by any sensor in the real environment at all, and any sensor in the real environment is not of the specific type.

[0111] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described herein. Other implementations may include a system of one or more computers and / or one or more robots including one or more processors operable to execute stored instructions to perform methods such as one or more of the methods described herein.

Claims

1. A method implemented by one or more processors, the method comprising: configuring a simulated environment of the robot simulator based on real environment state data based on one or more sensor-based observations of the real environment; determining, using the robot simulator, a first candidate sequence of robot actions and a second candidate sequence of robot actions for controlling a real robot in performing at least a portion of a robot task in a real environment; Using the configured simulation environment, applying a first candidate sequence of robot actions to a simulated robot of a robot simulator to generate first simulation state data; Using the configured simulation environment, applying a second candidate sequence of robot actions to a simulated robot of the robot simulator to generate second simulation state data; Based on the first simulation state data and the second simulation state data, determining to replace the second candidate sequence of robot actions with the first candidate sequence of robot actions for controlling the real robot in performing at least a portion of the robot task; as well as In response to determining to utilize a first candidate sequence of robot actions: Make the real robot perform the first candidate sequence of robot actions.

2. The method according to claim 1, wherein: Based on the first simulation state data and the second simulation state data, determining to use the first candidate sequence of robot actions to replace the second candidate sequence of robot actions comprises: generating one or more first features based on the first simulation data; generating one or more second features based on the second simulation data; and A first candidate sequence of actions utilizing the robot is determined based on comparing the first feature to the second feature.

3. The method according to claim 2, wherein: The one or more first characteristics include a first mission success characteristic and a first efficiency characteristic, and wherein the one or more second characteristics include a second mission success characteristic and a second efficiency characteristic.

4. The method according to claim 2, wherein: The one or more first features include a first mission success feature, and the one or more second features include a second mission success feature.

5. The method according to claim 2, wherein: The one or more first characteristics include a first efficiency characteristic, and wherein the one or more second characteristics include a second efficiency characteristic.

6. The method according to claim 1, wherein: The first candidate sequence of robot actions indicates a corresponding sequence of control commands to be implemented at corresponding time steps.

7. The method according to claim 6, wherein: In implementing the first candidate sequence of robot actions, the real robot sends control commands indicated by the first candidate sequence of robot actions to corresponding actuators of the real robot.

8. A method implemented by one or more processors, the method comprising: configuring a simulated environment of the robot simulator based on real environment state data based on one or more sensor-based observations of the real environment; Determining a candidate sequence of robot actions using a robot simulator for controlling a real robot in performing at least a portion of a robot task in a real environment, wherein determining the candidate sequence of robot actions comprises: Using simulated state data, the simulated state data simulates real state data that cannot be generated at all using any sensor in the real environment; and A candidate sequence of robot actions that causes a real robot to perform.

9. The method according to claim 8, wherein: The simulated state data is generated from a simulated viewpoint to simulate real state data that cannot be generated at all using any sensor in the real environment, and wherein any sensor in the real environment cannot capture real visual data from a corresponding real viewpoint in the real environment without human intervention.

10. The method according to claim 9, wherein: The simulated viewpoint is an overhead viewpoint.

11. The method according to claim 9, wherein: The simulated state data is generated from a simulated sensor of a specific type to simulate real state data that cannot be generated at all using any sensor in the real environment, and wherein any sensor in the real environment is not of the specific type.

12. The method according to claim 8, wherein: The simulated state data is generated from a simulated sensor of a specific type to simulate real state data that cannot be generated at all using any sensor in the real environment, and wherein any sensor in the real environment is not of the specific type.

13. The method according to claim 8, wherein: The candidate sequence of robot actions indicates a corresponding sequence of control commands to be implemented at corresponding time steps.

14. A system comprising: Memory, which stores instructions; One or more processors that execute instructions to: configuring a simulated environment of the robot simulator based on real environment state data based on one or more sensor-based observations of the real environment; determining, using the robot simulator, a first candidate sequence of robot actions and a second candidate sequence of robot actions for controlling a real robot in performing at least a portion of a robot task in a real environment; Using the configured simulation environment, applying a first candidate sequence of robot actions to a simulated robot of a robot simulator to generate first simulation state data; Using the configured simulation environment, applying a second candidate sequence of robot actions to a simulated robot of the robot simulator to generate second simulation state data; Based on the first simulation state data and the second simulation state data, determining to replace the second candidate sequence of robot actions with the first candidate sequence of robot actions for controlling the real robot in performing at least a portion of the robot task; as well as In response to determining to utilize a first candidate sequence of robot actions: Make the real robot perform the first candidate sequence of robot actions.

15. The system of claim 14, wherein: Based on the first simulation state data and the second simulation state data, determining to replace the second candidate sequence of robot actions with the first candidate sequence of robot actions, one or more of the processors will: generating one or more first features based on the first simulation data; generating one or more second features based on the second simulation data; as well as A first candidate sequence of actions utilizing the robot is determined based on comparing the first feature to the second feature.

16. The system of claim 15, wherein: The one or more first characteristics include a first mission success characteristic and a first efficiency characteristic, and wherein the one or more second characteristics include a second mission success characteristic and a second efficiency characteristic.

17. The system of claim 16, wherein: The one or more first features include a first mission success feature, and the one or more second features include a second mission success feature.

18. The system of claim 16, wherein: The one or more first characteristics include a first efficiency characteristic, and wherein the one or more second characteristics include a second efficiency characteristic.

19. The system of claim 15, wherein: The first candidate sequence of robot actions indicates a corresponding sequence of control commands to be implemented at corresponding time steps.

20. The system of claim 19, wherein: In implementing the first candidate sequence of robot actions, the real robot sends control commands indicated by the first candidate sequence of robot actions to corresponding actuators of the real robot.

Images