Simulation of multiple robots in a virtual environment

Abstract

Implementations are provided in which multiple robot controllers are operably coupled to a single virtual environment to generate training examples, for example, for training a machine learning model. In various implementations, a virtual environment can be simulated that includes an interactive object and multiple robot avatars that are independently and simultaneously controlled by corresponding multiple robot controllers outside the virtual environment. Sensor data generated from the perspective of each robot avatar of the multiple robot avatars can be provided to the corresponding robot controller. Joint commands that actuate one or more joints of each robot avatar can be received from the corresponding robot controller. The joints of each robot avatar can be actuated according to the corresponding joint commands. The actuation can cause two or more of the robot avatars to interact with the interactive object in the virtual environment.

Description

[Background technology]

[0001] Robots are often equipped with various types of machine learning models that are trained to perform various tasks and / or enable the robot to interact with dynamic environments. These models are sometimes trained by having a physical robot in the real world repeatedly perform a task, and the results of the repeated tasks are used as training examples to adjust the model. However, a large number of iterations may be required to sufficiently train a machine learning model to perform a task in a satisfactory manner.

[0002] The time and expense involved in training a model with real-world physical robot actions can be reduced and / or avoided by simulating robot actions in a simulated (i.e., "virtual") environment. For example, a three-dimensional virtual environment can be simulated with various objects acted upon by the robot. The robot itself can also be simulated within the virtual environment, and the simulated robot can operate to perform various tasks on the simulated objects. Machine learning models can be trained based on the results of these simulated tasks. In some cases, the simulation can be good enough to train, or at least partially train, a robotic machine model, for example, in a quick and / or inexpensive manner, so that the model can then be further trained, or "refined," using a limited number of real-world robot actions. Summary of the Invention

[0003] Different types of robots with different capabilities, constraints, hardware, etc. may operate in the same real-world environment. Thus, each robot, and more specifically, each robot's machine learning model, may need to be trained separately to enable the different robots to interact with dynamic aspects of the real-world environment, such as other robots, humans, and / or physically moved or otherwise modified objects.

[0004] Described herein are implementations for operably coupling multiple robot controllers to a single simulated environment, such as to generate training examples for use during training of machine learning models. These multiple robot controllers may operate independently of one another to generate and control corresponding multiple robot avatars within the simulated environment. The multiple robot avatars may be controlled via the external robot controllers to perform various tasks within the virtual environment, such as interacting with virtual objects within the virtual environment. Results of these interactions and performance within the virtual environment may be used, for example, as training data for training one or more machine learning models.

[0005] A robot controller coupled to a virtual environment can take a variety of forms. In some implementations, the robot controller can be hardware that is integral with and / or detachably removed from an actual, physical robot. For example, the robot controller can include a “brain” that receives sensor data from one or more sensors, processes the sensor data (e.g., using one or more “perception” machine learning models), and, based on the processing of the sensor data, provides joint commands to one or more joints (i.e., “actuators”) to actuate those joints. The “brain” can take the form of one or more processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (“FPGAs”), etc., and may or may not include other components such as memory for buffering, buses for connecting sensors / joints, etc.

[0006] In other implementations, the robot controller may be a computing device configured (e.g., programmed) to simulate as closely as possible the hardware of a real-world robot controller. For example, robot components may be simulated in the virtual environment at frequencies corresponding to the real-world frequencies at which the elements on which they operate operate. If a real-world vision sensor installed on the robot captures images at 100 MHz, a vision sensor may be simulated in the virtual environment that also captures images (or provides simulated images in the simulated environment) at 100 MHz. The programmed computing device may provide joint commands as output at the same frequency as the real-world control commands that the simulated robot sends to its real-world joints.

[0007] Multiple robot controllers may be operatively coupled to a virtual environment in various ways. In some implementations, the virtual environment may be simulated by one or more servers forming part of what is often referred to as a “cloud” infrastructure, or simply the “cloud.” For example, a commercial entity may simulate a virtual environment and allow its “clients” to couple robot controllers to the virtual environment, e.g., using an application programming interface (API). The clients may then virtually train their robot machine learning models without the clients having to create their own simulated environments. In many cases, the robot controllers may not be physically co-located with these servers and thus may be coupled to the servers via one or more local and / or wide area networks using various wired and / or wireless communication technologies.

[0008] In various implementations, each of multiple external robot controllers coupled to the virtual environment may be provided with sensor data generated from the perspective of a robot avatar in the virtual environment controlled by the robot controller. Similarly, each robot controller may provide joint commands that actuate one or more joints of a respective robot avatar in the virtual environment controlled by the robot controller. One or more joints of each robot avatar may be actuated according to these corresponding joint commands. In some implementations, this actuation may result in two or more of the robot avatars acting on a single interactive object in the virtual environment.

[0009] In some implementations where it is desirable for robots to cooperate with each other, robot avatars may cooperate in the virtual environment to interact with the same interactive object, for example, one robot avatar may pick up an interactive object while another robot avatar acts on a portion of the interactive object that was previously inaccessible.

[0010] As described above, different robots may operate at different frequencies. More specifically, different components of different robots may operate at different frequencies. Additionally, in the real world, it may occur that a robot deviates from its frequency and / or "lags," e.g., temporarily stops, i.e., its operation is delayed. Similarly, in virtual environments, such as those described herein, in which robot controllers are linked via one or more networks, it is possible that factors external to the robot controllers and / or the virtual environment, such as network latency / traffic, may cause a robot avatar to deviate from its expected frequency in a manner that cannot be easily reproduced. However, for training purposes, it may be desirable to be able to reproduce repetitions of a robot task in a deterministic manner.

[0011] Thus, in some implementations, the "simulated world clock" of the virtual environment can be selectively advanced (and paused in time) to account for deviations of individual avatars from their expected operating frequency. In some implementations, the simulated world clock can operate at a frequency equal to or greater than the highest frequency achievable by any of the robotic controllers. Multiple robotic avatars in the virtual environment can operate at frequencies corresponding to the real-world frequencies of the respective robotic controllers controlling the multiple robotic avatars.

[0012] Suppose the highest frequency robot avatar operates at 200 MHz and the lower frequency robot avatar operates at 20 MHz. The frequency of the simulated world clock can be set to 200 MHz, so that each iteration of the simulated world clock corresponds to one atomic motion by the highest frequency robot. Meanwhile, the lower frequency robot avatar performs an atomic motion at one-tenth the frequency of the simulated world clock, i.e., once every 10 cycles of the simulated world clock. In other words, for every 10 cycles of the simulated world clock, the highest frequency robot performs 10 atomic motions and the lower frequency robot performs one motion.

[0013] If a particular robot avatar falls behind for some reason (likely not easily reproducible), that robot avatar may not be ready to execute its respective action in sync with the simulated world clock. Therefore, to ensure that actions by a robot avatar can be deterministically reproduced, the simulated world clock may be paused until the lagging robot avatar can “catch up.” Taking the previous example, assume that a lower frequency (20 MHz) robot avatar falls behind by, for example, three clock cycles due to the network connection between its external robot controller and the server running the virtual environment being temporarily overwhelmed with network traffic. The simulated world clock may be paused until the lagging robot avatar is able to execute its action. Once the lagging robot avatar executes its action, the simulated world clock may be restarted, thereby allowing other robot avatars to resume their actions as well.

[0014] In some implementations, directed acyclic graphs (also called "robot device graphs") may be generated to represent sensors and / or other components of a robot within a virtual environment. Each graph may include nodes that represent components of a robot simulated by and / or for the benefit of a robot avatar, including, but not limited to, the robot's sensors, joints, etc. In some such implementations, a directed acyclic graph may be a dependency graph, in which edges between nodes represent dependencies between the components represented by the nodes.

[0015] For example, a child node may represent a visual sensor that supplies the visual data it captures to a parent node, which in turn may represent a processor or module that applies a machine learning model to the visual data captured at the visual sensor to generate additional output (e.g., object recognition and / or pose prediction). This output may be passed further up the graph until it reaches a root node that represents a final processing unit. This processing unit may process any data it receives from its child node and then generate commands that are propagated down the graph to other nodes, e.g., representing the joints of a robot.

[0016] In some implementations, each node in the acyclic graph may include information about the operational capabilities and / or constraints of the node's corresponding robot component. For example, a node representing a visual sensor may specify, e.g., the resolution of the visual sensor, the frequency at which the visual sensor captures visual data, etc. If one or more components represented by one or more nodes above the visual sensor's node operate at a higher frequency than the visual sensor, those nodes may nevertheless wait several cycles of the simulated world clock for the visual sensor node to generate its output before those higher-level nodes execute their own actions based on the visual data. In other words, if the visual sensor is a bottleneck for the robot in the real world, the acyclic graph may ensure that the bottleneck in the virtual world ensures that the robot's machine learning model is trained based on how the robot actually operates.

[0017] Nodes of the acyclic graph are not limited to robotic components such as sensors, joints, or processing units. In some implementations, one or more nodes of the directed acyclic graph may represent simulated environmental conditions of the virtual environment. For example, a node adjacent to a visual sensor node (e.g., above or below) may be configured to simulate particulates in the virtual environment that are captured in (and add noise to) visual data captured by the visual sensor. As another example, a node may be configured to impose a crosswind on a drone under certain circumstances, such as when the drone is attempting to land. Yet other nodes of the directed acyclic graph may represent simulated conditions of a simulated sensor of a robotic avatar, such as a visual sensor that has accumulated a layer of dust from simulated particulates in the virtual environment.

[0018] In some implementations, a computer-implemented method may be provided that includes: simulating a three-dimensional environment including an interactive object, the simulated environment including a plurality of robotic avatars independently and simultaneously controlled by a corresponding plurality of robotic controllers outside the simulated environment; providing, to each robot controller of the plurality of robotic controllers, sensor data generated from a perspective of a respective robot avatar of the plurality of robotic avatars controlled by the robot controller; receiving, from each robot controller of the plurality of robotic controllers, joint commands that actuate one or more joints of each robot avatar of the plurality of robotic avatars controlled by the robot controller; and actuating the one or more joints of each robot avatar in accordance with the corresponding joint commands, wherein the actuation causes two or more of the plurality of robotic avatars to interact with an interactive object in the simulated environment.

[0019] In various implementations, a given robot controller of the plurality of robot controllers may be integral with a robot operatively coupled to one or more processors. In some such implementations, sensor data provided to the given robot controller may be injected into a sensor data channel between one or more real-world sensors of the given robot and one or more processors of the robot controller integral with the given robot, and / or joint commands received from the given robot controller may be intercepted from a joint command channel between one or more processors of the robot controller and one or more joints of the given robot.

[0020] In various implementations, the method may further include generating, for each robot avatar of the plurality of robot avatars, a directed acyclic graph with nodes representing components of the robot avatar. In various implementations, the directed acyclic graph is a dependency graph in which edges between nodes represent dependencies between the components represented by the nodes. In various implementations, at least one node representing a simulated sensor of the robot avatar may impose a delay on the sensor output passed to the directed acyclic graph, the delay corresponding to the frequency of the real-world sensor corresponding to the simulated sensor. In various implementations, one or more nodes of the directed acyclic graph represent simulated environmental conditions of the simulated environment. In various implementations, one or more nodes of the directed acyclic graph represent simulated conditions of the simulated sensor of the robot avatar.

[0021] In various implementations, the actuating may include operating the plurality of robotic avatars at a frequency corresponding to a real-world frequency of a respective robot controller controlling the plurality of robotic avatars. In various implementations, the method may further include operating a simulated world clock of the simulated environment at a given frequency and operating a first robotic avatar of the plurality of robotic avatars at a first frequency that is lower than the given frequency. In various implementations, the method may further include pausing the simulated world clock until a robotic avatar of the plurality of robotic avatars advances to a next operation stage.

[0022] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by a processor to perform a method, such as one or more of the methods described above. Yet another implementation may include a control system including a memory and one or more processors that execute the instructions stored in the memory to implement one or more modules or engines that, singly or collectively, perform a method, such as one or more of the methods described above.

[0023] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0024] [Figure 1A] 1 illustrates a schematic diagram of an exemplary environment in which the disclosed techniques may be used, according to various implementations. [Figure 1B] 1 illustrates an exemplary robot, according to various implementations. [Figure 2] 1A-1C illustrate, in accordance with various implementations, examples of how a robot controller may interface with a simulation engine to facilitate the generation of a virtual environment including a robot avatar controlled by the robot controller. [Figure 3A] 1 illustrates an example of how the techniques described herein can be used to generate deterministically reproducible training sets, according to various implementations. [Figure 3B] 1 illustrates an example of how the techniques described herein can be used to generate deterministically reproducible training sets, according to various implementations. [Figure 4] 1 illustrates an example of an acyclic graph that may be used in various implementations to represent a robot and / or its components. [Figure 5] 1 illustrates exemplary methods for implementing selected aspects of the present disclosure. [Figure 6] 1 illustrates a schematic diagram of an exemplary architecture of a computer system. DETAILED DESCRIPTION OF THE INVENTION

[0025] 1A is a schematic diagram of an exemplary environment in which selected aspects of the present disclosure may be implemented according to various implementations. The various components shown in FIG. 1A, particularly those components forming simulation system 130, may be implemented using any combination of hardware and software. In some implementations, simulation system 130 is one or more servers that form part of what is often referred to as a "cloud" infrastructure, or simply "the cloud."

[0026] The robot 100 may be in communication with a simulation system 130. The robot 100 may take various forms, including, but not limited to, a telepresence robot (which may be as simple as a vehicle equipped with a display and a camera), a robotic arm, a humanoid robot, an animal, an insect, an aquatic creature, a wheeled device, a submersible vehicle, or an unmanned aerial vehicle ("UAV"). One non-limiting example of a robotic arm is shown in FIG. 1B. In various implementations, the robot 100 may include logic 102. The logic 102 may take various forms, including a real-time controller, one or more processors, one or more field programmable gate arrays ("FPGAs"), one or more application specific integrated circuits ("ASICs"), or the like. In some implementations, the logic 102 may be operably coupled to a memory 103. Memory 103 may take various forms, such as random access memory ("RAM"), dynamic RAM ("DRAM"), read-only memory ("ROM"), magnetoresistive RAM ("MRAM"), resistive RAM ("RRAM"), NAND flash memory, etc.

[0027] In some implementations, the logic 102 may communicate with one or more joints 104, for example, via one or more buses 110. 1~n , one or more end effectors 106, and / or one or more sensors 108 1~m As used herein, a "joint" 104 of a robot may broadly refer to an actuator, motor (e.g., servo motor), shaft, gear train, pump (e.g., gas or liquid), piston, drive, propeller, flap, rotor, or other component that may produce and / or experience propulsion, rotation, and / or motion. Some joints 104 may be independently controllable, but this is not required. In some cases, the more joints a robot 100 has, the more degrees of freedom of movement it may have.

[0028] As used herein, "end effector" 106 may refer to various tools that can be manipulated by the robot 100 to accomplish various tasks. For example, some robots may be equipped with an end effector 106 in the form of a claw with two opposing "fingers" or "digits." Such a claw is a type of "gripper" known as an "impact" gripper. Other types of grippers may include, but are not limited to, "penetration" (e.g., physically penetrating an object using a pin or needle), "astrictive" (e.g., picking up an object using suction or vacuum), or "proximity" (e.g., picking up an object using surface tension, freezing, or adhesive). More generally, other types of end effectors may include, but are not limited to, drills, brushes, force-torque sensors, cutting tools, deburring tools, welding torches, containers, trays, etc. In some implementations, the end effector 106 may be removable, and various types of modular end effectors may be installed on the robot 100 depending on the situation. Some robots, such as some telepresence robots, may not be equipped with end effectors. Instead, some telepresence robots may include a display for rendering a visual representation of a user controlling the telepresence robot, as well as speakers and / or microphones that facilitate the telepresence robot "acting" like a user.

[0029] The sensors 108 may take a variety of forms, including, but not limited to, a 3D laser scanner or other 3D visual sensor configured to provide depth measurements (e.g., a stereo camera used to perform stereo visual odometry), a two-dimensional camera (e.g., RGB, infrared), a light sensor (e.g., passive infrared), a force sensor, a pressure sensor, a pressure wave sensor (e.g., a microphone), a proximity sensor (also called a "distance sensor"), a depth sensor, a torque sensor, a barcode reader, a radio frequency identification ("RFID") reader, a radar, a rangefinder, an accelerometer, a gyroscope, a compass, a position coordinate sensor (e.g., a global positioning system, or "GPS"), a speedometer, an edge detector, etc. 1~m is shown as being integral to the robot 100, but this is not meant to be limiting.

[0030] The simulation system 130 may include one or more computing systems connected by one or more networks (not shown). An example of such a computing system is shown schematically in FIG. 6. In various implementations, the simulation system 130 may operate to simulate a virtual environment in which multiple robot avatars (not shown in FIG. 1, see FIG. 2) are simulated. Each robot avatar may be controlled by a robot controller. As previously mentioned, the robot controller may include, for example, the logic 102 and memory 103 of the robot 100.

[0031] Various modules or engines may be implemented as part of simulation system 130 as software, hardware, or any combination of the two. For example, in FIG. 1A , simulation system 130 includes a display interface 132 controlled, for example, by a user interface engine 134 to render a graphical user interface (“GUI”) 135. A user may interact with GUI 135 to trigger and / or control aspects of simulation system 130 (e.g., to control a simulation engine 136 that simulates the aforementioned virtual environment).

[0032] The simulation engine 136 may be configured to execute selected aspects of the present disclosure to simulate a virtual environment in which the aforementioned robotic avatars can operate. For example, the simulation engine 136 may be configured to simulate a three-dimensional environment including interactive objects. The virtual environment may include multiple robotic avatars independently and simultaneously controlled by corresponding multiple robot controllers (e.g., a combination of 102 and 103 of the robot 100) external to the virtual environment. Note that the virtual environment need not be visually rendered on a display. In many cases, the virtual environment and the operation of the robotic avatars therein may be simulated without a visual representation provided as output on a display.

[0033] The simulation engine 136 may be further configured to provide, to each robot controller of the plurality of robot controllers, sensor data generated from the perspective of a robot avatar controlled by the robot controller. As an example, assume that a visual sensor of a particular robot avatar is pointed toward a particular virtual object in a virtual environment. The simulation engine 136 may generate and / or provide, to the robot controller controlling the particular robot avatar, simulated visual sensor data that depicts the particular virtual object as seen from the perspective of the particular robot avatar (and more specifically, its visual sensor) in the virtual environment.

[0034] The simulation engine 136 may also be configured to receive, from each robot controller, joint commands that actuate one or more joints of a respective robot avatar controlled by the robot controller. For example, the external robot controller may process the sensor data received from the simulation engine to make various decisions, such as recognizing an object and / or its pose (perception), planning a path to the object and / or a grasp to be used to interact with the object, etc. The external robot controller may make these decisions and generate joint commands for one or more joints of the robot associated with the robot controller (execution).

[0035] In the context of a virtual environment simulated by the simulation engine 136, these joint commands may be used, for example, by the simulation engine 136 to actuate the joints of a robot avatar controlled by an external robot controller. Given that there may be multiple robot avatars in the virtual environment at any given moment, actuating two or more joints of the robot avatars may, in some cases, cause two or more robot avatars to act on interactive objects in the virtual environment, for example, one after the other (e.g., one robot avatar moves an object so another robot avatar can clean under it), simultaneously (e.g., one robot avatar lifts a vehicle so another robot avatar can change the vehicle's tires), etc.

[0036] The robot avatar and / or its associated components may be generated and / or organized for use by the simulation engine 136 in a variety of ways. In some implementations, the graph engine 138 may be configured to represent the robot avatar and / or its components, and possibly other environmental factors, as nodes / edges in a graph. In some implementations, the graph engine 138 may generate these graphs as acyclic directed graphs. In some cases, these acyclic directed graphs may take the form of dependency graphs that define dependencies between various robot components. An example of such a graph is shown in FIG. 4.

[0037] Representing robotic avatars and other components as an acyclic, directed dependency graph may provide various technical advantages. One advantage is that the robotic avatar may be effectively portable, in that its graph may be moved from one virtual environment to another. As one non-limiting example, different rooms / areas of a building may be represented by separate virtual environments. When a robotic avatar "leaves" a first virtual environment corresponding to a first room of a building, for example, by opening and entering a doorway leading to the second room, the robotic avatar's graph may be moved from the first virtual environment to a second virtual environment corresponding to the second room. In some such implementations, the graph may be updated to include nodes corresponding to environmental conditions and / or factors associated with the second room that may not be present in the first room (e.g., different temperature, humidity, particulates in the area, etc.).

[0038] Another advantage is that components of a robot avatar can be easily swapped out and / or reconfigured, for example, for testing and / or training purposes. For example, to test two different light detection and ranging ("LIDAR") sensors on a real-world physical robot, it may be necessary to obtain the two LIDAR sensors, physically swap them out, update the robot's configuration / firmware, and / or perform various other tasks to fully test the two different sensors. In contrast, using the graph and virtual environment techniques described herein, a LIDAR node in a robot avatar's graph representing a first LIDAR sensor can simply be replaced with a node representing a second LIDAR sensor.

[0039] Yet another advantage is that various delays that occur between various robot components in the real world can be easily replicated using the graph. For example, one node representing a simulated sensor on a robot avatar can be configured to impose a delay on the sensor's output being passed up the directed acyclic graph. This delay can correspond, for example, to the frequency of the real-world sensor that corresponds to the simulated sensor. Nodes above the sensor's node may have to "wait" for the sensor output even if they are operating at a higher frequency, such as a higher-frequency real-world robot equipped with a lower-frequency sensor.

[0040] Yet another advantage of using a graph as described herein is that external influences on the behavior of a real-world robot can be represented as nodes and / or edges of the graph that can correspondingly influence the behavior of a robot avatar in a virtual environment. In some implementations, one or more nodes of the directed acyclic graph can represent simulated environmental conditions of the virtual environment. These environmental condition nodes can be coupled to sensor nodes, such that the environmental condition nodes can exert or have their environmental influence on sensors corresponding to the connected sensor nodes. The sensor nodes can then detect this environmental influence and provide sensor data indicative thereof to higher nodes in the graph.

[0041] As one non-limiting example, a node coupled to (and therefore configured to affect) a visual sensor may represent particulates, smoke, or other visual obstructions present in an area. As another example, a node configured to simulate realistic crosswind patterns may be coupled to a wind sensor node of an unmanned aerial vehicle ("UAV") avatar to simulate crosswinds that may affect the flight of a UAV in the real world. Additionally, in some implementations, a node coupled to a sensor node may represent simulated conditions of that sensor of a robotic avatar. For example, a node connected to a visual sensor may simulate dirt and / or debris collecting on the lens of the visual sensor, e.g., using Gaussian blur or other similar blurring techniques.

[0042] 1B shows a non-limiting example of a robot 100 in the form of a robotic arm. An end effector 1061 in the form of a gripper claw is removably attached to the sixth joint 1046 of the robot 100. In this example, the sixth joint 104 1~6 However, this is not meant to be limiting, and the robot may have any number of joints. Robot 100 also includes a base 165, which is shown in a particular selected configuration or "pose."

[0043] FIG. 2 illustrates how the simulation engine 136 simulates three different types of real-world robots 200 in a virtual environment 240. 1~3 The movements of the three corresponding robot avatars are shown in 200's. 1~3 1 shows an example of how three real-world robots 200 can be simulated as 1~3 may take different forms, operate under different constraints, and / or have different capabilities. Although three robots / robot avatars are shown in Figure 2, any number of robots (or, more specifically, robot controllers) may be coupled to and simulated within the virtual environment 240 rendered by the simulation engine 136.

[0044] In this example, the first robot 2001 takes the form of a telepresence robot that can move around an environment using one or more trajectories, but does not include an end effector or tool. The second robot 2002 takes the form of a robot having two movable arms 242L and 242R. The movable arm 242L includes an end effector in the form of a sprayer 244, and the movable arm 242R includes an end effector in the form of a gripper 246. The first and second robots 200 1~2 Each of these includes a visual sensor 248. The third robot 2003 takes the form of a robot arm similar to the robot 100 of Figures 1A-1B.

[0045] Robot 200 1~3Each of the robots includes a respective robot controller (not shown in FIG. 2), which may correspond, for example, to the logic 102 and memory 103 of the robot 100 of FIG. 1A. In the real world (i.e., a non-simulated environment), the robot controller receives sensor data, e.g., from one or more sensors (e.g., 108), that informs the robot controller about various aspects of the environment in which the robot operates. 1~M ) from the various joints (e.g., 104 in FIG. 1B). The robot controller may process the sensor data (perception) and make various determinations and / or decisions (planning), such as path planning, grasp selection, localization, mapping, etc. Many of these determinations and / or decisions may be made by the robot controller using one or more machine learning models. Based on these determinations and / or decisions, each robot controller may control the various joints (e.g., 104 in FIG. 1B). 1~6 ) to actuate these joints.

[0046] When these robot controllers are coupled with the virtual environment 240 simulated by the simulation engine 136, three robot avatars 200' 1~3 can be similarly manipulated by a robot controller. In many cases, the robot controller is "connected" to the virtual environment 240 or is controlled by the actual joints 104. 1~n rather, a robot avatar 200' in a virtual environment 240 1~3 The robot may not even "know" that it is actually controlling a virtual joint.

[0047] However, instead of receiving actual sensor data from actual sensors (e.g., 108), the simulation engine 136 may simulate sensor data in the virtual environment 240, for example, based on the perspective of each robot avatar 200′ within the virtual environment 240. As indicated by the arrows in FIG. 2 , this simulated sensor data may be injected by the simulation engine 136 into a sensor data channel between one or more real-world sensors of each robot 200 and a robot controller integral with the robot 200. Thus, from the perspective of the robot controller, the simulated sensor data may be indistinguishable from actual sensor data.

[0048] Additionally, as indicated by the arrows in FIG. 2, the joint commands generated by the robot controller based on this simulated sensor data are provided to the simulation engine 136, which then controls the joints of the robot avatar 200' instead of the actual robot joints. 1~3 For example, joint commands received from a given robot controller may be intercepted from a joint command channel between the robot controller and one or more joints of the given robot.

[0049] A fully functional robot need not be coupled to the simulation engine 136 to simulate a corresponding robot avatar. In FIG. 2 , for example, the third robot 2003 is not coupled to the simulation engine 136. Instead, a simulated robot controller 250 is configured to simulate inputs (e.g., sensor data) and outputs (e.g., joint commands) for the third robot 2003. The simulated robot controller 250 may take various forms, such as a computing device with one or more processors and / or other hardware. The simulated robot controller 250 may be configured to provide inputs and receive outputs in a manner as similar as possible to an actual robot controller integral with the third robot 2003. Thus, for example, the simulated robot controller 250 may output joint commands at the same frequency as the joint commands output by the actual robot controller of the third robot 2003. Similarly, the simulated robot controller 250 may capture sensor data at the same frequency as the actual sensors of the third robot 2003. Additionally or alternatively, in some implementations, aspects of the robot that form the robot controller, such as logic 102, memory 103, and / or various buses to joints / sensors, may be physically extracted from the robot and coupled to the simulation system 130 as a standalone robot controller.

[0050] Robots (e.g., 200 1~3), standalone robot controllers, and / or simulated robot controllers (e.g., 250) may be coupled, or "connected," to the virtual environment 240 via the simulation engine 136 using various communication technologies. If a particular robot controller or simulated robot controller is co-located with the simulation system 130, it may be coupled to the simulation engine 136 using one or more personal area networks (e.g., Bluetooth), various types of universal serial bus ("USB") technologies, or other types of wired technologies. If a particular robot controller (simulated, standalone, or integral to a robot) is remote from the simulation system 130, it may be coupled to the simulation engine 136 via one or more local area networks and / or wide area networks, such as the Internet. As described with respect to FIGS. 3A and 3B, when using remote robot controllers, network interruptions and / or traffic can cause delays and / or other issues that can be addressed using various techniques described herein.

[0051] Different robots and / or robot components may operate at different frequencies. For example, one robot's visual sensor may generate visual sensor data at a frequency of 500 MHz. Even if the controller for the same robot operates at 1 GHz, it may be limited by this lower frequency visual sensor. Another robot's controller may also operate at a different frequency, such as 200 MHz. When these different frequency robot controllers are connected to the virtual environment 240, the simulation engine 136 can simulate their operating frequencies relative to each other to maintain a degree of realism, which is beneficial in generating training examples for training machine learning models.

[0052] In the real world, time progresses without interruption while robots operate at their respective frequencies. If a particular robot falls behind, the other robots continue operating without slowing down. This makes replicating robot task performance difficult because replicating this phenomenon may not be possible or realistic, regardless of the cause of a particular robot's delay. In contrast, in a simulated environment, such as simulated environment 140, time can be paused as needed, for example, to allow a robot that is lagging for some reason to "catch up" with the other robots. As a result, it is easier to deterministically replicate robot task performance in virtual environment 240.

[0053] In some implementations, the simulation engine 136 may operate a simulated world clock that can be advanced, paused, and even rewound as needed. In various implementations, the simulated world clock may operate at a frequency that matches or exceeds the highest frequency of the robot simulated by the robot avatar in the virtual environment 240.

[0054] FIG. 3A illustrates how two different robot avatars (ROBOT_A and ROBOT_B), corresponding to two different real-world robots (not shown) operating at different frequencies, may operate over time in a virtual environment, such as virtual environment 240. A timeline 360 ​​running from left to right in time includes tick marks corresponding to clock cycles of a simulated world clock. As seen in FIG. 3A, ROBOT_A performs operations at a frequency of every 5 clock cycles. ROBOT_B operates at half the frequency of ROBOT_A, i.e., 10 clock cycles. ROBOT_A and ROBOT_B are shown as performing operations synchronously with each other for simplicity, although synchronization is not required.

[0055] Assuming that neither ROBOT_A nor ROBOT_B experiences any delays due to, for example, heavy network traffic, network interruptions, etc., both robot avatars may operate as shown in FIG. 3A. However, this is not always the case. For example, FIG. 3B shows a similar timeline 360. In FIG. 3B, ROBOT_A and ROBOT_B operate at their normal frequency until the portion of the timeline labeled "ROBOT_B Delay," at which point ROBOT_B experiences a delay of three cycles. This delay may be caused by a variety of factors, such as the network problem mentioned above, or it may be because the robot controller controlling ROBOT_B is experiencing a problem that causes the delay, such as a memory overload (e.g., caused by an entropy spike in the virtual environment 240), an overheating processor, an interruption (e.g., a scheduled software update), etc.

[0056] If the simulated world clock continues to tick without slowing down, ROBOT_A will continue to operate, performing actions on the same clock cycle as it did in FIG. 3A . However, ROBOT_B may temporarily freeze. When ROBOT_B resumes, (e.g., from ROBOT_A's perspective) ROBOT_B and / or objects it interacts with may appear to have "jumped" or "warped" from one pose / position to another. In some cases, such as when ROBOT_A and ROBOT_B are performing tasks collaboratively, this may cause problems for ROBOT_A and / or ROBOT_B. Even if ROBOT_A and ROBOT_B are not collaborating and are performing unrelated tasks in different parts of the virtual environment 240, it may be difficult or impossible to recreate this situation as a training example.

[0057] Thus, in various implementations, the simulation engine 136 may be configured to pause the simulated world clock until a lagging robot avatar of the multiple robot avatars can proceed to the next stage of action. For example, in FIG. 3B , the time interval labeled “ROBOT_B Late” has no tick marks because the simulated world clock is paused. After a time interval that appears to be three clock cycles, ROBOT_B can again perform its action. As a result, at that time, or at some point shortly thereafter (e.g., the next clock cycle), the simulated world clock may be restarted, and ROBOT_A and ROBOT_B may resume their respective actions, as shown in FIG. 3B . In particular, comparing the right halves of the timeline 360 ​​in FIGS. 3A and 3B , the robot avatar in FIG. 3B performs the same action three clock cycles later than in FIG. 3A .

[0058] 4 illustrates an exemplary acyclic directed graph 400 that may be generated, for example, by graph engine 138 of simulation system 130, according to various implementations. In this example, graph 400 takes the form of a dependency graph that includes nodes representing components of a robot (not shown), environmental conditions, sensor conditions, etc. The particular layout and arrangement of FIG. 4 is not meant to be limiting. The various components shown in FIG. 4 may be arranged differently relative to other components in other implementations. Furthermore, only a few exemplary components are shown. Many other types of components are contemplated.

[0059] The graph 400 includes a robot controller 402, which is external to the virtual environment 240, as a root node. In other implementations, the robot controller may not be represented as a node; instead, the root node may act as an interface between the robot controller and child nodes (which may represent sensors and / or other robot controllers simulated within the virtual environment). The robot controller 402 may be implemented with a variety of hardware and software and may include components such as logic 102, memory 103, and possibly the bus of FIG. 1A. From a logical perspective, the robot controller 402 may include a perception module 403, a planning module 406, and an execution module 407. While shown as part of the root node in FIG. 4, in some implementations, one or more of these modules 403, 406, and 407 may be represented as their own standalone nodes connected to other nodes via edges. Modules 403, 406, and / or 407 may operate, in part, using machine learning models, such as an object recognition module, a model to assist path planning, or a model to assist grasp planning. One or more of these machine learning models may be trained using training data generated by operating multiple robotic avatars in a single virtual environment, as described herein.

[0060] The perception module 403 may receive sensor data from any number of sensors. In the real world, this sensor data may come from real-life sensors of the robot with which the robot controller 402 is integrated. In the virtual environment 240, this sensor data may be simulated by and propagated upward from various sensor nodes 4081, 4082, 4083, ..., which represent virtual sensors simulated by the simulation engine 136. For example, a vision sensor 4081 may provide simulated visual data, an anemometer 4082 may provide simulated data regarding wind speed, a torque sensor 4083 may provide simulated torque data, e.g., captured at one or more robot joints 404, etc.

[0061] In some implementations, simulated environmental conditions may also be represented as nodes in graph 400. These environmental conditions may be propagated upward from their respective nodes to sensors that would normally sense them in real life. For example, airborne particles (e.g., smoke) desired to be simulated in virtual environment 240 may be represented by airborne particle node 411. In various implementations, aspects of the desired airborne particles to be simulated, such as density, average particle size, etc., may be configured within node 411 by, for example, a user defining node 411.

[0062] In some implementations, environmental conditions may affect the sensor apart from what is observed by the sensor. This is demonstrated by the Gaussian blur node 415, which may be configured to simulate the effect of particulate debris collected on the lens of the visual sensor 4081. To this end, in some implementations, the lens of the visual sensor 4081 may be represented by its own node 413. In some implementations, having a separate node for a sensor component, such as a lens, may allow that component to be swapped out and / or configured separately from other components of the sensor. For example, different lenses may be deployed for the visual sensor node 4081 by simply replacing the lens node 413 with a different lens node, e.g., having a different focal length. Instead of the arrangement shown in FIG. 4, in some implementations, a particular floating node 411 may be a child node of the lens node 413.

[0063] As another example of an environmental condition, assume that the robot represented by graph 400 is a UAV configured, for example, to pick up and / or deliver packages. In some such implementations, a crosswind node 417 may be defined that simulates, for example, a crosswind that the UAV may experience when at a particular altitude, within a particular area, etc. Crosswind node 417 may be a child node of anemometer node 4082 such that the simulated crosswind may be propagated upward to and detected by the anemometer represented by node 4082.

[0064] The perception module 403 may be configured to collect sensor data from various simulated sensors represented by nodes 4081, 4082, 4083, ... during each iteration of the robot controller 402 (which may be done at the operating frequency of the robot controller relative to the simulated world clock mentioned above). The perception module 403 may then generate, for example, a current state. Based on this current state, the planning module 406 and / or the execution module 407 may make various decisions and generate joint commands to actuate the joints of the robot avatar represented by the graph 400.

[0065] The planning module 406 may perform what is sometimes referred to as "offline" planning to define, at a high level, a series of waypoints along a path to satisfy one or more reference points for the robot. The execution module 407 may, for example, generate joint commands that actuate the robot avatar joints to satisfy these waypoints (as closely as possible), taking into account sensor data received during each iteration. For example, the execution module 407 may include a real-time trajectory planning module 409 that generates joint commands taking into account the most recent sensor data. These joint commands are then used to control the various simulated robot avatar joints 404. 1~M to actuate various types of joints.

[0066] In some implementations, the real-time trajectory planning module 409 can provide data, such as object recognition and / or pose data, to a grasp planner 419. The grasp planner 419 then generates joint commands that cause the simulated robot gripper to take various actions, such as grasping, releasing, etc., and controls the gripper joints 404. 1~N In other implementations, the grasp planner 419 may not be represented by its own node, but may be incorporated into the execution module 407. Additionally or alternatively, the real-time trajectory planning module 409 may provide the grasp planner 419 with the other robot joints 404. N+1~M The robot generates joint commands to operate the robot joints 404 in various ways, and sends these to other robot joints 404. N+1~M can be provided to.

[0067] Referring now to FIG. 5 , an exemplary method 500 for implementing selected aspects of the present disclosure will be described. For convenience, the operations of the flowcharts are described with reference to a system that performs the operations. This system may include various components of various computer systems. For example, some operations may be performed in the robot 100, while other operations may be performed by one or more components of the simulation system 130. Furthermore, although the operations of the method 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0068] At block 502, the system may simulate, e.g., by simulation engine 136, a three-dimensional virtual environment (e.g., 240) including one or more interactive objects. As previously described and shown in FIG. 2, any number of external robot controllers (integrated with a robot, standalone, or simulated) may be connected to the virtual environment. At block 504, the system may generate, e.g., by graph engine 138, a directed acyclic graph for each external robot controller connected to the virtual environment. In some implementations, the system may poll the external robot controllers for information that can be used to generate the graph. In other implementations, the system may have a database of graphs that can be matched to robot controllers.

[0069] At block 506, the system may simulate, for example, multiple robot avatars independently and simultaneously controlled by multiple robot controllers external to the virtual environment by the simulation engine 136 and based on the graph generated at block 504. At block 508, the system may provide, to each robot controller of the multiple robot controllers, sensor data generated from the perspective of the robot avatar controlled by the robot controller. At block 510, the system may be configured to receive, from each robot controller of the multiple robot controllers, joint commands to actuate one or more joints of a respective robot avatar controlled by the robot controller.

[0070] At block 512, the system may actuate one or more joints of each robot avatar according to the corresponding joint commands. In some implementations, the actuation causes two or more of the robot avatars to interact with interactive objects in the virtual environment. In various implementations, data generated as a result of the actuations at 512 may be used as training data for training one or more machine learning models used by various components of the robot controller, such as the perception module 403, the planning module 406, and / or the execution module 407. For example, negative results may be used as negative training examples. Positive results may be used as positive training examples. In implementations where reinforcement learning is used, outcomes may be used to determine rewards, which may then be used to train a reinforcement learning policy.

[0071] 6 is a block diagram of an exemplary computer system 610. Computer system 610 typically includes at least one processor 614 that communicates with multiple peripheral devices via a bus subsystem 612. These peripheral devices may include, for example, a storage subsystem 624, such as a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices enable user interaction with computer system 610. Network interface subsystem 616 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.

[0072] The user interface input devices 622 may include a keyboard, a pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 610 or communications network.

[0073] The user interface output devices 620 may include a display subsystem, a printer, a fax, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 610 to a user or to another machine or computer system.

[0074] The storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include logic for executing selected aspects of the method 500 and / or implementing one or more aspects of the robot 100 or the simulation system 130. The memory 625 used by the storage subsystem 624 may include multiple memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program and data files and may include a hard disk drive, CD-ROM drive, optical drive, or removable media cartridge. Modules that implement the functionality of a particular implementation may be stored by the file storage subsystem 626 in the storage subsystem 624 or on another machine accessible by the processor 614.

[0075] Bus subsystem 612 provides a mechanism for allowing the various components and subsystems of computer system 610 to communicate with each other as intended. Although bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0076] The computer system 610 can be of various types, such as a workstation, a server, a computing cluster, a blade server, a server farm, a smartphone, a smartwatch, smart glasses, a set-top box, a tablet computer, a laptop, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computer system 610 shown in Figure 6 is intended only as a specific example intended to illustrate some implementations. Many other configurations of the computer system 610 are possible, having more or fewer components than the computer system shown in Figure 6.

[0077] While several implementations have been described and illustrated herein, various other means and / or structures may be used to perform the functions and / or obtain one or more of the results and / or advantages described herein, and each such variation and / or modification is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application in which the teachings are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. Accordingly, it should be understood that the foregoing implementations are presented by way of example only, and that, within the scope of the appended claims and their equivalents, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

Claims

1. 1. A method implemented using one or more processors, comprising: simulating a three-dimensional virtual environment including interactive objects, the three-dimensional virtual environment including a plurality of robotic avatars independently and simultaneously controlled by a corresponding plurality of robotic controllers external to the three-dimensional virtual environment, a given robotic controller of the plurality of robotic controllers being integral with a physical robot and operatively coupled to the one or more processors; providing, to each robot controller of the plurality of robot controllers, sensor data generated from a viewpoint of the respective robot avatar of the plurality of robot avatars controlled by the robot controller; receiving, from each robot controller of the plurality of robot controllers, joint commands to actuate one or more joints of the respective robot avatar of the plurality of robot avatars controlled by the robot controller; actuating one or more joints of each robot avatar of the plurality of robot avatars according to corresponding joint commands, wherein said actuating causes two or more of the plurality of robot avatars to interact with the interactive object in the three-dimensional virtual environment, said actuating including operating a robot avatar controlled by the given robot controller integral with the physical robot at the same real-world frequency as the physical robot; simulating the three-dimensional virtual environment includes generating, for each robot avatar of the plurality of robot avatars, a directed acyclic graph using nodes representing sensors of the robot avatar, the directed acyclic graph including at least one child node representing a simulated sensor of the robot avatar and a parent node representing the robot controller to which sensor data is output from the child node; at least one child node representing a simulated sensor of the robot avatar imposes a delay on sensor data output from the child node to the parent node in the directed acyclic graph, the delay corresponding to a frequency of a real-world sensor corresponding to the simulated sensor; method.

2. 10. The method of claim 1, wherein the sensor data provided to the given robot controller is injected into a sensor data channel between one or more real-world sensors of the physical robot and one or more processors of the robot controller integral with the physical robot.

3. 3. The method of claim 1, wherein the joint commands received from the given robot controller are intercepted from a joint command channel between one or more processors of the robot controller and one or more joints of the physical robot.

4. The method of claim 1 , wherein the directed acyclic graph is a dependency graph in which edges between nodes represent dependencies between the child node and the parent node represented by the node.

5. The method of claim 1 or 4, wherein one or more nodes of the directed acyclic graph represent simulated environmental conditions of the three-dimensional virtual environment.

6. The method of claim 4 or 5, wherein one or more nodes of the directed acyclic graph represent simulated conditions of simulated sensors of the robot avatar.

7. operating a simulated world clock of said three-dimensional virtual environment at a given frequency; The method of claim 1 , further comprising: operating a first robotic avatar of the plurality of robotic avatars at a first frequency that is lower than the given frequency.

8. The method of claim 1 or 7, further comprising pausing a simulated world clock until one of the plurality of robotic avatars advances to a next movement stage.

9. 1. A system comprising one or more processors and a memory storing instructions, the instructions causing the one or more processors to, in response to execution of the instructions by the one or more processors: simulating a three-dimensional virtual environment including interactive objects, the three-dimensional virtual environment including a plurality of robotic avatars independently and simultaneously controlled by a corresponding plurality of robotic controllers external to the three-dimensional virtual environment, a given robotic controller of the plurality of robotic controllers being integral with a physical robot and operatively coupled to the one or more processors; providing, to each robot controller of the plurality of robot controllers, sensor data generated from a viewpoint of the respective robot avatar of the plurality of robot avatars controlled by the robot controller; receiving, from each robot controller of the plurality of robot controllers, joint commands to actuate one or more joints of the respective robot avatar of the plurality of robot avatars controlled by the robot controller; and Actuating one or more joints of each robot avatar of the plurality of robot avatars according to corresponding joint commands, wherein the actuating causes two or more of the plurality of robot avatars to interact with the interactive object in the three-dimensional virtual environment, and the actuating includes operating a robot avatar controlled by the given robot controller integral with the physical robot at the same real-world frequency as the physical robot. Execute simulating the three-dimensional virtual environment includes instructions for generating, for each robot avatar of the plurality of robot avatars, a directed acyclic graph with nodes representing sensors of the robot avatar, the directed acyclic graph including at least one child node representing a simulated sensor of the robot avatar and a parent node representing the robot controller to which sensor data is output from the child node; at least one child node representing a simulated sensor of the robot avatar imposes a delay on sensor data output from the child node to the parent node in the directed acyclic graph, the delay corresponding to a frequency of a real-world sensor corresponding to the simulated sensor.

10. 10. The system of claim 9, wherein the sensor data provided to the given robot controller is injected into a sensor data channel between one or more real-world sensors of the physical robot and one or more processors of the robot controller integral with the physical robot.

11. 11. The system of claim 9 or 10, wherein the joint commands received from the given robot controller are intercepted from a joint command channel between one or more processors of the robot controller and one or more joints of the physical robot.

12. 10. The system of claim 9, wherein the directed acyclic graph is a dependency graph in which edges between nodes represent dependencies between the child node and the parent node represented by the node.

13. At least one non-transitory computer-readable storage medium containing instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations: simulating a three-dimensional virtual environment including interactive objects, the three-dimensional virtual environment including a plurality of robotic avatars independently and simultaneously controlled by a corresponding plurality of robotic controllers external to the three-dimensional virtual environment, a given robotic controller of the plurality of robotic controllers being integral with a physical robot and operatively coupled to the one or more processors; providing, to each robot controller of the plurality of robot controllers, sensor data generated from a perspective of the respective robot avatar of the plurality of robot avatars controlled by the robot controller, the sensor data provided to each robot controller being recognized by the robot controller as real-world sensor data; receiving, from each robot controller of the plurality of robot controllers, joint commands to actuate one or more joints of the respective robot avatar of the plurality of robot avatars controlled by the robot controller; and Actuating one or more joints of each robot avatar according to corresponding joint commands, where the actuating causes two or more of the plurality of robot avatars to interact with the interactive object in the three-dimensional virtual environment, and the actuating includes operating a robot avatar controlled by the given robot controller integral with the physical robot at the same real-world frequency as the physical robot. Execute simulating the three-dimensional virtual environment includes instructions for generating, for each robot avatar of the plurality of robot avatars, a directed acyclic graph with nodes representing sensors of the robot avatar, the directed acyclic graph including at least one child node representing a simulated sensor of the robot avatar and a parent node representing the robot controller to which sensor data is output from the child node; At least one child node representing a simulated sensor of the robot avatar imposes a delay on sensor data output from the child node to the parent node in the directed acyclic graph, the delay corresponding to a frequency of a real-world sensor corresponding to the simulated sensor.

Images