Coordinating Multiple Robots Using Graph Neural Networks

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, for controlling a plurality of robots, one of the methods including obtaining state data representative of a current state of an environment, generating graph data from the state data representative of a graph of the current state of the environment, processing the graph data using a graph neural network to generate a graph output including respective updated feature representations for each of the robot nodes of the graph, and selecting respective actions to be performed by each of the robots based on the graph output.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 252,799, filed October 6, 2021, the entire disclosure of which is incorporated herein by reference.

[0002] This specification relates to robotics, and more particularly to planning robotic movements of multiple robots. [Background technology]

[0003] Robotics planning refers to scheduling the physical movements of robots to perform tasks. Certain applications require the coordination of multiple robots within complex environments. For example, in industrial applications, multiple robot arms may operate simultaneously within a complex workspace to maximize production throughput. Each of the multiple robot arms may be controlled to move along a respective motion trajectory to reach one of multiple specified target locations and perform an operation. Coordination of the motion paths and schedules of the multiple robot arms is crucial to accomplishing the operation task within an optimal time frame without colliding with each other or obstacles in the work environment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 20200143239 [Patent Document 2] International Publication No. 2021 / 152515 [Non-patent literature]

[0005] [Non-Patent Document 1] Silver et al., "Mastering the game of go without human knowledge," Nature, 550:354–359, October 2017 [Non-patent document 2] Kocsis et al., "Bandit based monte-carlo planning," European conference on machine-learning, pp. 282-293, Springer, 2006. Summary of the Invention [Means for solving the problem]

[0006] This specification describes technology related to using graph neural network (GNN) processing to coordinate the behavior of multiple robots in an environment.

[0007] In one innovative aspect, a method for planning actions to be performed by multiple robots in an environment using a graph neural network is described. A computing system can repeatedly execute the method to generate actions to be performed by each of the robots at each of a plurality of time steps. The generated robot actions can be communicated to the robots to control their movement in the environment at each time step. A goal of generating the robot actions includes controlling the robots to accomplish a specified task within an optimal time frame without colliding with each other or other objects in the environment. As an example, a specified task may include moving each of the robots to a respective target location among a plurality of target locations. As another example, a specified task may include moving one or more of a plurality of robots along a specified path, such as controlling a robot arm holding a milling bit to strictly follow a milling pattern. As another example, a specified task may include multiple robots cooperatively performing a task, such as controlling two robot arms to hold two workpieces and a third robot arm to perform welding.

[0008] In some implementations, the environment can be a physical environment in which one or more robots operate, e.g., a physical workcell. In some other implementations, the environment can be a virtual representation of a physical environment, e.g., a simulated operating environment in which robot motion can be simulated. In the case of a simulated operating environment, the system can interactively plan robot behavior in a simulator that receives planned actions generated by a planning system and outputs updated state observations of the environment.

[0009] The planning process may begin by obtaining state data representing a current state of the environment, the computing system generating graph data representing a graph of the current state of the environment, the computing system processing the graph data using a graph neural network to generate a graph output, and selecting respective actions to be performed by each of the robots based on the graph output.

[0010] The graph representing the current state of the environment includes a plurality of nodes and a plurality of edges. Each edge connects a respective pair of nodes from the plurality of nodes. The plurality of nodes includes a respective robot node for each of the robots and a respective goal node for each of the goal locations. The graph data includes a respective initial feature representation for each of the robot nodes and each of the goal nodes. By way of example, the initial feature representation for each robot node may include one or more coordinates of a tooltip, coordinates of each of the joints, and a current joint angle. The initial feature representation for each goal node may include coordinates of the goal location.

[0011] In some implementations, the initial feature representation of one or more of the goal nodes may further include compatibility information for the corresponding one or more goals. The compatibility information for a goal node may, for example, identify a subset of robots that are compatible for operating on the corresponding goal. For example, multiple robots may be configured with various tool tips that are compatible or incompatible for operating on a particular goal, and the compatibility information for a goal node corresponding to a particular goal may indicate which robots are compatible.

[0012] In some implementations, the initial feature representation of one or more of the goal nodes may further include information regarding orientation constraints for the corresponding one or more goals. For example, to operate on a particular goal (e.g., to perform a spot weld on a goal location), a robot arm may need to be constrained to a particular orientation or a particular range of orientations. The orientation constraint information of a goal node corresponding to a particular goal may indicate the constrained orientation or the range of constrained orientations.

[0013] In some implementations, the graph may further include one or more obstacle nodes corresponding to one or more obstacles present in the environment, and a respective initial feature representation for each of the obstacle nodes is included in the graph data.

[0014] The graph's connection topology can be used to characterize the interaction relationships between robot nodes, goal nodes, and obstacle nodes. For example, each robot node can be connected by edges to each of the other robot nodes, each of the goal nodes, and each of the obstacle nodes to indicate the robot's interactions with another robot and with goal objects or obstacles. In some implementations, none of the goal nodes or obstacle nodes are connected to any other goal nodes or obstacle nodes. This configuration indicates that there is no interaction between goal objects and obstacles and simplifies graph processing.

[0015] In a particular implementation of the method, GNN processing of a graph representation of the state of the environment is incorporated into an interactive planning process. In particular, a computing system performs multiple planning iterations starting from a current state of the environment to generate planning data. Each planning iteration includes traversing the state of the environment, starting from the current state, until a leaf state of the environment is reached. Next, from the state data characterizing the leaf states, the computing system generates graph data representing a graph of the leaf states of the environment and processes the graph data using the GNN to generate a graph output including updated feature representations of each of the robot nodes. The computing system then updates the planning data according to the updated feature representations for future iterations. Finally, after a planning iteration, the computing system uses the planning data to select a robot behavior.

[0016] An implementation of the planning iterations may include, for example, a look-ahead search of a state tree having nodes representing states of the environment, starting from a root node representing the current state and traversing the state tree until a leaf node is reached. This process may include selecting one of multiple edges connecting to the first node based on the edge's action score to identify the next node in the tree. The action score may optionally be adjusted by an amount dependent on the prior probability of the action and / or a value score representing the predicted value of being in the state for successfully completing the task. The action probability and value score may be generated based on the updated feature representation output by the GNN.

[0017] The subject matter described herein may be implemented in particular embodiments to achieve one or more of the following advantages: The use of graph representations and graph neural network processing to plan robot behavior increases accuracy, efficiency, and the feasibility of scaling up to many robots working in parallel in complex environments.

[0018] Robotics planning has traditionally required extensive manual programming to direct how robotic components should move to accomplish specific tasks. Manual programming is tedious, time-consuming, and error-prone. Furthermore, robot behavior control schedules manually created for one task environment generally cannot be used for other tasks or different tasks. Machine learning techniques have shown great promise in improving the adaptability of robot control. However, significant challenges exist in using machine learning techniques in applications involving multiple robots (e.g., robotic arms) working closely together in relatively small cells with many tasks, a configuration commonly found in welding, assembly, and object manipulation. In such dense motion planning applications, the robots are close to each other, their work volumes overlap significantly, and the tasks are numerous and lack a prescribed arm allocation or completion order. Motion planning must take into account other robots moving simultaneously to avoid collisions. Furthermore, the optimal allocation and scheduling of robotic arms to goals is not known a priori and must be determined in conjunction with collision avoidance. These characteristics make the scalability of existing systems and approaches a major challenge, as the time complexity of most conventional algorithms is exponential in the number of arms and targets. As a result, in real-world applications, dense motion planning (i.e., motion planning for multiple robots that cooperate closely) is typically performed manually in a slow and repetitive process, and any changes to the task or environment require time-consuming replanning.

[0019] Implementations that use multi-layer perceptron (MLP) networks to represent policies and value functions in reinforcement learning suffer from the same exponential scaling as conventional methods, because the models do not exploit the structure of the scene and do not reuse learned knowledge between robots and robot-robot pairs (for collision avoidance), and between robot-target pairs (for motion planning).

[0020] In contrast, a graph representation of the state of the environment encodes the structure of the scene into a network architecture, enabling knowledge reuse through weight sharing and providing an efficient representation of complex interaction relationships among multiple robots and between robots and objects in the environment in a structured data space. GNN processing allows each robot node to efficiently learn an embedding containing information about its interactions with other objects in the environment to generate planned behaviors to be executed in response to the state of the environment. Advantages of such an approach include the ability to effectively learn in very large / complex state spaces. Specifically, this leads to machine learning systems that achieve significantly improved performance in planning the movements of multiple robots while simultaneously significantly reducing the amount of processing power and / or processing time. The reduction in processing power can, in some implementations, lead to a significant reduction in power consumption or the amount of specialized hardware required to execute the planning operations. The reduction in processing time enables real-time control of multiple robots in complex working environments. Furthermore, by combining GNN processing of the graph representation of the environment's state with look-ahead search, the quality of multi-robot planning and coordination can be significantly improved, including generating robot behaviors that significantly reduce the time it takes to complete multiple tasks with multiple robots.

[0021] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 illustrates an exemplary multi-robot control system. [Figure 2A] FIG. 1 illustrates an example of a graph representation of an environment including multiple robots interacting with goal locations and obstacles. [Figure 2B] FIG. 10 illustrates processing graph data of a robot node to generate an updated feature representation of the robot node. [Figure 3] FIG. 1 illustrates processing graph data to generate value and tactic predictions for robot behavior planning. [Figure 4A] FIG. 1 illustrates an example of a multi-robot planning and control system. [Figure 4B] FIG. 1 illustrates a data flow diagram for an example iterative planning process for generating planning data for multiple robots. [Figure 5] 1 is a flow diagram of an example process for generating robot behavior from state data. [Figure 6] 10 is a flow diagram of another example process for generating robot behaviors from state data in planning iterations. DETAILED DESCRIPTION OF THE INVENTION

[0023] Like reference numbers and designations in the various drawings indicate like elements.

[0024] This specification generally describes a system that uses graph neural network (GNN) processing to coordinate the actions of multiple robots in an environment. The goal of behavior planning is to generate action plans and schedules for all robots to perform specified tasks within optimal time frames without colliding with each other or other objects in the environment.

[0025] As used herein, a robot refers to a machine having one or more movable components that can be controlled by command signals to move the robot to a desired position, orientation, or both. One example of a robot is a remotely operated vehicle that can be controlled to reach a target location along a specified motion path. Another example of a robot is a robot arm having a base position, one or more joints, and optionally, a tool tip. One or more of the joints may include motor-driven actuators that change the joint angle to control the orientation and tip position of the robot arm with its motion path. A robot arm may have many degrees of freedom depending on the number of joints and / or the type of tool tip.

[0026] As used herein, a task is an operation performed by a robot. An example of a task for multiple teleoperated vehicles is to drive each of the vehicles to a respective target location among multiple target locations. Another example of a task for multiple robotic arms is for each robotic arm to reach a respective target object and, optionally, perform an operation on the target object. Operations may include picking up, moving, part positioning, welding, and surface polishing of target objects, to name a few.

[0027] As used herein, a generated behavior for a robot is a data structure that includes values ​​for some or all of the robot's controllable degrees of freedom. When the robot receives the generated behavior, the generated behavior controls the robot's movement. For example, a generated behavior that includes updated joint positions and / or joint angles sent to a robot arm can control the robot arm to perform a movement specified by the parameter values ​​in the behavior. As another example, a generated behavior may include respective torques to be applied to each of the robot arm's joints.

[0028] FIG. 1 illustrates an exemplary multi-robot control system 110.

[0029] Multi-robot control system 110 is an example of a system in which the systems, components, and techniques described below are implemented as a computer program on one or more computers at one or more locations.

[0030] The multi-robot control system 110 controls multiple robots 104a-104n within an environment 102 that includes multiple target locations 106a-106n. The target locations are locations related to the robots' tasks, and may be locations intended to be reached by one or more of the robots depending on the task. The environment 102 may further include one or more obstacles 108. An obstacle 108 may be any object in the environment whose position is not controlled by the system 110, i.e., any object in the environment other than the robots 104a-104n.

[0031] In some implementations, the environment 102 may be a physical environment in which one or more robots operate, such as a physical work cell, while in some other implementations, the environment 102 may be a virtual representation of a physical environment, such as a simulated operating environment in which robot motion may be simulated.

[0032] At each of a number of time steps, multi-robot control system 110 controls robots 104a-104n by collecting state data of the environment and selecting a respective behavior for each of the robots at the time step.

[0033] After the multi-robot control system 110 selects an action for a particular time step, the system updates the state data and obtains a reward signal according to the updated state data. The reward signal is associated with an objective to be achieved by the system 110 by controlling the robots. For example, the objective may be to accomplish the task of moving each of the robot arms to reach a respective one of the target locations within the shortest time frame without the robot arms colliding with each other or one of the objects in the environment. Thus, the system 110 may assign a large positive value to the reward signal when the task is accomplished and terminate the episode; when a collision occurs, assign a large negative value to the reward signal and terminate the episode; and account for the passage of time and assign a small negative value or discount factor to each additional time step. In another example, the system may use a shaped reward signal, for example, by assigning a positive value to the reward signal for each time step in which the robot is approaching the target and a negative value to the reward signal for each time step in which the robot is moving away from the target. Using a shaped reward signal may allow the system to overcome challenges associated with sparse reward signals.

[0034] In one particular example, to calculate the reward, the system first calculates the score of the state as the average of the scores of each goal. The goal score is set to 1 for goals that are solved (e.g., reached by the robot tool tip at a particular orientation) or to a shaped reward up to 0.5 for goals that are not solved. The goal score can be a combination of the distance score and the orientation score. The score may be calculated as follows:

number

[0035] The combination of the two scores ensures that the orientation score is only taken into account when the end effector is near the goal. The reward at each time step can be defined as the difference in score between that state and the previous step, with a small penalty for joint space acceleration to promote smoothness.

[0036] The system 110 can use the expected cumulative future reward to guide a section of behavior at each of multiple time steps. The cumulative future reward for a particular time step can be, for example, the time-discounted sum of future rewards received beginning after the behavior at the time step is performed by the robot. The "expected" cumulative future reward for a given behavior at a particular time step is an estimate generated by the system 110 of the cumulative future reward that would be received if the given behavior were performed at the particular time step.

[0037] The multi-robot control system 110 includes a state data collection engine 112 that collects state data 122 characterizing the state of the environment 102 at each of multiple time steps. If the environment 102 is a physical environment, the state data 122 may be generated by sensors, such as cameras and position sensors, within the environment 102. The state data collection engine 112 may receive the state data 122 directly from the sensors or process sensor data to generate the state data. The state data may include information about the robots, target locations, and obstacles, including, for example, the position coordinates of the robots, target locations, and obstacles. The robot information may also include each robot's tool tip coordinates, joint coordinates, and joint angles, as well as motion parameters such as the velocity and acceleration of the robot or robot components. If the environment is a simulated environment, the state data 122 may be generated by a simulator that receives actions generated by the planning system and outputs updated state observations of the environment.

[0038] The multi-robot control system 110 further includes a graph data generation engine 114 that generates graph data 124 representing a graph from the state data at each of a plurality of time steps. The graph includes a plurality of nodes and a plurality of edges, each edge connecting a respective pair of nodes in the graph.

[0039] The nodes in the graph may be categorized as robot nodes representing the robots 104a-104n, goal nodes representing goal locations 106a-106n, and obstacle nodes representing obstacles 108.

[0040] The graph data generation engine 114 maps the robots 104a-104n to robot nodes, the goal locations 106a-106n to goal nodes, and the obstacles 108 to obstacle nodes. The mapping of robot nodes and the mapping of goal nodes can be one-to-one mapping, i.e., the graph data generation engine 114 maps each robot to a corresponding robot node and each goal location to a corresponding goal node.

[0041] The mapping of obstacle nodes can also be a one-to-one mapping, i.e., the graph data generation engine 114 maps each obstacle to a corresponding object node. Alternatively, the graph data generation engine 114 may perform a geometrical decomposition of obstacles with non-trivial sizes and / or shapes and map the obstacle to multiple obstacle nodes corresponding to multiple locations.

[0042] In mapping the robots, goal locations, and obstacles to respective nodes of the graph 200, the graph data generation engine 114 encodes state data for the robots, goal locations, and obstacles into initial feature vectors for the corresponding nodes. For example, the initial feature vector associated with a robot node may include the position coordinates, joint position coordinates, and joint angles of the corresponding robot. In one particular example, the feature vector for a robot node (e.g., a node representing a robot arm) may be a concatenation of multiple parameters specifying one or more of joint configurations, joint velocities, home positions and orientations, tool center point (TCP) positions and orientations, and / or dwelling statuses.

[0043] The initial feature vector associated with the goal node or obstacle node may include the position coordinates of the corresponding goal position or obstacle. The initial feature vector is included in the graph data.

[0044] In some implementations, the initial feature representation of one or more of the goal nodes may further include compatibility information for the corresponding one or more goals. The compatibility information for a goal node may, for example, identify a subset of robots that are compatible for operating on the corresponding goal. For example, multiple robots may be configured with various tool tips that are compatible or incompatible for operating on a particular goal, and the compatibility information for a goal node corresponding to a particular goal may indicate which robots are compatible.

[0045] In some implementations, the initial feature representation of one or more of the goal nodes may further include information regarding orientation constraints of the corresponding one or more goals. For example, to operate on a particular goal (e.g., to perform a spot weld on a target location), a robot arm may need to be constrained to a particular orientation or a particular range of orientations. The orientation constraint information of a goal node corresponding to a particular goal may indicate the constrained orientation or the range of constrained orientations. In one particular example, the feature vector of a goal node may be a concatenation of multiple parameters specifying the target's location and orientation, as well as the status of the goal (e.g., 0 for pending, 0-1 for dwelling progress, 1 for completed).

[0046] The edges of the graph are used to characterize the interactive relationships between the robots, the target locations, and the obstacles. The graph data generation engine 114 can encode information about the edges in an adjacency matrix or an adjacency list, which is also included in the graph data. In some implementations, the graph data further includes an initial feature vector indicating edge features for each edge. The edge features for a particular edge may encode the distance, relative position, and / or relative orientation of the two nodes connected by the particular edge. In one particular example, the feature vector of a robot-robot edge can be a concatenation of multiple parameters specifying one or more of the relative positions and orientations of the TCPs, the distance score of the pair of robots, and the orientation score of the pair of robots. In another example, the feature vector of a robot-target edge can be a concatenation of multiple parameters specifying the relative positions and orientations of the TCPs and the target.

[0047] Each robot node is connected to each of the other robot nodes by an edge so that potential interactions between pairs of robots can be characterized by the graph. Each edge connecting two robot nodes can indicate interactions, such as potential collisions and communications, between the robot nodes. For example, a particular edge connecting two robot nodes can indicate information for coordinating the two robot nodes, such as indicating which of the robot nodes is reaching which goal location. Each robot node is further connected by an edge to each of the goal nodes to characterize the robot's interaction with the corresponding goal, e.g., approaching or reaching the goal. Each robot node is further connected by an edge to each of the obstacle nodes to characterize the robot's interaction with the corresponding obstacle, e.g., approaching or colliding with the obstacle.

[0048] In some implementations, no goal node or obstacle node is connected to any other goal node or obstacle node. The absence of goal node-to-goal node, obstacle node-to-obstacle node, and goal node-to-obstacle node connections in the graph indicates no interaction between goal locations and obstacles, simplifying the graph.

[0049] After the graph data is generated, the multi-robot control system 110 processes the graph data 124 using the graph neural network 116 to generate a graph output 126. The GNN may include one or more message-passing layers that operate on feature representations of the robot nodes, goal nodes, and obstacle nodes. The graph output may include updated node features for the multiple robot nodes and, optionally, updated feature representations of the goal nodes and obstacle nodes. In some implementations, the graph output further includes updated feature representations of the edges. When processing the graph data 124 using the GNN 116, the system 110 can efficiently embed the robot nodes, goal nodes, and potential interaction relationships of the goal nodes, as well as initial node features, into the updated node features of the robot nodes in the graph output 126.

[0050] The multi-robot control system 110 further includes a robot behavior generation engine 118 that uses the graph output 126 to generate behaviors 128a-128n to be performed by the robots 104a-104n at each of a plurality of time steps.

[0051] More specifically, in some implementations, the robot behavior generation engine 118 generates behaviors by processing each updated node feature using a behavior decoder neural network, e.g., a fully connected neural network that outputs a selection of a behavior to be performed by the corresponding robot.

[0052] In some other implementations, instead of directly predicting behaviors 128a-128n from the graph output 126, the robot behavior generation engine 118 may perform multiple planning iterations to generate plan data and generate behaviors 128a-128n according to the plan data. For example, in each planning iteration, the robot behavior generation engine 118 may traverse the state of the environment starting from the current state until a leaf state of the environment is reached, process the state data at the leaf state to generate updates, and use the generated updates to update the plan data. Planning iterations enable the robot behavior generation engine 118 to perform look-ahead search, which can significantly improve the performance of a system for coordinating multiple robots.

[0053] The multi-robot control system 110 can further perform neural network training to update the parameters of the neural networks used by the GNN 116 and the robot behavior generation engine 118 based on the graph output 126 and / or the expected revenue associated with the generated behaviors 128a-128n.

[0054] 2A shows an example of a graph 200 representing an environment including two robots interacting with two target locations and an obstacle. The graph 200 includes a robot node 210a representing the first robot, a robot node 210b representing the second robot, a target node 220a representing the first target location, a target node 220b representing the second target location, and an obstacle node 230a representing the obstacle.

[0055] The graph 200 further includes edges #1 through #8. These edges are indicated by arrowheads to indicate the direction of message passing in the GNN process. Edges #1 and #2 connect two robot nodes 210a and 210b, which pass messages to each other. Edges #3 and #4 connect the goal node 220a to the robot nodes 210a and 210b, respectively. The goal node 220a passes messages to each of the robot nodes 210a and 210b, but does not pass messages to the other goal node 220b or the obstacle node 230a. Similarly, edges #5 and #6 connect the goal node 220b to the robot nodes 210a and 210b, respectively. The goal node 220b passes messages to each of the robot nodes 210a and 210b, but does not pass messages to the other goal node 220a or the obstacle node 230a. Edges #7 and #8 connect obstacle node 230a to robot nodes 210a and 210b, respectively. Obstacle node 230a passes messages to each of robot nodes 210a and 210b, but does not pass messages to either goal node 220a or 220b.

[0056] 2A , in graph 200, each robot node is connected by an edge to each of the other robot nodes and receives messages from each of the other robot nodes, so that potential interactions between pairs of robots can be characterized by GNN processing, i.e., message passing performed by the GNN. Each robot node is further connected by an edge to each of the goal nodes and receives messages from each of the goal nodes to characterize the robot's interaction with a corresponding goal location, e.g., approaching or reaching the goal location. Each robot node is further connected by an edge to each of the obstacle nodes and receives messages from each of the obstacle nodes to characterize the robot's interaction with a corresponding obstacle, e.g., approaching or colliding with the obstacle.

[0057] In contrast, neither the goal node nor the obstacle node is connected to any other goal node or obstacle node. Furthermore, as indicated by the direction of the edge arrows, neither the goal node nor the obstacle node receives input from any other node. The absence of goal node-to-goal node, obstacle node-to-obstacle node, and goal node-to-obstacle node connections in the graph indicates no interaction between the goal locations and obstacles, simplifying GNN processing.

[0058] The graph in Figure 2A is for illustrative purposes only. In general, a multi-robot control system can generate and process larger graphs than those shown in Figure 2A for robot control applications with more robots operating in complex environments containing more obstacles, more target locations, or both.

[0059] 2B is a diagram illustrating GNN processing of graph data of graph 200. Various architectures can be used for GNNs. As an example, FIG. 2B illustrates a process using a GNN with an interaction network architecture.

[0060] As shown in FIG. 2B , the graph data includes node features for two robot nodes 210a and 210b, two goal nodes 220a and 220b, and one obstacle node 230a in graph 200. For each robot node, message passing is performed to gather information from the other nodes. For robot node A, the node features of robot node B, goal node A, goal node B, and obstacle node A are collected (as indicated by the bold lines with arrows), processed using a relation function 250, and aggregated to form an updated embedding 260. Examples of relation function 250 include a multilayer perceptron with learnable weights or an attention layer. System 110 then performs an update to generate updated features 270 for robot node A based on robot node A's current features 265 and updated embedding 260.

[0061] The system 110 performs a message passing process for all robot nodes. Optionally, the system 110 performs the message passing multiple times using additional message passing layers, each of which contains network parameters that are learned during training.

[0062] A GNN can employ any suitable network architecture, such as a message-passing neural network (MPNN), a non-local neural network (NLNN), a CommNet, or a Relation Network. While different types of architectures can be used for a GNN, for any given node, in each message-passing layer, the system 110 updates the node's feature representation based only on the feature representations of nodes connected to the node by edges in the graph. Furthermore, in the particular implementation illustrated by FIGS. 2A and 2B, only the robot node is updated in the message-passing layer using inputs from other robot nodes, goal nodes, and obstacle nodes. As indicated by the direction of the arrows on the edges of the graph 200, neither the goal node nor the obstacle nodes receive inputs from any other nodes.

[0063] 3 illustrates processing graph data to generate policy and value predictions. Graph data 310 may include feature representations of robot nodes, goal nodes, and obstacle nodes. GNN 320 performs one or more rounds of message passing between nodes to generate graph output 330. Graph output 330 includes updated robot node feature representations 330a-330n, and optionally further includes updated feature representations of goal nodes and obstacle nodes (not shown in FIG. 2B).

[0064] The behavior generation engine 118 of the system 110 further processes the graph output 330 using a policy prediction neural network 340, e.g., a multilayer perceptron or a recurrent neural network, to generate a predicted policy output including multiple policies 345a-345n for multiple robots. Each of the policies 345a-345n may define a probability distribution of possible actions to be performed by the corresponding robot. For example, a policy may include a respective action probability for each action in a set of possible actions that may be performed by the robot to interact with the environment. In another example, a policy may include parameters for the distribution of the set of possible actions. The system 110 may select an action using the highest probability or by sampling from the probability distribution.

[0065] The behavior generation engine 118 can further generate summary features 335 from the graph output 330 and process the summary features 335 using a value prediction neural network 350, e.g., a multi-layer perceptron or a recurrent neural network, to predict a value score 355. For example, the behavior generation engine 118 can combine the updated features of the robot nodes, e.g., by averaging, summing, or processing using a multi-layer perceptron, to generate the summary feature. The value score 355 defines the expected revenue, i.e., the expected cumulative future reward, as a result of the robot interacting with the goal location and obstacles according to the selected behavior.

[0066] In some implementations, the behavior generation engine 118 can directly use the predicted strategies 345a-345n to identify an optimal action from a set of possible actions to be performed by the corresponding robot based on current state data. For example, the strategies may identify joint velocities or torques to be applied to one or more joints of the corresponding robot in the optimal action. In one particular example, the identified optimal action may specify normalized joint velocities (e.g., with a range of [-1, 1] in each dimension) at a given time step. To increase the resolution and move closer to zero to enable finer control when approaching a goal, the behavior generation engine 118 discretizes the normalized joint velocities and converts them into

number

[0067] In these implementations, system 110 can further learn the parameters of policy prediction network 340, graph neural network 320, and, when used, value prediction network 350, by training the networks through reinforcement learning. After training, system 110 can discard value prediction network 350, because the output of value prediction network 350 is used only to improve the training of policy prediction network 340 and graph neural network, and not directly to select an action.

[0068] The system 110 can train the networks 340 and 350 using any suitable reinforcement learning technique, such as actor-critic techniques, policy gradient-based techniques, etc. In some implementations, the system can train the networks from recent experiences (i.e., trajectories including observations, actions, and optionally rewards from previous time steps) stored in a replay memory. Generally, trajectories can be derived from experience information generated as a result of the robot's interaction with the environment or another instance of the environment for use in training the model. Each trajectory represents information about the agent's interaction with the environment.

[0069] 4A shows an example of a multi-robot planning and control system 400. Compared to the multi-robot and control system 110 shown in FIG. 1, the system 400 merges GNN processing with the planning process. In particular, the system 400 includes a robot behavior planning engine 420, which includes a graph data generation engine 422, a graph neural network 424, a prediction engine 426, and a behavior sampling engine 428. The system further includes a state data collection engine, a robot behavior selection engine 440, and a training engine 430.

[0070] 1, the planning and control system 400 controls multiple robots 404a-404n within an environment 402 that includes multiple target locations 406a-406n. The target locations are locations related to the robots' tasks, and may be, for example, locations intended to be reached by one or more of the robots depending on the task. The environment 402 may further include one or more obstacles 408.

[0071] At each of a plurality of time steps, the planning and control system 400 selects an action 442a-442n to be performed by the robot 404a-404n, respectively. The state of the environment 402 at a time step depends on the state of the environment at the previous time step and the action performed by the robot at the previous time step. A state data collection engine 410 of the system 400 receives current observations 409 that characterize the current state of the environment 402 and generates state data 412 based on the current observations 409. The state data collection engine 410 can further receive a reward signal 414 based on the current observations 409. The reward signal 414 for a given time step indicates the progress the robot is making toward achieving its goal.

[0072] The robot behavior planning engine 420 performs multiple planning iterations to generate plan data 429. The plan data 429 may include data indicating the respective values ​​of the robot performing each of a set of possible actions in the environment 402 when the environment is in its current state.

[0073] In particular, planning engine 420 performs multiple iterations and updates planning data 429 with the results of each iteration.

[0074] In each planning iteration, the robot behavior planning engine 420 generates a sequence of actions that will advance the environment 402 to a new predicted future state, starting from the state represented by the current observations 409. The planning data 429 enables the system 400 to effectively select the actual actions to be performed by each of the robots 404a-404n in response to the current observations 409, starting from the current state of the environment and taking into account possible future states of the environment.

[0075] The planning engine 420 can generate plan data 429 by performing a look-ahead search guided by data indicating the value of each of a set of possible actions. The look-ahead search can be a tree search, e.g., a Monte Carlo tree search, in which a state tree includes nodes representing states of the environment 402 and directed edges connecting the nodes of the tree. An outgoing edge from a first node to a second node of the tree represents an action taken in response to an observation characterizing the first state that resulted in the environment transitioning to the second state.

[0076] In particular, in each planning iteration, the engine 420 starts from a root node representing the current state and traverses the state tree until it reaches a leaf node of the state tree, i.e., a node with no outgoing edges. The engine 420 then expands the leaf node and uses the results of the expansion and traversal to update statistics of the traversed nodes, including an estimate of the value of the root node. The tree traversal process is further illustrated by FIG. 4B and the accompanying description. Generally, the planning engine 420 includes a graph data generation engine 422, a graph neural network 424, a transition model 425, a value and / or policy prediction engine 426, and optionally, a behavior sampling engine 428. The planning engine 420 may include a reward prediction model for predicting rewards for actions.

[0077] 1, the graph data generation engine 422 generates graph data that maps the state data 412 to a graph. The graph includes a number of nodes representing robots, goal locations, and optionally goal objects, and a number of edges connecting the nodes. In particular, the graph data generation engine 422 maps the robots 404a-404n to robot nodes, the goal locations 406a-406n to goal nodes, and optionally, the obstacles 408 to obstacle nodes in the graph.

[0078] The transition model 425 generates, for each outgoing edge from a first node to a second node in the search tree, a next state of the second node to which the environment will transition based on the current state represented by the first node and the action taken. The transition model 425 may be implemented as a GNN.

[0079] In certain implementations, the inputs and outputs of the transition model 425 are in the "real state" data space of the environmental observations, i.e., the transition model 425 operates on data that represents physically meaningful parameters of the environmental observations, including, for example, the robot's joint positions and angles, the location of targets, and the positions of obstacles.

[0080] In some implementations, the transition model 425 may operate on "hidden state" data corresponding to the state of the environment. That is, the inputs and outputs of the transition model 425 are lower-dimensional representations of the environment observations. For example, the inputs of the transition model 425 may include data output by the graph neural network 424, which computes embeddings (i.e., lower-dimensional representations) of the observations. The outputs of the transition model 425 can be in the same data space as the inputs, i.e., the same data space as the data output by the graph neural network 424.

[0081] The planning engine can use the GNN 424 to process graph data corresponding to state representations at leaf nodes of the search tree and generate graph output using embeddings of the state representations of the leaf nodes. When the transition model 425 operates on a "real state" data space, the system 400 can use the graph data generation engine 422 to map the states of the leaf nodes to graph data representations as input to the GNN of the transition model 425. Alternatively, when the transition model 425 operates on a "hidden state" data space, the GNN of the transition model 425 can use embeddings of the state representations output by the GNN 424.

[0082] The prediction engine 426 is configured to process the graphical output generated by the GNN 424 to predict quantities most directly related to planning, including action selection strategies, value functions, and, when appropriate, rewards. The predicted strategy output can define a score distribution for a set of possible robot action combinations that can be performed by each robot. Each action combination includes actions performed by multiple robots. For example, the predicted strategy output can include a respective numerical probability value for each combination of robot actions in the set of possible action combinations. Each numerical probability value can be the probability that the corresponding action combination is the best to take given a state. The value output can also specify a numerical value representing the robot's overall progress toward accomplishing the task when the environment is in a given environmental state.

[0083] The prediction engine 426 may include a neural network with any suitable architecture to process the graph output. For example, the neural network of the prediction engine 426 may include an input layer that receives hidden state inputs, followed by one or more convolutional layers or one or more fully connected layers, and an output layer that outputs a score distribution. The prediction engine 426 may also include separate policy prediction networks and value prediction networks for predicting policies and values, respectively.

[0084] The behavior sampling engine 428 is configured to receive as input the predicted strategy output of the predictive model 150 and process the input to generate as output data defining a sampling distribution. The sampling distribution can be a distribution of some or all of the possible actions that can be performed by the robot and may, for example, include respective numerical probability values ​​for each of multiple actions within the entire set of possible actions. During planning, the planning engine 420 can sample some actions from the sampling distribution to generate an appropriate subset of actions that are used to expand the state tree, i.e., to be added as outgoing edges to leaf nodes.

[0085] After planning, the system 400 proceeds to select actual actions 442a-442n to be performed by each robot based on the plan data 429. The plan data 429 may include statistical data gathered during planning for each outgoing edge of the root node of the state tree, i.e., the node corresponding to the state characterized by the current observations 409, and the robot action selection engine 440 may select actual actions 442a-442n based on the statistical data for the node-edge pair corresponding to the root node.

[0086] For example, the robot behavior selection engine 440 may make this selection proportional to the number of visits of each outgoing edge of the root node of the state tree. That is, the action from the set of all possible actions that was most frequently selected during planning when the environment 402 was in the state characterized by the current observations 409, i.e., the action corresponding to the outgoing edge from the root node with the most visits in the planning data, may be selected as the actual behavior 442a-442n to be performed by the robot in response to the current observations. As another example, the robot behavior selection engine 440 may make this selection by first determining, from the sequences of actions in the planning data, the sequence of actions with the greatest associated value, and then selecting the first behavior in the determined sequence of actions as the actual behavior 442a-442n to be performed by the respective robot in response to the current observations 409.

[0087] The training engine 430 trains the model included in the planning engine 420 to learn the model parameters 425, so that the planning engine 420 generates plan data based on which an optimal robot behavior can be effectively selected. The optimal robot behavior can be the behavior that maximizes the expected cumulative reward received by the system 400. The expected cumulative reward can be the long-term, time-discounted sum of rewards received by the system 400 for performance by the robot when interacting with an environment.

[0088] The training engine 430 trains the GNN 422 and prediction engine 426 models using a suitable training technique, for example, an end-to-end backpropagation-through-time technique, to jointly and iteratively adjust the values ​​of a set of model parameters 425.

[0089] In some implementations, the training engine 430 trains the model included in the planning engine 420 from recent experiences (i.e., trajectories including observations, actions, and optionally rewards from previous time steps) stored in the replay memory 415. Generally, trajectories may be derived from experience information generated as a result of the robot's interaction with the environment or another instance of the environment for use in training the model. Each trajectory represents information about the agent's interaction with the environment.

[0090] 4B illustrates the data flow of an iterative planning process for generating plan data for multiple robots. In each planning iteration, the planning engine 420 uses the transition model 425 to traverse the states of the environment represented by the state tree 413, starting from a current state represented by current state data 412a, through one or more next states represented by next state data 412b, and so on, until a leaf state of the environment (represented by leaf state data 412n) is reached. While traversing the state tree, the behavior planning engine 420 can select the next state according to statistics associated with corresponding node-edge pairs in the state tree 413.

[0091] When the transition model 425 operates in the "actual state" data space, the planning engine 420 uses a graph data generation engine 422 to map leaf state data to a graph feature data representation 452 of the leaf state. The GNN 424 processes the graph feature data representation 452 to generate an updated feature representation 454. The policy prediction network 426a of the prediction engine 426 processes the updated feature representation 454 to generate a predicted robot policy 456, and the value prediction network 426b of the prediction engine 426 processes an updated feature summary 457 generated from the updated feature representation 454 to generate a predicted value score 458. The planning engine 420 uses the predicted robot policy 456 and the predicted value score 458 to update statistics for one or more corresponding node-edge pairs in the state tree 413 traversed to reach the leaf node. The planning engine 420 may also include statistics of multiple node-edge pairs of the state tree 413 in the planning data, so that the robot behavior selection engine 440 can use the statistics to select optimal behavior for the robot.

[0092] The statistical data for each of some or all of the node-edge (i.e., state-action) pairs may be data gathered as a result of repeatedly running the planning engine to generate different outputs starting from a node that represents the current state of the environment. For example, the planning data may include, for each outgoing edge of the root node of the state tree, (i) an action score Q for the action represented by the edge, (ii) a visit number N for the action represented by the edge, representing the number of times the action was selected during multiple planning iterations, and (iii) a prior probability P for the action represented by the edge. During planning, the root node of the state tree corresponds to the state characterized by the current observations.

[0093] For any given node representing a given state of the environment, the action score Q of the action represents a current estimate of the revenue received if the action is executed in response to the observations characterizing the given state. Revenue refers to a cumulative measure of the "reward" received by the agent, e.g., a time-discounted sum of rewards. The agent can receive a respective reward at each time step, which is specified by a scalar numeric value and characterizes, e.g., the agent's progress toward completing an assigned task. The visit count N of the action is the current number of times the action has been executed by the agent in response to the observations characterizing the given state. And the prior probability P represents the likelihood that the action should be executed in response to the observations characterizing the given state, i.e., the action that maximizes the revenue received compared to all other actions that could be executed in response to the observations.

[0094] The system may maintain the planning data in a memory device accessible to the system. Although logically described as a tree, the planning data generated by using the planning engine may be represented by any of a variety of convenient data structures, for example, as multiple triples or as an adjacency list.

[0095] In each planning iteration, the system performs the planning according to the collected statistics on the corresponding node-edge pairs starting from the node-edge pair corresponding to the root node, for example, by finding the maximum for the upper confidence bound, i.e.,

number

[0096] Exemplary look-ahead search algorithms, including action selection, state tree expansion, and statistics update algorithms, are described in more detail in U.S. Patent Publication No. 20200143239, entitled "Training action selection neural networks using look-ahead search," by Simonyan et al., filed May 28, 2018, and published May 7, 2020; and in Silver et al., "Mastering the game of go without human knowledge," Nature, 550:354-359, October 2017; and Kocsis et al., "Bandit based Monte-Carlo planning," European conference on machine-learning, pp. 282-293, Springer, 2006.

[0097] Once a leaf node is reached, the system can use the GNN 424 to expand the state tree from the leaf node and update the planning data. Specifically, the system can use the GNN 424 to process the graph feature data representation 452 of the leaf node's state data to generate an updated feature representation 454 of the leaf state. The system can process the updated feature representation 454 using a predictive model to generate an output including a predicted policy output defining a score distribution for the set of actions. The system can sample an appropriate subset of the set of actions, for example, by generating data defining a sampling distribution from the score distribution and sampling a fixed number of samples from the sampling distribution. The system then updates the state tree by adding each outgoing edge from the leaf node representing each sampled action to the state tree. The system can then update statistics for each sampled action by associating each outgoing edge representing the sampled action with a prior probability of the sampled action derived from the predicted policy output.

[0098] For each edge traversed during a planning iteration, the system may increment the visit count N of the edge by a predetermined constant value, for example, by 1. The system may also update the behavior score Q of the edge using the predicted value for the leaf node by setting Q equal to the new average of the predicted values ​​of all explorations that included traversal of the edge.

[0099] An exemplary process for updating statistics and expanding leaf nodes is described in detail in PCT Patent Application Publication WO2021152515, entitled "Planning for agent control using learned hidden states," to Schrittwieser et al., filed January 28, 2021, and published August 5, 2021, the entire disclosure of which is incorporated herein by reference.

[0100] 5 is a flow diagram of an exemplary process for generating robot behavior from state data. For convenience, process 500 is described as being performed by one or more computer systems located at one or more locations. For example, the multi-robot control system shown in FIG. 1 , suitably programmed in accordance with this specification, may perform process 500.

[0101] The system repeatedly performs the following steps at each of a number of time steps:

[0102] First, the system obtains state data representing the current state of an environment in step 510. The environment includes multiple robots, one or more goal locations, and optionally, one or more obstacles. The state data includes information about the current states of the robots, goal locations, and obstacles.

[0103] Next, in step 520, the system generates graph data from the state data, representing a graph of the current state of the environment. The graph includes a plurality of nodes and a plurality of edges. The plurality of nodes includes a respective robot node for each of the robots, a respective goal node for each of the goal locations, and optionally a respective obstacle node for each of the goal objects. The graph data includes initial feature representations for each of the robot nodes, each of the goal nodes, and optionally each of the obstacle nodes.

[0104] By way of example, the initial representation of each robot node may include one or more coordinates of the tool tip, the coordinates of each of the joints, and the current joint angle. The initial representation of each goal node may include the coordinates of the goal location.

[0105] In some implementations, the initial feature representation of one or more of the goal nodes may further include compatibility information for the corresponding one or more goals. The compatibility information for a goal node may, for example, identify a subset of robots that are compatible for operating on the corresponding goal. For example, multiple robots may be configured with various tool tips that are compatible or incompatible for operating on a particular goal, and the compatibility information for a goal node corresponding to a particular goal may indicate which robots are compatible.

[0106] In some implementations, the initial feature representation of one or more of the goal nodes may further include information regarding orientation constraints for the corresponding one or more goals. For example, to operate on a particular goal (e.g., to perform a spot weld on a goal location), a robot arm may need to be constrained to a particular orientation or a particular range of orientations. The orientation constraint information of a goal node corresponding to a particular goal may indicate the constrained orientation or the range of constrained orientations.

[0107] Each edge of the graph connects a respective pair of nodes from the plurality of nodes. The graph includes an edge between each robot node and each goal node, an edge between each robot node and each other robot node, and optionally an edge between each robot node and each obstacle node. In some implementations, the graph does not include any edges between any two goal nodes or between any two obstacle nodes, and does not include any edges between any obstacle nodes in the graph and any goal nodes.

[0108] In some implementations, the graph data further includes an initial feature vector indicating edge features for each edge, which may encode the distance, relative position, and / or relative orientation of two nodes connected by the edge.

[0109] After generating the graph data from the state data, the system processes the graph data using a graph neural network to generate a graph output in step 530. The graph output includes an updated feature representation for each of the robot nodes. Optionally, the graph output includes a respective updated feature representation for each of the robot nodes and / or each of the goal nodes. In some implementations, the graph output further includes updated feature representations for the edges. The graph neural network includes one or more message passing layers, and for any given node, each message passing layer updates the feature representation of the node based only on the feature representations of nodes connected to the node by edges in the graph.

[0110] Next, in step 540, the system selects a respective action to be performed by each of the robots based on the graph output. In certain implementations, the system processes the graph output directly using a neural network to select an action for each robot. In certain other implementations, the system performs one or more planning iterations using the graph output to generate plan data, and uses the plan data to select an action for each robot.

[0111] 6 is a flow diagram of an exemplary process for generating robot behaviors from state data in planning iterations. For convenience, process 600 is described as being performed by one or more computer systems located at one or more locations. For example, a multi-robot and planning control system shown in FIG. 4A, suitably programmed in accordance with this specification, may perform process 600.

[0112] The system controls multiple robots to perform tasks by repeatedly performing the following actions:

[0113] First, the system performs multiple planning iterations starting from the current state of the environment to generate planning data. The planning data may include a sequence of actions for the robot. The planning data may further include statistics about the state of the environment related to the sequence of actions. Each planning iteration includes steps 620 through 650.

[0114] In step 620, the system performs a look-ahead iteration by starting from the current state and traversing the states of the environment until a leaf state of the environment is reached.

[0115] In step 630, the system generates graph data representing a graph of the leaf states of the environment from the state data characterizing the leaf states. The graph includes a plurality of nodes and a plurality of edges. The plurality of nodes includes a respective robot node for each of the robots, a respective goal node for each of the goal locations, and optionally a respective obstacle node for each of the goal objects. The graph data includes initial feature representations for each of the robot nodes, each of the goal nodes, and optionally each of the obstacle nodes.

[0116] Each edge of the graph connects a respective pair of nodes from the plurality of nodes. The graph includes an edge between each robot node and each goal node, an edge between each robot node and each other robot node, and optionally an edge between each robot node and each obstacle node. In some implementations, the graph does not include any edges between any two goal nodes or between any two obstacle nodes, and does not include any edges between any obstacle nodes in the graph and any goal nodes.

[0117] In step 640, the system processes the graph data using a graph neural network to generate a graph output. The graph output includes an updated feature representation for each of the robot nodes. Optionally, the graph output includes a respective updated feature representation for each of the robot nodes and / or each of the goal nodes. The graph neural network includes one or more message passing layers, and for any given node, each message passing layer updates the node's feature representation based only on the feature representations of nodes connected to the node by edges in the graph.

[0118] In step 650, the system updates the planning data according to the graph output. The system may generate updates to the planning data and update the planning data using the generated updates. In generating the updates, the system may generate summary features of the leaf states from the graph output and process the summary features using a value prediction neural network to predict a value score representing the predicted value of being in the leaf state for successfully completing the task. Furthermore, for each robot node, the system may process the updated feature representation of the robot node using a policy neural network to generate a policy output that defines a probability distribution of a set of possible actions to be performed by the corresponding robot node when the environment is in the leaf state.

[0119] After performing the planning iterations, the system uses the planning data to select an action to be performed by each robot in step 660. For example, the system may make the selection by first determining, from the sequences of actions in the planning data, the sequence of actions that has the greatest associated value, and then selecting the first action in the determined sequence of actions as the actual action to be performed by each robot according to the current observations.

[0120] Embodiments and functional operations of the subject matter described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or a combination of one or more of them, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0121] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatuses, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus can also be or further include special-purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Optionally, in addition to hardware, an apparatus can include code that creates an execution environment for a computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0122] A computer program (which may also be called or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program at hand, or in multiple organized files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and connected together by a communications network.

[0123] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by performing operations on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0124] A computer suitable for running a computer program can include, and by way of example, can be based on, a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to these mass storage devices, or both. However, a computer need not have such devices. Furthermore, a computer can be incorporated into another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0125] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0126] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0127] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having an associated graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (“LANs”) and wide area networks (“WANs”), e.g., the Internet.

[0128] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0129] While this specification contains many specific implementation details, these should not be considered limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features that are described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in a particular combination, and may even be initially claimed as such, one or more features of a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0130] Similarly, while operations are shown in a particular order in the figures, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all of the operations shown be performed, to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the division of various system modules and components in the above-described embodiments should not be understood as requiring such division in all embodiments, and it should be understood that the described program components and systems generally can be integrated together in a single software product or packaged in multiple software products.

[0131] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0132] 102 Environment 104a Robot 104n Robot 106a Target location 106n Target location 108 Obstacles 110 Multi-robot control system 112 State Data Collection Engine 114 Graph Data Generation Engine 116 Graph Neural Networks, GNN 118 Robot Behavior Generation Engine 122 Status Data 124 Graph Data 126 Graph Output 128a Action 128n Action 150 Predictive Models 200 graphs 210a Robot Node 210b Robot Node 220a Target node 220b Target node 230a Obstacle Node 250 Relational Functions 260 Updated Embeds 265 Current Features 270 Updated Features 310 Graph Data 320 GNN, Graph Neural Network 330 Graph Output 330a Updated robot node feature representation 330n Updated robot node feature representation 335 Summary Features 340 Policy Prediction Neural Network 345a Measures 345n measures 350 Value Prediction Neural Network 355 Value Score 400 Multi-Robot Planning and Control System 402 Environment 404a Robot 404n robot 406a Target location 406n Target location 408 Obstacles 409 Current Observations 410 State Data Collection Engine 412 State Data 412a Current State Data 412b Next State Data 412n Leaf condition data 413 State Tree 414 Reward Signal 415 Regenerative Memory 420 Robot Behavior Planning Engine 422 Graph Data Generation Engine 424 Graph Neural Networks (GNNs) 425 Transition Model, Model Parameters 426 Prediction Engine, Value and / or Strategy Prediction Engine 426a Policy Prediction Network 426b Value Prediction Network 428 Behavioral Sampling Engine 429 Planning Data 430 Training Engine 440 Robot Action Selection Engine 442a Action 442n Action 452 Graph Feature Data Representation 454 Updated feature representation 456 Predicted Robot Policy 457 Updated Feature Summary 458 Predicted Value Score

Claims

1. 1. A method executed by one or more computers for controlling a plurality of robots to move to a plurality of target locations in an environment, the method comprising the following operations: obtaining state data representing a current state of the environment; generating, from the state data, graph data representing a graph of the current state of the environment, the graph including a plurality of nodes and a plurality of edges, each edge connecting a respective pair of nodes from the plurality of nodes, the plurality of nodes including a respective robot node for each of the robots and a respective goal node for each of the goal locations, the graph data including a respective initial feature representation for each of the robot nodes and each of the goal nodes; processing the graph data using a graph neural network to generate a graph output including a respective updated feature representation for each of the robot nodes; selecting a respective action to be performed by each of the robots based on the graph output; and The method includes repeatedly performing

2. 2. The method of claim 1 , wherein the environment includes one or more obstacles, the graph includes a respective obstacle node for each of the one or more obstacles, and the graph data includes a respective initial feature representation for each of the obstacle nodes.

3. The method of claim 2 , wherein the graph includes an edge between each robot node and each obstacle node.

4. The method of claim 3 , wherein the graph does not include any edges between any two obstacle nodes.

5. The method of claim 3 , wherein the graph does not include any edges between any obstacle nodes and any goal nodes of the graph.

6. The method of claim 1 , wherein the graph includes an edge between each robot node and every other robot node.

7. The method of claim 1 , wherein the graph includes an edge between each robot node and each goal node.

8. The method of claim 1 , wherein the graph does not include any edges between any two goal nodes.

9. The method of claim 1 , wherein the graph data includes edge data representing the edges of the graph.

10. selecting the respective actions to be performed by each of the robots; The method of claim 1 , further comprising predicting a strategy or value for each of the actions by processing the graphical output.

11. selecting the respective actions to be performed by each of the robots; performing one or more planning iterations using the graphical output to generate planning data; selecting an action using the planning data; and 2. The method of claim 1, comprising:

12. 1. A method executed by one or more computers for controlling a plurality of robots to perform a task involving moving said robots to a plurality of target locations in an environment, the method comprising the following acts: performing a plurality of planning iterations starting from a current state of the environment to generate planning data, wherein performing each planning iteration includes: traversing the states of the environment starting from the current state until a leaf state of the environment is reached; generating graph data from state data characterizing the leaf states representing a graph of the leaf states of the environment, the graph including a plurality of nodes and a plurality of edges, each edge connecting a respective pair of nodes from the plurality of nodes, the plurality of nodes including a respective robot node for each of the robots and a respective goal node for each of the goal locations, the graph data including a respective initial feature representation for each of the robot nodes and each of the goal nodes; processing the graph data using a graph neural network to generate a graph output including a respective updated feature representation for each of the robot nodes; generating updates to the planning data from the graphical output; and performing a plurality of planning iterations, including updating the planning data using the generated updates; selecting an action using the planning data after executing the planning data; The method includes repeatedly performing

13. generating updates to the planning data from the graphical output; generating summary features of the leaf states from the graph output; and processing the summary features using a value prediction neural network to predict a value score representing the predicted value of being in the leaf state for successfully completing the task; and 13. The method of claim 12, comprising:

14. Generating updates to the planning data from the graph output includes, for each robot node:

13. The method of claim 12, comprising processing the updated feature representation of the robot node using a policy neural network to generate a policy output that defines a probability distribution over a set of possible actions to be taken by the corresponding robot node when the environment is in the leaf state.

15. one or more computers; one or more storage devices storing instructions that, when executed by said one or more computers, cause said one or more computers to perform the operations of the method of any one of claims 1 to 14; Including, the system.

16. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 14.

Images