A graph neural network system for behavior prediction and reinforcement learning in multi-agent environments

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, for predicting actions of, or effects on, agents in an environment having multiple agents, particularly for reinforcement learning. In one aspect, a relational forward model (RFM) system receives agent data representing agent actions for each of a plurality of agents, and implements: an encoder graph neural network subsystem that processes the agent data into graph data to provide encoded graph data; a recurrent graph neural network subsystem that processes the encoded graph data to provide processed graph data; a decoder graph neural network subsystem that decodes the processed graph data to provide decoded graph data; and an output that provides representational data of node attributes and / or edge attributes of the decoded graph data related to predicted actions of one or more agents. The reinforcement learning system includes the RFM system.

Description

Background Art

[0001] The present specification relates to neural networks for predicting actions of or effects on agents in an environment with multiple agents, and in particular neural networks for reinforcement learning.

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs from received inputs. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, the next hidden layer or output layer. Each layer of the network generates an output from the received inputs based on the current values ​​of the corresponding set of parameters.

[0003] Some neural networks exhibit a graph structure comprising nodes connected by edges; the graph may be a multigraph in which nodes may be connected by multiple edges. Nodes and edges may have associated node features and edge features; these may be updated using node functions and edge functions implementable by the neural network. Summary of the invention

[0004] This specification describes neural network systems and methods implemented as a computer program for processing data representing the behavior of multiple agents on one or more computers in one or more locations for predicting the actions of the agents or for determining the impact on the operation of the agents. The agents may be robots on a factory floor, or autonomous or semi-autonomous vehicles. The described neural network systems may be used for reinforcement learning, for example, to improve performance by anticipating the actions of other agents, or for learning cooperative behavior.

[0005] Thus, in one aspect, a relational forward model (RFM) neural network system for predicting or explaining the actions of multiple agents in a shared environment comprises: an input, receiving agent data representing agent actions of each of the multiple agents; and one or more processors. The one or more processors are configured to implement processing of the agent data into graph data (in conjunction with graph data processing) to provide encoded graph data, which may include data representing at least nodes and edges of the graph; the edges may be directed or undirected. Each agent may be represented by a node. Non-agent entities in the environment may also each be represented by a node. The nodes have, for example, node attributes for determining the actions of each agent. The nodes may each have a set of the same attributes. The graph data provided to the encoder may lack edge attributes. Edges may connect agents to each other and to non-agent entities. The encoded graph data may include node attributes and edge attributes representing an updated version of the graph data.

[0006] The one or more processors may also be configured to implement a processing graph neural network subsystem, particularly a recurrent graph neural network subsystem. The recurrent / processing graph neural network subsystem may include a recurrent neural network that processes the encoded graph data and provides processed graph data, the processed graph data including updated versions of the node attributes and edge attributes of the encoded graph data. The one or more processors may also be configured to implement a decoder graph neural network subsystem that decodes the processed graph data and provides the decoded graph data, the decoded graph data including updated versions of the node attributes and edge attributes of the processed graph data. The system may have a system output that provides representation data, the representation data including a representation of one or both of the node attributes and edge attributes of the decoded graph data for one or more agents (e.g., all) agents. The representation may be related to a predicted action or an interpreted action of one or more agents, for example, derived from the node attributes or edge attributes of the decoded graph data, respectively.

[0007] In some embodiments, the action may include the motion of the agent. Thus, the agent data captured by the system may include agent position and motion data for each agent. Node attributes may then include attributes of the position and motion of each agent. In some embodiments, each agent may be connected to every other agent by an edge, and every non-agent entity may be connected to every agent by an edge. However, in embodiments, non-agent entities, such as stationary entities, are not connected by edges.

[0008] In some embodiments, the system output includes one or more output neural network layers, such as a multilayer perceptron (MLP). Node attributes can be represented as vectors. The one or more output neural network layers can combine the node attributes of the nodes in the decoded graph data to output representation data. The representation derived from the node can include the predicted action of the agent represented by the node. The representation data can define a spatial graph, such as a heat map, of data derived from the node attributes of one or more nodes representing one or more agents. In such a graph, the data derived from the node attributes can be represented at the location of the corresponding node. For example, in the case where the action can include the movement of the agent, the graph can represent the probability of each agent represented being at a certain location.

[0009] In some embodiments, the representation data includes a representation of edge attributes of the decoded graph data for edges connected to one or more nodes. The representation of the edge attributes of the edge can be determined from a combination of the edge attributes of the edge. For example, the edge attributes can be represented as a vector, and the combination of the edge attributes of the edge can be a vector norm, such as a p-norm, where p is an integer.

[0010] In an embodiment, edges are directional (although nodes can be connected by edges in both opposite directions). Thus, an edge can connect from a starting node to an end node, such as from a proxy or non-proxy entity node to a proxy node. The representation of the edge, such as the vector norm, can represent the importance or influence of the starting node on the proxy node to which it is connected.

[0011] The edge representation data can define a spatial graph, such as a heat map. In the spatial graph, the representation of the edge attribute of the edge, such as the vector norm of the attribute, can be located at the starting node position of the edge. Such a spatial graph can be defined on a per-agent basis, i.e., there can be one graph for each agent considered.

[0012] It is assumed that the representation of the edge, such as the vector norm, indicates the importance of the edge. Thus, for example, by indicating which nodes influenced the action or by identifying which other(s) nodes were most influential on a particular agent (e.g., by ranking the other nodes), the edge properties of the decoded graph can encode information that can be used to explain the agent's behavior. Changes in this explanatory information can be tracked over time.

[0013] A neural network system as described above can be trained by supervised training, for example, based on observations of the behavior of multiple agents in a shared environment.

[0014] In a neural network system including one or more of an encoder graph neural network subsystem, a processing graph neural network subsystem, and a decoder graph neural network subsystem as described above, one or more of the encoder graph neural network subsystem, the processing graph neural network subsystem, and the decoder graph neural network subsystem can be configured to implement the following graph network processing: For each edge, an edge neural network is used to process edge features to determine an output edge feature. For each node, the output edge features of the edges connected to the node are aggregated to determine the aggregated edge features of the node. For each node, a node neural network is used to process the aggregated edge features and the node features to determine the output node features. This process can be performed once or iteratively multiple times. Processing edge features can include: for each edge, providing the edge features and the node features of the nodes connected by the edge to the edge neural network to determine the output edge features.

[0015] One or more of the encoder graph neural network subsystem, the processing graph neural network subsystem, and the decoder graph neural network subsystem may also be configured to use a global feature neural network to determine a global feature vector. The global feature vector may represent output edge features and output node features. Where the encoder graph neural network subsystem determines the global feature vector, the subsequent processing graph neural network subsystem and the decoder graph neural network subsystem may also operate on the global feature vector. A graph neural network subsystem, such as a processing (circular) graph neural network subsystem, may include a circulatory graph network. Thus, one or more of the edge neural network, the node neural network, and the global feature neural network (described below) may include a circulatory neural network, such as a GRU (Gated Recurrent Unit) neural network.

[0016] In some embodiments, the system may be included in a reinforcement learning system. The reinforcement learning system may be configured to select an action to be performed by one of the agents interacting with the shared environment. The reinforcement learning system may include inputs to obtain state data and reward data, the state data representing the state of the shared environment and the reward data representing the reward received by the agent for performing the action. For example, the state data may be derived by capturing one or more observations (such as images) from the environment and processing these observations using an observation processing neural network (such as a convolutional neural network). The reinforcement learning system may also include an action selection policy neural network that processes the state data and the reward data to select an action. The action selection policy neural network may be configured to receive and process the representation data to select an action.

[0017] Observations can also be used to train the system to predict / explain the actions of an agent. For example, the observation processing neural network or another neural network can be configured to recognize the actions of an agent in an environment. The observation processing neural network or another neural network can additionally or alternatively be configured to recognize agents and / or non-agent entities, for example for adapting a graph based on the entities present.

[0018] In general, the reinforcement learning system can be of any type. For example, it can be a policy-based system that directly parameterizes the policy and value functions, such as an Advantage Actor Critic (A2C or A3C) system (e.g., Mnih et al. 2016). Alternatively, it can be a Q-learning system in which the output approximates the action-value function and optionally approximates the value of the state for determining the action, such as a deep Q-learning network (DQN) system or a Double-DQN system. In another alternative, it can be a distributed reinforcement learning system, such as IMPALA (Importance-Weighted Actor-Learner), Espholt et al., arXiv:1802.01561. In a continuous control setting, it can directly output an action, such as a torque or acceleration value, using, for example, a DDPG (Deep Deterministic Policy Gradient) technique (arXiv 1509.02971) or a variant thereof.

[0019] In some embodiments, the reinforcement learning system can be configured to train a neural network system to predict or explain the actions of the agent from observations of a shared environment during training of the reinforcement learning system. Thus, the two systems can be trained together. The neural network system used to predict or explain the behavior of the agent may be specific to the reinforcement learning system, i.e., it does not need to rely on data that is not available to the reinforcement learning system for training. In some embodiments, the two systems can be co-located in one of the agents and can learn in conjunction with each other.

[0020] Optionally, the graph network processing may include self-referencing functions implemented by a neural network. For example, node updates may depend on the initial features of the node. Such processing may be advantageous in facilitating the agent to predict its own actions, which may enhance the information on which action selection is based. Optionally, a neural network system for predicting or explaining the agent's actions may be derived for multiple time steps to provide additional information to the reinforcement learning system. Thus, the system may be used for imagination-based planning (Weber et al., arXiv:1707.06203).

[0021] A method of predicting or explaining the actions of multiple agents in a shared environment is also described. The method may include receiving agent data representing the actions of each of the multiple agents. The method may also include processing the agent data in conjunction with graph data to provide encoded graph data. The graph data may include data representing at least nodes and edges of a graph, wherein each agent is represented by a node, wherein non-agent entities in the environment are each represented by a node, wherein the nodes have node attributes for determining the actions of each agent, wherein the edges connect the agents to each other and to the non-agent entities, and wherein the encoded graph data includes node attributes and edge attributes representing updated versions of the graph data. The method may also include processing the encoded graph data using a recurrent graph neural network to provide processed graph data, the processed graph data including updated versions of the node attributes and edge attributes of the encoded graph data. The method may also include decoding the processed graph data to provide decoded graph data, the decoded graph data including updated versions of the node attributes and edge attributes of the processed graph data. The method may also include outputting a representation of one or both of the node attributes and edge attributes of the decoded graph data for one or more agents, wherein the representation is related to the predicted behavior or explained behavior of the agent.

[0022] The behavior may include an action of the agent. Outputting the representation may include processing node attributes of a node of the decoded graph data to determine a predicted action of the agent represented by the node.

[0023] The method may be used to interpret the actions of an agent. Outputting the representation may then include processing edge attributes of an edge of the decoded graph data (the edge connecting an influence (origin) node, which may be an agent or a non-agent node, to an agent node) to determine data representing the importance of the influence node to the agent node.

[0024] The method can be used for reinforcement learning. For example, reinforcement learning can learn a policy for selecting actions to be performed by an agent depending on the state of an environment shared by multiple agents. The reinforcement learning method can include using the described method to predict the behavior of agents in the environment and then using the predicted behavior to learn a policy.

[0025] In general, embodiments of the described systems and methods can be used with real or simulated agents and environments. The environment can be a real environment and the agent can be a mechanical agent, such as a robot that interacts with the real environment. In some other embodiments, the agent can include control devices that operate in a manufacturing or service facility and work together to control aspects of the facility (e.g., the operating temperature of a server farm). Other examples are described later.

[0026] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.

[0027] Some embodiments of the described neural network systems are capable of characterizing the behavior of multiple autonomous agents, such as robots in a factory or autonomous or semi-autonomous vehicles. This information can then be used to train agents to complete tasks more effectively or efficiently. For example, agents such as mechanical agents can be trained to complete tasks with improved performance and / or using less data and processing resources. More specifically, training is faster and uses less data and computing resources than some other techniques, and the described systems can predict both actions and effects, which enables them to learn in situations where other techniques may find difficulty, thereby facilitating better performance.

[0028] The described neural network systems may also be used by an agent so that the agent can collaborate with other agents to perform a task. The other agents may or may not be of the same type; for example, they may be human-controlled or computer-controlled, or they may be from different manufacturers. An example of such collaborative behavior is the control of multiple robots on a factory or warehouse floor; equipping such robots with a neural network system of the type described herein may facilitate improved robot control because the behavior of the other robots may be taken into account. In another example, an autonomous or semi-autonomous vehicle may use a neural network system of the type described to predict the behavior of other road users or pedestrians, thereby improving safety.

[0029] Importantly, because the described system for predicting / explaining the actions of multiple agents requires less experience to learn, the described system can achieve good results when trained together with a reinforcement learning system, even if other agents are still learning and their behaviors are changing.

[0030] In some other applications, the described neural network system can provide information about the motivation or intention of the agent. That is, the information derived from the edges of the processed graph can be used to determine why the agent acted in this way or how it might act. This can provide insight into the way an autonomous agent under the control of a machine learning system acts. This information can be useful for regulatory and other purposes, because it is easier to trust an autonomous system if the motivation for its behavior can be identified.

[0031] Configuring a system of one or more computers to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operation or action. Configuring one or more computer programs to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 An example neural network system for predicting or explaining the actions of multiple agents in a shared physical system environment is shown.

[0033] Figure 2a and Figure 2b A graph neural network subsystem and the operation of the graph neural network subsystem are shown.

[0034] Figure 3 Details of a graph processing neural network system for predicting or explaining the actions of multiple agents in a shared physical system environment are shown.

[0035] Figure 4 Shows the use of Figure 3 The process of using graph processing neural network systems to provide representation data for predicting agent actions.

[0036] Figure 5a and Figure 5b Shown are the positioning of agents on the grid (top) and (bottom), example heatmaps of action predictions and agent-entity importance relations, respectively.

[0037] Figure 6 Shown includes the combination Figure 3 Graph processing neural network systems as agents for reinforcement learning systems.

[0038] Like reference numbers and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION

[0039] Figure 1 An example neural network system 100 is shown for predicting or explaining the actions of multiple agents in a shared physical system environment 102. Neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0040] Agents may include mechanical agents, such as robots or vehicles in real or simulated environments, and the system may predict or explain their behavior, for example, for safety or control purposes. For example, the neural network system 100 may be used in a robot or vehicle control system to control the behavior of the robot / vehicle based on a prediction or explanation of the behavior of one or more other agents (which may be other robots / vehicles and / or personnel). The control system may include a reinforcement learning system. The explanation of the behavior of a particular agent may include data defining the importance or influence of one or more other agents or non-agent entities in the environment on the particular agent. The data may similarly be used by the control system to control the agent in response to an explanation of the behavior of one or more other agents (e.g., other robots / vehicles or personnel). Further examples are given later.

[0041] Non-agent entities in the environment may include: objects to be collected, manipulated, moved and / or dropped; static objects or entities, such as obstacles or markers; entities that define allowed or preferred movements or positions, such as routes, lanes, passages, avenues, parking spaces (which may be occupied or unoccupied), etc.; entities that provide instructions or control commands, such as signs, etc.

[0042] The neural network system 100 receives a semantic description of the state of an environment 104 formatted as a graph as explained below, and processes the data to provide an output that includes representation data for predicting and / or explaining the behavior of an agent in the environment. The neural network system 100 includes a graph processing neural network system 106, which includes one or more graph neural network blocks 110 that process an input graph to provide output data 108 in the form of an output graph, from which representation data can be derived to predict and / or explain the behavior of the agent. In some embodiments, such as reinforcement learning applications, the neural network system 100 may include a training engine 120 to train the graph processing neural network system 106, as described later. For example, in a reinforcement learning application, an agent may include a graph processing neural network system 106 to enhance observation of the environment with predictions of the behavior of other agents.

[0043] A semantic description of the state of an environment can be one in which data about the environment is explicit rather than inferred, for example, from observations. A semantic description of the state of an environment can include agent data as well as data relating to non-agents (e.g., static entities). For each agent and non-agent entity, and where applicable, the data can include one or more of: position, movement (e.g., velocity vector), type (e.g., one-hot encoded), state or configuration, and (for agents) the last action taken by the agent. A semantic description of the state of an environment can also include global information, such as scores or rewards associated with the environment, such as scores in a game or scores or rewards associated with completing one or more tasks.

[0044] The semantic description of the environment state is compiled into a graph data representing the environment state as the input graph In particular, directed graphs. In the input graph, each of the agents and non-agent entities can be represented by a node, and edges can connect each agent to every other agent and to every non-agent entity (non-agent entities do not need to be connected). The semantic description of the state of the environment can be used to provide attributes for the nodes of the input graph. The input graph may have no edge attributes; agent cooperation / non-cooperation (teammate) attributes can be indicated in the edge attributes of the input graph. The output graph has edge attributes; each of these edge attributes provides a representation of the influence of the sender node on the receiver node.

[0045] Figure 2a The graph neural network subsystem 110 is shown. The subsystem accepts an input directed graph G, which includes: a set of node features Where N n is the number of nodes, each n i is a vector of node features; a set of directed edge features Where N e is the number of edges, each e j is the vector of edge features, s j and r j are the indices of the sender and receiver nodes, respectively; and the vector of global features g. For the input graph, the vector of node features can be derived from node attributes (such as agent position and velocity); in cases where the attributes are not applicable, such as the state of non-agent entities, they can be assigned zero values. For the input graph, edges can have no attributes, and the vector of global features can represent the global score or reward when applicable.

[0046] The graph neural network subsystem 110 processes the input graph G=(g, {n i},{e j ,s j , r j}) to determine the output graph Typically, although not necessarily, the input graph and the output graph may have different characteristics.

[0047] The graph neural network subsystem 110 has three sub-functions: edge-wise function f e , node-wise function n and the global function f g Each of these is implemented using a different corresponding neural network (i.e. a neural network with different parameters (weights), i.e. an edge neural network, a node neural network and a global feature network, respectively). In variations, some neural networks and / or updates described later may be omitted.

[0048] In some embodiments, each of these functions is implemented using a corresponding multilayer perceptron (MLP). In some embodiments, a recurrent neural network can be used to implement one or more of these functions. In this case (not shown), the function, i.e., the recurrent neural network, converts the additional hidden graph G h As input, and also provide the updated hidden map as output. The input graph and hidden graph can be combined, for example, using a GRU (Gated Recurrent Unit) style or LSTM (Long Short-Term Memory) style gating scheme. The output can be split, for example by replication, to obtain an updated hidden graph and output graph.

[0049] In an embodiment, the graph neural network subsystem 110 is configured to first apply the edge function f e to update all edges (in each specified direction) and then apply the node-wise function f n To update all nodes, and finally apply the global function f g The input graph is processed by updating the global feature vector.

[0050] Figure 2b The operation of the example graph neural network subsystem 110 is shown. In step 200, for each edge {e j ,s j , r j}, which collects the characteristics of the sender and receiver nodes As well as edge feature vectors and global feature vectors, and use edge neural networks to calculate the output edge vector Then in step 202, for each node {n i}, which aggregates the edge vectors of the node as receivers using an aggregation function to determine a set of aggregated edge features The aggregation function should be invariant with respect to the permutation of edge vectors. For example, it may include determination of an average, maximum, or minimum value. In some embodiments, the aggregation function may include an elementwise summation of edges, e.g. where r j =i. Then, using the node neural network Calculate the output node vector from the set of aggregated edge features, the current node feature vector and the global feature vector Finally, for each graph, the process performs an element-wise summation at step 204, for example Aggregate all edge feature vectors and all node feature vectors, as well as the current global feature vector g, using the global feature neural network Calculate the output global eigenvector g * .

[0051] Figure 3 An embodiment of a graph processing neural network system 106 is shown. The embodiment includes an encoder graph neural network subsystem (GN encoder) 302 coupled to a recurrent graph neural network subsystem (e.g., graph GRU) 304, which is coupled to a decoder graph neural network subsystem (GN decoder) 306. Each graph neural network subsystem may be as described with reference to FIG. 2. Input graph at time step t The GN encoder 302 processes the encoded graph data to provide the encoded graph data. The encoded graph data is processed by the recurrent graph neural network subsystem 304 together with the hidden graph of the previous time step. Processed to provide an updated hidden graph for time steps The latent graph is decoded by the GN decoder 306 to provide an output graph for the time step

[0052] The representation data may be derived directly from one or more attributes of the output graph, or the output graph may be further processed, for example, by MLP 308, to provide the representation data. For example, the node attributes of each agent node of the output graph may be processed by MLP 308 (which may be a single layer "MLP") to reduce the set of node attributes to a set of output values ​​for predicting the agent's actions. The set of output values ​​may include, for example, a set of logarithms of available actions for the agent, or may define, for example, a distribution of continuous actions. In general, the graph processing neural network system 106 may be applied to discrete or continuous actions.

[0053] The graph processing neural network system 106 can be trained by jointly training the graph neural network subsystems; the graph neural network subsystems can be trained based on any output attribute. Supervised training is used based on a dataset of input-output pairs, back-propagating the gradient of any suitable loss function, and training the recurrent graph neural network subsystem using back-propagation through time. For example, in one embodiment, gradient descent is used in batches of, for example, 128 rounds to minimize the cross-entropy loss between the agent's predicted actions and ground-truth actions.

[0054] Figure 3 The architecture of three separate graph neural network subsystems, encoder, recurrent block, and decoder, allows the system to perform relational reasoning on the raw input data before including the time loop, and then again on the output of the recurrent subsystem. This allows the recurrent graph neural network subsystem to construct a memory of the relations between entities, rather than just a memory of their current state.

[0055] In one example implementation, Figure 3 The architecture is as follows: the graph encoder neural network subsystem 302 includes a separate 64-unit MLP with one hidden layer and ReLU activations for each of the edge, node, and global neural networks; each of the edge, node, and global neural networks of the recurrent graph neural network subsystem 304 includes a GRU with a hidden state size of 32; the decoder neural network subsystem 306 is the same as the encoder; the aggregation function includes summation.

[0056] Figure 4 A process is shown for using a graph processing neural network system 106 to provide representation data for predicting agent actions. For each time step t, the process inputs data defining a (semantic) description of the state of the environment and uses this data to construct an input graph (400). The process then encodes the input graph using the GN encoder 302 to provide encoded graph data (402), processes the encoded graph data using the graph GRU 304 to provide a latent graph (404), and decodes the latent graph using the GN decoder to provide an output graph (406). The process can then determine representation data from the output graph, such as by using an MLP 308 and / or processing the output graph as described below, to predict or explain the actions of one or more agents (408).

[0057] Figure 5a (Bottom) shows an example representation of action predictions from the graph processing neural network system 106 as a heat map. Figure 5a(Top) shows the positioning of agents 500a, 500b on an x, y position grid. In this example, possible agent actions include the agent not moving and the agent moving to one of four adjacent positions on the grid (one of five actions). The bottom graph shows, for each agent position, the probability of the agent moving to each of the four adjacent positions on the grid and not moving. Each position is encoded with a predicted logarithm for the corresponding action (e.g., using a grayscale representation).

[0058] Figure 5b (Bottom) shows an example representation of agent-entity importance relationships as a heat map from the graph processing neural network system 106. These relationships can be used to explain agent behavior and can be derived from the edge attributes of the output graph. For example, to determine the strength of the relationship between a particular agent and other entities, the Euclidean norm of each edge vector connecting the node of the agent to the node of each other entity can be determined. Figure 5b (lower), the position of each other entity is encoded with the strength of the relationship between the particular agent 500a and the entity (e.g., using grayscale). Figure 5b Upper, with Figure 5a Same as above). Therefore, this representation can indicate the degree of influence of the entity on this particular agent.

[0059] Figure 6 Agent 600 is shown, which is one of the agents in shared environment 102. For example, agent 600 can be a robot or vehicle in an environment with other robots or vehicles and optionally with non-agent entities. Figure 3 The graph processing neural network system (relational forward model, RFM) 106 of is incorporated into a reinforcement learning (RL) system, represented here as the action selection policy network (Π) 602. Some, all, or none of the other agents may include similar RL systems.

[0060] In the illustrated example, for example, RFM 106 receives a semantic description of the state of the environment in the form of an environment graph, and also receives the last action taken by the agent for one or more other agents (e.g., teammates). This information is compiled into an input graph (pre-processing not shown). In other embodiments, the agent may infer some or all of this information from observations of the environment.

[0061] The agent also receives observations, such as egocentric observations of the environment of the reinforcement learning system. The observations may include still or moving images, and / or any form of sensor data, and / or any other data characterizing the current state of the environment. The RL system also receives reward data, such as defined numerical values, from the environment related to the rewards received by the agent for performing actions. The representation data output 108 from the RFM 106 is provided to the RL system and may be combined with the observations of the environment by a combiner 604 to provide enhanced observations 606.

[0062] The representation data may include Figure 5a (Bottom) A prediction of the action of one or more other agents in the form of a heat map as shown. When both the observation and the heat map (or other representation) are represented egocentrically, the action prediction can include an additional image plane attached to the observation. Each other agent can have a separate additional image plane, or the data in the image planes can be combined into a single additional image plane.

[0063] The representation data may also or instead include the salience of some or all other entities to the agent's actions (or to other agents' actions) (e.g., in the form of Figure 5b (lower) in the form of a heat map as shown in Figure 1. Other entities can be agents and / or non-agent entities. When both the observation and the heat map (or other representation) are represented egocentrically, the influence of other entities on the agent or the influence of other entities on other agents can similarly include additional image planes attached to the observation.

[0064] The reinforcement learning system processes the reinforced observation and reward data and selects actions to be performed by the agent in order to perform the task. The task may be implicitly defined by the reward data. The actions may be selected from a discrete set of actions, or may be continuous control actions; examples of tasks and actions are given later. Typically, an action selection policy network 602 is used to select actions; this may output a Q value for selecting an action, or it may define a distribution for selecting an action, or may output an action directly. The action selection policy network 602 may be trained using any suitable reinforcement learning technique (e.g., a Q-learning technique or an actor-critic technique).

[0065] In more detail, in one example, the action selection output of the action selection policy network 602 may include a corresponding numerical probability value for each action in a set of possible actions that may be performed by the agent. The RL system may select an action to be performed by the agent, for example, by sampling actions according to their probability values ​​or by selecting the action with the highest probability value.

[0066] In another example, the action selection output may directly define the action to be performed by the agent, for example by defining torque values ​​that should be applied to the joints of the robotic agent or acceleration values ​​to be applied to the robot or vehicle driving.

[0067] In another example, the action selection output may include a corresponding Q-value for each action in a set of possible actions that can be performed by the agent. The RL system may process the Q-values ​​(e.g., using a soft-max function) to generate a corresponding probability value for each possible action, which may be used to select an action to be performed by the agent. The RL system may also select the action with the highest Q-value as the action to be performed by the agent. The Q-value of an action is an estimate of the reward that would result from the agent performing the action in response to the current observation and subsequently selecting future actions to be performed by the agent based on the current values ​​of the action selection policy network parameters. Here, reward refers to a cumulative measure of the rewards received by the agent, such as a time-discounted sum of rewards.

[0068] In some cases, the RL system can select an action to be performed by the agent according to an exploration strategy. For example, the exploration strategy can be a greedy exploration strategy, in which the RL system selects an action to be performed by the agent according to the action selection output with probability 1-∈, and randomly selects an action to be performed by the agent with probability ∈. In this example, ∈ is a scalar value between 0 and 1.

[0069] In an embodiment, the training engine 120 can train both the RFM 106 and the RL system. They can be trained jointly, i.e., the agent 600 can have a private RFM 106 that is trained together with the action selection policy network 602. The RFM 106 can be trained using supervised learning as described above to predict action-observation-action pairs. Figure 6 As shown, in an embodiment, gradients are not back-propagated through the RL system into the RFM 106 .

[0070] In one embodiment, the training engine 120 uses an actor-critic technique to train the action selection policy network 602. In this embodiment, the action selection policy network 602 is configured to generate value estimates in addition to the action selection outputs. The value estimates represent estimates of rewards (e.g., the time-discounted rewards that would result from selecting a future action performed by the agent according to the current values ​​of the action selection network parameters given the current state of the environment). The training engine may use a reinforcement learning objective function given by the following formula: The gradient of is used to train the action selection policy network:

[0071]

[0072]

[0073]

[0074]

[0075] where α and β are positive constants, is the expected value relative to the current action selection strategy (i.e., defined by the action selection strategy network parameter θ), V(s t ,θ) refers to the action selection strategy network for observation s t The generated value is estimated, H(π(·|s t ,θ)) is a regularization term, which refers to the action selection network for the observation s t The entropy of the resulting probability distribution over possible actions, is, for example, the n-step-ahead return given by:

[0076]

[0077] where γ is a discount factor between 0 and 1, r t+i is the reward received at time step t+i, V(s t+n ,θ) refers to the value estimate at time step t+n.

[0078] In general, entities in the environment may be natural or artificial, and the environment may be a real environment or a simulated real environment or a virtual environment. Agents may include computer-controlled or human-controlled machines, such as robots or autonomous land, sea or air vehicles. Agents may also include humans and / or animals. Agents may also include stationary or mobile software agents, i.e., computer programs configured to operate autonomously and / or with other software agents or humans to perform tasks such as configuration or maintenance of a computer or communication network, or configuration or maintenance of a manufacturing plant or data center / server farm.

[0079] For example, some embodiments of the system can be used for prediction or control of vehicle traffic or pedestrian traffic (e.g., for traffic signal control to reduce congestion), or for prediction or control of a team of people performing a task or playing a game (e.g., by providing signals to the people based on output from the system, such as representative data). In some other embodiments, the system can be used for cooperative control of robots performing tasks such as warehouse or logistics automation, package delivery control (e.g., using robots), drone fleet control, etc.

[0080] Any autonomous or semi-autonomous agent of the foregoing type may include a reinforcement learning system that operates in conjunction with an RFM system to control the agent. Multiple autonomous or semi-autonomous agents of the types previously described may include an RFM system and a control system, such as a reinforcement learning system, that operates in conjunction with the RFM system to promote cooperative behavior in complex environments. In some embodiments, the agents may be of different types. For example, in an autonomous vehicle or warehouse in a warehouse setting, controlled robots from multiple different manufacturers may operate. In this case, equipping each of these robots / vehicles with a combination of a reinforcement learning system and an RFM system allows different entities to learn to work together. Similar benefits can be obtained using other types of agents.

[0081] Non-proxy entities may include any non-proxy object in the context of the proxy described above.

[0082] exist Figure 6 In other applications of agents of the type shown, in which a relational forward model is incorporated into a reinforcement learning system, the environment is a real environment and the agent is an electromechanical agent that interacts with the real environment. For example, the agent can be a robot or other stationary or mobile machine that interacts with the environment to complete a specific task (e.g., locating an object of interest in the environment, or moving an object of interest to a specified location in the environment, or navigating to a specified destination in the environment); or the agent can be an autonomous or semi-autonomous land or air or sea vehicle that navigates in the environment.

[0083] In these embodiments, observations may include, for example, one or more of images, object position data, and sensor data (e.g., from images, distance or position sensors, or sensor data from actuators) that capture observations as the agent interacts with the environment. In the case of a robot or other mechanical agent or vehicle, observations may similarly include one or more of the position, linear or angular velocity, force, torque, or acceleration, and overall or relative posture of one or more parts of the agent. Observations may be defined in 1, 2, or 3 dimensions, and may be absolute and / or relative observations. For example, in the case of a robot, observations may include data characterizing the current state of the robot, such as one or more of the following: joint positions, joint velocities, joint forces, torques, or accelerations, and overall or relative postures of a part of the robot (such as an arm and / or an object held by the robot). Observations may also include, for example: sensed electronic signals, such as motor currents or temperature signals; and / or image or video data, such as from a camera or LIDAR sensor, such as data from a sensor of the agent or from a sensor positioned separately from the agent in the environment.

[0084] In these embodiments, the action may be: a control input to control a robot, such as a torque or higher-level control command for a robot joint; or a control input to control an autonomous or semi-autonomous land or air or sea vehicle, such as a torque or higher-level control command for a control surface or other control element of the vehicle; or, for example, motor control data. In other words, the action may include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or part of another mechanical agent. The action data may include data of these actions and / or electronic control data (such as motor control data) or more generally, data for controlling one or more electronic devices in the environment, the control of which has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land or air or sea vehicle, the action may include actions to control the navigation of the vehicle (e.g., steering and movement (e.g., braking and / or acceleration)).

[0085] For example, the simulation environment can be a simulation of a robot or vehicle agent, and the reinforcement learning system can be trained on the simulation. For example, the simulation environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent is a simulated vehicle that navigates in the motion simulation. In these embodiments, the action can be a control input that controls a simulated user or a simulated vehicle. Before using the system in the real environment, the simulation environment can be useful for training the reinforcement learning system. In another example, the simulation environment can be a video game, and the agent can be a simulated user playing the video game. Generally, in the case of a simulation environment, observations can include simulated versions of one or more previously described observations or observation types, and actions can include simulated versions of one or more previously described actions or action types.

[0086] In the case of an electronic agent, observations may include data from one or more sensors (e.g., current, voltage, power, temperature, and other sensors) that monitor a portion of a plant or service facility, and / or electronic signals representing the functions of electronic and / or mechanical objects of the device. In some applications, the agent may control actions in a real environment including an object of the device (e.g., in a facility such as a data center, server farm, or power grid mains power or water distribution system), or operations in a manufacturing plant or service facility. Then, the observations may be related to the operation of the plant or facility. For example, in addition to or in lieu of the foregoing, they may include observations of the use of electricity or water by the device, or observations of power generation or distribution control, or observations of resource use or waste generation. The agent may, for example, control actions in the environment to improve efficiency by reducing resource use, and / or, for example, reduce the environmental impact of operations in the environment by reducing waste. For example, the agent may control the use of electricity or other power consumption or water in the facility, and / or the temperature of the facility and / or objects in the facility. Actions may include actions that control equipment items of the plant / facility or impose operating conditions on equipment items of the plant / facility, and / or actions that result in changes in settings in the operation of the plant / facility (e.g., adjusting or turning on / off components of the plant / facility).

[0087] In other applications, the environment is a real environment, and the agent manages the distribution of tasks across computing resources (e.g., on mobile devices and / or in data centers). In these embodiments, an action can include allocating a task to a specific computing resource.

[0088] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more thereof. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-temporary program carrier to be run by a data processing device or to control the operation of a data processing device. Alternatively or additionally, program instructions can be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical signal, optical signal, or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device to be executed by a data processing device. Computer storage media can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. However, computer storage media are not propagation signals.

[0089] The term "data processing apparatus" covers various apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an operating environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0090] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language (including compiled or interpreted languages) or declarative or procedural languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to run on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.

[0091] As used in this specification, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs that are different from inputs. An engine can be a coded block of functionality, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device (e.g., a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, laptop or desktop computer, PDA, smart phone, or other fixed or portable device) including one or more processors and a computer-readable medium. In addition, two or more engines can be implemented on the same computing device or on different computing devices.

[0092] The processes and logic flows described in this specification may be performed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by a dedicated logic circuit (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)), and the apparatus may also be implemented as the dedicated logic circuit. For example, the processes and logic flows may be performed by a graphics processing unit (GPU), and the apparatus may also be implemented as a graphics processing unit (GPU).

[0093] For example, a computer suitable for running a computer program can be based on a general or special microprocessor or both, or based on a central processing unit of any other kind. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic element of a computer is a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more large-capacity storage devices (for example, a disk, a magneto-optical disk or an optical disk) for storing data, or be operatively coupled to receive data from the one or more large-capacity storage devices or to transmit data thereto, or both. However, a computer need not have such a device. In addition, a computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device (for example, a universal serial bus (USB) flash drive) or the like.

[0094] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0095] To support interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to support interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound input, voice input, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0096] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification) or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0097] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between client and server is generated by computer programs running on the respective computers and having a client-server relationship to each other.

[0098] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the content claimed, but should be interpreted as descriptions of features peculiar to specific embodiments of specific inventions. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Moreover, although features may be described above as working in certain combinations and even initially claimed as such, one or more features in the claimed combination may be cut out from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the combination.

[0099] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in a continuous order or that all of the operations shown be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0100] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented reinforcement learning system, comprising a neural network system trained by supervised training based on observations of the behavior of multiple agents in a shared environment for predicting or explaining actions of multiple agents in a shared environment, wherein the reinforcement learning system is configured to select an action to be performed by one of the agents interacting with the shared environment, wherein the shared environment is a real-world environment and wherein the agent is a mechanical agent, or comprises a control device that performs actions for controlling equipment items in a manufacturing or service facility and for working together to control the equipment items, or the agent manages the distribution of tasks across computing resources by performing actions to assign tasks to specific computing resources, or the shared environment is a simulated environment including a motion simulation environment and the agent is a simulated vehicle navigating the simulated environment, the neural network system comprising: an input for receiving agent data representing an agent action for each of a plurality of agents; as well as One or more processors configured to implement: The encoder graph neural network subsystem processes the proxy data as graph data to provide encoded graph data, Wherein, the agent data represents an agent action of each of the plurality of agents; The graph data includes (i) data representing at least the nodes and edges of the graph and (ii) node attributes of at least some of the nodes in the graph, where nodes represent agents and one or more non-agent entities in the environment, The edges connect the nodes in the graph. where the node attributes represent the agent's agent actions, and wherein the encoded graph data includes node attributes and edge attributes representing an updated version of the graph data; a recurrent graph neural network subsystem, comprising a recurrent neural network that processes the encoded graph data and provides processed graph data, the processed graph data comprising updated versions of node attributes and edge attributes of the encoded graph data; a decoder graph neural network subsystem that decodes the processed graph data and provides decoded graph data including updated versions of node attributes and edge attributes of the processed graph data; and The system output provides representation data, the representation data comprising a representation of one or both of node attributes and edge attributes of the decoded graph data for one or more agents, wherein the representation is related to a predicted action or an interpreted action of the one or more agents, The reinforcement learning system includes: an input for obtaining and processing state data representing observations of the state of a shared environment and reward data representing rewards received as a result of an agent performing an action, wherein the reward data implicitly defines a task to be performed; a policy neural network; and a combiner for combining the representation data and the state data to provide an enhanced observation, wherein the policy neural network is configured to process the enhanced observation to select an action.

2. The reinforcement learning system according to claim 1, wherein: The agent data representing the agent action includes agent position and motion data for each of the plurality of agents, and wherein the node attributes used to determine the action of each agent also include attributes of the position and motion of each agent.

3. The reinforcement learning system according to claim 1, wherein: Each agent is connected to every other agent by an edge, and wherein each non-agent entity is connected to every agent by an edge.

4. The reinforcement learning system according to claim 1, wherein: The system output includes one or more output neural network layers that combine node attributes of nodes in the decoded graph data to output representation data, and wherein the representation includes a predicted action for the agent represented by the node.

5. The reinforcement learning system according to claim 4, wherein: The representation data defines a spatial graph of data derived from node attributes of one or more nodes representing one or more agents, and wherein the data derived from the node attributes are represented at or adjacent to the location of the corresponding node in the spatial graph.

6. The reinforcement learning system according to claim 1, wherein: The representation data comprises a representation of edge attributes of the decoded graph data for edges connected to the one or more nodes, and wherein the representation of the edge attributes for the edge is determined from a combination of the edge attributes of the edge.

7. The reinforcement learning system according to claim 6, wherein: The representation data defines a spatial graph, and wherein, in the spatial graph, a representation of edge attributes for the edge is represented at a starting node position of the edge.

8. The reinforcement learning system according to claim 1, wherein: One or more of the encoder graph neural network subsystem, the recurrent graph neural network subsystem, and the decoder graph neural network subsystem are configured as: For each edge, an edge neural network is used to process the edge features to determine the output edge features. For each node, aggregate the output edge features of the edges connected to the node to determine the aggregate edge feature of the node, and For each node, a node neural network is used to process the aggregated edge features and node features to determine the output node features.

9. The reinforcement learning system according to claim 8, wherein: Processing the edge features includes: for each edge, providing the edge features and node features of nodes connected by the edge to an edge neural network to determine an output edge feature.

10. The reinforcement learning system according to claim 8, wherein: One or more of the encoder graph neural network subsystem, the recurrent graph neural network subsystem, and the decoder graph neural network subsystem are also configured to use the global feature neural network to determine a global feature vector, the global feature vector representing output edge features and output node features, and wherein a subsequent graph neural network subsystem is configured to process the global feature vector when determining the output edge features and the output node features.

11. A method of predicting or explaining the actions of multiple agents in a shared environment using a neural network system trained by supervised training based on observations of the behavior of the multiple agents in the shared environment, wherein the agents interact with the shared environment, wherein the shared environment is a real-world environment, and wherein the agents are mechanical agents, or include control devices that perform actions for controlling equipment items in a manufacturing or service facility and for working together to control the equipment items, or the agents manage the distribution of tasks across computing resources by performing actions to assign tasks to specific computing resources, or the shared environment is a simulated environment including a motion simulation environment, and the agents are simulated vehicles navigating in the simulated environment, the method comprising: receiving agent data representing an action of each agent in a plurality of agents; Processing the proxy data into graph data to provide encoded graph data, wherein the graph data includes data representing at least nodes and edges of the graph, wherein each agent is represented by a node, wherein non-agent entities in the environment are each represented by a node, wherein the nodes have node attributes for determining an action for each agent, wherein the edges connect the agents to each other and to the non-agent entities, and wherein the encoded graph data includes node attributes and edge attributes representing an updated version of the graph data; processing the encoded graph data using a recurrent graph neural network to provide processed graph data comprising updated versions of node attributes and edge attributes of the encoded graph data; decoding the processed graph data to provide decoded graph data comprising updated versions of the node attributes and edge attributes of the processed graph data; and outputting a representation of one or both of node attributes and edge attributes of the decoded graph data of the one or more agents, wherein the representation is related to the predicted behavior or explained behavior of the one or more agents, Wherein, the method further comprises: obtaining and processing observed state data representing a state of a shared environment and reward data representing a reward received as a result of the agent performing an action, the reward data implicitly defining a task to be performed; combining the representation data and the status data to provide enhanced insight; The enhanced observations are processed to select an action.

12. The method according to claim 11, wherein: The behavior comprises actions of the agent, and wherein outputting the representation comprises processing node attributes of a node of the decoded graph data to determine a predicted action of the agent represented by the node.

13. The method according to claim 11, for interpreting the actions of an agent, wherein: Outputting the representation includes processing edge attributes of edges of the decoded graph data connecting the influence node to the proxy node to determine data representing importance of the influence node to the proxy node.

14. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to implement a reinforcement learning system comprising a neural network system trained by supervised training, the neural network system comprising: an input for receiving agent data representing an agent action for each of a plurality of agents; The encoder graph neural network subsystem processes the proxy data into graph data to provide encoded graph data. Wherein, the agent data represents an agent action of each of the plurality of agents; The graph data includes (i) data representing at least the nodes and edges of the graph and (ii) node attributes of at least some of the nodes in the graph, where nodes represent agents and one or more non-agent entities in the environment, The edges connect the nodes in the graph. where the node attributes represent the agent's agent actions, and wherein the encoded graph data includes node attributes and edge attributes representing an updated version of the graph data; a recurrent graph neural network subsystem, comprising a recurrent neural network that processes the encoded graph data and provides processed graph data, the processed graph data comprising updated versions of node attributes and edge attributes of the encoded graph data; a decoder graph neural network subsystem that decodes the processed graph data and provides decoded graph data including updated versions of node attributes and edge attributes of the processed graph data; and The system output provides representation data, the representation data comprising a representation of one or both of node attributes and edge attributes of the decoded graph data for one or more agents, wherein the representation is related to a predicted action or an interpreted action of the one or more agents, wherein the reinforcement learning system comprises: an input for obtaining and processing state data representing observations of a state of a shared environment and reward data representing rewards received as a result of an agent performing an action, the reward data implicitly defining a task to be performed; a policy neural network; and a combiner for combining the representation data and the state data to provide an enhanced observation, wherein the policy neural network is configured to process the enhanced observation to select an action, wherein the reinforcement learning system is configured to select an action to be performed by one of the agents interacting with the shared environment, wherein the shared environment is a real-world environment and wherein the agent is a mechanical agent, or comprises a control device that performs actions for controlling equipment items in a manufacturing or service facility and for working together to control the equipment items, or the agent manages the distribution of tasks across computing resources by performing actions to assign tasks to specific computing resources, or the shared environment is a simulated environment including a motion simulation environment and the agent is a simulated vehicle navigating the simulated environment.

15. One or more non-volatile computer-readable storage media according to claim 14, wherein: The agent data representing the agent action includes agent position and motion data for each of the plurality of agents, and wherein the node attributes used to determine the action of each agent also include attributes of the position and motion of each agent.

16. The one or more non-volatile computer-readable storage media of claim 14, wherein: Each agent is connected to every other agent by an edge, and wherein each non-agent entity is connected to every agent by an edge.

17. The one or more non-volatile computer-readable storage media of claim 14, wherein: The system output includes one or more output neural network layers that combine node attributes of nodes in the decoded graph data to output representation data, and wherein the representation includes a predicted action for the agent represented by the node.

18. One or more non-volatile computer-readable storage media according to claim 17, wherein: The representation data defines a spatial graph of data derived from node attributes of one or more nodes representing one or more agents, and wherein the data derived from the node attributes are represented at or adjacent to the location of the corresponding node in the spatial graph.

19. The one or more non-volatile computer-readable storage media of claim 14, wherein: The representation data comprises a representation of edge attributes of the decoded graph data for edges connected to the one or more nodes, and wherein the representation of the edge attributes for the edge is determined from a combination of the edge attributes of the edge.

20. The one or more non-volatile computer-readable storage media of claim 19, wherein: The representation data defines a spatial graph, and wherein, in the spatial graph, a representation of edge attributes for the edge is represented at a starting node position of the edge.

Images