Performing control based on multi-participant interactions

Abstract

A computer-implemented method of obtaining, for a multi-participant interaction is provided, in which each of a plurality of participants is able to perform one of a respective set of actions and each participant receives a reward defined by a respective reward function of the corresponding actions performed by the plurality of participants, corresponding action embeddings for one or more of the actions which one or more of the participants can perform.

Description

DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application PERFORMING CONTROL BASED ON MULTI-PARTICIPANT INTERACTIONS CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to GR Patent Application No.20230100786, filed on September 29, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND

[0002] This specification relates to processing data using machine learning models.

[0003] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0004] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output. SUMMARY

[0005] This specification describes methods which may be performed by one or more computer systems in one or more locations. Some of these methods enable resources (e.g. robots or other hardware resources) to be efficiently allocated to tasks (e.g. which of multiple computer systems, with different respective capacities, is allocated a computing task). Other of the methods allow actions by agents to be controlled in a way which is predicted to be optimal. In particular, some of the methods make it possible for multiple agents (e.g. robots) to cooperate with one another to perform a task in a way that improves their collective efficiency.

[0006] As explained below in more detail, many control tasks of this kind have a formal correspondence (e.g. a logical or mathematical correspondence) to a normal form game (NFG) which is an interaction between multiple participants (also called “players” of the NFG, though in some cases, as described below, the control task itself can involve controlling individuals called players, who do not necessarily have a one-to-one mapping with the players of the NFG). Each of the plurality of participants is able to perform one of a respective set of actions and each participant receives a reward defined by a respectiveDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application reward function of the corresponding actions performed by the plurality of participants. Thus, information about the NFG (e.g. information about optimal participant behaviors) can be used to allocate resources or control agents.

[0007] The correspondence between the participants and items or roles in a system to be controlled varies widely from one application of the present techniques to another. In one case, the participants may be correspond to respective agents (e.g. electromechanical agents (robots) in the real world, or software agents in a computational environment) which themselves perform one of a set of actions (“agent actions”) corresponding to the respective set of actions of the corresponding participants in the multi-participant interaction. Alternatively, in another case, the actions of a first of the participants may correspond to respective roles (e.g. the white / black side in a game of chess, the home / away side in football, P1 / P2 in a symmetric game; or the role of performing a certain mechanical / computational task), and the actions of a second of the participants correspond to respective options for filling that role (e.g. the identity of a chess player; a specific football player; a specific game player; or a specific mechanical agent or computational agent or other resource). Thus, a combination of the actions by the two participants amounts to the allocation to a specific role (defined by the action of the first participant) of one of the options for filling that role (defined by the action of the second participant), e.g. assigning a specific robot or computer agent to perform a mechanical / computational task; or assigning a particular data storage device to store a particular load of data). In yet another case, the actions of two (or more) participants correspond to respective options for choosing two (or more) parties (e.g. two or more teams / players / artificial agents) from a set of available parties, so that a combination of the actions by two (or more) participants amounts to a selection of a set of two (or more) of the parties (e.g. parties who could compete against each other in a competition).

[0008] As noted, strategic interactions between multiple participants can be formulated as normal-form games (NFGs) in which multiple participants simultaneously select resepective actions (the respective actions for the participants are collectively called a “joint action”), and the participants receive respective rewards (“payoffs”) according to the joint action. The strategic interactions are presented as payoff tensors, in which a corresponding reward value to each participant is tabulated under every possible joint action. The payoff tensor can also be considered as a reward function, which defines a respective reward to each participant as a function of the joint action. In some cases, the payoff tensor may contain omissions, asDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application described below. Any permutation of participant actions would define a strategically equivalent game.

[0009] It would be valuable to form an embedding function to generate representation of aspects of an NFG, such that the presentation can be processed (“an inquiry”) to obtain information about the NFG. The embedding function may be implemented as a neural network defined by a plurality of numerical parameters which are learned.

[0010] It would be desirable for the representations to have several qualities. Firstly, it is desirable for the representations to be versatile, allowing for information to be obtained at the level of individual actions, joint actions, per-participant or describing the entire game. Also, it would be desirable for the representation to be “invariant”: that is, for any per- action inquiry, the output of the inquiry for two actions is exchanged if the positions of two actions is exchanged in the pay-off tensor. Thirdly, it is undesirable if the embedding function which generates the representation of an action assumes that outcomes of all joint actions are known (have been observed). That is, the representations should be generated in a way which accommodates NFGs which are “incomplete”, in the sense that the payoff tensor contains omissions. For example, in practical applications such as generating rankings in Go and Chess, it is infeasible to evaluate and test all pairs of players, and it would be desirable if the representation of the game enables inquiries which make predictions about the game based on incomplete information from a subset of the match- ups. Fourth, it would be desirable if the embedding function is usable for games of different sizes; for example, it would be desirable if the number of parameters does not depend on the size of the game (in game theory, the “size” of a game is commonly measured in any of a number of ways, such as the number of unique game states possible, the number of decision points, and the total number of possible actions over all decision points). Finally, it would be desirable for the neural network architecture to be interpretable, allowing for different stages of the network to be inspected.

[0011] In general terms, the present technique proposes that a representation of a NFG is formed in which neural network(s) are used to form corresponding representations for each of the possible actions each participant can take.

[0012] For that purpose an action embedding system (also called an NfgTransformer) is used which includes one or more one or more action encoder neural networks (also called NfgTransformer blocks). The one or more action encoder neural networks are used (successively, if there is more than one action encoder neural network) to refine initial action embeddings (e.g. action embeddings which are a default value such asDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application having all components being zero, or random action embeddings) for one or more of the actions which one or more of the participants can perform. In an implementation, the action encoder neural network each comprise multiple successive attention mechanisms, and are each conditioned on the reward function.

[0013] To make an “inquiry” about the NFG, a decoder network (e.g. another trained neural network) may process one or more of the action embedding(s) to obtain a specific type of information relating to the interaction. Based on the information, the resources / agents are controlled. The information may, for example, be data indicative of whether action(s) by participants maximize an expectation of their rewards. Alternatively, the information may be estimates for values of the reward function which are not known prior to the generation of the action embeddings.

[0014] The action embeddings may be used as the input to a decoder network which has been trained to receive one or more of the action embeddings, and to generate an output providing a specific type of information relating to the interaction (e.g. data indicative of whether action(s) by participants maximize an expectation of their rewards, or, as explained below, estimates for values of the reward function which are not known prior to the generation of the action embeddings), and based on which the resources / agents can be controlled. Many decoder network architectures are possible, and a decoder network of a certain selected architecture can be trained to give information of a desired type.

[0015] In a first example, the participants in the interaction may formally correspond to respective agents (e.g. robots) which collectively perform at least one task in an environment. The action for each participant may correspond to one of the tasks, or a component of one of the tasks, to which the corresponding agent is allocated. Alternatively, each action for a participant may correspond to an “agent action” (i.e. an action performed by one of the agents, such as an action which contributes to solving a task, or a component of one of the tasks).

[0016] In another example, the participants in the game may again correspond to respective agents which collectively perform at least one task, and the actions may correspond to different proportions of resources of the agent (e.g. different proportions of the time of the agent) which are allocated to various functions required to perform the task. For example if the task is keeping a real-world environment clean, a given action for a given participant may indicate a respective proportion of the agent’s time or other resources which is spent on each of a number of tasks or sub-tasks (e.g. washing, collecting litter, etc.).DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application

[0017] In both examples, the reward function may be indicative of one or more of a metric of the quality of performance of the task(s), and a physical cost of performing the task(s) by the agents (e.g. the wear experienced by the robot and the power consumption by the robot). In typical implementations the ability of a first agent to perform a component of the task(s) in the environment is affected by the one or more other agents performing the one or more other components of the task in the environment.

[0018] In a third example, there are situations in which a plurality of resources (e.g. multiple robots, or multiple computer-system-implemented computational models such as large language models (LLMs), or data storage spaces) are available for carrying out a plurality of tasks (e.g. manipulation and / or navigation tasks in a real-world environment, or computational tasks performed by the computational models or using the data storage spaces), and it is known, for certain combinations of resources and tasks, how suitable a certain resource is for performing that task, but the suitability of some of the resources for some of the tasks is not known. In other words, if there is a numerical measure of the suitability called the “performance value”, the performance value is known for some combinations of resources and tasks (e.g. as a measured performance of the task using the resource (“a performance measure”)), but the performance value for other combinations of resources and tasks is not known.

[0019] This is formally equivalent to a two-participant interaction in which one of the participants can take actions (“first actions”) which correspond to choices of one of the resources, and the other participant takes actions (“second actions”) which correspond to choices of the task, and the reward function for a given combination of actions corresponds to the performance value for the corresponding combination of resources and tasks. Thus, the value of the reward function is known for some combinations of first and second actions, and not for others. Although the values of the reward function for some combination of actions are not known, attention embeddings for the first actions can still be generated by the techniques explained here (the fact that the values of the reward function for some of the combination(s) of the first actions and second actions are not known, can be dealt with by “masking” certain “keys” in the action encoder neural network), and a decoder network can be used to give estimates (predictions) of the performance values for those combinations of resources and tasks for which the performance values were not already known. This makes it possible to derive a performance value (known or estimated) for any given task (i.e. any second action) and any resource (i.e. any first action). Thus, the suitability (performance values) of each of the resources for a certain task may beDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application established, and using this, a given resource can be chosen for the task (e.g. as the resource for which the performance value is highest; or in a more complicated way if there are additional constraints on the choice, e.g. if a given resource can only be used for a specified proper subset of the tasks, e.g. for only one of the tasks). In some implementations, the tasks may have the same character, e.g., they may all be routing tasks, or they may all be scheduling tasks, or they may all be navigation and / or manipulation tasks, or they may all be manufacturing tasks.

[0020] In terminology explained below, the decoder network may be a “per payoff decoder network”, since the predicted performance value is the reward to a single participant for performing the first action, given that the second participant performs the second action. Alternatively, another form of decoder network may be used, particularly if other aspects of the application (e.g. resource allocation) are to be determined, e.g. a “per joint action decoder network” or a “per action decoder network” (e.g. for NE solving).

[0021] The process for generating action embeddings may be performed for only a subset of the participants (e.g. only a first of the participants in an example given below), and / or optionally only for a subset of the actions the one or more participants can perform (e.g. in an application given below, action embeddings are only needed for actions for which there is incomplete information about the reward function).

[0022] To generate the action embedding for a given action, an initial embedding for the action is constructed, e.g. as a vector having a number of components denoted D. The initial embeddings for each of the actions may be the same, e.g. with all the components of all of them being the same value, e.g. zero. The number of action encoder neural networks is denoted K. They may be arranged in a stack, i.e. in an ordered sequence, with each of the action encoder neural networks except the first, receiving the output of the preceding action encoder neural network. The initial embedding is processed by a first of the action encoder neural networks in the stack, to give an updated embedding which is input to the next action encoder neural network of the stack, and so on, until the action embedding is output by the last (K-th) action encoder neural network of the stack. For each action encoder neural network, the input to that action encoder neural network is a set of the current version of the embeddings (“the current embeddings”) for each of the actions for which an action embedding is being generated, i.e. the initial embeddings in the case of the first action encoder neural network of the stack, and in all other cases the corresponding embeddings output by the immediately preceding action encoder neural network in the sequence. TheDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application outputs of the action encoder neural network (e.g. at successive corresponding times) are a set of updated embeddings of each of the actions for which an action embedding is to be generated. The set of updated embeddings generated by each action encoder neural network may be independent of the order of the current embeddings in the input to the action encoder neural network. In an implementation, rather than there being a stack of K action encoder neural networks implemented by separate computing units, a single computing unit (e.g. an element of software run by a single computer system, or a hardwired circuit) may be applied successively K times to implement the K action encoder neural networks.

[0023] In an implementation, each action encoder neural network may comprise a sequence of attention mechanisms. A first attention mechanism of each of the action encoder neural networks may be a first self-attention mechanism. The action encoder neural network takes the current embeddings it receives, e.g. in respect of actions by a corresponding one of the participants, and for each current embedding generates “key vectors” for each of a set of actions performed by the whole set of participants (“joint actions”). A joint action can be considered as the action to which the current embedding relates and also a plurality of “joint action sets”, where each joint action set is one of the respective set of actions for each of the other participants. In other words, a given “joint action” (i.e. set of N actions for each of the respective N participants) may be considered as an action of any one of the participants, and a “joint action set” of the actions of the other participants.

[0024] The key vector for each joint action may comprise (e.g. may be composed of) the reward function for the corresponding actions by all the participants, and the current embedding for the actions of the joint action. Thus, if the number of actions each actor can perform is denoted by T, the number of key vectors may be up to TN. However, as mentioned, key vectors may not be constructed for all TNpossibilities, e.g. because certain combinations of actions are not possible (e.g. there may be a constraint that not all the participants can perform the same action, or because only action embeddings are only needed for a subset of the actions, or because the value of the reward function is not known for some combinations of possible actions of the participants).

[0025] The attention mechanism may be single, or multi-head, attention mechanism. In general an attention mechanism determines a relationship between two sequences; a self- attention mechanism is a special case of an attention mechanism, and is configured to relate different positions in the same sequence to determine a transformed version of the sequence as an output. An attention mechanism input may comprise a vector for each element of aDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application sequence, and the attention mechanism output comprises a vector for each element of an output sequence, which may correspond to a respective element of the input sequence.

[0026] In the case of a self-attention mechanism, the same input sequence provides two inputs to the self-attention mechanism (a first input, which as explained below may, if the self-attention mechanism is a QKV attention mechanism, be “key-values pairs” comprising a key vector and an identical value vector; and a second input which termed a query vector) and the inputs are used by the self-attention mechanism to determine a new representation of the same sequence for the attention layer output, which similarly comprises a vector for each element of the input sequence.

[0027] The output of the first self-attention mechanism may be a number of vectors equal to the number of vectors it received. For each action by a participant for which an action embedding is being formed, there may be a respective output for some (or all) of the possible joint action sets for the other participant(s).

[0028] The output of the first self-attention mechanism may be input to a second attention mechanism of the action encoder neural network, which may apply a different attention mechanism (e.g. a cross-attention mechanism, not a self-attention mechanism). If the attention mechanism is a QKV attention mechanism, the key-value pairs (key vector and value vector) may be based on the output of the first attention mechanism; and the current embedding for the action for which the action embedding is being generated may be used as the query vector. This gives a respective modified embedding for each of the current embeddings.

[0029] In one possibility, each modified embedding may be output from the action encoder neural network as the updated embedding. Alternatively, each action encoder neural network may successively apply a stack of a number (e.g. a number denoted A, which is an integer greater than or equal to one) of further (second) self-attention mechanisms to the modified embeddings, to generate the updated embedding. That is, the first further self- attention mechanism receives the output of the second attention mechanism, and each of the other further self-attention mechanisms receives the output of the preceding further self- attention mechanism. Any one or more of the further self-attention mechanisms may be a QKV attention mechanism, and may use the input it receives to form identical key vectors, value vectors and query vectors.

[0030] As noted, not all of the possible vectors which a given one of the attention mechanism could in principle receive may actually be generated. For example, for certain combinations of the actions of the N participants, the corresponding values of the rewardDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application function may not be known, so key vectors for those combinations are not created. Furthermore, certain combinations of actions may be excluded. In this case, in each of the attention mechanism one or more of the vectors may be “masked”, e.g. treated as if they existed but were all zeros.

[0031] The action embeddings, once generated, may be used by inputting them to a decoder network. The decoder networks are used to obtain specific information relating to the NFG represented by the reward function on which the action encoder neural networks were conditioned. Multiple decoder networks may be provided to obtain different corresponding forms of information. Each decoder network is configured to operate based on one or more of the action embeddings. Optionally, the action embeddings generated by a single action embedding system may be suitable for use as inputs to more than one of the decoder networks; alternatively, a respective action embedding system may be provided for each decoder network.

[0032] A first sort of decoder network is a “per action decoder network” which receives one of the action embeddings, and can generate an output relating to the action corresponding to the action embedding. For example, the per action decoder could be used to provide a value indicating if the action is, or is close to (i.e. less than a threshold amount away from according to a distance metric in the space of actions), an action for the same participant which is an equilibrium for the NFG. Or it could provide an estimated typical value of rewards received by that participant if that action is taken (i.e. averaged over the possible actions of other participants). In the case of an equilibrium, the equilibrium may be a Nash equilibrium. In the following discussion, a Nash equilibrium is assumed throughout, but in game theory a Nash equilibrium is a special case of a correlated equilibrium, which in turn is special case of a coarse correlated equilibrium, and the present techniques may in principle be used for any of these equilibria, so disclosure herein which references a Nash equilibrium is to be understood in the more general sense as a disclosure of the present techniques in relation to any form of equilibrium.

[0033] A per action decoder network may be used to process (e.g. successively) all (or as discussed below, some of) the actions a participant may perform (e.g. all actions of the corresponding the set of actions for the participant), to obtain a respective output for each action, which provides information about the action. In cases in which the participants correspond to agents (as discussed above), an agent corresponding to the participant may then be controlled based on the information. For example, the output of the embedding network indicates that a certain action for the participant is part of, or close to, an actionDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application which is part of a Nash equilibrium for the interaction (NFG), or has the highest typical reward value, or that the action should be a certain proportion of the actions which the participant performs to achieve the Nash equilibrium. In these cases, the agent may be controlled to perform that action according to the output of the decoder network.

[0034] Note that to save computing requirements, the per action decoder network may not be supplied with action embeddings for all the actions a participant may perform; that is, there may be a trade-off between the compute and the quality of the action which is selected. For example, action embeddings may be selected from a pre-computed cache of actions, as described below (or actions may be selected from a distribution, and corresponding action embeddings for selected actions may be generated), and input to a per action decoding network trained to give an estimate of the action received. This process may be repeated until a termination criterion is reached (e.g. the expected reward for one of the action embeddings is above a threshold), and then the action with the highest estimated reward could be performed. As an alternative to selecting pre-computed action embeddings at random, or selecting actions from a distribution at random, there could be a search over the space of actions for actions for which the per action decoder network predicts a high expected reward based on the corresponding action embedding. Typically, similar actions may have similar action embeddings and similar expected rewards, so this search could progressively approach a local maximum in the action space of the expected reward value. For example, it may sample a respective example action from each of several corresponding regions of the action space, determine which of these actions is best, and then further sample for actions in the corresponding region of the action space. This process may be repeated to progressively approach an action having a (e.g. locally maximal) expected reward.

[0035] If, e.g. using a per action decoder network, it has been determined that a controlled agent should perform a certain task, or component of the task, the agent (e.g. robot or autonomous vehicle) may be provided with an (e.g. pre-computed) action policy which programs it to perform that task, or component of the task, e.g. by receiving sensor readings (e.g. captured images) of a real-world environment, and based on those sensor readings to take corresponding agent actions (manipulation and / or navigation actions), chosen by the action policy based on the sensor readings, which contribute to solving the task, or the component of the task. In other examples, the participant may correspond to a computing job scheduling agent, a router for a communications network, a controller for routing electrical power, or a controller for a manufacturing operation. Further discussion ofDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application applications of the present technique is given below. A per action decoder network may comprise a feedforward network, such as a multi-layer perceptron (MLP).

[0036] A second sort of decoder network is a “per joint action decoder network”. This receives multiple ones (e.g. two) of the action embeddings, and provides information about the combination of the actions. For example, one possible per joint action decoder network receives respective actions for each of the participants, and outputs one or more values describing consequence(s) of the combination of those actions, for example the (in some cases unknown) reward function (e.g. a total of the rewards to all participants, or a deviation of the total rewards to all participants from those there would be under a NE) for that combination of actions. A per joint action decoder network may comprise a summation unit (or concatenation unit) to sum the received vectors, followed by a feedforward network, such as a multi-layer perceptron (MLP).

[0037] Based on this new information, as discussed above, a resource may be selected to perform a certain task. The resource may be: a robot or autonomous vehicle; a computing hardware resource; or a routing resource, as discussed more fully below.

[0038] Another sort of decoder network is a “per payoff” decoder network, which may receive multiple ones of the action embeddings, and obtain information about possible reward (payoff) values if those actions are taken. A per payoff decoder network, like a per joint action decoder, receives information about the respective actions of multiple participants (e.g. a joint action, which is a respective action by all participants), but generates information relevant to a specific one of the participants, e.g. the reward a specific participant would receive if all participants perform the joint action. (By contrast, a per joint action decoder network is used to predict information which is not specific to a single participant, e.g. a deviation gain prediction). A per payoff decoder network may comprise an attention layer to apply an attention mechanism in which one of the received vectors is used as a query vector, and other(s) of the received vectors is / are used as key and value vectors. The output of the attention mechanism may be processed by a feedforward network, such as a multi-layer perceptron (MLP).

[0039] An example of the use of a per payoff decoder network is an implementation in which there is a two-participant interaction, and the actions of a first participant correspond to respective choices of a first player (or a first team of players) from a set of players (or a set of teams of players), and the actions of a second participant correspond to respective choices of a second player (or a second team of players) from a set of players (or a set of teams of players); note that the first and second players / teams may be chosen fromDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application the same set of teams / players, or from different respective sets of teams / players. A per payoff decoder network may be trained, upon receiving action embeddings for an action of the first participant and an action of the second participant (i.e. a specific choice for the first team / player and the second team / player), to generate data indicative of which player / team would win a competition between the two teams / players (e.g. a football game or a video game, or any other sort of competitive activity) and / or a confidence level for that prediction, e.g. data predicting a likelihood that one of the players / teams would win a competition the other player / team, and / or predicting a margin by which one player / team would win the competition. The possibility that both participants choose the same player / team would be excluded (e.g. pairs of action embeddings for such combinations of actions may never be input to the per joint action decoder network).

[0040] Based on this data, a pairing may be selected between one player / team and another player / team (e.g. as a pair of players or pair of teams such that the likelihood of one winning the competition against the other is within a certain numerical range around 50%, or for which the predicted margin by which one player / team would win is less than a threshold). The selected pair of players / teams may then be caused to perform an actual competition. For example, two teams of football players may be selected from a set of (e.g. many) football teams to play a football competition; or two human players may be selected from a set of (e.g. many) human players to compete in a video game competition; or one human player and one artificial agent (e.g. computer program or robot; other sorts of agent to which the present techniques are applicable are discussed below) may be selected from a set of human players and a set of artificial agents, to compete in a competition; and / or two artificial agents may be selected (from a set, or from respective sets, of artificial agents), to compete against each other. Many systems for training the control systems of artificial agents involve pairing artificial agents with each other, or with human players, in a competition, and the control system(s) are modified after or during the competition to improve the corresponding artificial agent(s) (a “self-play training process”). The present techniques can be applied to such a situation. In some cases, the training process is speeded up by picking pairs of artificial agent(s) which has a chance of winning close to 50%, or pair of an artificial agent and a human for which each has a chance of winning close to 50%.

[0041] It is to be understood that although the above description, for simplicity, is for the case of two competing parties (e.g. two players or two teams), the number of competing parties may be any integer greater than two, and the discussion above may be generalized to the selection of more than two parties to compete. For example, “three sidedDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application football” is a version of association football for three teams played on a hexagonal pitch with three goals, one for each team. The present techniques may be used to select groups of three, roughly equally strong, football teams to compete, from a longer list of teams (e.g.3 teams out of 30 available teams). More generally, to select players / teams for a competition, the number of the participants could be equal to the number of parties (players / teams) in the competition (e.g. three), and each action of each participant would correspond to a choice of the list of players / teams who might be chosen to compete. The decoder network could be arranged to receive a set of respective action embeddings for each of the participants (i.e. a number of action embeddings equal to the number of parties in the competition, e.g. three) referred to an as “action embedding set”, with each action embedding of the action embedding set corresponding to a different respective choice of a player / team. The decoder network may be trained to output data indicative of the results of a competition between the corresponding players (e.g. a level of confidence that one of the players / teams would win such a competition), and set of players / teams could be chosen to actually compete based on the action data, e.g. a set of players / team for whom the level of confidence was low.

[0042] Similarly, any number of roughly equally strong artificial agents (and optionally human agents) may be chosen from a larger set of artificial agents (and optionally human agents), to compete against each other, and / or against other artificial and / or human agents, e.g. in a self-play training process.

[0043] Another sort of decoder network is a “per game” decoder network, which may receive the action embeddings for all possible actions by all participants, and provide information about the game as a whole. A per game decoder network may comprise a summation unit (or concatenation unit) to sum the received vectors, followed by a feedforward network, such as a multi-layer perceptron (MLP).

[0044] In some implementations, the generation of the action embeddings (using a an action encoder system including a pre-existing stack of action encoder neural networks) is the most computationally expensive part of the process of obtaining information describing a certain multi-participant interaction using the action encoder neural networks and one or more of the decoder(s) (e.g. the action encoder system may have a much larger number of parameters than the decoder network(s)). This may require an excessive amount of computational resources for a computing device with low computational power (e.g. a mobile device, e.g. a smartphone), especially if only a short time is available when making a decision about how an agent / resource control operation is to be performed (“inference time”).DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application

[0045] For that reason, optionally some or all the action embeddings may be pre- computed (e.g. at a time when a computer system which computes them is not busy, and at a time before the action embeddings are being used to control agents / resources etc.) and stored in a cache, and / or the generation of the action embeddings for a first computer system to use to make control decisions may be performed by an auxiliary computer system (e.g. a server), e.g. one which is more powerful than the first computer system (e.g. is capable of performing a larger number of computing operations (e.g. floating point operations) per second). At inference time (e.g. a time when the first computer system is controlling agents or other resources based on the formally-equivalent multi-participant interaction), the first computer may retrieve the pre-computed action embeddings from a cache (e.g. its own cache, or a cache of the auxiliary computer system), and process them using one or more decoder networks. In other words, the generation of the action embeddings may be “front- loaded” and / or outsourced.

[0046] For example, at inference time (e.g. a time when a computer system is responsible for making control decisions to control agent(s) and / or other resources), the environment and / or state of the agents define a context. This context may comprise a state of an environment in which the agents operate, determined from observations, and / or a state of the agents and / or a stage of a control task which has been reached.

[0047] The context may imply that certain actions or action combinations are inapplicable (e.g. not consistent with the context). For example, for an agent which is a mechanical agent, certain agent actions (and therefore corresponding actions by a participant corresponding to the mechanical agent) may only be applicable if an object in the environment to be moved is less than a size threshold. If the context comprises the information that the size of the object is above the size threshold, those agent actions are inapplicable, and the part of the formally equivalent NFG game which deals with the situation of objects below that size threshold is irrelevant. Accordingly, these actions, and their corresponding action embeddings, may not be used in controlling the agent, given this context.

[0048] In another example, certain actions may be applicable to a first portion of a control task (e.g. picking up an object in the environment) and certain actions may be applicable to a second portion of the control task (e.g. throwing the object). To perform the first part of the control task, only the action embeddings for the former actions are needed. To perform the second part of the control task, only the action embeddings for the latter actions are needed.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application

[0049] In both these situations, the control system would control the agent(s) or other resources based on the action embeddings for some or all of the actions which are relevant given the context.

[0050] In this case, to determine how control should be carried out in the context, the computer system responsible for control decisions may retrieve from a cache (its own, or that of an auxiliary computer or an external database) some or all of the action embeddings for actions selected based on the context, e.g. the actions which are compatible with the context (e.g. not necessarily all the compatible actions, because of the trade-off mentioned above). These are thus pre-computed action embeddings (e.g. computed before the context was known and / or before the control of an agent / resource was being performed). The action embeddings for actions which are not compatible with the context do not need to be retrieved, leading to a reduction in the memory requirement at inference time.

[0051] Note that the action embeddings provide a compressed representation of a game defined by a certain reward function. For example, for N participants each capable of T actions, the number of possible action combinations for which the reward function defines a reward is TN, and the reward function defines a respective reward received by each of the N participants for each of this combination, so in other words the reward function (payoff tensor) has up to N×NTnumerical values. By contrast, the set of all action embeddings is only N×T×D numerical values, where D is the number of components (numerical values) of each action embedding (i.e. one D-dimensional action embedding for each of T actions by each of N participants). This may be much fewer numerical values than the payoff tensor. Furthermore, adding another participant (i.e. increasing N by one) increases the payoff tensor exponentially, but only increases the number of numerical values defining the action embeddings linearly. Thus, storing action embeddings, rather than full reward functions (payoff tensors) may give an enormous data saving. Experiments have shown that in many game classes the amount of data which has to be stored until inference time is under 10% of the amount of data required to store the full payoff tensor. In other words, if a control action is to be performed based on a formally equivalent NFG, an efficient way of doing this is to compute the reward function (payoff tensor) at a time when computational operations are cheaply available (e.g. using an auxiliary computer), then calculating the action embeddings, and discarding the reward function, thereby freeing up memory. The action embeddings may then be stored in a cache until inference time (i.e. as pre-computed action embeddings), ready for some or all of them to be retrieved at interference time and used to make control decisions.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application

[0052] The action encoder neural network(s) (e.g. each of the attention mechanisms of each of the action encoder neural networks) and the decoder network(s) are each defined by respective pluralities of numerical parameters. They may be trained by supervised learning, e.g. by putting the action encoder neural network(s) in series with one or more of the decoder networks (i.e. such that action embedding(s) output by the (stack of) action encoder neural networks, are input to the one or more decoder networks) and training them jointly (i.e. with updates to the action encoder neural network being substantially simultaneous with or interleaved with updates to the decoder networks), e.g. by a supervised learning process, based on knowledge of what the output of the decoder network(s) should be. The training may be to successively reduce a loss function.

[0053] In this process, multiple “training reward functions” may be obtained (e.g. sampled from a probability distribution for reward functions), and successively, for different ones of the training reward functions in turn, modifications may be made to the action encoder neural network and the one or more decoder networks. In other words, the loss function may vary over time, based on current one(s) of the training reward functions. In other words, the stack of action encoder neural networks and the decoder network(s) are trained on a large number of training reward functions, and then used to provide information about an interaction (NFG) defined by just one reward function. This is more successful if the one reward function is representative of the training reward functions.

[0054] For example, upon the stacks of action encoder neural networks receiving initial embeddings for one or more of the actions of one or more of the participants, the stack of action encoder neural network(s) may together generate, conditioned on the current training reward function, one or more corresponding action embeddings. The action embeddings may be supplied to the decoder network(s), to generate information about a multi-participant interaction defined by the current training reward function. Modifications are then made to the action embedding neural networks and / or decoder networks to bring the outputs of the decoder networks closer to knowledge (e.g. ground truth knowledge) about multi-participant interaction defined by the current training reward function (e.g. by reducing the value of a loss function indicative of the discrepancy, e.g. by back-propagation to the parameters of the action encoding neural networks and the decoder networks). Thus, this is a supervised learning process.

[0055] A decoder network used for this training may be one for which “ground truth” information is available. For example, as noted above, in a two-player game one possible decoder network is a decoder network trained to predict the value (“game payoff”)DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application to either (or both) of the participants of the reward function for a combination of two action embeddings it receives, which embed actions by the two respective participants (in the example given above, this value was equivalent to the performance value for a resource corresponding to an action of a first of the participants, and a task corresponding to an action of a second of the participants). This information is available for the sampled game, because it is the value of the sampled reward function for the corresponding action of the first participant and the corresponding action for the second participant. Thus, the training may be carried out by supervised learning with the stack of action encoder neural networks in series with a decoder network which, during the supervised learning, is trained to predict values of the reward function. In other words, the stack of action encoder neural networks and decoder network may be trained jointly such that when the stack of action encoder neural networks generates an action embedding for an action of the first participant, and (subsequently) an action embedding for an action of the second participant, the decoder network is configured to receive these two action embeddings and correctly predict a corresponding value of the reward function (e.g. the reward received by either or both of the first and second participants based on the corresponding pair of actions). For example, the decoder network may receive the action embeddings (e.g. as a concatenation of the action embeddings in a certain order, e.g. the action embedding of the first participant’s action may be first in the concatenation), and predict the reward received by the first of the participants, and / or the reward received by the second of the participants. Both of those rewards are known since they are defined by the sampled reward function. Note that in order for the decoder network to perform successfully, the stack of action encoder neural networks learns to represent, in the action embeddings it produces, relevant information from the reward functions (game payoffs). The decoder network may be a per payoff neural network (e.g. for predicting a reward to a specific participant) or a per joint action neural network (e.g. for predicting a total of the rewards over all participants).

[0056] In the case of games with more than two participants, this is generalized, such that for a respective action generated by each of the participants, the action encoder neural network generates respective action embeddings for the respective actions of each of the participants. The decoder network receives the action embeddings (e.g. as a concatenation of the action embeddings), and predicts the corresponding reward(s) received by any one or more of the participants, as defined by the sampled reward function. In the iterative training, successive updates are made to the stack of action embedding neural networks and the decoder network, such that the prediction of the decoder networkDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application gradually becomes closer to the ground truth reward defined by the sampled reward function.

[0057] Alternatively, a per action decoder network (i.e. an encoder trained to receive a single action embedding, embedding a single action of one of the participants) may be used in the joint training instead of, or in addition to, a per payoff decoder network or a per joint action decoder. The per action decoder network may, for example, be trained to produce data indicating how far the action is from the Nash equilibrium. The Nash Equilibrium can, in some cases, be calculated analytically, or estimated by a known method. There is no ground truth label per se, but a ground truth loss function can be computed given the sampled reward function (game payoff) and the decoder output. During the joint training, the per action decoder network is iteratively modified (trained) to give a higher per-action output upon receiving an action embedding which the stack of action encoder neural networks generates for an action correspondingly near to a (true) Nash Equilibrium of the game defined by the sampled reward function.

[0058] After the joint training, any desired further decoder network which was not trained as part of the joint learning may be trained by a subsequent supervised learning process, by putting it in series with the trained stack of action encoding neural networks, and repeating the process above with the parameters of the action encoding neural networks frozen.

[0059] In one implementation, outputs of one or more of the decoder networks may be used (e.g. iteratively) to modify the reward function of the multi-participant interaction for which the decoder network(s) were used to provide information. For example, if the reward function is defined by multiple numerical parameters, and if the action(s) of one (or more) of the participants, as predicted by one or more of the decoder network(s) (e.g. assuming that the participants take respective actions which the per action decoder network indicates are a Nash equilibrium of the interaction), are not as desired, an “interaction loss function” may be formed indicating the difference between the predicted action(s) and the desired actions. The numerical parameters of the reward function may be iteratively modified to reduce the value of the interaction loss function (e.g. by back-propagating updates through the (frozen) action encoder neural networks and (frozen) decoder network(s) to the numerical parameters of the reward function). By this technique, a reward function may be iteratively generated such that the participants, who act to maximize their own rewards (e.g. using a control system other than one suggested here), are encouraged toDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application perform the desired actions in order to maximize their respective reward (“mechanism design”).

[0060] Particular embodiments of the subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0061] First, the present methods provide a rational (and in ideal cases, optimal) way of allotting resources to multiple tasks to give an efficient performance of the tasks. This may be achieved without the expense of determining the suitability of every resource to every task; in order words, partial information about the suitability of some of the resources for some of the tasks is used to infer the suitability of other combinations of the resources and tasks. Experimentally, this has been found to be very accurate, when the partial information is sufficiently great (even if it provided information about only a small proportion, e.g. under 20%. or even under 10%, of the resource-task combinations). This greatly reduces the expense which has to be undertaken to obtain the suitability information (known performance values).

[0062] Secondly, the present method permits control strategies for one or more agents to be selected in a rational (and in some cases optimally efficient) manner. Although other methods of controlling multiple agents exist by maximizing a collective result, in some cases this can result in solutions which are not egalitarian. For example one or more agents may sacrifice themselves by taking no reward to improve the collective reward. This is typically undesirable in an engineering context, e.g. in a communications system it could result in no data packets being transmitted by an agent which is a certain router, so that communication for certain users fails completely. The techniques described herein appear to avoid such undesirable scenarios.

[0063] Third, the described techniques are particularly suitable for implementation by computational devices having limited computing power but which are required to make control decisions (i.e. perform a control operation). This is because the computationally expensive task of generating action embeddings can be performed in advance, and / or by a more powerful auxiliary computer, and the relatively computationally less expensive operation of processing the action embeddings using decoder network(s) to generate information about the multi-participant interaction needs to be performed at run time.

[0064] The pre-computing of the action embeddings allows, in an implementation, for a parallelization of downstream tasks (e.g. at interference time), since these are independent of each other. For example, for multi-agent control situations, actionDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application embeddings for multiple respective actions for all the agents may be pre-computed, e.g. at a time before the multi-agent control begins and e.g. by a server. Later, one or more other computer systems (e.g. less computationally powerful computing systems, e.g. mobile devices) may use some or all of the pre-computed action embeddings relating to the actions of a respective one of the participants to control a respective agent, as a downstream control task.

[0065] The parallelization of the downstream tasks means that they may be performed using data-parallel hardware (e.g. GPUs, TPUs). For example, the downstream control tasks for each of a number of agents can be performed in parallel using data-parallel hardware (e.g. devices supporting Single Instruction Multiple Data (SIMD) operations). Examples include GPUs, TPUs and CPUs having support for SIMD instructions sets such as Streaming SIMD Extensions (SSE), Advanced Vector Extensions (AVX), and / or ARM® NEON™ .

[0066] The described techniques can be used in environments where the space of agent actions is very large, e.g., combinatorial. For example each agent may have a large number of possible actions (agent actions) that the agent can perform at each time step. This results in a vast space of possible actions that multiple agents can perform. The described techniques can be used effectively in such action spaces.

[0067] Furthermore, because the described techniques can be implemented using deep neural networks, they can be used in any one or more of the following ways: - as a sub-network for a larger neural network system, e.g. ones trained to drive cars, recommend items, negotiate contracts, etc.. - to generalize across scenarios, e.g. make item recommendations to an individual based on other individuals’ demographics; or estimate payoffs when interacting with other drivers, but also take into account traffic signals. - backpropagation through the trained deep neural network to the parameters of a game is possible, which in other words makes it possible to train the mechanisms of a game (“mechanism design”, i.e. the process of adapting a game to incentivize certain behavior).

[0068] The described techniques are applicable, e.g., to many different types of routing, scheduling, and control tasks, such as routing robots or autonomous vehicles, electrical power, or data packets; scheduling computing jobs; and controlling robots or manufacturing units or machines that interact to manufacture a product.

[0069] In particular the described techniques are applicable to any situation which has a formal equivalence to a normal-form game. Many examples are given below. In theseDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application examples, a suitable decoder network can ask inquiries (questions) relating to the situation, but rephrased as questions about the formally equivalent normal-form game. One such inquiry is “what is the equilibrium strategy in the game”? For example, in a situation described below in which making many AI models available to a user is formally equivalent to corresponding actions of a first participant, the answer to this inquiry indicates what combination of models should be presented to users such that it will be robust on multiple benchmark tasks, e.g. benchmark translation tasks.

[0070] In another example, the inquiry may be designed to answer the question: “how can we understand the different strategies / actions for each player in a game”? For example, if two benchmark translation tasks are similar, in the sense that they disambiguate LLMs in the same way, then actions representing them should have similar respective embeddings.

[0071] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. Brief description of the figures

[0072] Exemplary implementations of the present techniques will now be described with reference to the following drawings, in which: Fig.1, is composed of Fig.1A which illustrates an action embedding system for encoding actions, and Fig.1B which illustrates an action encoder neural network for use in the action encoder system; Fig.2 illustrates a first “per action” decoder network for processing action embeddings; Fig.3 illustrates a second “per joint action” decoder network for processing action embeddings; Fig.4 illustrates a third “per payoff” decoder network for processing action embeddings; Fig.5 shows experimental results from the use of the third decoder network; Fig.6 illustrates a method for generating action embeddings; Fig.7 illustrates a method of controlling at least one participant of a multi- participant interaction;DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application Fig.8 illustrates a method of obtaining data indicative of the reward to one of the participants of performing a first action in a multi-participant interaction; Fig.9 illustrates a method of modifying a multi-participant interaction; Fig.10 illustrates a method selecting a first number of parties to compete with each other in a competition; Fig.11 illustrates a method of allocating a resource, from a plurality of resources, to a given task; and Fig.12 illustrates a method of training an action embedding system including one or more action encoder neural networks to generate corresponding action embeddings for actions which one of the participants in a multi-participant interaction can perform. Detailed description of the embodiments

[0073] Computer implemented methods will now be described which are examples of implementations of the present disclosure. These methods may be implemented by one or more computer systems, in one or more locations.

[0074] The examples employ action embedding systems which, for an action which a participant in a multi-participant game may perform, generate a corresponding embedding of the action.

[0075] Firstly, notation used to describe a multi-participant interaction (NFG game) is introduced. The notation for a game used here assumes that that there are multiple participants (sometimes called “players”, but the word “player” has other uses in this document, as described below), indexed by respective values of an index variable p. The number of participants may be denoted N, and p=1,…N. For simplicity in the following explanation it is sometimes assumed that each of the N participants can make the same number of actions, denoted T, but the present concepts are not limited in that respect. Nor is it required that each participant can make the same T actions as other participants. The set of actions which participant p can make is denoted ^^^ൌ ^ ^^^^ , … , ^^^்^, from which participant p selects a specific action ^^^. For this action it receives a real valued payoff (reward) which is a function of the joint action denoted a performed by all the N participants, i.e. ^^ ൌ^^^^, … . , ^^ே^. a is an element of an action space noted ^^, so the reward function indicating the reward received by player p in the case of the joint action a can be writtenℝ, where ℝ is the set of real numbers.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application

[0076] The joint action a can also be written as ^^ ൌൌis a “joint action set” which denotes the actions of all the participants except p.

[0077] ^^^^^^ൌ ^^൫ ^^^,is a “strategy” which denotes the probability of the participants playing the joint action a. A “pure” strategy is an action distribution ^^^ ^^^ which is deterministic, whereas a “mixed” strategy may be stochastic. In some cases, ^^^ ^^^ may be referred to as a mixed joint strategy.

[0078] The value to participant p under strategy ^^ is given by the expectation value of ^^^, i.e.The term “payoff tensor” refers to the tabulation of ^^^for all values of the action and the participant, and is denoted G. It may thus have N entries for each of the ^^ேpossible joint actions (though as noted below some of those entries may be omitted and thus “unknown”).

[0079] If the mixed joint strategy ^^ factorises into an output product of marginals, i.e. ^^ ൌ⊗^^^^for a set of marginals ^ ^^^^, i.e. an individual strategy ^^^for each respective participant p, participant p’s unilateral deviation incentive is defined as:

[0081] A Nash equilibrium (NE) is a situation where no participant could gain by changing their own strategy. A factorizable mixed joint strategy is said to be ^^ െ ^^ ^^ if and only if ^^^^ ^^^ ൌ ^^ ^^ ^^^^^^^ ^^^ ^ ^^. This quantity is referred to as the NE gap, as it intuitively measures the distance from ^^ to a NE of the game. A mixed-strategy NE is guaranteed to exist for a finite game, but exactly computing a normal-form NE beyond two-participant zero-sum games is PPAD ("Polynomial Parity Arguments on Directed graphs") -complete. In the case that ^^ is deterministic, a is a pure-strategy NE when ^^ ^ ^^^ ൌ 0.

[0082] The implementations of the present disclosure employ an action encoder system, including at least one action encoder neural network. Each action encoder neural network depends on, and is obtained using, the payoff tensor G. The embedding produced for an action ^^^for participant p is denoted ^^^(i.e. in bold font). Thus, the embeddings of all T actions for all N participants is the set ^ ^^^,… , ^^ே^, where ^^^ൌ ^ ^^^^ , … , ^^^்) are the embeddings of the T possible actions for participant p. Accordingly, the embedding f produced by the action encoder neural network may be written as ^^: ^^ → ^ ^^^,… , ^^ே^.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application

[0083] A participant permutation may be denoted ^^: ^^ → ^^′, and a participant action permutation is denoted ^^^: ^^^^→ ^^^^ᇲ. Considering an isomorphism ^^: ^^ → ^^ᇱwith ^^=( ^^^, ^^ ∈^^^^, ^^), an encoder function f is said to be “equivariant” if

[0084] Intuitively, permutation equivariance implies that ^^ and f commute, or shown that for an equivariant embedding function, if ^^^൫ ^^^^, ^^^^൯ ൌ, then ^^^^ ൌ ^^^^ . This guarantees that repeated actions are treated identically in any downstream applications of the embeddings.

[0086] Also, it can be shown that if participant p and participant q are symmetric, and if f is deterministic and equivariant with f(G)=(… , ^^^,… , ^^^, … ^, then ^^^and ^^^are identical up to permutation. This implies that participant symmetry is reflected in the action embedding space.

[0087] Referring to Fig.1A, an example action embedding system is depicted. The action embedding system iteratively updates a set of ^^ ^^ initial action embeddings, respectively for each the T actions for each of the N participants. These initial action embeddings may be set in any way, for example randomly, but in the experimental results presented below, they were zero-initialized, i.e. ^^௧^ ൌ ^^ for all t=1,….T and all p=1,…N. Here each ^^௧^ has D components, i.e. ^^௧^ ∈ ℝ^where D is an integer.

[0088] The initial NT D-dimensional embeddings are updated in each of K iterations by a respective action encoder neural networks (also called NfgTransformer blocks) 111, 112,…,11k,…11K, indexed by an integer variableto give final action embeddings 12. Here K is a positive integer, e.g. greater than 1. Each action encoder neural network also receives data indicative of the payoff tensor G as an input. Thus, after the K iterations, the payoff tensor (which has a dimensionality of NTNsince it indicates the payoff to each of the N participants of the joint action which is a choice from among T actions for each of N participants) has been represented as NT sets of D-dimensional action embeddings. This is typically a considerable compression of the payoff tensor.

[0089] Fig.1B shows the structure of each of the action encoder neural networks 111, 112,…, 11k, …., 11K. The action encoder neural network includes a first self-attention mechanism 14. It also includes a second attention mechanism 16 which is a cross-attentionDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application mechanism. The second attention mechanism 16 generates modified embeddings for each of the NT actions.

[0090] Optionally, the modified embeddings output by the second attention mechanism are successively processed by a stack of a number (denoted A) of further self- attention mechanisms 171, 172, …17A. The outputs of the last of the A further attention mechanisms (if present) form the updated embeddings generated by the action encoder neural network. If the further attention mechanisms are not present, the output of the second attention mechanism 16 may constitute the updated embeddings generated by the action encoder neural network.

[0091] The processing by each action encoder neural network will now be explained in more detail. As noted, the inputs to the action encoder neural networks are NT D- dimensional “current embeddings”, i.e. an embedding for each of the T actions for each of the N participants.

[0092] A first key vector formation unit 13 processes the current embeddings. For each of the TNjoint actions (i.e. each possible action by each participant), the key vector formation unit generates N corresponding key vectors, one for each of the participants. For the p-th participant, the action in the joint action a has a current representation ^^^, and, according to the payoff tensor, the reward for this participant of the joint action is denoted ^^^^ ^^^. The corresponding key vector is a concatenation of ^^^and ^^^^ ^^^. Thus, for a given joint action, the N corresponding key vectors are ^^^௩≡ ^ ^^ ^^ ^^ ^^ ^^ ^^ In aspecific attention mechanism described below, these are used as all of key vectors, value vectors and query vectors. This yields one vector output per action for every joint action.

[0093] The first self-attention mechanism 14 generates an output for a given key vector which can be denotedreferred to as a “play” of action ^^^under the joint action a and the payoff tensor G. The first attention mechanism thus provides “player- to-coplayer self-attention”.

[0094] A second key vector formation unit 15 rearranges the ^^ ൈ ^^ேplays generated by the first self-attention mechanism 14, to form, for each of the possible actions for each of the T actions each of the N participants can perform, a corresponding key vector for each of the possible sets of actions the other players can perform. In other words, for each of the ^^ ^^ ^^ actions ^^^௧a given participant can perform, the second key vector formation unit 15 generates corresponding key vectors ^^ீ൫ ^^^, for all ^^^^.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application

[0095] These provide key-vector and value-vector inputs to the second attention mechanism 16, which also receives the corresponding current action embedding as a query vector Q. The second attention mechanism thus provides “action-to-play cross attention”.

[0096] The self-attention mechanisms 171, 172, ….17A provide “action-to-action self-attention”, in which action embeddings interact with each other directly.

[0097] A description is now given of the operation of a known attention mechanism, which may be used to implement the first and second attention mechanisms 14, 16, and the self-attention mechanisms 171, 172, ….17A.

[0098] In general terms, a given attention mechanism typically has one or more “heads”. Each head of the attention mechanism is typically implemented by an attention neural network layer. Often the attention mechanism is followed by one or more feed- forward neural network layers (not shown) to generate the attention layer output. The outputs of the heads are combined (e.g. summed) to provide the output of the attention mechanism. This enables the attention layer to aggregate multiple streams of information in one forward pass.

[0099] Several sorts of attention mechanism are known, but one which may be employed in the present embodiment applies a QKV (query-key-value) attention operation. These are explained for example in: A. Vaswani, et al., “Attention is all you need”, in Advances in neural information processing systems, 30, 2017; A. Dosovitskiy, et al., “An image is worth 16x16 words: Transformers for image recognition at scale”, in International Conference on Learning Representations, 2021; and A. Jaegle, “Perceiver: General perception with iterative attention”, in International conference on machine learning, pp. 4651–4664. PMLR, 2021. [000100] In summary, each head of a QKV attention operation maps a query vector (“query”), and a set of key vector and value vector pairs (“key-value pairs”) to an output. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function e.g. a dot product or scaled dot product, of the query with the corresponding key. [000101] A self-attention mechanism is a type of attention mechanism in which the queries and key-value pairs are based on the same input vector. In some implementations, a given head of a self-attention mechanism is configured to apply each of a query transformation e.g. defined by a matrix a key transformation e.g. defined by a matrix ^^^, and a value transformation e.g. defined by a matrix ^^^, to the attention layer input forDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application each element denoted by a vector x of the input sequence X (where X is a matrix in which each row is one of the elements x of the sequence; note that the number of rows of X may be limited to a value N+1, so that the output of the attention mechanism for a given input x may only be based on that input x and the N preceding inputs) to derive a respective query matrix (formed of query vectors) ^^ ൌ ^^ ^^ொ, a respective key matrix (formed of key vectors) ^^ ൌ ^^ ^^^, and a respective value matrix (formed of value vectors) ^^ ൌ ^^ ^^^, which are used determine an attended sequence for the output. For example the attention mechanism may be a dot product attention mechanism applied by applying each query vector to each key vector to determine respective weights for each value vector, then combining the value vectors using the respective weights to determine the attention layer output for each element of the input sequence. The attention layer output may be scaled by a scaling factor e.g. by the square root of the dimensions of the queries and keys, to implement scaled dot product attention. Thus, for example, an output of the attention chanism may be determined as ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^me^ ^^√ௗ^V where d is a dimension of the key (and the query and, in some implementations, the value) vector. A summation over the value vectors included in V is assumed here, weighted by the respective values ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^Optionally, a multi-layer perceptron (MLP) may be provided in the attention mechanism to process the ^^ ^^ ^^ ^^ ^^ ^^ ^^and the output of the attention mechanism is the output of the MLP, rather thanitself. [000102] The self-attention mechanism 14 and the self-attention mechanisms 171-17A each apply a self-attention mechanism of this kind. In each case, they receive key vectors, i.e. in the case of the first self-attention mechanism 14, from the first key vector formation unit 13; in the case of the first further self-attention mechanism 171 from the cross-attention mechanism 16; and in the case of each further self-attention mechanism except the first 172, …17A, from the preceding one of the further self-attention mechanisms 171,….17(A-1)), and use those keys vectors also as the value and query vectors. Thus, for the self-attention mechanism 14, the key vectors, value vectors and query vectors are the same at any given time (i.e. each time the attention mechanism is performed to generate an output from the inputs to the self-attention mechanism); at each time, they are a different one of ^ ^^ ^^ ^^ ^^ ^^ ^^ In each self-attention mechanism 171,…17A, the key vectors,DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application value vectors and query vectors are also the same at any given time; i.e. at each time, they are a corresponding one of ^ ^^௧^ ^^∈^ே^,௧∈^்^. [000103] In the case that an attention mechanism is not a self-attention mechanism, the attention mechanism is referred to as a cross-attention mechanism. Thus, each attention layer input x is applied to another data matrix Y having rows composed of corresponding vector elements y. The key matrix and value matrix may be multiplied by Y rather than X, to give the corresponding matrix of key vectors and value vectors. [000104] The unit 16 performs a cross-attention mechanism of this kind. The key- value pairs are the playsoutput by the second key formation unit 16 (each “key-value pair” means a key vector and an identical value vector), and the query vectors are the current action embeddings ^ ^^௧^ ^. The output of the unit 16 is a modified action embedding for each respective current action embedding. [000105] In either case, the attention operation is: 1) order-invariant with respect to the key-value pairs; and 2) equivariant with respect to the query vectors. These properties are desirable in the embodiment of Fig.1 to achieve the permutation equivariance property discussed above. [000106] Note that the zero-initialization optionally employed in the present implementation is in contrast to that usually employed in QKV architectures (e.g. A. Jaegle, op. cit.), in which a set of latent positional embeddings are randomly initialized and optimized along with the rest of the network, allowing the model to take advantage of spatial structure inherent in input data. This is crucial for data modalities such as images and text. In the present implementation, however, it is desirable to avoid any specialization based on the position information of an action or a participant. Instead, in the present example, the action embedding is driven by data, based solely on its outcomes in a strategic interaction with other actions. [000107] It will be understood that the attention mechanism performed by the self- attention mechanism 14, the cross-attention mechanism 16 and the self-attention mechanisms 171-17A, are defined (for each head, if there is more than one) by the respective matrices K, V and Q, and by any other neural network (not shown, but as mentioned this may be an MLP) which is present at the input or output of the attention mechanism. As explained below, these may be trained jointly with one or more decoder networks (sometimes called just “detectors”) which are also neural networks. Here “jointDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application training” refers to a training process in which updates to multiple neural networks are substantially simultaneous or interleaved. [000108] Many variations in the mechanisms illustrated in Figs.1A and 1B are possible within the scope of the present disclosure. For example, if the payoff tensor is only available for a subset of all joint actions, only that subset of the joint actions may be employed. [000109] Before turning to a discussion of how the action embeddings generated by the system of Fig.1A are used, a discussion is given here of how, in a variation, the scheme above may be varied in two particular cases in which certain joint actions are “invalid” (e.g. the payoff tensor does not include a payoff value for them). [000110] In the first case, the payoff tensor is incomplete (in other words, there are some joint actions for which the payoff / reward is unknown). For example, the costs of evaluating outcomes for every joint-action may be high, e.g. in the case in which each evaluation involves human studies. An example is ranking human individuals at a competitive task, since the joint-actions are evaluated by organizing competitions. In the case that only the results of a subset of joint actions are available, it may nevertheless be desirable to accurately predict the outcomes of unobserved ones. [000111] In the second case, the games are “restricted” in the sense that during the training which produced the training data the participants may have different numbers of available actions from how the actions they have in the situation which is to be predicted. [000112] These two cases may be implemented in the scheme of Fig.1 by “masking out” invalid joint-actions from the sets of key vectors attend to in each attention mechanism, so that they have no influence on the learned representations. The masking operation amounts to enforcing invalid query-key pairs to have zero support (e.g. following the softmax operation discussed below). Experimental results given below demonstrate that the architecture of Fig.1A is highly effective for representing incomplete games and predicting payoffs for unobserved joint actions. [000113] There is now a discussion of how the action embeddings generated by the system of Fig.1A for a given NFG defined by a payoff tensor G, are used by decoder networks to obtain information about the given NFG. [000114] For each decoder network, experimental results are summarized below in relation to “synthetic games” of varying sizes, for example for deriving Nash Equilibria (NE) and deviation gain estimation. The “synthetic games” are drawn from a distribution of games. (As described below, the action embedding systems and decoder networks are alsoDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application jointly trained, using such synthetic games.) To generate synthetic games with broad coverage, the technique is used which is explained in L. Marris, et al., “Turbocharging solution concepts: Solving NEs, CEs and CCEs with neural equilibrium solvers”, Advances in Neural Information Processing Systems, 35:5586–5600, 2022. Each game’s payoff tensor G has zero-mean over other participant strategies and Frobenius norm ‖ ^^‖ிൌ √ ^^ே. The results are compared to baseline MLP networks with a number of parameters which is at least 4.95M parameters, which was the number of parameters in the largest implementation of an embodiment of the present disclosure used in the present experiment. Another baseline was an equivarant MLP network that preprocesses input payoff tensors such that actions for each participant are re-arranged in descending order of average payoffs with its outputs reversed accordingly. Unlike the conventional MLP models, the number of parameters used by the embodiments of the current disclosure was identical for all game sizes. [000115] Fig.2 shows a first form of decoder network: a “per action” decoder network. The per action decoder network receives, at a given time, the action embedding (indicated as 21) for a respective action t of a respective participant p, and generates a corresponding output. Thus, at different times, the per action decoder network can generate respective outputs for each action of each participate, from corresponding action embeddings[000116] The per action decoder network may, for example, be implemented as a multi-layer perceptron, MLP, 22. Upon processing an action embedding for a certain action of a certain participant, the MLP 22 generates respective data ^^^௧ൌ ^^ ^^ ^^൫ ^^௧^ ൯ (indicated as 23) which is a logit for an actionIn this way, respective data 23 is generated for all NxT actions that the N participants can perform. The MLP 22 may have multiple outputs, e.g. T outputs, with each corresponding to one of the actions. Whether this is preferable may depend on the downstream application. Alternatively, it may have a single output, e.g. a single scalar value ^^^௧. This ensures, advantageously, that the ordering of the actions for each participant does not matter. [000117] The result can be used by a unit 24 to derive a strategy ^^^^for each of the N participants. For example, for each of N participants, the unit 24 may perform a determination of a marginal action distribution ^^^^ൌ ^^ ^^ ^^ ^^ ^^ ^^ ^^൫ ^^^^, … , ^^^்൯. This defines a strategy for participant p to use in the game which the decoder network predicts is a Nash equilibrium.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application [000118] The action embedding system of Fig.1A and the decoder network of Fig.2 may be trained jointly. This may be done by supervised learning by iteratively updating the parameters of the system of Fig.1A and the decoder network of Fig.2 with simultaneous or interleaved updates to minimize a loss function, using training data from a set of synthetic games for which the NE is known. According to the technique of Z. Duan, et al., “Is Nash equilibrium approximator learnable? In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems”, pp.233–241, 2023, in the experiments the loss function used in the training is ^^ ^^ ^^^^^^^ ^^^^ which is a NE gap, is minimized. Here the joint action is ^^^ ൌ⊗^^^^^(the outer product of all the marginals) and, as mentioned above, ^^^^ ^^^^ ൌ ^^ ^^ ^^^^ᇲ∈ ^^^^^^~ఙ^^ ^^^൫ ^^^ᇱ, ^^^^൯ െ ^^^^ ^^^൧. During the training a loss function calculation unit 25 is provided to calculate ^^ ^^ ^^^^^^^ ^^^^, and the training can be performed by iteratively varying numerical parameters defining the MLP 22 to reduce the ^^ ^^ ^^^^^^^ ^^^^ generated by the loss function calculation unit 25. [000119] Following the training, the action embedding system of Fig.1 can then be used to generate action embeddings for a new game defined by a payoff tensor G, and the decoder network of Fig.2 (without the loss function calculation unit 25) can be used to process the results to estimate a strategy which is a NE. [000120] Experiments were performed above averaged across 5 independent runs, for each of the (N, T) possibilities (2, 16), (2, 64), (3, 8) and (3, 16), using the three baseline algorithms (EquivariantMLP, MLP (see Duan et al, 2023) and NES (see Marris et al, 2022)), and also using implementations of the system of Fig.1A in which the hyper- parameters (D, K, A) took various possible values: (32, 2, 1), (32, 4, 1), (32,8,1), (64, 8, 0), (64, 8, 0), (64, 8, 2), (128, 2, 1), (128, 4, 1), (128, 4, 1) and (128, 8, 1). EquivariantMLP outperformed MLP, but remained ineffective at solving the task. NES significantly outperforms both MLP variants in a two-participant setting but trails behind in 3-participant games. The implementation of Fig.1A is also equivariant by construction but learns to capture relevant information without handcrafted payoff feature vectors, and all variants tested, most with fewer parameters than the baselines, significantly outperformed all the three baselines across game sizes with near-zero NE gap. [000121] Increasing the number K of action encoder neural networks 111, …, 111K improves performance, especially as the game becomes larger. This makes intuitive sense, as it added to the number of times the actions embeddings can be usefully refined. In contrast, the benefit of increasing the embedding size D is muted. In the case of A=0 (i.e.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application each action encoder neural network does not include any action-to-action self-attention), the performance is not as good. A=2 improved on A=1 in more complex games, but not in smaller games. This suggests that action-to-action self-attention facilitates learning, especially in tasks that require iterative reasoning. [000122] Figure 3 shows an example of a second form of decoder network: a “per joint action” decoder network. The per joint action decoder network receives, at a given time, the action embeddings ^^௧^ (indicated as 31) generated by the system of Fig.1A for a respective action t of each participant p for a joint action a. For each of the ^^ேjoint-actions a, a summation unit 32 calculates the respective sum∑^^^௧^ . Each sum is input to a MLP 33, togenerate for each joint action a corresponding value^^^ ൌ ^^ ^^ ^^ ^∑ ^^ ^^^^^^^ indicated as 34. Thisis the estimate of the maximum deviation gain for the corresponding joint action. [000123] During the training, the action embedding system of Fig.1A and the per-joint action decoder network of Figure 3 may be trained jointly, e.g. by supervised learning, based on a database of data about synthetic games which provide training examples, to collectively estimate the “maximum deviation gain”, i.e. the maximum incentive for any participant to deviate from a joint action, such as a (deterministic) joint pure-strategy ^^. This quantity is informative about the stability of a joint behavior, and in particular whether a joint pure-strategy ^^ has a maximum deviation gain ^^ ^ ^^^ of zero. If so, then by definition, the joint strategy is a pure-strategy NE. Specifically, the systems of Fig.1A and Fig.3 are jointly trained (e.g. by gradient descent) to regress towards the maximum deviation-gain ^^ ^ ^^^ for every joint pure-strategy ^^ for the synthetic games in the training database, such asby minimizing a loss function such as ^^ଶ^ ^^^^^^^, ^^ ^^ ^^^ ^^^^ ^^^^ where ^^^^^^^ൌ^^ ^^ ^^^^ᇲ∈ ^^^^ ^^^൫ ^^^ᇱ,The loss function is calculated by a loss function calculation unit 35. [000124] Following the training, the action embedding system of Fig.1A can then be used to generate action embeddings for a new game defined by a payoff tensor G, and the decoder network of Fig.3 (without the loss function calculation unit 35) can be used to process the results to estimate the maximum deviation gain. [000125] Experimentally, when using games in which components of the payoff tensor are sampled from an equilibrium-invariant subspace at different game sizes (see Marris et al., 2023), it was found that implementations of the present disclosure significantly outperformed the baselines. The number K of transformer blocks played a role in the final performance, but a more important parameter was the dimensionality D of each actionDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application embedding. Variants with high D (e.g. D=128) were competitive even for comparatively shallow models (e.g. K as low as 4). This can be explained by the lack of structure in the underlying game: payoffs of one joint-action do not provide any information about outcomes of another. [000126] Fig.4 shows an example of a third form of decoder network: a “per payoff value” decoder network, which may be used for payoff reconstruction in the case that the payoff tensor is incomplete (i.e. does not specify the reward for some joint actions). In the case of games having an underlying structure (i.e. a correlation between components of the payoff tensor G), a per payoff value decoder network can be used to estimate payoff values for each joint action based on a (proper) subset of the joint actions. The per payoff value decoder network receives, at a given time, the action embeddings ^^௧^ (indicated as 41) generated by the system of Fig.1A for a respective action t of each participant p, collectively forming a joint action for which the per payoff decoder network generates corresponding payoff data for each given participant. [000127] A self-attention unit 42 attends to the set of action embeddings 41 of a given joint action at any corresponding time, to generate for one or more (or all) of the possible^^ேjoint-actions a, an estimate ^^^^ (indicated as 43) of the corresponding payoff values(reward) to participant p of each joint action. This may include joint actions for which payoff values which were unavailable in producing the action embeddings 41. The self- attention unit 42 is similar to the action-to-joint action self-attention units 14 of Fig.1B, but does not include appending the payoff values to the action embeddings, since these are to be reconstructed. Note that, as explained above, a self-attention mechanism, such as the self-attention mechanism 42, may include a MLP perceptron at its output, so that ^^^^ may be theoutput of that MLP. An alternative but equivalent representation of this diagram would be to show the later layer(s) of the MLP as a separate MLP unit, which is trained jointly with the self-attention mechanism 42. [000128] During the training the action embedding system of Fig.1A and the per- payoff decoder network of Figure 4 may be trained jointly, by supervised learning based on a database of data about synthetic games for which the full payoff tensor is available. Specifically, the systems of Fig.1A and Fig.4 may be jointly trained to minimize a lossfunction such as ^^ଶ^A loss function calculation unit 44 may be provided tocalculate this loss.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application [000129] Following the training, the action embedding system of Fig.1A can then be used to generate action embeddings for actions in a new game defined by an (incomplete) payoff tensor G in the manner explained above, and the decoder network of Fig.3 (withouth the loss function calculation unit 44) can be used to process the results to estimate the missing values 43 of the payoff tensor G. [000130] Experiments were conducted using a payoff tensor which was structured, specifically DISC games (D. Balduzzi, et al., “Open-ended learning in symmetric zero-sum games”, in International Conference on Machine Learning, pp.434–443. PMLR, 2019) which are a class of structured, symmetric two-participant zero-sum games having a complexity described by a parameter Z, which describes the dimensionality of two (unknown) latent vectors. In these games, payoffs under one joint-action are informative about the outcomes of others. [000131] Specifically, experiments were performed in which implementations of the present disclosure were used in cases that the payoff tensor G had valid “observed” values for a random proportion p of possible joint actions. The goal of the implementation was to predict the outcome for each participant under the other “unobserved” joint-actions. A rich body of literature has been dedicated to solving this problem since is relevant to real-life competitions, e.g. in which certain pairs of a set of players have competed one or more times and the results are known, but other pairs of players have not. Known algorithms for rating players in these circumstances are Elo ratings (A.E. Elo. The Rating of Chessplayers: Past and Present. Ishi Press International, 2008), mElo ratings (D. Balduzzi, et al., “Re- evaluating evaluation”. Advances in Neural Information Processing Systems, 31, 2018) and xElo (Q. Bertrand, et al, “On the limitations of the Elo, real-world games are transitive, not additive”, in International Conference on Artificial Intelligence and Statistics, pp.2905– 2921. PMLR, 2023). [000132] Figure 5 shows experimental results from an implementation of the present disclosure (the result marked “ours”) and from these three baselines (Elo, mElo, xElo). The vertical lines show the payoff prediction error averaged over all the players across the unobserved joint-actions. The results were sampled over 32 randomly sampled empirical DISC games in each game configuration. The implementation of the present disclosure outperforms Elo and its variants significantly across all settings. In particular, the implementation predicted the latent variable game (i.e. DISC games with Z=1) near perfectly as soon as 10% of joint actions were observed, with an error an order of magnitude lower than the second best competing method. This is remarkable as the baseline methodsDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application were partly inspired by the construction of DISC games, and the implementation of the present disclosure was not. [000133] As noted above, another sort of decoder network is a “per game” decoder network, which may receive the action embeddings for all possible actions by all participants, and provide information about the game as a whole. Similar to a joint-action decoder network of Fig.3, a per game decoder network may comprise a summation unit (or concatenation unit) to sum the received vectors, followed by a feedforward network, such as a multi-layer perceptron (MLP). Its training and usage are analogous to the decoder networks mentioned above. [000134] The described techniques are applicable, e.g., to many different types of routing, scheduling, and control tasks, such as routing robots or autonomous vehicles, electrical power, or data packets; scheduling computing jobs; and controlling robots or manufacturing units or machines that interact to manufacture a product. [000135] In particular the described techniques are applicable to any situation which has a formal equivalence to a normal-form game. Many examples are given below. In all these examples, a suitable decoder network can ask questions relating to the situation, but rephrased as questions about the formally equivalent normal-form game. For example, “what is the equilibrium strategy in the game”? For example, in a situation described below in which making many AI models available to a user is formally equivalent to corresponding actions of a first participant, what combination of models should be presented to users such that is will be robust on multiple benchmark tasks, e.g. benchmark translation tasks? Or “how can we understand the different strategies / actions for each player in a game”? For example, if two benchmark translation tasks are similar, in the sense that they disambiguate LLMs in the same way, then actions representing them should have similar respective embeddings. [000136] In some implementations, some or all of the agents described above may be animate (e.g. human) agents, having different capabilities. For example, they may be technicians having different skills, who cooperate to perform technical task(s). Using the present techniques the agents (e.g. comprising a mixture of human agents and inanimate agents) may be allotted to corresponding tasks, or components of the tasks (e.g. although only the performance of the human agents on other tasks has been measured). This makes it possible, for example, for the inanimate agents to provide the maximum contribution to the performance of the technical task(s). In other applications, however, all the agents areDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application inanimate agents, e.g. electro-mechanical apparatus or computer models, as described in the following. [000137] In some implementations the environment is a real-world environment, and each of the agents comprises a robot or autonomous or semi-autonomous land or air or sea vehicle. The tasks performed by the agent or agents may then each comprise manipulation of objects in the environment and / or navigating a path through the environment. The agent actions may comprise agent actions controlling movements of the agents in the environment, e.g., agent actions controlling a speed or direction of movement, e.g., steering, of an agent, or agent actions performed by the robot or autonomous vehicle such as collecting or delivering a payload such as cargo or passengers. The rewards may relate to a metric of performance of the task, such as a time or delay, or a distance to reach a destination (e.g., from the location of the agent) and / or to a physical cost of performing the task such as a fuel or energy use to reach a destination. For example the reward may be include rewards received for moving along a path, e.g., within a particular time or dependent on a speed of motion of the agent or for less than a particular energy use by the agent. The state of the environment, which includes the agents, may be characterized by state representations that are derived from observations of the environment using any type of sensor, e.g., an image or video sensor (which here includes a LIDAR sensor), or sensors of the agent state or configuration, or sensors of the environment such as sensors of positions of one or more other agents, of average speed of agent movement, a congestion sensor, sensors of the locations of static or dynamic objects such as trees, buildings, pedestrians, or traffic lights, or map data, e.g., defining possible routes an agent may take. [000138] As an example, tasks performed by the agents may comprise delivery of a payload such as cargo, e.g., one or more packages, or passengers. For example the agents may cooperate to deliver the payload to multiple different destinations in an energy-efficient manner and / or avoiding mutual congestion in transit or on arrival (e.g., by defining a minimum time interval between deliveries to a particular destination). The present techniques may be used to determine which agent is to deliver which item and / or to which destination. [000139] As one particular example the agents may comprise robots or vehicles performing a task such as warehouse, logistics, or factory automation, e.g., collecting, placing, or moving stored goods or goods or parts of goods during their manufacture. [000140] As another example the agents may comprise electric taxis and the agents may determine collective behavior that enables the taxis to cooperate in collection andDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application delivery of passengers in a time- and energy-efficient manner, e.g., taking account of available range. [000141] As another example the agents may comprise cars or trucks and the agents may determine collective behavior that enable the vehicles to cooperate when moving on a road, e.g., to facilitate smooth and safe traffic flow by defining the rewards that aim to maintain a maximum, minimum or desired distance between the vehicles. [000142] In an example, autonomous driving may be provided based on the formally equivalent game in which, for each of a series of times (e.g. at successive road intersections) a driver has multiple agent actions available, e.g. go forwards, yield or wait, each with respective outcomes. Multiple participants may correspond to multiple respective drivers on the road. Each given participant may receive a reward based on the combination of agent actions of all the drivers (e.g. a higher reward if, for this combination of agent actions, the given participant reaches the destination more quickly, and / or with less fuel consumption; and a lower reward if the combination of agent actions causes an accident, or uses more fuel, or violates traffic rules). The reward function depends on the specific road network. Using a per action decoder network as described above, it can be determined for each agent action by each driver (participant), whether a given action is close to an equilibrium, i.e. is the corresponding one of the combination of actions by all the corresponding participants which is (at least locally) optimal. If so, all the vehicles can be controlled to perform these agent actions. [000143] This situation can be generalized to provide a system to control any one or more agents (e.g. robots) in an e.g. real-world environment, to provide an (at least locally) optimal collective performance of a task which is translated into a reward function, e.g. encoding success at the task, and / or undesirable events (e.g. collisions; high power consumption; or behaviors which cause wear). [000144] As another example the vehicles may be airborne vehicles such as drones. In an agricultural setting, for example, the task(s) may then comprise delivery of payloads such as pesticide, seed, water, or fertilizer to plants. In an emergency response setting the tasks may comprise searching for a person. In another setting the tasks may comprise package delivery tasks. [000145] In some implementations, the described technology may be used to manage large numbers of robots or autonomous or semi-autonomous vehicles operating on land, in the air, or on water. The agent actions may be low-level, intermediate-level, or high-level actions. Examples of high level actions include agent actions that specify locations that aDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application robot or vehicle should move to or avoid; examples of intermediate level actions include agent actions that specify which robot or vehicle is to move first, or has priority, at a junction or intersection of routes. The rewards defining the expected return may be based on an event or an aspect of the environment. For example, a reward may indicate whether an agent has accomplished a task as described above, or the progress of the agent towards accomplishing the task (the particular rewards correspond to the above tasks and are therefore not repeated for conciseness). [000146] In some implementations the environment is an environment is a computing environment, and each of the agents comprises a computing job scheduling agent, e.g., a software agent that manages distribution of compute tasks across computing resources, e.g., on a mobile device or in a computer or on a computer cluster, or in a data center. The computing resources may comprise various hardware resources. The resources may comprise different low level resources such as processors with different capabilities, e.g., one or more general purpose processors (CPUs), or GPUs (Graphics Processing Units), or TPUs (Tensor Processing Units), or different intermediate level resources such as different computers that work together in a computer cluster, or different higher level resources such as servers in a data center. The hardware resources may, but need not have, different computational or memory capabilities. The tasks performed by the agents may comprise executing computer programs. The agent actions may comprise actions allocating one or more computing jobs associated with the tasks to one or more computing hardware resources. The rewards may relate to a metric of performance of a task such as a time to complete a task, or the computational cost of performing the task (e.g., where this depends on the hardware on which it is implemented), or the power or energy cost of performing a task, or the reliability of performing a task (e.g., where a task is repeated for fault- tolerance). The state of the environment may be characterized by state representations that are derived from observations of, e.g., a computational workload on a hardware resource, a physical state, such as temperature, of a hardware resource, or whether a hardware resource is exhibiting a fault. A simulator may simulate the effects of the agent actions in the environment, e.g., the effects of allocating particular computing jobs to particular resources. [000147] In an implementation of the present techniques, software agents may be allotted to corresponding tasks based on partial information about the suitability of the software agents to perform those tasks. For example, if multiple software agents (e.g. large language models, LLMs) are available to perform a certain computing task, and experiments have been performed to measure how each of the software agents performs for a respectiveDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application subset of the computing tasks, the present technique permits a rational selection of a corresponding one of the software agents for each task, e.g. selecting for a certain task an LLM or other software agent, which, out of those of the software agents which are capable of performing the task to an acceptable level, has the lowest computational overheads. [000148] In an example, the present techniques may be used for ranking AI models (LLMs, visual language models (VLMs), etc.) given a suite of benchmark tasks (e.g. question-and-answer (Q&A), summarization, translation, captioning, etc.). For example, a first participant (player) may choose an AI model. A second participant may choose one of the benchmarks. The reward (payoff) received by the first participant may be the performance of the chosen AI model on the chosen benchmark. The reward to the second participant may the negated payoff of the first participant. If the present techniques are used to derive a predicted strategy for the first participant, the predicted strategy may be to find an AI model which performs well on all the benchmarks, which may suggest that it is suitable for other applications also. [000149] One particular challenge to LLMs is presenting information to the public on matters of grave concern (e.g. climate change). Using LLMs to address serious information needs raises safety concerns, due to their limitations in analysing factual information. In dialogues with users, a LLM which provides fluent, grammatical responses and advanced linguistic dialogue behaviors is trusted by users, making evaluation of LLMs by non-human evaluators treacherous. Research on how to evaluate systems that may achieve or exceed human abilities (“scalable oversight”) has also been largely theoretical. The present techniques can be used to overcome both these problems, by analysing NFGs which are formally equivalent to the required LLM evaluation. For example, at least one participant may perform actions which correspond to choices of LLMs (e.g. choices for trade-offs between the parameters of the LLM, subject to computational constraints), with another participant choosing actions corresponding to questions which are difficult to answer, and rewards being defined by human raters with relevant educational background. [000150] In another example, each Blockchain contract / protocol is formally equivalent to a NFG, since is defines the payoffs to each participant of the contract, i.e. the rewards to each participant. Thus, each of multiple parties agreeing to a Blockchain (or other) contract may use the present techniques to choose (e.g. using a per action decoder network as described above) an action (a set of stipulated contractual conditions) which gives all participants an (at least locally) optimal payoff.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application [000151] This (and the multi-driver / multi-robot situation described above) are examples of using the present techniques to perform a negotiation to select a collective action which formally equivalent to a Nash Equilibrium of a corresponding game. The board game “Diplomacy” ® is a representative simplification of real world negotiations, and it has been demonstrated that this can be solved by providing a learning signal to a neural network policy in the form of NFG equilibria in a decision process at each stage of the game (e.g. to determine whether at a given stage of a game a player should move troops to a certain location). The present techniques may generalize this to more realistic models of multi-participant interaction, and provide, with very modest computational cost, not just a target equilibrium but a more nuanced target, e.g. an equilibrium which maximizes total welfare of all players. This allows learning a policy involving additional, more human-like criteria. [000152] In some implementations the environment is a packet communications network environment and the agents each comprise a router to route packets of data over the communications network. The tasks performed by the agents may then comprise packet routing tasks. The agent actions may comprise routing actions to route the packets of data, e.g., actions to determine a next or further destination for one or more data packets. The rewards may relate to one or more packet routing metrics, e.g., relating to a path from the router to a next or further node in the data packet communications network. Such metrics may include, e.g., an estimated time, delay, speed, distance, or hop count for a group of one or more routed data packets to travel from the router to the next or further node in the data packet communications network, a measurement of route utilization, a route packet loss or other reliability measure, a route bandwidth or throughput, or a route maximum transmission unit (MTU). The state of the environment may be characterized by state representations that are derived from observations of available data packet routes and their characteristics, e.g., characteristics corresponding to the aforementioned metrics. For example the observations may include observations of a routing table of the router. A simulator may simulate the effects of different choices of route. [000153] In some implementations the environment is an electrical power distribution environment, and the agents are each configured to control routing of electrical power from an electrical power distribution node, e.g., a node associated with the agent, to one or more other nodes over one or more power distribution links, e.g., in a “smart grid”. [000154] As electrical power distribution grids become more decentralized, for example because of the addition multiple smaller capacity, and potentially intermittentDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application renewable power generators, the additional interconnections amongst the power generators and consumers can destabilize the grid and can cause overload of a link in the grid, particularly because of phase differences between connected points. [000155] The tasks performed by the agents comprise tasks that distribute power from a power generator to power consumers. The agent actions comprise control actions to control the routing of electrical power between the nodes. For example the agent actions may comprise actions to connect or disconnect sources or sinks of electrical power to the grid, or actions controlling the routing of electrical power over links between nodes of the grid, or actions controlling a frequency or phase of an electrical power generator, or actions controlling a frequency mismatch or phase mismatch between elements of the grid, e.g., connected via a node. The rewards may relate to one or more electrical power routing metrics, e.g., relating to an electrical power loss within the grid, or to a frequency or phase mismatch over one or more of the power distribution links, or to a risk of overloading or failure or to a risk of exceeding a threshold on one or more of the power distribution links. [000156] The state of the environment may be characterized by state representations that are derived from observations from sensors sensing a state of the power grid, e.g., sensing frequency, phase, current, voltage, electrical loading, resistance or impedance associated with one or more connections between the nodes, or static observations of a configuration of the power grid, e.g., of elements coupled to the grid and power distribution links. A simulator may simulate the effects of different choices of electrical power control or routing actions. [000157] In some implementations the environment is a real-world manufacturing environment for manufacturing a product, such as a chemical, biological, or mechanical product, or a food product. As used herein a “manufacturing” a product also includes refining a starting material to create a product, or treating a starting material, e.g., to remove pollutants, to generate a cleaned or recycled product. The manufacturing plant may comprise a plurality of manufacturing units such as vessels for chemical or biological substances, or machines, e.g., robots, for processing solid or other materials. The manufacturing units are configured such that an intermediate version or component of the product is moveable between the manufacturing units during manufacture of the product, e.g., via pipes or mechanical conveyance. [000158] In implementations each of the agents comprises a control system configured to control manufacture of a mechanical, chemical, or biological product. For example the control system may be configured to control one or more of the manufacturing units orDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application machines or to control movement of an intermediate version or component of the product between the manufacturing units or machines. In general the manufacturing units or machines interact to manufacture the product. The task(s) performed by the agent or agents may comprise a task to manufacture the or another mechanical, chemical, or biological product or intermediate or component thereof. That is the agents may interact to control manufacture of the same product, or they may control the manufacture of different products. As used herein manufacture of a product includes manufacture of a food product by one or more kitchen robots. [000159] The agent actions may comprise control actions to control the use of a machine or a manufacturing unit for processing a solid or liquid material to manufacture the product or intermediate or component thereof, or to control movement of an intermediate version or component of the product within the manufacturing environment, e.g., between the manufacturing units or machines. In general the agent actions may be any actions that have an effect on the observed state of the environment, i.e., actions configured to adjust any of the sensed parameters described below. These may include agent actions to adjust the physical or chemical conditions of a manufacturing unit, or agent actions to control the movement of mechanical parts of a machine or joints of a robot. [000160] The state of the environment may be characterized by state representations that are derived from observations from sensors sensing a state of the manufacturing environment, e.g., sensors sensing a state or configuration of the manufacturing units or machines, or sensors sensing movement of material between the manufacturing units or machines. As some examples such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow / movement rate or flow / movement path of one or more materials; physical or chemical conditions, e.g., a physical state, shape or configuration or a chemical state such as pH; configurations of the units or machines such as the mechanical configuration of a unit or machine, or valve configurations; image or video sensors to capture image or video observations of the manufacturing units or of the machines or movement; or any other appropriate type of sensor. In the case of a machine such as a robot the observations from the sensors may include observations of position, linear or angular velocity, force, torque or acceleration, or pose of one or more parts of the machine, e.g., data characterizing the current state of the machine or robot or of an item held or processed by the machine or robot. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal, or image or videoDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application data for example from a camera or a LIDAR sensor. Sensors such as these may be part of or located separately from the agent in the environment. A simulator may simulate the effects of the different choices of the aforementioned agent actions in the manufacturing environment. [000161] The agents may further comprise static or mobile software agent(s), e.g. computer programs configured to operate autonomously and / or with other software agents or people to perform a task. For example the environment may be an integrated circuit routing environment and each agent of the group of agents may be configured to perform a routing task for routing interconnection lines of an integrated circuit such as an ASIC. The first or each machine learning, e.g. reinforcement learning, objective-defining value may be dependent on one or more routing metrics such as an interconnect resistance, capacitance, impedance, loss, speed or propagation delay, physical line parameters such as width, thickness or geometry, and design rules. The objectives may include one or more objectives relating to a global property of the routed circuitry e.g. component density, operating speed, power consumption, material usage, or a cooling requirement. The agent actions may comprise component placing actions e.g. to define a component position or orientation and / or interconnect routing actions e.g. interconnect selection and / or placement actions. [000162] Some other applications relate to making a personalized recommendation for a user. Suppose it is desired to recommend items (e.g. movies) suitable for one or more individuals, based on data characterizing the items the individuals have used (e.g. watched) in the past. This may be formulated as a game as follows. A first participant chooses an item (e.g. a movie from a database). A second participate chooses a second item (e.g. another movie from a database). The game randomly picks an individual (or a demographic) and asks the individual to rate the two items. The first participant wins / loses if the individual gives a higher ranting to its chosen item than to the second participant’s. [000163] As a further example, the agent actions may include presenting advertisements, the observations may include advertisement impressions or a click-through count or rate, and the reward may characterize previous selections of items or content taken by one or more users. [000164] As another example, the agents may operate in a simulated environment which is a chemical synthesis or protein folding environment such that each state is a respective state of a protein chain or of one or more intermediates or precursor chemicals. The agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, the agent actions are possible folding actions for folding theDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application protein chain or actions for assembling precursor chemicals / intermediates and the result to be achieved may include, e.g., folding the protein so that the protein is stable and so that it achieves a particular biological function or providing a valid synthetic route for the chemical. As another example, the agent may be a mechanical agent that performs or controls the protein folding actions or chemical synthesis steps selected by the system automatically without human interaction. The observations may comprise direct or indirect observations of a state of the protein or chemical / intermediates / precursors and / or may be derived from simulation. [000165] In a similar way the environment may be a drug design environment such that each state is a respective state of a potential pharmaceutically active compound pharmaceutically active compound and the agent is a computer system for determining elements of the pharmaceutically active compound and / or a synthetic pathway for the pharmaceutically active compound. The drug / synthesis may be designed based on a reward derived from a target for the pharmaceutically active compound, for example in simulation. As another example, the agent may be a mechanical agent that performs or controls synthesis of the pharmaceutically active compound. [000166] Note that in some implementations, certain of the agents may have a common purpose, e.g. a single entity (or multiple cooperating entities), and other agents represent parties which are antagonistic to the first agents. For example, the first agents may be used to perform security operations to secure a computing environment or a physical facility, and the other agents may be controlled by parties which wish to compromise the security. In this case, the present techniques may be used to control the first agents (or the other agents) and / or to allot resources to computational tasks which achieve the common purpose, or which are antagonistic to it. The present techniques, and in particular selected decoder networks, may be used to understand the behavior of either group of agents, and / or to predict (e.g. locally) optimal actions for either group of agents. [000167] Fig.6 is a flow diagram of an example process 60 for obtaining action embeddings for actions which one of more participants of a multi-participant interaction can perform. Each of a plurality of participants is able to perform one of a respective set of actions and each participant receives a reward defined by a respective reward function of the corresponding actions performed by the plurality of participants. The process 60 can be performed by a system of one or more computers located in one or more locations. For example, the action embedding system of FIG.1A (e.g. comprising one of more of theDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application action encoder neural networks of Fig.1B), appropriately programmed in accordance with this specification, can perform the process 60. [000168] In a first step 61, a respective initial embedding is defined for one or more of the corresponding sets of actions one or more of the participants can take. For example, the initial embedding may be selected at random or set to a default value such as all zeros. A dataset is defined comprising the initial embeddings, such as the dataset 10 of Fig.1A. [000169] In step 62, the dataset is processed successively by a sequence of one or more (in fact, a number K where K is an integer) action encoder neural networks, such as the action encoder neural networks 111, 112, .., 11K of Fig.1A. The first action encoder neural network of the sequence processes the dataset defined in step 61, and each of the subsequent ones of the action encoder neural networks (if any) processes the output of the preceding action encoder neural network of the sequence. In one implementation, multiple ones of the action encoder neural networks may be implemented by a single software routine, or a single portion of an integrated circuit, which is employed multiple time steps, and in each time except the first time step (when it processes the dataset defined in step 61) processes its output in the preceding time step. In step 62, a set of final action embeddings may be generated, one for each respective one of the initial embeddings, such as the final embeddings 12 of Fig.1A. [000170] Fig.7 illustrates a flow diagram of an example process 70 for controlling at least one participant of a multi-participant interaction. Each of a plurality of participants are able to perform one of a respective set of actions and each participant receives a reward defined by a respective reward function of the corresponding actions performed by the plurality of participants. The process 70 can be performed by a system of one or more computers located in one or more locations. [000171] In step 71, an action embedding is obtained for one or more of the corresponding set of actions which the at least one participant can perform. For example, the action embedding system of FIG.1A (e.g. comprising one of more of the action encoder neural networks of Fig.1B), appropriately programmed in accordance with this specification, can perform step 71, for example by the method 60 explained above with reference to Fig.6. [000172] In step 72, the action embeddings are input to a decoder network to obtain corresponding reward information for each of the corresponding set of actions which the at least one participant can perform. In an example, step 72 may be carried out using the decoder network of Fig.2. The reward information may describe a strategy for performingDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application actions for each of the participants which provides rewards which are at a Nash equilibrium. For example, according to the strategy each participant may perform an action for which the reward information indicates that a corresponding predicted reward for the participant is greatest (i.e. greatest out of all the actions the participant can perform). [000173] In step 73, the at least one participant is controlled to perform one or more of the actions based on the corresponding reward information for the set of actions (i.e. one or more actions according to the strategy). For example, each participant may be controlled to perform actions according to a strategy specified by the reward information, e.g. so that the participants as a whole act according to the Nash equilibrium. [000174] Examples are given above of technical situations in which the method 70 may be used, e.g. ones in which the participants correspond to robots or autonomous vehicles, performing manipulation or navigation tasks. [000175] Fig.8 illustrates a flow diagram of a process 80 of obtaining data indicative of the reward to one of the participants of performing a first action in a multi-participant interaction, given that the other participant(s) perform corresponding second actions. [000176] In step 81, an action embedding is obtained for the first and second actions. For example, the action embedding system of FIG.1A (e.g. comprising one of more of the action encoder neural networks of Fig.1B), appropriately programmed in accordance with this specification, can perform step 81. [000177] In step 82, the respective action embeddings for the first and second actions a processed using a decoder network, configured to output the data indicative of the reward. [000178] For example, for some pairs of first and second actions, the corresponding rewards may be defined by the payoff tensor. The method 80 may be useful to predict (estimate) unknown rewards, i.e. ones not specified by the payoff tensor. For example, step 82 may be performed using the per payoff decoder network described above with reference to Fig.4. [000179] In another example, the data may be data indicating whether the first action and second actions are close to a Nash equilibrium of the interaction, e.g. by indicating the incentive (amount of reward) for any of the participants (e.g. a first participant) to deviate from a NE strategy by performing the first action. For example, the data indicative of the reward may indicate a difference between a reward received by the first participant upon performing the first action, given that the other participants perform a second action of a NE, and the reward the first participant would receive in the NE. e.g. summed over all possible choices for the first participant out of the set of multiple participants. In this case,DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application step 82 may be performed using the joint action decoder network described above with reference to Fig.3. [000180] Fig.9 illustrates a flow diagram of a method 90 of modifying a multi- participant interaction. Each of a plurality of participants is able to perform one of a respective set of actions and each participant receives a reward defined by a respective reward function of the corresponding actions performed by the plurality of participants. The reward function is defined by one or more numerical parameters, e.g. components of a payoff tensor as described above. The process 90 can be performed by a system of one or more computers located in one or more locations. The modification is performed by modifying the one or more numerical parameters of the payoff tensor (reward function) G, so that if the agents are controlled to act in such a way as to produce a NE (optionally, but not necessarily, by a control mechanism as described above, e.g. in relation to Fig.7), the modification causes the actions of the participants to become closer (according to some distance metric) to desired actions. [000181] In a first step 91, which may be performed by a method 80 as explained above with reference to Fig.8, data is determined which is indicative of a reward received by at least one of the participants for one or more of the set of possible actions of the at least one of the participants, for current values of the numerical parameters. [000182] In a second step 92, the numerical parameters of the reward function G are modified to reduce the value of an interaction loss function indicative of a discrepancy between desired actions of the at least one of the participants and actions of the at least one of the participants which are indicated by the determined data as having the highest reward value. [000183] Fig.10 is a flow diagram of a process 100 for selecting a first number of parties to cooperate together in a multi-participant interaction (e.g. compete with each other in a competition, or cooperate to perform a task together) out of a set of available parties (e.g. selecting a football team from a larger group of possible players, or selecting a number of robots from a larger group of possible robots). For convenience, the process 100 will be described as being performed by a system of one or more computers located in one or more locations. [000184] The selection is performed by mapping the problem to a NFG game with a number of participants which may be equal to the first number of parties. Each of the participants are able to select and perform one of a respective set of actions whichDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application correspond to a selection of one of the parties from the available parties. In step 101, the multi-participant interaction is defined. [000185] In step 102, for each of the participants, a corresponding plurality of action embeddings are obtained for each of a plurality of actions from the respective set of actions for the participant. Step 102 may for example be performed using the action embedding system of Fig.1A. [000186] In step 103, a plurality of action embedding sets are defined. Each action embedding set has, for each of the participants, one of the corresponding plurality of action embeddings. Thus, each of the action embedding sets corresponds to a respective selection of a subset of the parties from the set of available parties, where the subset includes the first number of parties. [000187] In step 104, the action embedding sets are supplied to a decoder network (for example, one having the general form of Fig.4), trained to output, for each action embedding set, corresponding prediction data predicting results of a competition between the parties corresponding to the action embeddings of the action embedding set. Note that for some action embedding sets, this data may be available already (e.g. obtained experimentally), and in this case the existing data may be used instead of obtaining an output from the decoder network. [000188] In step 105, based on the prediction data, one of the action embedding sets is selected. [000189] In step 106, the parties to perform the interaction (e.g. to compete) are selected as the parties which correspond to the action embeddings of the selected action embedding set. Step 106 may comprise causing the selected parties to perform the interaction, e.g. by initiating the multi-participant interaction, such as causing them to cooperate to perform a task. [000190] Fig.11 is a flow diagram of a process 1100 of allocating a resource, from a plurality of resources, to a given task. For convenience, the process 1100 will be described as being performed by a system of one or more computers located in one or more locations. The method is based on a mapping of the allocation of resources-to-tasks to an NFG interaction between a first participant which is able to perform one of a first set of actions corresponding to a respective selection of one of the resources, and a second participant which is able to perform a second set of actions corresponding to the tasks. A reward function of the corresponding actions performed by the two participants is a performance value (a value of a performance measure) of the corresponding resource in performing theDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application corresponding task. The performance value may be based on a numerical value indicating how well the resource performs the task, and may also be based on any cost of using the resource in the corresponding task. [000191] In step 1101, the performance value is obtained for each of the resources on the given task. In the case of a resource for which the value of the performance measure for the given task is known, the known value of the performance measure may be used. In the case of a resource for which the value of the performance measure for the given task is not known, an estimate is obtained, for example by using the decoder network of Fig.4, of the reward to the first participant of performing the action of selecting the resource given that the action of the second participant is to perform the action corresponding to the given task. [000192] In step 1102, the resource for which the performance value obtained in step 1101 is highest, is allocated to performing the task. [000193] Fig.12 illustrates a method 1200 of training an action embedding system (such as the one of Fig.1A) including one or more action encoder neural networks to generate corresponding action embeddings for actions which one of the participants in a multi-participant interaction can perform. The training may be performed jointly with the training of at least one decoder network (such as one(s) having a form as shown in a corresponding one of Figs.2-4). For convenience, the process 1200 will be described as being performed by a system of one or more computers located in one or more locations. The method 1200 may be performed repeatedly, i.e. the set of steps 1201-1205 may be performed multiple times to effect the training. [000194] In step 1201, a training reward function is selected from a distribution of reward functions. Examples of possible distributions are given above. [000195] In step 1202, for at least one action of one or more of the participants, a corresponding initial embedding of the action is defined. For example, the initial action embeddings may have a default value such as having all components being zero, or may be random action embeddings. [000196] In step 1203, the at least one initial embedding is processed successively by the one or more action encoder neural networks conditioned on the selected training reward function, to generate at least one corresponding action embedding. [000197] In step 1204, the at least one action embedding generated in step 1203 is processed with the at least one decoder network, to predict at least one parameter of a multi- participant interaction based on the selected training reward function. The parameter may be one which is known (e.g. it may be obtained from the selected reward function byDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application calculation (e.g. analytically), or estimated for the selected training reward function using another (e.g. known) system for predicting parameters of NFGs, or obtained experimentally by computationally playing multiple games using the selected reward function). [000198] In step 1205, one or more parameters defining the one or more action encoder neural networks are modified, to reduce a loss function characterizing a discrepancy between the predicted at least one parameter of a multi-participant interaction based on the selected reward function, and a corresponding known parameter of the multi- participant interaction based on the selected reward function. [000199] Optionally, there may additionally (or, alternatively, for some performances of the set of steps 1201-1204) be a step of modifying the parameters of the at least one decoder network to reduce a loss function, e.g. the same loss function. Thus, the training of the action encoder neural networks is performed jointly with the training of the at least one decoder network (i.e. with updates to the action encoder neural networks being substantially simultaneous with updates to the at least one decoder network, or interleaved with them). [000200] The loss function may for example, be one of the ones discussed above with reference to Figs.2 to 4. The loss function may be calculated by the corresponding one of the loss function calculation units 25, 35, 44 of Figs.2 to 4. [000201] Thus, by repeated performances of the set of steps 1201-1205 the action encoder neural network can be trained, e.g. jointly with at least one of the decoder networks, to generate action embeddings which the at least one decoder network can use to generate parameters of the multi-participant interaction which are close to the known ones, for any training reward function selected from the distribution of reward functions. [000202] For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. [000203] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can beDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. [000204] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. [000205] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. [000206] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. TheDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). [000207] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. [000208] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. [000209] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application [000210] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a relationship graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet. [000211] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. [000212] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. [000213] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described programDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application components and systems can generally be integrated together in a single software product or packaged into multiple software products. [000214] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application CLAIMS 1. A computer-implemented method of obtaining, for a multi-participant interaction in which each of a plurality of participants is able to perform one of a respective set of actions and each participant receives a reward defined by a respective reward function of the corresponding actions performed by the plurality of participants, corresponding action embeddings for one or more of the actions which one or more of the participants can perform, the method comprising: for one or more of the participants, and for one or more of the corresponding sets of actions the one or more participants can take, defining an initial embedding of the action; defining a dataset comprising the initial embeddings; and processing the dataset successively by one or more action encoder neural networks to generate action embeddings corresponding to the one or more actions the one or more participants can take, each of the action encoder neural networks comprising a stack of successive attention mechanisms and being conditioned on the reward function.

2. A method according to claim 1 in which processing the dataset by one of the action encoder neural network comprises: forming an input to the action encoder neural network based on current embeddings, and processing the input using the action encoder neural network to generate updated embeddings, the set of updated embeddings being independent of the order of the current embeddings in the input to the action encoder neural network.

3. The method of claim 2, in which processing the input by the action encoder neural network comprises, for each of the one or more actions of each of the one or more participants: defining a plurality of joint action sets, each joint action set being a possible action for each of the other participants selected from the corresponding set of possible actions; for each joint action set, forming a corresponding key vector from the current embedding of the action, and the corresponding value of the respective reward function; and processing the key vectors by at least one first self-attention mechanism to generate a corresponding output for each of the key vectors.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application 4. The method of claim 3, wherein processing the input by the action encoder neural network further includes processing the outputs for the plurality of action sets and the current embeddings by at least one attention mechanism, to generate a modified embedding for each of the one or more actions which each of the one or more participants can perform.

5. The method of claim 4 further including processing the modified embeddings successively by one or more second self-attention mechanisms, to generate corresponding updated embeddings for each of the one or more actions which each of the one or more participants can perform.

6. The method of claim 5, in which the modified embeddings constitute corresponding updated embeddings for each of the one or more actions which each of the one or more participants can perform.

7. The method of any of claims 3 to 5, in which the one or more of the actions are all the actions the one or more participants could take, at least the first self-attention mechanism being configured to not to attend to any key vector corresponding to an action and joint action set for which the corresponding of the reward value is unknown.

8. A method of controlling at least one participant of a multi-participant interaction in which each of a plurality of participants are able to perform one of a respective set of actions and each participant receives a reward defined by a respective reward function of the corresponding actions performed by the plurality of participants, the method comprising: obtaining an action embedding for one or more of the corresponding set of actions which the at least one participant can perform; inputting the action embeddings to a decoder network to obtain corresponding reward information for each of the corresponding set of actions which the at least one participant can perform; and controlling the at least one participant to perform the one or more of the actions based on the corresponding reward information for the set of actions.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application 9. The method of claim 8, in which the obtained action embeddings were generated according to the method of any of claims 1 to 7.

10. The method of claim 8 or 9, in which each action embedding is a pre-computed action embedding retrieved from a cache.

11. The method of any of claims 8 to 10, in which the at least one participant is controlled to perform an action which the corresponding reward information indicates is a Nash equilibrium of the multi-participant interaction.

12. The method of any of claims 8 to 11, in which the one or more of the participants comprise a robot or autonomous vehicle, wherein the actions are performing respective components of at least one manipulation and / or navigation task in a real-world environment, and wherein the reward function relates to one or more of a metric of the value of the performance of the components by the participants to the performance of at least one task, and a physical cost of performing the components of the at least one task by the participants.

13. The method of any of claims 8 to 11, in which each of the participants comprises a computing job scheduling agent, wherein actions comprise actions allocating one or more jobs to one or more computing hardware resources, and wherein the reward function relates to one or more of a metric of time to perform a task of executing a computer program, an energy cost of performing the task, a computational cost of performing the task, and a reliability of performing the task.

14. The method of any of claims 8 to 11, in which each of the participants comprises a router to route packets of data over a communications network, the actions comprise routing actions to route the packets of data, and the reward function relates to one or more packet routing metrics.

15. The method of any of claims 8 to 11, wherein each of the participants is configured to control routing of electrical power from a node associated with the participant to one or more other nodes over one or more power distribution links of an electrical power distribution environment, the actions comprise control actions to control the routing of electrical power between the nodes, and the reward function relates to a loss on one of the power distribution links, or to a frequency or phase mismatch in relation to one of the power distribution links, or to overloading one of the power distribution links.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application 16. The method of any of claims 8 to 11, wherein the participants comprise control systems configured to control corresponding stages of manufacture of a mechanical, chemical, or biological product, wherein the actions comprise control actions to control the use of a machine or a manufacturing unit for processing a solid or liquid material to manufacture a product or intermediate or component thereof, or to control movement of the product or intermediate or component thereof within the manufacturing environment, and wherein the return function relates to a metric of one or more of: performance of the task; and a physical cost of performing the task.

17. A method of obtaining for a multi-participant interaction in which each of a plurality of participants are able to select and perform one of a respective set of actions and each participant receives a reward defined by a respective reward function of the corresponding actions performed by the plurality of participants, data indicative of the reward to one of the participants of performing an first action given that the other participants perform corresponding second actions, the method comprising obtaining action embeddings for the first and second actions; and processing the respective action embeddings for the first and second actions using a decoder network, configured to output the data indicative of the reward.

18. The method of claim 17, in which action embeddings have been generated according to the method of any of claims 1 to 7.

19. The method of claim 18, in which the data indicative of the reward is a reward, unspecified by the reward function, to the one of the participants of performing the first action given that the other participants perform the corresponding second actions.

20. A method according to claim 17 or 18, in which the data indicative of the reward is data indicating of the proximity of the first and second actions to a Nash equilibrium of the interaction.

21. The method of any of claims 17 to 19, in which the action embeddings are pre- computed action embeddings retrieved from a cache.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application 22. A method of modifying a multi-participant interaction in which each of a plurality of participants are able to select and perform one of a respective set of actions and each participant receives a reward defined by a respective reward function of the corresponding actions performed by the plurality of participants, the reward function being defined by one or more numerical parameters, the method comprising repeatedly: determining, by a method according to one of claims 17 to 19, data indicative of a reward received by at least one of the participants for one or more of the set of possible actions of the at least one of the participants, for current values of the numerical parameters; and modifying the numerical parameters to reduce the value of an interaction loss function indicative of a discrepancy between desired actions of the at least one of the participants and actions of the at least one of the participants which are indicated by the determined data as having the highest reward value.

23. A method selecting a first number of parties to participate in a multi-participant interaction, from a number of available parties which is greater than the first number, the method comprising: defining a multi-participant interaction in which each of a number of participants which is equal to the first number, are able to select and perform one of a respective set of actions, each of the actions corresponding to a selection of one of the parties from the available parties; for each of the participants, obtaining a corresponding plurality of action embeddings for each of a plurality of actions from the respective set of actions for the participant; defining a plurality of action embedding sets, each action embedding set having, for each of the participants, one of the corresponding plurality of action embeddings; supplying the action embedding sets to a decoder network configured to output, for each action embedding set, corresponding prediction data predicting results of a competition between the parties corresponding to the action embeddings of the action embedding set; based on the prediction data, selecting one of the action embedding sets; andDeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application selecting the parties to participate as the parties corresponding to the action embeddings of the selected action embedding set.

24. The method of claim 23, in which the available parties include one or more artificial agents, and the competition is a self-play training process for a control system of the artificial agents.

25. The method of claim 23 or claim 24, in which each of the artificial agents is a robot.

26. The method of any of claims 23 to 25, in which the obtained action embeddings were generated according to the method of any of claims 1 to 7.

27. The method of any of claims 23 to 26, in which each action embedding is a pre- computed action embedding retrieved from a cache.

28. A method of allocating a resource, from a plurality of resources, to a given task, which is one of a plurality of tasks, the allocation being based a model of an interaction between a first participant which is able to perform one of a first set of actions corresponding to a respective selection of one of the resources, and a second participant which is able to perform a second set of actions corresponding to the tasks, a reward function of the corresponding actions performed by the two participants being a performance measure of the corresponding resource in performing the corresponding task, the method comprising: obtaining a performance value for each of the resources on the given task as: in the case of a resource for which the value of the performance measure for the given task is known, the value of the performance measure, and in the case of a resource for which the value of the performance measure for the given task is not known, an estimate obtained by the method of claim 19 of the reward to the first participant of performing the action of selecting the resource given that the action of the second participant is to perform the action corresponding to the given task; and allocating the resource for which the performance value is highest to performing the task.DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application 29. The method of performing a task, which is one of a plurality of tasks, the method comprising allocating a resource to the task from a plurality of resources, by a method according to claim 28, and performing the task using the allocated resource.

30. The method of claim 28 or claim 29, in which the resources comprise robots or autonomous vehicles in a real-world environment, the tasks are manipulation and / or navigation tasks, and the reward function relates to one or more of: a metric of the value of the performance of the corresponding task using the corresponding resource; and a physical cost of performing the corresponding task using the corresponding resource.

31. The method of claim 28 or claim 29, in which the resources are computing hardware resources, the tasks are computing tasks, and the reward function relates to one or more of a metric of time to perform the corresponding task using the corresponding resource, an energy cost of performing the task, a computational cost of performing the task, and a reliability of performing the task.

32. The method of claim 28 or claim 29, in which the resources are routing resources to route packets of data over a communications network, and the reward function relates to one or more packet routing metrics.

33. The method of claim 28 or claim 29, wherein the resources are shares of an electrical power distribution environment to distribute power from a power generator to power consumers, and the reward function relates to a loss on one of the power distribution links, or to a frequency or phase mismatch in relation to one of the power distribution links, or to overloading one of the power distribution links.

34. A method of training one or more action encoder neural networks to generate, for a multi-participant interaction in which each of a plurality of participants are able to select and perform one of a respective set of actions and each participant receives a reward defined by a respective reward function of the corresponding actions performed by the plurality of participants, corresponding action embeddings for actions which one of the participants perform, each of the action encoder neural networks comprising a stack of successive attention mechanisms and being conditioned on the reward function; the method comprising:DeepMind Technologies Limited F&R Ref.: 45288-0372WO1 PCT Application training the one or more action encoder neural networks by repeatedly: selecting a training reward function from a distribution of reward functions; for at least one action of one or more of the participants, defining a corresponding initial embedding of the action; processing the at least one initial embedding successively by the one or more action encoder neural networks conditioned on the selected training reward function, to generate at least one corresponding action embedding; processing the at least one action embedding with at least one decoder network, to predict at least one parameter of a multi-participant interaction based on the selected training reward function; and modifying one or more parameters defining the one or more action encoder neural networks, to reduce a loss function characterizing a discrepancy between the predicted at least one parameter of a multi-participant interaction based on the selected reward function, and a corresponding known parameter of the multi-participant interaction based on the selected reward function.

35. A method according to any of claims 1 to 33, which is performed using one or more action encoder neural networks trained using a method according to claim 34.

36. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of any one of claims 1-35.

37. A computer-program product containing instructions that when executed by one or more computers cause the one or more computers to perform the method of any one of claims 1-35.

Images