Contract negotiation for agent collaboration in multi-agent systems

Through the negotiation contract mechanism between agents and neural network prediction, agents achieve efficient cooperation in multi-agent systems, solving the problem of low cooperation efficiency in the existing technology, and are suitable for complex environments and large action spaces.

CN119072701APending Publication Date: 2024-12-03GDM HOLDING LLC
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202380038476.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-11
Filing Date
2023-05-10
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In existing multi-agent systems, the cooperation between agents is inefficient and it is difficult to achieve optimized tasks through independent actions. Especially in complex environments, there is a lack of effective cooperation mechanisms and communication means.

Method used

Through the negotiated contract mechanism between agents, agents can interact in multiple time steps, simulate future state changes and action impacts, determine contracts that are conducive to both parties, and then select actions that are executed, and use neural networks to predict actions and Value assessment, realize action selection and contract conclusion.

Benefits of technology

Improves the efficiency of cooperation among agents, can achieve faster and more efficient task completion in complex environments, enhances the possibility of cooperation, and does not rely on central coordination agencies, is suitable for large-scale action space environments, providing robustness and low traffic.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072701A_ABST
    Figure CN119072701A_ABST
Patent Text Reader

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for enabling agents to cooperate with each other in a manner that improves their collective efficiency. The agent may modify its behavior by taking into account the behavior of other agents, so that a better overall result than each agent acts independently may be achieved. This is accomplished by enabling the agents to mutually negotiate contracts that limit their respective actions.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] This specification relates to cooperation of agents in multi-agent systems.

[0002] The system uses a neural network. A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input according to the current values ​​of the corresponding set of parameters. Summary of the invention

[0003] This specification describes techniques that enable agents to cooperate with each other in a way that improves their collective efficiency. Agents can take into account the actions of other agents to modify their behavior, thereby achieving better overall results than each agent acting independently.

[0004] In one aspect, a computer-implemented method of controlling a first agent to perform a task in an environment in which the first agent interacts with one, two, or more other agents performing one or more other tasks in the environment is described.

[0005] The first agent obtains a state representation representing a state of the environment at each of a plurality of time steps, processes the state representation using an action selection subsystem to generate a policy output, uses the state representation to determine predicted actions for other agents, and uses the policy output to select an action to perform depending on the predicted action.

[0006] At one or more time steps, a first agent negotiates a contract with a second agent that defines restrictions on actions that can be selected by the first agent and restrictions on actions that can be selected by the second agent.

[0007] This involves a first agent determining whether to propose a contract by determining one or more scores representing expected values ​​of the state of the first agent's environment at subsequent time steps with and without the proposed contract. This is done by simulating the effects of future actions of the first agent and other agents i) without the proposed contract and ii) taking into account the effects of restrictions on actions in the proposed contract. The proposed contract is transmitted to a second agent dependent on the scores, and in response to receiving an indication from the second agent to accept the proposed contract, an action is selected for the first agent to perform taking into account the restrictions on the first agent's actions in the proposed contract.

[0008] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0009] Equipping agents with the ability to negotiate contracts as described above helps the agents learn to cooperate to perform their respective tasks. Agents equipped in this way may be able to perform tasks that they would not otherwise be able to perform, or they may be able to perform tasks more efficiently, such as faster or with fewer resources than otherwise possible. In general, communicating agents outperform non-communicating agents, and there can be substantial gains through cooperation.

[0010] The technology helps to reach an agreement on a contract that is beneficial to both parties. For example, if a contract is only beneficial to the first agent, the second agent is unlikely to reach the contract. Because the technology helps to identify contracts that are beneficial to both parties, the possibility of beneficial cooperation among agents is enhanced. Agents can also impose sanctions to discourage the violation of the agreed contract. In an embodiment, agents can act simultaneously in the environment.

[0011] Implementation of the technology does not rely on a central coordinating mechanism that may represent a single point of failure. Instead, the technology can be implemented in a decentralized, e.g., peer-to-peer, setting. This facilitates deployment of the technology in a wide range of applications and can also help provide robustness. In addition, the amount of communication between agents to reach a contract can be relatively small.

[0012] The described techniques can be used in environments where the action space is very large, such as combinatorial environments. For example, each agent may have a large number of possible actions that can be performed at each time step. This leads to a huge space of potential contracts. The described techniques can be used effectively in such action spaces.

[0013] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 An example of a multi-agent system is shown.

[0015] Figure 2 is a flow chart of an example process for controlling a first agent.

[0016] Figure 3 is a flow chart of a first example process for determining whether to propose a contract.

[0017] Figure 4 is a flow diagram of a second example process for determining whether to propose a contract.

[0018] Figure 5is a flow chart of an example process for selecting contracts to evaluate for proposal.

[0019] Figure 6 The benefits that can be achieved using the first example process are shown.

[0020] Figure 7 Benefits that may be achieved using the second example process are illustrated.

[0021] Like reference numbers and names in the various drawings indicate like elements. DETAILED DESCRIPTION

[0022] This specification generally describes a computer-implemented method for controlling a first agent to perform a task in an environment in which the first agent interacts with one, two, or more other agents in the environment that perform one or more other tasks. The operations of the other agents may be controlled by a computer-implemented method, such as similar to the method for controlling the first agent, or some or all may be partially or wholly under human control.

[0023] Typically, the tasks may have the same characteristics, for example, they may all be routing tasks, or they may all be scheduling tasks, or they may all be manufacturing tasks. Other tasks may be the same as or different from the task of the first agent. The tasks performed by the agents may contribute to achieving a common goal. In a typical implementation, the ability of the first agent to perform a task in the environment is affected by one or more other agents that perform one or more other tasks in the environment. Some example tasks will be described in more detail later.

[0024] The first agent, and in embodiments other agents, are enabled to reach a contract for a joint plan, and in this way can outperform agents operating independently. The contract imposes constraints on the actions of the agents that enter into the contract, and agreeing on the contract involves negotiation, i.e., communication between the agents. The contract can reflect a balance between cooperation and competition, and the described techniques facilitate operation in many real-world domains where agents interact, i.e., where the actions of one agent affect another, and where the goals of the agents may only partially coincide.

[0025] Such multi-agent systems are common in the real world and include, for example, systems of robots or autonomous or semi-autonomous vehicles that interact while performing tasks, such as systems of warehouse robots; factory or plant automation systems; and computer systems. Thus, as examples, agents may include robots or vehicles, facility items in a factory or plant, or software agents in a computer system that, for example, control the allocation of tasks to hardware items or the routing of data on a communications network.

[0026] Figure 1 An example of a multi-agent system is shown, which includes agents 102a..n, each of which is controlled by a corresponding agent control and contract negotiation system 100a..n. Each agent control and contract negotiation system 100a..n can be implemented as one or more computer programs on one or more computers in one or more locations. In the example shown, each agent is computer controlled, but in other examples, one or more agents can be human controlled.

[0027] Agents 102a..n operate in a common environment 104, with each agent performing a respective task. The tasks performed by the respective agents may be the same or may be different from one another. Generally speaking, the way one agent performs its task may affect the way another agent is able to perform its task. For convenience, the following description refers to the agent control and contract negotiation system 100a of agent 102a (also referred to as the first agent); in embodiments, other agent control and contract negotiation systems are similar.

[0028] At each of the plurality of action selection time steps, the agent control and contract negotiation system 100a selects an action to be performed by the agent 102a in response to a state representation representing a state of the environment. In an embodiment, from the observation of the environmental state Deriving a state representation. For example, the observations may include images of the environment and / or other sensor data, such as data representing the location, state, or configuration of one or more agents, or data representing the state or configuration of the environment including objects in the environment; or generally input data from the environment. In some embodiments, the observations may be processed, such as by a neural network, to obtain a state representation; in some embodiments, the observations themselves may be used as a state representation.

[0029] The agent control and contract negotiation system 100a can also perform actions Receiving rewards Typically, rewards are numerical, i.e. scalar, quantities and can be based on any event or aspect of the environment. For example, the reward It may indicate whether the agent 102a has completed the task, or the progress of the agent 102a in completing the task (eg, physical configuration of one or more objects, or navigation to a target location in the environment).

[0030] In an embodiment, the agent control and contract negotiation system 100a includes an action selection subsystem 110a for selecting an action to be performed by the agent 102a. The action selection subsystem 110a is configured to process a state representation representing the state of the environment at a (current) time step, such as a state representation derived from an observation of the environment, to generate a policy output for selecting an action to be performed by the agent. As previously described, each agent 102 may have an associated action selection subsystem 110.

[0031] There are many ways in which a policy output can be used to select actions. For example, a policy output can define a probability distribution over a set of actions that can be performed by an agent. The probability distribution can then be used to select an action, for example, by sampling or selecting the action with the highest probability from the probability distribution. A policy output can parameterize such a probability distribution, or it can define a probability distribution as a set of scores from which actions can be selected, such as a score for each action in a set of possible actions. As another example, a policy output can define an action directly, such as by identifying the speed or torque of a mechanical action. Typically, actions can be continuous or discrete; optionally, continuous actions can be discretized. An action can include multiple individual or primitive actions to be performed at a time step, such as a mixture of continuous and discrete actions. In some embodiments, a policy output can include multiple outputs, such as multiple outputs from multiple heads on a neural network for selecting multiple actions at a particular time step.

[0032] In general, any type of action selection subsystem can be used for the action selection subsystem 110a. In some embodiments, but not necessarily, the action selection subsystem 110a includes an action selection policy neural network. For example, the action selection subsystem can use an action selection policy neural network that is configured to process a state representation characterizing an environment state according to action selection policy neural network parameters to generate an action selection neural network output that can be a policy output. In some embodiments, the action selection neural network output of the action selection policy neural network can be indirectly used to generate a policy output, such as by using the action selection policy neural network to simulate possible futures for determining a policy output. For example, the action selection subsystem can implement the "Sampled Best Response" algorithm, which is described in arXiv:2006.04635v4 by Anthony, Eccles, Tacchetti, Kramár, Gemp, Hudson, Porcel, Lanctot, Julien, Everett, Singh, Graepel, and Yoram.

[0033] As further described later, the agent control and contract negotiation system 100a determines predicted actions for other agents, particularly using state representations. In an embodiment, the action selection subsystem 110a or a version of the action selection subsystem 110a (e.g., with different learnable parameter values, such as learnable parameter values ​​from an earlier training iteration) is used to predict actions of other agents 102b..n, e.g., predict actions selected by the action selection subsystems 110b..n of other agents.

[0034] The predicted actions of other agents can be determined in a variety of ways, and the manner in which they are implemented can depend on the specific technical application. For example, in one approach, the agents have the same or corresponding task goals, such as navigating to, for example, a target location of an object in the environment or achieving a specific state or configuration of the environment. Then, from observations of the environment The state representation derived and processed by the action selection subsystem to predict the action of a particular agent can represent the environment associated with the particular agent. For example, if the task is to navigate to a target location, the state representation can include a representation of the location of the particular agent. In other cases, such as when the environment is a computing or communication environment, the state representation can include a representation of the state or configuration of the computing or communication environment associated with the agent. In some other embodiments, the state representation processed by the action selection subsystem to predict the action of a particular agent can be combined with, for example, observations from the particular agent. Obtained separately.

[0035] As another example, the action selection subsystem 110a may be goal-conditioned, e.g., it may also process inputs identifying a task to be achieved. In such a case, the first agent may obtain information, e.g., from another agent defining the goal of the other agent, and process that information and inputs from the observation to predict the actions of the other agent. As a further example, the agent control and contract negotiation system 100a can maintain a copy of the action selection subsystem of one or more other agents and use it to predict the actions taken by the other agents.

[0036] In an embodiment, the agent control and contract negotiation system 100a also includes a value neural network 120a, which is configured to process the state representation according to the value neural network parameters to determine a state value. The state value can define the expected reward of the first agent from the state of the environment represented by the state representation. In some embodiments, the value neural network 120a determines the state value of each agent, such as by processing the state representation for each agent and providing a separate "head" to the value neural network 120a; or in other ways, such as by using the same value neural network for each of the agents, or by obtaining a copy of the value neural network of each of the other agents.

[0037] Typically, the expected return from a state of the environment may include an estimate of the cumulative, time-discounted sum of rewards for performing a task starting from that state. Rewards may include negative rewards, i.e., costs. In an embodiment, each agent has a value neural network. Typically, embodiments of the techniques described herein may attempt to maximize the expected return of the first agent 102a and / or may attempt to minimize the corresponding cost of the first agent 102a. However, the techniques described herein promote cooperation among agents, thereby achieving improved returns for all agents, including the first agent 102a.

[0038] In some embodiments, each agent's action selection subsystem 110a, such as each agent's action selection policy neural network and value neural network, has been pre-trained to perform the task, and the described techniques use these pre-trained neural networks to negotiate the contract. There are many methods for such pre-training, and the techniques described herein do not rely on any particular method. As just some examples, one or more of imitation learning, such as behavioral cloning, regret minimization, and reinforcement learning (based on rewards), can be used.

[0039] As a specific example, imitation learning followed by reinforcement learning can be used to train and improve the action selection subsystem 110a and value neural network 120a of each agent. Such an approach is described in arXiv:2006.04635v4 by Anthony et al. (supra). Broadly, this can involve, for each of a plurality of training iterations: generating training data for the training iteration by controlling an agent with an improved policy that selects an action in response to an input state representation by performing a best response calculation using (i) a candidate policy generated from a policy neural network since one or more previous iterations and (ii) a candidate value neural network. A candidate value neural network can be generated from the value neural network since each of the one or more previous iterations. The policy neural network value neural network can be updated at each of the plurality of training iterations by training on the training data.

[0040] In some embodiments of the technology, the learnable parameters of the action selection subsystem, such as the action selection subsystem of the action selection policy neural network, and the value neural network can be frozen. In some other embodiments, the learnable parameters of the action selection subsystem, such as the action selection subsystem of the action selection policy neural network, and the value neural network can be trained, such as fine-tuned, during operation of the process described herein.

[0041] In general, the action selection subsystem 110a, e.g., the action selection policy neural network, and the value neural network 120a can have any suitable architecture. For example, the action selection policy neural network and the value neural network 120a can include, for example, one or more feed-forward neural network layers, one or more convolutional neural network layers, one or more attention neural network layers, or one or more normalization layers. The neural network can be pre-trained by back-propagating the gradient of the objective function to update the values ​​of neural network parameters such as weights, e.g., using an optimization algorithm such as Adam. As an example, the reinforcement learning objective function can be based on a received reward, e.g., based on a Bellman error, or based on a policy optimization objective.

[0042] The agent control and contract negotiation system 100a further includes a communication subsystem 130a for communication between agents. In an embodiment, each agent has such a system, and this enables the agents, more specifically the agent control and contract negotiation systems of the agents, to communicate with each other to negotiate a joint plan of actions. This typically involves negotiating an agreement or contract with one or more other agents according to a protocol, as described in more detail later. Typically, the contract defines restrictions on the actions that each agent may take in the future. For example, such a contract may define restrictions on actions that can be selected by a first agent and restrictions on actions that can be selected by another second agent. The communication subsystem 130a can implement any form of communication, such as wired or wireless communication, point-to-point or point-to-multipoint; it can implement low-level data communication, or it can use natural language.

[0043] An embodiment of the agent control and contract negotiation system 100a also includes a simulator 140a, which is configured to simulate the effects of future actions of agents, particularly the first agent 102a and other agents. The simulator 140a processes state representations that characterize the state of the environment at the current time step, as well as the agents' supposed actions, such as candidate actions for the first agent and predicted actions for each of the other agents. The simulator 140a determines a predicted subsequent state of the environment, such as the next state of the environment, and particularly a state representation that characterizes the predicted next state of the environment. Thus, the simulator 140a can predict the effects of a contract, for example, by simulating the effects of future actions of the first agent and other agents in i) the absence of a proposed contract and ii) taking into account the effects of restrictions on actions in the proposed contract.

[0044] Figure 2 is a flow chart of an example process for controlling a first agent, such as agent 102a, to perform a task in an environment in which the first agent interacts with one or more other agents that perform one or more other tasks. Figure 2 The process may be implemented on one or more computers at one or more locations, such as Figure 1 is implemented on the intelligent agent control and contract negotiation system 100a, and optionally on each other intelligent agent control and contract negotiation system.

[0045] At each time step in the plurality of time steps, the first agent 102a obtains a state representation characterizing the state of the environment at the current time step, for example from observation of the environment, and processes this using the action selection subsystem 110a for the agent to generate a policy output for the agent (step 200). The state representation is also used to predict the actions of other agents, and the policy output and these predicted actions are used to select an action to be performed by the first agent.

[0046] Typically, an action of a first agent is selected depending on the predicted actions of the other agents. For example, an action may be selected from candidate actions by determining, for each candidate action, the state value of the next state of the environment given the predicted actions of the other agents. An action may be selected based on the expected reward from the next state of the environment. Each of the other agents may similarly select an action in this manner.

[0047] The next state of the environment can be determined from a simulation of the environment, such as by processing candidate actions and predicted actions using simulator 140a. The expected value of the state of the environment, such as the expected reward from the next state of the environment, can be determined by processing a state representation representing the state of the environment using value neural network 120a.

[0048] In an embodiment, the process includes, in one or more of the time steps, a first agent negotiating a contract with a second agent, where the second agent is one of the other agents (step 202). The contract defines restrictions on actions that can be selected by the first agent and restrictions on actions that can be selected by the second agent.

[0049] In an embodiment, negotiating a contract includes: a first agent determines whether to propose a contract (step 204) by determining the value of one or more scores, for example using an SVE value estimate described later, which represents the expected value of the state of the first agent's environment at a subsequent time step with and without the proposed contract. This can be performed by simulating the impact of (potential) future actions of the first agent and other agents in embodiments of all agents i) without the proposed contract and ii) considering the impact of restrictions on actions in the proposed contract. Potential future actions can be determined using an action selection subsystem or a version thereof, for example with different learnable parameter values, and the impact of future actions can be evaluated using a value neural network 120a to determine the expected reward from the next state of the environment, for example as determined by a simulator 140a. In an embodiment, the method uses the scores to determine whether to propose a contract, in some embodiments to determine whether to propose any contract, and in some embodiments to determine whether to propose a specific contract.

[0050] The proposed contract may then be transmitted to the second agent depending on the scores, for example when one or more scores indicate that the expected return with the proposed contract is greater than the expected return without the proposed contract (step 206).

[0051] The proposed contract may or may not be accepted by the second agent. In the absence of an acceptance, a non-acceptance may be assumed, or a non-acceptance may be explicitly communicated (step 208).

[0052] When the contract is accepted by the second agent, such as indicated by a communication from the second agent to the first agent, the process selects an action for the first agent to perform, taking into account the restrictions on the first agent's actions in the contract (step 210). Correspondingly, the second agent can select actions taking into account the restrictions on the actions that can be selected by the second agent. The second agent can determine whether to accept the proposed contract by determining the expected return (for the second agent) with and without the proposed contract, such as by simulating the impact of restricted and unrestricted actions as described for the first agent. When the proposed contract is not accepted, the first agent can select actions in any suitable manner based on the policy output, without being constrained by the proposed contract with the second agent. Nevertheless, these actions may be constrained by one or more contracts with other agents, if any.

[0053] The process continues by obtaining and processing a state representation characterizing the environment at the next time step (step 200 ) according to any suitable criteria until the task is completed (step 212 ).

[0054] When there are more than two agents, Figure 2 The process may be used by a first agent to propose a contract to each of the other agents (although not all other agents in the system may be able to reach a contract). In an embodiment, if there are multiple reached contract actions, the constraints of all contracts reached by the first agent may be applied to the first agent's action.

[0055] As described in more detail later, there are various protocols that can be used to negotiate, i.e., communicate and reach, contracts. For example, a "Mutual Proposal" protocol imposes restrictions on the actions of both the first agent and the second agent, and a "Propose-Choose" protocol enables each of the first agent and the second agent to reach a specific corresponding action. The specific required or disallowed actions may depend on the application. For example, in the "Mutual Proposal" protocol, there may be restrictions on actions that may cause a collision or conflict, or a partial collision or conflict, or the risk of such in the environment; or restrictions on actions that would support a third agent. In the "Propose-Choose" protocol, in the presence of more than one other agent, a single "second agent" can be selected as a partner to reach a contract, one of which, for example, provides the most favorable contract or the largest expected return based on a score value.

[0056] In some embodiments of the process, the first agent may receive one or more proposed contracts from a second agent or other agents in the agent. Negotiating the contract may then involve both the first agent and the second agent accepting the proposed contract before the contract is used to restrict the actions of the first agent (and the second agent). For example, in some embodiments of the "Propose-Select" protocol, both the first agent and the second agent need to select (accept) the same contract in order for it to be implemented. In some embodiments, the first agent may accept (and may be able to indicate that they are willing to accept) two contracts, namely the proposed contract proposed by the first agent and the proposed contract from the second agent. Either contract may then be implemented to restrict actions, such as the contract with the maximum expected return.

[0057] As an example only, when an action is unconstrained, the action can be determined directly from the output of the action selection policy neural network, or the action can be determined using the "sampled best response" (SBR) algorithm (described above). The current action selection policy of (e.g., defined by its action selection policy neural network) is represented as , and the action selection strategies of other agents are expressed as (e.g. defined by the same or a different action selection policy neural network), will be The vector representation of other agent actions sampled in is (where subscript Indicates that except for the agent ), and A sample of such action vectors is represented as , which can be used for intelligent agents Candidate actions for and the state of the environment The estimated future reward, i.e., the action value estimate Defined as: in, is given by the (all) agent actions represented by the simulator 140 The vector of is the transition function of the environment state; and is a state value, such as a state value from the value neural network 120. The multiple—e.g., many—candidate actions sampled in the y-axis determine the action value estimate , and one of these—eg, the candidate action with the highest value estimate—can be selected as the action to be performed.

[0058] As described above, determining whether to propose a contract can involve determining the value of one or more scores that represent expected values ​​for the state of the first agent's environment at a subsequent time step to the current time step.

[0059] In an embodiment, determining the expected value of the state of the environment for the first agent at a subsequent time step involves sampling one or more possible actions of the first agent at the time step using the action selection subsystem 110a of the first agent. For each of the one or more actions and for each of the other agents, the process can then sample multiple possible actions of the other agents at the time step and use the value neural network to determine the state value of the predicted new state of the environment (at the subsequent time step) determined by the sampled actions of the agent.

[0060] As previously described, the action selection subsystem of the first agent or a version thereof may be used to determine a plurality of possible actions of the other agents, for example by processing a current state representation using the action selection subsystem. The predicted new state of the environment may be determined by simulating the effects of the possible actions using simulator 140a, for example to determine a state representation of the predicted new state of the environment. Simulating the effects of future actions may include averaging the state values ​​of each predicted new state (determined by the possible actions) over sampled actions for each of the other agents to determine an expected value of the state of the environment for the first agent, particularly for each of one or more sampled possible actions of the first agent.

[0061] Simulating future actions of (all) agents may include, for each agent, selecting a set of one or more candidate actions for the agent using an action selection subsystem and / or, if a proposed contract applies to the agent, by applying restrictions on actions in the proposed contract. In some embodiments, the proposed contract defines a restricted set of actions; in some embodiments, the proposed contract defines a single action to be performed by an agent, such as the first agent or the second agent.

[0062] In some embodiments, by Partial Action Profile Sampling is performed to calculate the agent Take Action Monte Carlo estimate of the value of Indicates that for the agent All agents except Available) the corresponding action selection strategy of its corresponding action selection subsystem The sampled actions are then The agent can then be calculated as follows Take Action Estimation of the value of , which is later referred to as Simulated Action Value Estimation (SAVE): in . Similarly, you can use Rather than For intelligent agents Compute the SAVE estimate (and the STAVE estimate described later) where Agents can be used The value of a neural network (assuming this is a useful enough approximation) or a version of it with different learnable parameters such as weights, or by having access to the agent The value of a neural network or its copy is determined.

[0063] Based on a restricted set of possible actions, it is possible to determine, for example, the agent by averaging the actions of the agent sampled from the restricted policy. As an example, for the simulated value estimate (SVE) represented by The contract, in which the agent Action selection strategy of the action selection subsystem Limited to a set of actions , and the agent Action selection strategy Limited to a set of actions , then we can The average of the actions sampled in . This value can be estimated as , where each is the complete action profile, i.e., it includes all the actions of the agent, ,in Represents the agent and Agent and from the unrestricted policies of other agents, and ,Pick Monte Carlo samples. This can be done by sampling from an unrestricted strategy, when the sampled action exceeds the limit (or ) to sample a restricted set of actions. When the actions of the first agent and the second agent are constrained by the proposed contract, the SVE value estimate can be used to determine the expected value of the state of the first agent's environment.

[0064] It is convenient but not essential that a given capture strategy and the set of restricted actions The neural network can be constructed by masking the The logit probability of all actions in — for example, by replacing the actions that are not in The weights of all actions in are set to zero—from the restricted policy (It can also be expressed as ) in the sampling action.

[0065] You can also determine the agent to consider by doing the following Action and Agent Action The simulated dual action value estimation (STAVE) of the combined impact of : averages the actions of other agents sampled from their corresponding policies. This can be done by averaging the actions of the other agents except and Agent All agents except of Samples And use the full action profile To estimate the value of the next state to complete.

[0066] Figure 3 is a flow chart of an example process for determining whether to propose a contract in an embodiment of a process using a "mutual proposal" protocol. Figure 3 The process may be implemented on one or more computers at one or more locations, such as Figure 1 The invention is implemented on the intelligent agent control and contract negotiation system 100a, and optionally on each of the intelligent agent control and contract negotiation systems.

[0067] In some embodiments of the process, the contract is a "generic" contract that defines a class of actions that defines the same or corresponding restrictions on actions that can be selected by both the first agent and the second agent. As an example, corresponding restrictions on actions may define that one agent, such as the first agent, is prohibited from moving to a space occupied by another agent, such as the second agent. That is, the restrictions may correspond, but the specific actions that are restricted may be different. Such a contract may be represented as .

[0068] The example process involves determining a first score representing an expected value of the state of the environment of the first agent at a subsequent time step with a proposed contract (step 300); and determining a second score representing an expected value of the state of the environment of the first agent at a subsequent time step without a proposed contract (step 302). The first score and the second score can then be compared, and the comparison result can be used to determine whether to propose a contract, for example, transmitting a proposed contract when the first score exceeds the second score (step 304). Broadly speaking, the first agent proposes a contract when it expects to achieve greater utility when the contract is agreed upon, otherwise it avoids proposing a contract. As an example, each of the first score and the second score can include a simulated value estimate (SVE) as described above.

[0069] The first agent can propose such a contract to each of the other agents, for example using Figure 3 The process is carried out and multiple general contracts can be reached, in which case the action restrictions of all reaching contracts apply.

[0070] In some embodiments, the first agent and the second agent will only accept such a "mutually proposed" contract if each of them proposes a contract to the other that defines a restricted set of actions, i.e., if both parties agree to restrict their actions. If no agreement is reached, each agent is free to choose actions that are not restricted by the proposed contract (although they may be restricted by the contract reached with other agents). If an agreement is reached, the first agent and the second agent choose actions according to the agreed restrictions.

[0071] Figure 4 is a flow chart of an example process for determining whether to propose a contract in an implementation of a process using an "offer-take" protocol. Figure 4 The process may be implemented on one or more computers at one or more locations, such as Figure 1 The invention is implemented on the intelligent agent control and contract negotiation system 100a, and optionally on each of the intelligent agent control and contract negotiation systems.

[0072] In some embodiments, the contract defines specific actions that must be taken by the first agent and the second agent. For example, the contract may define a first action that must be selected by the first agent and a second action that must be selected by the second agent, i.e., each agent is restricted to a single action under the contract. A "proposal-pick" protocol may then be used to determine which contracts (actions) to propose during the proposal phase. Agents may choose from all contracts (actions) involving them, whether they are contracts proposed by them or contracts proposed to them. The group of all contracts involving, for example, the first agent may be referred to as the first agent's On The Table contracts.

[0073] In some implementations of the Propose-Take protocol, two agents can only reach an agreement, i.e., accept a contract, if they both choose the same contract. Or both ,in Representing an Agent Towards an Agent In some embodiments, agents can indicate that they are willing to accept and (If the agent sorts them in a different order, it can choose one at random). agents, each of which can potentially propose and receive contracts, so the first agent who wants to reach an agreement with the second agent is competing with other agents. If the first agent proposes a contract that is most beneficial to itself, the second agent is unlikely to choose that contract, and the implementation of the protocol takes into account which contracts other agents are likely to accept.

[0074] In an embodiment, one or more scores used to determine whether to propose a contract include an agreement score, which represents the expected value of the environment state of both the first agent and the second agent at a subsequent time step when the first agent and the second agent respectively select the first action and the second action (if there are other agents, the actions of the other agents may be unrestricted).

[0075] refer to Figure 4At step 400, the process involves determining respective first and second baseline values, the first and second baseline values ​​representing expected values ​​of the state of the environment for the first agent and for the second agent at subsequent time steps without restricting the second agent to the second action and without restricting the first agent to the first action, respectively. The expected values ​​of the state of the environment for the first agent and for the second agent at subsequent time steps may be determined using a value neural network, for example, it may include a state value or an average of state values ​​from the value neural network.

[0076] The process then determines an agreement score using (ie, relative to) the baseline value (step 402 ), and determines whether to propose a contract based on the agreement score (step 404 ).

[0077] In an embodiment, multiple candidate contracts may be evaluated and a candidate contract with a highest agreement score may be proposed. That is, the process may include determining a proposed contract for the second agent by determining multiple candidate proposed contracts and selecting a candidate proposed contract based on the candidate's agreement score, such as by selecting the candidate proposed contract with the highest agreement score as the proposed contract.

[0078] In some embodiments, the agreement score includes the product of: i) the expected value of the environment state of the first agent at a subsequent time step when the first agent and the second agent select the first action and the second action, respectively ( ) and the first baseline value ( ) and ii) the expected value of the environment state of the second agent at the subsequent time step when the first agent and the second agent select the first action and the second action respectively ( ) and the second baseline value ( ). For example, the agreement score can be determined as ,in express In an embodiment, the agreement score is equal to or greater than zero.

[0079] In some embodiments, the (highest) agreement score approximates a so-called "Nash Bargaining score" or "Nash Bargaining Solution" (e.g., as described in J.F. Nash, "The bargaining problem," in The Essential John Nash, pp. 37-48, Princeton University Press, 2016). Determining the agreement score in this way helps identify contracts that are mutually beneficial to the first agent and to the second agent relative to the situation without a contract. Figure 5 The method for selecting contracts described in this paper also provides an approximation to Pareto optimality (where, e.g., between two agents, neither agent will choose a contract if they can find another contract that has higher utility for both agents) and symmetry (expected utility for both agents).

[0080] In some implementations, the first baseline value comprises an expected value for the state of the first agent's environment at a subsequent time step when the first agent selects a first (required) action and the action selected by the second agent is unrestricted. The second baseline value may comprise an expected value for the state of the second agent's environment at a subsequent time step when the second agent selects a second (required) action and the action selected by the first agent is unrestricted.

[0081] In some embodiments, determining the first baseline value and the second baseline value takes into account whether a contract can be reached between other pairs of agents other than the first agent and the second agent. This can be done by simulating the negotiation process of other agents. Therefore, for each pair of agents other than the pair including the first agent and the second agent, determining the first baseline value may include identifying whether they will reach a contract for a subsequent time step. This can be done in the manner previously described for the first agent and the second agent. When they reach a contract, the first baseline value can be determined as the expected value of the state of the environment of the first agent at the subsequent time step as if the contract reached by the pair has been established (in place). Similarly, determining the second baseline value may include, for each pair of agents other than the pair including the first agent and the second agent, identifying whether they will reach a contract for a subsequent time step. When they reach a contract, the second baseline value may be determined as the expected value of the state of the environment of the second agent at the subsequent time step as if the contract reached has been established. As previously described, the expected value of the state of the environment can be determined by using the value neural network 120 to process the state of the environment at the subsequent time step, for example, determined by the simulator. Typically, the expected value can be determined by performing Monte Carlo sampling over the possible actions of the agent, determined, for example, by the action selection subsystem, to obtain an average.

[0082] Figure 5 is a flow chart of an example process for selecting contracts to evaluate for proposal, such as in an "offer-pick" protocol as described above. Figure 5 The process may be implemented on one or more computers at one or more locations, such as Figure 1 The invention is implemented on the intelligent agent control and contract negotiation system 100a, and optionally on each of the intelligent agent control and contract negotiation systems.

[0083] Figure 5 The purpose of the process is to be able to propose contracts that approximate the solution to the Nash game. One difficulty is that the action space of multiple agents 102a..n is combinatorial and may be very large, for example, larger than , and the space of possible contracts grows quadratically as it combines the actions of the first agent and the second agent. One approach to solving this problem is to sample many actions from the action selection policies of the first agent and the second agent. Such an approach may involve determining a proposed contract for the second agent by determining a plurality of candidate proposed contracts, and then selecting one of the candidate proposed contracts as the proposed contract.

[0084] Determining the candidate proposed contract may include: determining a first candidate action set for the first agent by processing the state representation at the current time step using the action selection subsystem to generate a policy output for the first agent (step 500). Determining the candidate proposed contract may also include (the first agent) determining a second candidate action set for the second agent, for example, by processing the state representation at the current time step using the action selection subsystem or a version thereof to generate a policy output for the second agent (step 502). The process may then generate an action pair, the action pair comprising a first candidate action from the first candidate action set and a second candidate action from the second candidate action set (step 504).

[0085] Then, the candidate proposed contracts may be determined by selecting a set of candidate action pairs via the following operations: ranking the action pairs according to the metric and selecting The top ranked action pairs (step 506). Each candidate proposed contract may include one of the candidate action pairs.

[0086] The pair metric for an action pair may include a combination, e.g., a sum, of an expected value for the state of the first agent's environment at a subsequent time step when the first agent performs a first candidate action in the action pair, and an expected value for the state of the second agent's environment at a subsequent time step when the second agent performs a second candidate action in the action pair. The combination—e.g., a sum—can be a weighted combination used to weight the value of the second agent's proposed contract, e.g., to adjust the likelihood of accepting the proposed contract. The expected values ​​for the first agent and for the second agent's environment state can be scaled to approximately match each other.

[0087] An agreement score may then be calculated for each of the candidate proposed contracts (step 508 ), and one of the candidate proposed contracts may then be selected as the proposed contract based on the agreement scores calculated for the candidate proposed contracts—eg, by selecting the contract with the highest agreement score (step 510 ).

[0088] Optionally, the process may also include: the first agent determines whether to accept the contract proposed by the first agent to the second agent; or whether to accept an upcoming contract proposed by the second agent or one of the other agents (step 512). For example, this may involve the first agent determining one or more scores, such as an agreement score for the proposal or upcoming contract, as described above, and determining whether to accept the proposal or upcoming contract based on the determined scores. In some embodiments, the first agent selects from their contracts for consideration and selects a single other agent to enter into a contract with.

[0089] As a specific example, this process can be accomplished by Sampling Agents of Candidate actions Collection and from strategy Sampling Agents of Candidate actions Collection ,, determine the Cartesian product , and select the top ranked pairs of metrics as above Thus, multiple candidate proposed contracts are generated.

[0090] It can be assumed that the other agents (if any) are unrestricted and follow unrestricted policy profiles Select actions to determine the value of the metric, from which we can derive Action Profiles .action The pair metric can include the agent The value of the next state of the environment and for the agent The value of the next state of the environment The weighted sum of . The value can be scaled to Similar ratios make it easier to negotiate contracts between agents with high versus low estimates.

[0091] For example, the scaling factor can be determined as ,in and This can be calculated as follows: for all agents sampling A collection of action profiles, and then use SAVE for the agent and Agent Select the best actions and combine them using STAVE to estimate the agent’s performance when the contract is not reached. The value of ; and for the agent Therefore, in one example implementation, the metric may be determined as , where the weight can be in the range [0,1], and The low value of emphasizes the importance of the proposer, the agent , the benefits of, and vice versa.

[0092] Agent Can be sent to the agent The multiple candidate proposal contracts provided are represented as This process can be done using Medium Action and The agreement score determines the proposal which contract to enter into, which in an embodiment is based on a “no deal” first baseline value and a second baseline value against which the gain of entering into the contract is measured.

[0093] In one approach, it is assumed that the other agents do not form an agreement among themselves and that they Select actions from the list so that (continue with the previous notation) and These values ​​can be estimated using the STAVE algorithm described above.

[0094] In another approach, an estimate of the baseline value is determined by considering what other agents might be able to do better by forming an agreement among themselves. Thus, the process can consider for each agent what contracts it might have with other agents, and then use the best desired state value among them as the baseline. An algorithm for this, "BATNA" (Best Alternative to Negotiated Agreement), is given below:

[0095] here (and ); ; and the returned baseline value is The algorithm iterates through the agents step by step. , Agent to find the action pairs that maximize the agreement score using the no-deal estimate ("find Nash game deals"), and then for each pair, if a higher agreement score improves the no-deal estimate ("find BATNA"), take that higher agreement score as the baseline. A damping factor, such as exponential smoothing, can be applied to the returned agreement values.

[0096] Each agent can choose from their contracts for consideration by calculating an agreement score for each agent, e.g., using a previously calculated baseline value, and selecting a single other agent to enter into an agreement with. If some agreement scores are positive, then the contract with the highest score can be selected; otherwise, the agent can select the contract that is most beneficial to itself based on the expected value of the agent's environment state at the next time step, provided that this value is greater than a no-deal baseline (calculated as described above, e.g., using BATNA). In some embodiments, this process requires that the contract selected in this manner be selected by both the first agent and the second agent, i.e., both choose Or both (Use actions in each , The corresponding pairs are evaluated) for reaching a contract.

[0097] In some practical situations, such as when not all agents are of the same type or when humans are involved, the constraints in the agreed contract may be violated. It is useful to provide an agent with the ability to sanction the behavior of another "deviating" agent to encourage consistency. For example, a deviating agent may break a contract when it determines that it expects to gain from breaking the contract. Typically, the sanctioning agent can modify its actions in an attempt to reduce the reward of the deviating agent.

[0098] In some embodiments, the first agent can detect when the second agent has violated the contract, for example from observations of the state of the environment (which includes the agent). The violation may occur at a time step marked as a violation time step. In response to detecting the violation, the first agent can sanction the second agent. Sanctioning the second agent may include the first agent modifying the contract negotiation with the second agent in one or more time steps after the violation time step. For example, the first agent may stop the contract negotiation with the second agent after the violation time step, or may modify the determination of the expected value of the environment state of the first agent at a subsequent time step, such as adding a term representing the negative value of the expected value of the environment state to the second agent at a subsequent time step (in an attempt to reduce the reward of the deviating agent).

[0099] As an example, for using weights Sanctioning Agents The modified value estimate of the next state can be determined as In the "mutual proposal" protocol, this can be used to determine the modified simulated action value estimate (SAVE), which is used to compute the simulated value estimate (SVE). In the "proposal-take" protocol, a similar modification can be used when choosing whether to accept a contract, for example, instead of The modified value estimation of the next state can be extended to sanction multiple deviating agents, such as each using weights .

[0100] Figure 6 and 7 Examples of the benefits that can be obtained by implementing these techniques are shown. For an example task, each graph shows the total number of interacting agents on the x-axis and then the improvement expressed as a ratio relative to a single agent on the y-axis. Figure 6 It is the way in which the mutual proposal agreement is implemented; Figure 7 is an implementation of the proposal-selection protocol. These advantages are significant and grow as the number of agents increases.

[0101] The techniques are applicable, for example, to many different types of routing, scheduling, and control tasks, such as routing robotic or autonomous vehicles, power or data packets; scheduling computing jobs; and controlling robotic or manufacturing cells or machines that interact to manufacture products.

[0102] In some embodiments, the environment is a real-world environment, and at least the first agent, and in embodiments each agent, comprises a robot or an autonomous or semi-autonomous vehicle. The tasks performed by the agent or agents may then each comprise navigating a path through the environment and / or manipulating objects in the environment. The actions may comprise actions to control the movement of the agent in the environment, such as actions to control the speed or direction of movement (e.g., steering) of the agent, or actions performed by the robot or autonomous vehicle, such as collecting or delivering a payload, such as cargo or passengers, or manipulating objects in the environment.

[0103] The contract can constrain (or require) any such actions, for example with the goal of improving the overall efficiency of the task. For example, by restricting specific agents to specific routes or by limiting the number of agents on any specific route, congestion or delivery time or energy usage on the route can be reduced. By defining actions for agents, propose-take type protocols can be used to jointly optimize routes for multiple agents.

[0104] As another example, actions associated with objects in an environment, such as objects to be transported by agents, may include actions that require agents to meet and transfer one or more objects from one agent to another, such as in exchange for goods. Contracts associated with such actions, such as mutually beneficial mutual proposal type contracts, may be used to optimize the transportation of objects to their respective destinations.

[0105] The expected return and reward may be related to a measure of task performance such as time or delay or distance to a destination (e.g., from the agent's location) and / or a physical cost of performing the task such as fuel or energy usage to reach the destination and / or another cost such as costs associated with congestion on one or more specific routes. In embodiments, the method may attempt to maximize the reward or expected return, or minimize the cost. For example, the expected return may depend on, for example, the reward received for moving along a path within a specific time, or on the speed of movement of the agent, or on the energy used by the agent being less than a specific amount of energy.

[0106] The state of an environment including an agent may be characterized by a state representation derived from observations of the environment using any type of sensor, such as an image or video sensor (here including a LiDAR sensor); or a sensor of the agent's state, position, or configuration, or an environmental sensor, such as a sensor of the state, position, or configuration of one or more other agents; or, for example, a sensor of the average speed at which the agent is moving; or a congestion sensor, such as for sensing congestion on one or more possible routes that the agent could take; or a sensor of the location of static or dynamic objects such as trees, buildings, pedestrians, or traffic lights; or map data, such as for defining routes that the agent may take; or generally, a sensor or device that captures data related to the state, position, or configuration of objects in the environment.

[0107] As an example, the tasks performed by the agents may include delivering a payload such as cargo, e.g., one or more packages, or a passenger. For example, the agents may cooperate to deliver the payload to multiple different destinations in an energy-efficient manner and / or avoid congestion with each other in transit or upon arrival (e.g., by defining a minimum time interval between deliveries to a particular destination). The contract may involve actions such as determining which agent is to deliver which item and / or to which destination, or which routes to take or avoid.

[0108] As a specific example, an agent may include a robot or vehicle that performs tasks such as warehouse, logistics, or factory automation, such as collecting, placing, or moving stored goods or goods or parts of goods during their manufacturing process.

[0109] As another example, the agents may include electric taxis, and the agents may determine contracts that enable the taxis to cooperate in picking up passengers in a way that is time and energy efficient, for example taking into account available range.

[0110] As another example, the agents may include cars or trucks, and the agents may determine contracts that enable the vehicles to cooperate while traveling on a road, such as by defining rewards designed to maintain a maximum, minimum, or target distance between vehicles to promote smooth and safe traffic flow.

[0111] As another example, the vehicle may be an aerial vehicle, such as a drone. For example, in an agricultural environment, the mission may then include delivering a payload to plants, such as pesticides, seeds, water, or fertilizer. In an emergency response environment, the mission may include searching for personnel. In another environment, the mission may include a package delivery mission.

[0112] In general, the techniques may be used to manage large numbers of robots or autonomous or semi-autonomous vehicles operating on land, air, or water. In general, the simulators described previously may simulate the effects of actions of agents, such as vehicles, in an environment. For example, the simulator may take a current state and agent actions, and output data representing a next state. For example, the current state and agent actions may include the current location of the agent, as well as choices regarding the route taken by the agent (e.g., a left turn at an intersection may cause the vehicle to use one road instead of another), and the simulator may then determine the level of congestion on each road and the resulting travel time.

[0113] Actions and contracts may be associated with low-level, mid-level, or high-level actions. Examples of high-level actions include actions that specify locations that a robot or vehicle should move to or avoid; examples of mid-level actions include actions that specify which robot or vehicle moves first or has priority at a junction or intersection of a route. Low-level actions may specify a specific speed or direction for the agent. Typically, rewards, which define the expected return, may be based on an event or aspect of the environment. For example, a reward may indicate whether the agent has completed a task as described above, or the agent's progress toward completing a task (the specific rewards correspond to the tasks described above, and are therefore not repeated for the sake of brevity).

[0114] In some embodiments, the environment is a computing environment, and each agent includes a computing job scheduling agent, such as a software agent that manages the distribution of computing tasks across computing resources, such as on a mobile device, in a computer, in a computer cluster, or in a data center. Computing resources may include various hardware resources. Resources may include the following: different low-level resources, such as processors with different capabilities, such as one or more general-purpose processors (CPUs) or GPUs (graphics processing units) or TPUs (tensor processing units); or different intermediate resources, such as different computers working together in a computer cluster; or different higher-level resources, such as servers in a data center. Hardware resources may, but need not, have different computing or memory capabilities. Tasks performed by the agent may include executing a computer program. Actions may include the action of assigning one or more computing jobs associated with a task to one or more computing hardware resources. Contracts may restrict such actions, such as limiting the availability of hardware computing resources to specific agents, such as by time or resource type, or requiring specific agents to use specific resources. Expected returns and rewards may be related to task performance measures, such as the time to complete the task, or the computational cost of performing the task (e.g., which depends on the hardware implementing the task), or the power or energy cost of performing the task, or the reliability of performing the task (e.g., repeating the task to achieve fault tolerance). The state of the environment can be characterized by state representations derived from observations of the computing environment, such as the amount of computing work on hardware resources, memory usage / availability of hardware resources, physical state of hardware resources such as temperature, or whether a hardware resource has failed. Contracts may involve scheduling computing jobs, such as controlling the allocation of computing jobs to hardware resources. Simulators can simulate the effects of actions in the environment, such as the effects of allocating specific computing jobs to specific resources.

[0115] In some embodiments, the environment is a packet communication network environment, and the agents each include a router to route data packets through the communication network. The tasks performed by the agents may then include packet routing tasks. The actions may include routing actions to route data packets, such as actions for determining the next or further destination of one or more data packets. The contracts may restrict these actions, such as for coordinating packet transmission and reducing congestion, and generally control the routing of data packets through the communication network. The expected returns and rewards may be related to one or more packet routing metrics, such as the path from the router to the next or further node in the data packet communication network. Such metrics may include, for example, the estimated time, delay, speed, distance or number of hops for a group of one or more routed data packets to travel from the router to the next or further node in the data packet communication network, a measurement of routing utilization, routing packet loss or other reliability measurements, routing bandwidth or throughput, or routing maximum transmission unit (MTU). The state of the environment may be characterized by a state representation derived from observations of available data packet routes and their characteristics, such as characteristics corresponding to the above-mentioned metrics. For example, observations may include observations of the routing table of the router. The simulator may simulate the impact of different routing options. The environment of a packet communication network is similar in some respects to the real-world environment in which the mobile agents described previously are routed, and the concepts of this field can be applied accordingly to the packet communication network.

[0116] In some embodiments, the environment is a power distribution environment, and the agents are each configured to control the routing of power from a power distribution node, such as a node associated with the agent, to one or more other nodes via one or more power distribution links, such as in a "smart grid."

[0117] As the electricity distribution grid becomes more decentralized, for example due to the addition of multiple smaller capacity and potentially intermittent renewable generators, additional interconnections among generators and consumers may destabilize the grid and may cause overloads of links in the grid, particularly due to phase differences between connection points.

[0118] The tasks performed by the agent include tasks to distribute electricity from generators to electricity consumers. These actions include control actions for controlling the routing of electricity between nodes. For example, these actions may include actions to connect or disconnect a power source or sink to the grid, or actions to control the routing of electricity through links between grid nodes, or actions to control the frequency or phase of power generators, or actions to control frequency mismatches or phase mismatches between grid elements connected, for example, via nodes. The contract may coordinate these actions, for example to improve the stability of the grid and reduce the risk of grid overload; and may generally be related to controlling the routing of electricity. The expected returns and rewards may be related to one or more power routing metrics, such as power loss within the grid, or frequency or phase mismatches on one or more power distribution links, or risk of overload or failure, or risk of exceeding a threshold on one or more power distribution links.

[0119] The state of the environment can be characterized by a state representation derived from observations from sensors that sense the state of the power grid, such as sensing frequency, phase, current, voltage, electrical load, resistance, or impedance associated with one or more connections between nodes, or static observations of the configuration of the power grid, such as elements coupled to the power grid and power distribution links. The simulator can simulate the effects of different power control or routing action choices.

[0120] In some embodiments, the environment is a real-world manufacturing environment for manufacturing products, such as chemical, biological or mechanical products or food. As used herein, "manufacturing" a product also includes refining starting materials to produce a product, or processing starting materials, such as to remove contaminants to generate a clean or recycled product. A manufacturing plant may include multiple manufacturing units, such as containers for chemical or biological substances, or machines such as robots for processing solids or other materials. The manufacturing units may be configured so that intermediate versions or components of a product can be moved between manufacturing units during product manufacturing, such as via pipes or mechanical transport.

[0121] In such an embodiment, at least the first agent, and in an embodiment each agent, includes a control system that is configured to control the manufacture of a mechanical, chemical or biological product. For example, the control system can be configured to control one or more manufacturing units or machines, or to control the movement of intermediate versions or components of a product between manufacturing units or machines. Typically, manufacturing units or machines interact to manufacture products. Typically, the tasks performed by the agent or agents include the tasks of manufacturing the mechanical, chemical or biological product or another mechanical, chemical or biological product or its intermediates or components. That is, agents can interact to control the manufacture of the same product, or they can control the manufacture of different products. As used herein, product manufacturing includes manufacturing food products by one or more kitchen robots.

[0122] Actions may include control actions for controlling the use of a machine or manufacturing unit to process solid or liquid materials to manufacture a product or intermediate or component thereof, or controlling the movement of intermediate versions or components of a product within a manufacturing environment, such as between manufacturing units or machines. In general, an action may be any action that has an effect on the observed state of the environment, i.e., an action configured to adjust any of the sensed parameters described below. These may include actions to adjust the physical or chemical conditions of a manufacturing unit, or actions to control the movement of mechanical parts of a machine or robotic joints, or actions to control the movement of materials between manufacturing units. Contracts may restrict such actions with the purpose of improving manufacturing efficiency, such as by reducing competition for the use of a manufacturing unit or machine; and manufacturing conditions and / or the movement of materials within a manufacturing environment may generally be controlled. Aspects of this area are again similar to those previously described, and corresponding concepts apply.

[0123] Expected returns and rewards may be related to measures of task effectiveness, such as the number of products manufactured, the quality of the products, the rate at which products are produced, or measures of the physical costs of performing the manufacturing task, such as the amount of energy, materials, or other resources used to perform the task.

[0124] The characteristics of the state of the environment may be a representation of the state derived from observations from sensors that sense the state of the manufacturing environment, such as sensors that sense the state or configuration of a manufacturing cell or machine, or sensors that sense the movement of materials between manufacturing cells or machines. As some examples, such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow / movement rate, or flow / movement path of one or more materials; physical or chemical conditions such as physical state, shape or configuration or chemical state such as pH; configuration of a cell or machine, such as a mechanical configuration or valve configuration of a cell or machine; image or video sensors for capturing image or video observations of a manufacturing cell or machine or movement; or any other suitable type of sensor. In the case of a machine such as a robot, observations from sensors may include observations of the position, linear or angular velocity, force, torque or acceleration, or posture of one or more parts of the machine, such as data characterizing the current state of the machine or robot or an item held or handled by the machine or robot. Observations may also include, for example, sensed electronic signals such as motor current or temperature signals, or, for example, image or video data from a camera or lidar sensor. Sensors such as these can be part of the agent in the environment or placed separately from the agent. The simulator can simulate the effects of different choices of the above actions in a manufacturing environment.

[0125] For a system of one or more computers configured to perform a specific operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which in action causes the system to perform the operation or action. For one or more computer programs configured to perform a specific operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0126] The embodiments and functional operations of the subject matter described in this specification can be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more computer program instruction modules, which are encoded on a tangible non-transitory program carrier for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiving device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0127] The term "data processing apparatus" refers to data processing hardware, covering all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an action system, or a combination of one or more of them.

[0128] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to a related program, or in multiple coordinated files, such as files storing one or more modules, subroutines, or portions of code. A computer program may be deployed for execution on one computer located at one site or on multiple computers distributed across multiple sites and interconnected by a communications network.

[0129] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuit systems, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0130] Computers suitable for executing computer programs include, for example, central processing units that can be based on general or special microprocessors or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to receive data from or transmit data to or to one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer need not have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0131] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CDROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in a special purpose logic circuit system.

[0132] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by a user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0133] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer having a relational graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0134] A computing system may include clients and servers. Clients and servers are generally remote from each other and generally interact through a communication network. The relationship of client and server arises through computer programs acting on the respective computers and having a client-server relationship to each other.

[0135] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on any invention or the scope that may be claimed, but should be interpreted as descriptions of the features of specific embodiments of specific inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described as working in certain combinations and even initially claimed, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may be directed to a sub-combination or a variant of the sub-combination.

[0136] Similarly, although the actions are depicted in a particular order in the figure, it should not be understood as requiring that the actions be performed in the particular order shown or in a sequential order, or that all of the actions shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged in multiple software products.

[0137] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method of controlling a first agent to perform a task in an environment in which the first agent interacts with one or more other agents performing one or more other tasks, in, the first agent having an action selection subsystem and a value neural network, the action selection subsystem being configured to process a state representation representing a state of the environment to generate a policy output for selecting an action to be performed by the agent, the value neural network being configured to process the state representation to determine a state value, wherein the state value defines an expected reward of the agent from the state of the environment represented by the state representation; Wherein, the first agent at each time step in a plurality of time steps: obtaining a state representation characterizing the state of the environment at a current time step; processing the state representation using the action selection subsystem of the agent to generate the policy output for the first agent; Determining a predicted action for the other agent using the state representation; and using the policy output for the first agent to select an action for the first agent to perform depending on the predicted actions of the other agents; The method comprises, at one or more of the time steps: The first agent negotiates a contract with a second agent, wherein the second agent is one of the other agents, the contract defining restrictions on actions that may be selected by the first agent and restrictions on actions that may be selected by the second agent; Wherein, negotiating the contract includes the first agent determining whether to propose a contract by performing the following operations: determining one or more scores representing expected values ​​of the state of the environment of the first agent at subsequent time steps with and without the proposed contract by simulating the effects of future actions of the first agent and the other agents (i) in the absence of the proposed contract and ii) taking into account the effects of the restrictions on the actions in the proposed contract, and transmitting the proposed contract to the second agent depending on the one or more scores; and In response to receiving an indication from the second agent to accept the proposed contract, an action is selected for the first agent to perform taking into account the restrictions on the action of the first agent in the proposed contract.

2. The method according to claim 1, further comprising: include: The first agent determines whether to accept the proposed contract; and wherein selecting the action of the first agent taking into account the restrictions on the action of the first agent in the proposed contract is further dependent on the first agent accepting the proposed contract.

3. The method according to claim 1 or claim 2, in, Determining the expected value of the state of the environment of the first agent at the subsequent time step includes: sampling one or more possible actions of the first agent at the time step using the action selection subsystem, and, for each of the one or more actions and for each other of the other agents: sampling a plurality of possible actions of the other agent at the time step; and determining, using the value neural network, a state value for a predicted new state of the environment determined by the sampled actions of the agent; and wherein Simulating the impact of the future action includes: averaging the state value of each predicted new state over the sampled actions of each of the other agents to determine the expected value of the state of the environment of the first agent for each of the one or more possible actions sampled by the first agent.

4. The method according to claim 1, claim 2 or claim 3, in, Simulating the future actions of the agents includes, for each agent: A set of one or more candidate actions is selected for the agent by i) using the action selection subsystem, and or ii) by applying the restrictions on the actions in the proposed contract if the proposed contract applies to the agent.

5. The method according to any one of claims 1 to 4, in, The contract defines a class of actions that define the same or corresponding restrictions on the actions that may be selected by both the first agent and the second agent; the method further comprising: determining a first score representing an expected value of the state of the environment of the first agent at a subsequent time step with the proposed contract and a second score representing an expected value of the state of the environment of the first agent at a subsequent time step without the proposed contract; comparing the first score and the second score; and When the first score exceeds the second score, it is determined to transmit the proposed contract.

6. The method according to any one of claims 1 to 4, in, The contract defines a first action that must be selected by the first agent and a second action that must be selected by the second agent; wherein the one or more scores include an agreement score, the agreement score representing an expected value of the state of the environment for both the first agent and the second agent at the subsequent time step when the first agent and the second agent select the first action and the second action, respectively, the method further comprising: determining respective first and second baseline values, the first and second baseline values ​​representing the expected values ​​of the state of the environment of the first and second agents at the subsequent time steps without restricting the second agent to the second action and without restricting the first agent to the first action, respectively; and The agreement score is determined relative to the baseline value.

7. The method according to claim 6, in, The agreement score comprises the product of: i) a difference between the expected value of the state of the environment of the first agent at the subsequent time step and the first baseline value when the first agent and the second agent select the first action and the second action, respectively; and ii) a difference between the expected value of the state of the environment of the second agent at the subsequent time step and the second baseline value when the first agent and the second agent select the first action and the second action, respectively.

8. The method according to claim 6 or claim 7, in, The first baseline value includes the expected value of the state of the environment of the first agent at the subsequent time step when the first agent selects the first action and the action selected by the second agent is not restricted; and wherein the second baseline value includes the expected value of the state of the environment of the second agent at the subsequent time step when the second agent selects the second action and the action selected by the first agent is not restricted.

9. The method according to claim 6 or claim 7, in, Determining the first baseline value includes: for each agent pair other than the pair including the first agent and the second agent, identifying for the subsequent time step whether they will reach a contract, and, when they will reach a contract, determining the first baseline value as the expected value of the state of the environment of the first agent at the subsequent time step as if the reached contract had been established; and Wherein, determining the second baseline value includes: for each pair of agents except the pair including the first agent and the second agent, identifying whether they will reach a contract for the subsequent time step, and, when they will reach a contract, determining the second baseline value as the expected value of the state of the environment of the second agent at the subsequent time step as if the reached contract has been established.

10. The method according to any one of claims 6 to 9, further comprising: include: A proposed contract for the second agent is determined by determining a plurality of candidate proposed contracts and selecting the candidate proposed contract with the highest agreement score as the proposed contract.

11. The method according to any one of claims 6 to 10, further comprising: include: Determining the proposed contract of the second agent by determining a plurality of candidate proposed contracts and selecting one of the candidate proposed contracts as the proposed contract, wherein determining the candidate proposed contract comprises: determining a first set of candidate actions for the first agent by processing the state representation at the current time step using the action selection subsystem to generate the policy output for the first agent; determining a second set of candidate actions for the second agent by processing the state representation at the current time step using the action selection subsystem to generate the policy output for the second agent; generating an action pair, the action pair comprising a first candidate action from the first candidate action set and a second candidate action from the second candidate action set; and Determining the candidate proposed contracts by selecting a set of candidate action pairs via: ranking the action pairs according to a pair metric and selecting K highest ranked action pairs; wherein each candidate proposed contract includes one of the candidate action pairs; and Wherein, the pair metric of the action pair includes a combination of the expected value of the state of the environment of the first agent at the subsequent time step when the first agent performs the first candidate action in the action pair, and the expected value of the state of the environment of the second agent at the subsequent time step when the second agent performs the second candidate action in the action pair.

12. The method according to claim 11, in, The combination is a weighted combination to weight the value of the proposed contract of the second agent.

13. The method according to any one of claims 1 to 12, further comprising: include: detecting, by the first agent, that the second agent has breached the contract at a breach time step; and, in response, Sanctioning the second agent, wherein the sanctioning includes the first agent modifying the negotiation of the contract with the second agent at a time step subsequent to the breach time step.

14. The method according to any one of claims 1 to 13, in, The environment is a real-world environment, wherein at least the first agent comprises a robot or an autonomous vehicle, wherein the task and each of the other tasks comprises navigating a path through the environment, wherein the actions comprise actions to control the movement of the agent in the environment, and wherein the expected reward is related to a measure of one or more of: the effectiveness of the task; and the physical cost of performing the task.

15. The method according to any one of claims 1 to 13, in, The environment is a computing environment, wherein each of the agents comprises a computing job scheduling agent, wherein the task and each of the other tasks comprises executing a computer program, wherein actions comprise actions of assigning one or more jobs to one or more computing hardware resources, and wherein the expected reward is related to a measure of time to perform the task, an energy cost to perform the task, a computational cost to perform the task, and a reliability of performing the task.

16. The method according to any one of claims 1 to 13, in, The environment is a packet communication network environment, wherein each of the agents comprises a router for routing data packets through the communication network, wherein the task and each of the other tasks comprises a packet routing task, wherein the action comprises a routing action for routing the data packets, and wherein the expected reward is related to one or more packet routing metrics.

17. The method according to any one of claims 1 to 13, in, The environment is an electric power distribution environment, wherein each of the agents is configured to control the routing of electric power from a node associated with the agent to one or more other nodes via one or more electric power distribution links, wherein the task and each of the other tasks comprises a task of distributing electric power from a generator to electric power consumers, wherein actions comprise control actions for controlling the routing of electric power between the nodes, and wherein the expected reward is related to a loss on one of the electric power distribution links, or to a frequency or phase mismatch on one of the electric power distribution links, or to an overload on one of the electric power distribution links.

18. The method according to any one of claims 1 to 13, in, The environment is a real-world manufacturing environment, wherein at least the first agent includes a control system configured to control the manufacture of a mechanical, chemical or biological product, wherein the task and each of the other tasks includes the task of manufacturing the mechanical, chemical or biological product or another mechanical, chemical or biological product or an intermediate or component thereof, wherein actions include control actions for controlling the use of a machine or manufacturing unit for processing solid or liquid materials to manufacture the product or its intermediate or component, or controlling the movement of the product or its intermediate or component in the manufacturing environment, and wherein the expected reward is related to a measure of one or more of: the effectiveness of the task; and the physical cost of performing the task.

19. Computer-readable instructions, or one or more computer storage media storing computer-readable instructions, which, when executed by one or more computers, cause the one or more computers to implement the method of any one of claims 1 to 18.

20. A system comprising one or more computers and one or more storage devices storing instructions, wherein when the instructions are executed by the one or more computers, the one or more computers are caused to perform corresponding operations of the method according to any one of claims 1 to 18.

Citation Information

Cited By

  • Unmanned cluster distributed intelligent routing method and device and electronic equipment

    CN119629697A