Optimal policy generation method for multi-agent cooperation based on two-stage intention sharing

Abstract

The application discloses a kind of optimal strategy generation method of multi-agent cooperation based on two-stage intention sharing, at each decision time, let each agent first generate intention action according to its own observation, and broadcast intention action, after broadcast ends, agent receives intention information from other agents, then each agent independently calculates the importance of other agent intention information, marks the agent with sufficient importance in it as dependent object, and broadcasts dependent object set, after broadcast ends, agent receives dependent object information from other agents, establishes dependent relationship graph according to dependent object information, removes circular dependency in dependent relationship graph using a greedy algorithm, and finally each agent redecides according to intention action information according to dependent relationship graph. Using this method can end-to-end automatically generate optimal strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of optimal strategy generation method of multi-agent cooperation based on two-stage intention sharing, belong to software development technical field. BACKGROUND

[0002] Under continuous decision problem, reinforcement learning technology has great potential, in some complex scenarios (such as go, electronic games), the agent trained by reinforcement learning technology has even exceeded the level of human experts. Multi-agent reinforcement learning technology is the presentation of reinforcement learning technology under multi-agent system, also has the potential without doubt, the process of training multi-agent strategy using multi-agent reinforcement learning technology can be end to end, it is helpful to automatically generate optimal strategy, so it is gradually favored by researchers.

[0003] But multi-agent reinforcement learning technology still has great defects in improving the cooperation ability of agent, and more research and innovation are needed in this regard. In the human world, people cooperate by exchanging intentions, and multi-agent reinforcement learning technology is still unclear about how to introduce intention exchange. SUMMARY

[0004] The present application provides an optimal strategy generation method for multi-agent cooperation based on two-stage intention sharing, which is a method for improving the cooperation ability between agents by exchanging intentions.

[0005] Technical scheme: an optimal strategy generation method for multi-agent cooperation based on two-stage intention sharing, at each decision time, each agent generates an intended action according to its own observation, and broadcasts the intended action, after the broadcast is over, each agent will receive intention information from other agents;

[0006] Each agent independently calculates the importance of other agent's intention information, marks the agent with sufficient importance as a dependent object, and broadcasts the dependent object set, after the broadcast is over, each agent will receive dependent object information from other agents, establishes a dependency relationship graph according to the dependent object information, removes the circular dependency in the dependency relationship graph using a greedy algorithm, and finally each agent redecides according to the intended action information according to the dependency relationship graph.

[0007] The above method specifically includes the following steps:

[0008] Step 1: each agent (agent needs to have observation ability, communication ability and computing ability) generates an intended action according to its own observation data, assuming that there are N agents in total, and the intended action set generated by all agents is P, P={p1,p2,…p i..., p N}, i = 1,..., N, where N is the number of agents, p i represents the intended action of the i-th agent;

[0009] Step 2: Each agent broadcasts its intended action p to other agents, and let M i represent the messages received by the i-th agent after the broadcast ends. i M i-1 = {p1, p2,..., p i+1 ,..., p N};

[0010] Step 3: Each agent calculates the importance of the received intention information, and let D i represent the importance calculation result of the i-th agent, D i = {d i1 , d i2 ,..., d ij ,..., d iN}, j = 1,..., i-1, i+1,..., N, where d ij represents the importance of the intention information of agent j to agent i;

[0011] Step 4: Introduce a hyperparameter γ, representing the importance threshold, and each agent filters out the dependent object set at this step according to D i and γ, and let V i represent the dependent object set of the i-th agent, V i = {j | j ≤ N ∧ d ij > γ};

[0012] Step 5: Each agent broadcasts its dependent object set V i to other agents, and let B represent the dependency graph obtained by each agent after the broadcast ends, B ij = 1 indicates that agent i depends on agent j, and the directed loop in B is removed to avoid circular dependency, and let the result be W.

[0013] Step 6: Each agent makes a new decision based on the dependency graph and the intended action information. To avoid intention misdirection, the agent being depended on in the dependency graph is prohibited from making a new decision, and its final action is its intended action; the agent not being depended on (assuming it is the i-th agent) makes a new decision based on W i and M i to obtain the final action, and let A represent the final joint action of all agents, A = {a1, a2,..., a i ,..., a N}, i = 1,..., N, A is the decision output of the agent at this step.

[0014] In step 1 above, each agent generates an intentional action based on its own observation data. Each agent maintains a value network C to estimate the expected cumulative reward that can be obtained by performing each action under the current observation data. The intentional action of the i-th agent is given by formula p. i =argmax a C i (o i a) is given, where C i The value network representing agent i, o i 'a' represents the data currently observed by the agent, and 'a' represents the action currently being performed by the agent.

[0015] In step 2 above, each agent broadcasts its intended action p to other agents. It is assumed that communication conditions are good and there are no communication obstacles. After the broadcast, the message received by the i-th agent is M. i ={p1,p2,…p i-1 ,p i+1 ,…,p N}

[0016] Step 3 above, where each agent calculates the importance of the received intent information, takes the calculation of the importance of agent j's intent information by agent i as an example, and specifically includes the following steps:

[0017] Step 3.1: Agent i maintains a value network D i D i (o i ,a - a) represents the observed data o i Below, the actions of the remaining agents are a - At that time, the expected reward for agent i to perform action a;

[0018] Step 3.2: Place M i p in j Replace with k, and denote the result as M. ijk M ijk ={p1,…,p j-1 ,k,p j+1 ,…,p N};

[0019] Step 3.3: Calculate in D i Estimate the optimal action u of agent i. ijk =argmax a D i (o i M ijk ,a);

[0020] Step 3.4: Repeat the calculation to obtain the optimal action set u ij = {u ijk | k e U}, where U represents the action space of the agent;

[0021] Step 3.5: For a given action k, calculate the maximum value a ijk = max u∈uij D i (o i , M ijk , u) and the minimum value b ijk = min u∈uij D i (o i , M ijk , u) when the agent i takes different optimal actions;

[0022] Step 3.6: Normalize a ijk and b ijk using the softmax function, and denote the normalized distribution as q ijk , q ijk = softmax[a ijk , b ijk ];

[0023] Step 3.7: Calculate the maximum value of the standard deviation of the distribution q ijk as the importance of agent j relative to agent i, denoted as d ij , d ij = max(std(q k∈U )).

[0024] Step 5 above, each agent broadcasts its dependency object set V i to other agents, assuming that the communication condition is good and there is no obstacle in communication. Denote the dependency graph obtained by each agent after broadcasting as the adjacency matrix B, B ij = 1 indicates that agent i depends on agent j, remove the directed loop in B to avoid circular dependency, and denote the result as W. The specific steps to remove the directed loop are as follows:

[0025] Step 5.1: Initialize the W matrix as the B matrix;

[0026] Step 5.2: Select the node set b with out-degree 0 in B;

[0027] Step 5.3: If b is empty, find the node b min with the smallest out-degree and the smallest number in B min , delete the outgoing edge of node b min from W, and then add b i to b;

[0028] Step 5.4: delete all nodes in b from B;

[0029] Step 5.5: repeat steps 5.2-5.4 until B is empty.

[0030] The above step 6, each agent according to the dependency graph and the intended action information to re-decision, in order to avoid the emergence of intention misleading, prohibit the dependent agent (assuming the first i agent) in the dependency graph to re-decision, its final action is its intended action, that is a i = p i . The agent (assuming the first i agent) is not dependent on the re-decision according to W i and M i to get the final action, in particular, each agent i maintains a value network Q i , Q i (o i , E i , a) represents the observation data o i , the agent i's dependent object's intention information is E i , the agent i executes the action a's expected reward, then the agent i's final action a i = argmax a Q i (o i , E i , a). Note that the final joint action of all agents is A, A = {a1, a2, … a i , …, a N}, i = 1, …, N, A is the decision output of the agent at this step.

[0031] The above-mentioned value network C, D, Q are updated by Double Q-Learning algorithm.

[0032] The agent is a vehicle, the vehicle has an observation data acquisition device and a communication device for signal transmission, and a computing device for running the optimal strategy generation method.

[0033] Beneficial Effects: At each decision-making moment, this invention allows each agent to generate an intentional action based on its own observations, and then broadcasts this intentional action. After the broadcast, the agent receives intentional information from other agents. Next, each agent independently calculates the importance of the intentional information from other agents, marks agents with sufficiently high importance as dependent objects, and broadcasts the set of dependent objects. After the broadcast, the agent receives dependent object information from other agents, builds a dependency graph based on the dependency object information, and uses a greedy algorithm to remove circular dependencies in the dependency graph. Finally, each agent re-decides based on the intentional action information according to the dependency graph. This invention utilizes intention communication to optimize multi-agent policy generation, enhancing the collaborative ability of trained agents. This method can automatically generate optimal policies end-to-end, representing a novel approach in the field of multi-agent reinforcement learning. Attached Figure Description

[0034] Fig. 1 This is a flowchart of the method of the present invention;

[0035] Fig. 2 This is a logic diagram of the present invention. Detailed Implementation

[0036] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0037] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0038] like Figs. 1-2 As shown, the optimal strategy generation method of the present invention based on two-stage intent sharing and multi-agent cooperation mainly includes the steps of agent intent generation and communication in each round of decision-making in a multi-agent system.

[0039] In the intention generation step, each agent generates an intended action according to the observation of the current time point, and broadcasts it as intention information to all other agents, completing the first round of communication. After the first round of communication, each agent has received the intentions of other agents, then the agent evaluates the importance of these intentions, marks the sender of the sufficiently important intention as the dependent object at the current decision time, and broadcasts the dependent object information, completing the second round of communication. After the second round of communication, each agent can establish a dependency relationship graph according to the dependent object information and remove the circular dependency, and finally make a new decision according to the intention information, so as to achieve the purpose of enhancing multi-agent cooperation through exchanging intentions.

[0040] It should be noted that the hardware environment of the present application includes: a router or switch capable of normal access to a wide area network for data transmission between agents, and a computing device capable of normal use for running the optimal strategy generation method. The software environment includes: Python3.9.

[0041] The embodiment of the present application takes the traffic intersection passing scene as an example, and there are several vehicles controlled by automatic driving (agents) in the scene. The observation data of the agent includes the position and direction of all vehicles near the intersection, the forward route (straight, left turn or right turn) of the agent, the action space of the agent U={1,2}, wherein 1 represents stopping and yielding, and 2 represents driving forward for a distance according to the route of the agent. The agents in the scene need to quickly and safely drive out of the intersection. The optimal strategy generation method for multi-agent cooperation based on two-stage intention sharing provided by the embodiment of the present application includes the following steps:

[0042] Step 1: Each agent decides whether to stop and yield at the decision time according to the observation data, assuming that there are N agents in total, and the set of intended actions generated by all agents is denoted as P, P={p1,p2,…p i ,…,p N},i=1,…,N, wherein N is the number of agents, p i represents the intended action of the i-th agent, and the value range is {1,2};

[0043] Step 2: Each agent broadcasts its intended action p to other agents, and the message received by the i-th agent after broadcasting is denoted as M i , M i ={p1,p2,…p i-1 ,p i+1 ,…,p N};

[0044] Step 3: Each agent calculates the importance of the received intention information, and the importance calculation result of the i-th agent is denoted as D i , D i={d i1 ,d i2 ,…d ij ,…,d iN}, j=1,…,i-1,i+1,…,N, where d ij This indicates the importance of agent j's intention information to agent i, with a value range of [0,1). The larger the value, the higher the importance.

[0045] Step 4: Introduce the hyperparameter γ = 0.1, representing the importance threshold. Each agent determines the importance threshold based on D. i The set of dependent objects for this step is selected by γ, and the set of dependent objects for the i-th agent is denoted as V. i V i ={j|j≤N∧d ij >0.1};

[0046] Step 5: Each agent broadcasts its set of dependent objects V to other agents. i Let the dependency graph obtained by each agent after the broadcast ends be the adjacency matrix B. ij =1 indicates that agent i depends on agent j. Remove the directed loop in B to avoid circular dependencies, and denote the result as W;

[0047] Step 6: Each agent makes a new decision based on the dependency graph and intention action information. To avoid intention misdirection, agents that are dependent on each other in the dependency graph are prohibited from making new decisions; their final action is their intention action (e.g., an agent that initially decided to stop and yield cannot continue moving forward); agents that are not dependent on each other (let's say the i-th agent) make new decisions based on W. i and M i A new decision is made to arrive at the final action (e.g., the original decision was to stop and yield, but after observing that other vehicles have stopped to yield, a new decision can be made to continue). Let the final joint action of all agents be A, where A = {a1, a2, ... a...}. i ,…,a N}, i = 1, ..., N, where A is the agent's decision output at this step, a x The value range is {1,2}. Then, each agent executes an action according to the final decision, either stopping to give way or continuing to move forward.

[0048] When applying the optimal policy generation method for multi-agent cooperation based on two-stage intent sharing disclosed in this invention, a certain amount of reinforcement learning training process is required before the trained agent policy can be applied to a specific multi-agent scenario to solve a specific multi-agent task.

[0049] A typical application scenario for this invention is providing decision-making support for autonomous vehicles. For example, in a traffic intersection, there are several autonomous vehicles that need to pass through the intersection quickly and safely. In this case, the method provided by this invention can be used for training, and then the strategy can be deployed to the vehicles. By communicating driving intentions between the vehicles, the goal of quickly and safely passing through the intersection can be achieved.

[0050] In this embodiment, in step 1, each agent decides whether to stop and yield based on its own observation data. Each agent maintains a value network C to estimate the expected cumulative reward for performing two different actions under the current observation. The intended action of the i-th agent is given by formula p. i =argmax a C i (o i a) is given, where C i The value network representing agent i, o i This represents the agent's current observation data, i.e., choosing the action that maximizes the expected reward.

[0051] In step 2, each agent broadcasts its intended action p to other agents. It is assumed that communication conditions are good and there are no communication obstacles. After the broadcast, the message received by the i-th agent is M. i ={p1,p2,…p i-1 ,p i+1 ,…,p N}

[0052] In step 3, each agent calculates the importance of the received intent information. Taking the calculation of the importance of the intent information of agent j by agent i as an example, the specific steps include the following:

[0053] Step 3.1: Agent i maintains a value network D i D i (o i ,a - ,a) represents the observation o i Below, the actions of the remaining agents are a - At that time, the expected reward for agent i to perform action a;

[0054] Step 3.2: Place M i p in j Replace them with 1 and 2 respectively (representing two different actions), and denote the result as M. ij1 and M ij2 M ij1 ={p1,…,p j-1 ,1,p j+1 ,…,p N}, Mij2 = {p1,..., p j-1 ,2, p j+1 ,..., p N} ;

[0055] Step 3.3: Calculate the optimal action u i for agent i respectively in D ij1 = argmax a D i (o i , M ij1 , a) and u ij2 = argmax a D i (o i , M ij2 , a) ;

[0056] Step 3.4: Get the optimal action set u ij = {u ij1 , u ij2} ;

[0057] Step 3.5: For a given action k (1 or 2), calculate the maximum value a ijk

[0058] = max u∈uij D i (o i , M ijk , u) and the minimum value b ijk = min u∈uij D i (o i , M ijk , u) ;

[0059] Step 3.6: Normalize a ijk and b ijk using the softmax function, and denote the normalized distribution as q ijk , q ijk = softmax[a ijk , b ijk ] ;

[0060] Step 3.7: Calculate the maximum value of the standard deviation of the distribution q ijk as the importance of agent j relative to agent i, denoted as d ij , d ij = max{std(q ij1 ), std(q ij2 )}.

[0061] In step 4, a hyperparameter γ = 0.1 is introduced to represent the importance threshold. Each agent determines the importance threshold based on D. i The set of dependent objects for this step is selected by γ, and the set of dependent objects for the i-th agent is denoted as V. i V i ={j|j≤N∧d ij >0.1}.

[0062] In step 5, each agent broadcasts its own set of dependent objects V. i Here, it is assumed that the communication conditions are good and there are no communication obstacles. Let the dependency graph obtained by each agent after the broadcast be the adjacency matrix B. ij =1 indicates that agent i depends on agent j. To avoid circular dependencies, we remove the directed cycle in B, and denote the result as W. The specific steps for removing the directed cycle are as follows:

[0063] Step 5.1: Initialize the W matrix as the B matrix;

[0064] Step 5.2: Select the set b of nodes in B with an out-degree of 0;

[0065] Step 5.3: If b is empty, find the node b in B with the smallest out-degree and the smallest node number. min , will node b min Remove the outgoing edge from W, then b min Add b;

[0066] Step 5.4: Delete all nodes in b from B;

[0067] Step 5.5: Repeat steps 5.2-5.4 until B is empty.

[0068] In step 6, each agent makes a new decision based on the dependency graph and intention action information. To avoid intention misdirection, the agent that is depended upon in the dependency graph (let's say the i-th agent) is prohibited from making a new decision; its final action is its intention action, which is a. i =p i (For example, an agent that initially decided to stop and yield cannot continue moving forward). The undependent agent (let's say the i-th agent) proceeds according to W. i and M i The final action is obtained through a re-decision process; specifically, each agent i maintains a value network Q. i Q i (o i E i ,a) represents the observation o i Below, the intent information of the dependent object of agent i is E. iThe expected reward of the agent i performing action a at this time, then the final action a of the agent i at this time i = argmax a Q i (o i ,E i ,a) (for example, originally the decision is to stop and give way, but it is found that other vehicles all stop and give way for themselves, so it can be re-decided to continue to go forward). Record the final joint action of all agents as A, A = {a1, a2, … a i ,…,a N}, i = 1, …, N, A is the decision output of the agent at this step, a x The value range of a is {1, 2}, then each agent performs the action according to the final decision, respectively stops and gives way or continues to go forward.

[0069] The value networks C, D, and Q mentioned above are updated by Double Q-Learning algorithm.

[0070] It is apparent for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all aspects as illustrative and not restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended to encompass all changes falling within the meaning and range of equivalents of the elements of the claims. Any reference signs in the claims should not be considered as limiting the involved claims.

[0071] In addition, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that those skilled in the art can understand.

Claims

1. A method for optimal policy generation based on two-stage intention sharing multi-agent collaboration, characterized in that, At each decision-making moment, each agent generates an intended action according to its own observation, and broadcasts the intended action, after the broadcast ends, each agent will receive the intention information from other agents; Each agent independently calculates the importance of the intention information of other agents, marks the agent whose importance meets the requirements as a dependent object, and broadcasts the dependent object set, after the broadcast ends, each agent will receive the dependent object information from other agents, establishes a dependency relationship graph according to the dependent object information, removes the circular dependency in the dependency relationship graph by using a greedy algorithm, and finally each agent redecides according to the intention action information according to the dependency relationship graph; The agent is a vehicle, the vehicle has an observation data acquisition device, a communication device for signal transmission, and a running optimal strategy generation method calculation device; In the traffic intersection passing scene, there are several vehicles controlled by automatic driving, the observation data of the agent includes the positions and directions of all vehicles near the intersection, and the forward route of the agent, the action space of the agent U={1,2}, wherein 1 represents stopping and giving way, and 2 represents driving forward for a distance according to the route of the agent; The agent in the scene needs to quickly and safely drive out of the intersection; At each decision-making moment, each agent generates an intended action according to its own observation, and broadcasts the intended action, after the broadcast ends, each agent will receive the intention information from other agents, including the following steps: Step 1: Each agent decides whether to stop and yield at the decision moment based on its own observation data. Assume there are a total of N agents, and let P be the set of intentional actions generated by all agents, where P = {p1, p2, ..., p...} i ,…,p N }, i = 1, ..., N, where N is the number of agents, p i This represents the intended action of the i-th agent, with a value range of {1,2}; Step 2: Each agent broadcasts its intended action p to other agents, and let M be the message received by the ith agent after the broadcast is over i , M i ={p1,p2,…p i-1 ,p i+1 ,…,p N} Each agent independently calculates the importance of the intention information of other agents, marks the agent whose importance meets the requirements as a dependent object, and broadcasts the dependent object set, after the broadcast ends, each agent will receive the dependent object information from other agents, establishes a dependency relationship graph according to the dependent object information, removes the circular dependency in the dependency relationship graph by using a greedy algorithm, and finally each agent redecides according to the intention action information according to the dependency relationship graph, including the following steps: Step 3: Each agent calculates the importance of the received intention information, and the importance calculation result of the i-th agent is denoted as D i , i D i1 ={d i2 ,d ij ,…d iN},j=1,…,i-1,i+1,…,N, where d ij represents the importance of the intention information of agent j to agent i; Step 4: Introduce a hyper-parameter γ, representing the importance threshold, each agent selects the dependent object set of this step according to D i and γ, and the dependent object set of the i-th agent is denoted as V i . i = {j | j ≤ N ∧ d ij > γ} ; Step 5: Each agent broadcasts its set of dependency objects V to other agents i , and the dependency graph obtained by each agent after broadcasting is denoted as an adjacency matrix B, B ij = 1 indicates that agent i depends on agent j, and the directed loop in B is removed to avoid circular dependency, and the result is denoted as W; Step 6: Each agent re-decides according to the dependency graph and the intended action information; in order to avoid the occurrence of intention misleading, the agent which is depended in the dependency graph is prohibited to re-decide, and its final action is its intended action; the agent i which is not depended re-decides according to W i and M i to obtain the final action, and the final joint action of all agents is denoted as A, A = {a1, a2, … a i , …, a N}, i = 1, …, N, and A is the decision output of the agent at this step.

2. The method of claim 1, wherein, In the step 1, each agent generates an intended action according to its own observation data, and each agent respectively maintains a value network C for estimating the expectation of the cumulative reward finally harvested by performing each action under the current observation data, and the intended action of the i-th agent is given by the formula p i = argmax a C i (o i ,a) where C i represents the value network of the agent i, o i represents the current observation data of the agent, and a is the action performed by the current agent.

3. The method of claim 1, wherein, In step 3, each agent calculates the importance of the received intention information, and for agent i, the importance of the intention information of agent j is calculated, including the following steps: Step 3.1: Agent i maintains a value network D i , D i (o i , a - , a) represents the expected reward for agent i to perform action a given observation data o i and the remaining agents' actions a - . Step 3.2: Place M i p in j Replace with k, and denote the result as M. ijk M ijk ={p1,…,p j-1 ,k,p j+1 ,…,p N }; Step 3.3: Compute the optimal action u for the next state s' in D i Estimate the optimal action u for the next state s' in D ijk = argmax a D i (o i ,M ijk ,a) Step 3.4: Repeat the calculation to obtain the optimal action set u ij = {u ijk | k e U}, where U represents the agent action space; Step 3.5: For a given action k, compute the maximum value a of the rewards for agent i taking different optimal actions ijk = max u∈ uij D i (o i ,M ijk ,u) and the minimum value b of the rewards for agent i taking different optimal actions ijk = min u∈uij D i (o i ,M ijk ,u). Step 3.6: Normalize α ijk and β ijk using the softmax function, let the normalized distribution be θ ijk , θ ijk = softmax[α ijk , β ijk ]; Step 3.7: Compute the maximum of the standard deviation of the distribution θ ijk as the importance of agent j relative to agent i, denoted d ij , d ij = max k∈U (std(θ ijk )). Step 3.7: Compute the maximum of the standard deviation of the distribution θ ijk as the importance of agent j relative to agent i, denoted d ij , d ij = max k∈U (std(θ ijk )).

4. The method of claim 1, wherein, In step 5, each agent broadcasts its dependency object set V to other agents i , and the dependency graph obtained by each agent after broadcasting is denoted as an adjacency matrix B, B ij = 1 indicates that agent i depends on agent j, and the directed loop in B is removed to avoid circular dependency, and the result is denoted as W. The specific steps of removing the directed loop are as follows: Step 5.1: initialize the W matrix as the B matrix; Step 5.2: select the node set b with out-degree 0 in B; Step 5.3: If b is empty, find the node b in B with the smallest out-degree and the smallest number min , and remove the out-edges of b min from W, then add b min to B and b to b; Step 5.4: delete all nodes in b from B; Step 5.5: repeat steps 5.2-5.4 until B is empty.

5. The method of claim 1, wherein, In step 6, each agent makes a new decision based on the dependency graph and intentional action information. To avoid intentional misdirection, agent i, which is depended upon in the dependency graph, is prohibited from making a new decision; its final action is its intentional action, which is a. i =p i The undependent agent i according to W i and M i The final action is obtained through a re-decision process; specifically, each agent i maintains a value network Q. i Q i (o i E i a) represents the observed data o i Below, the intent information of the dependent object of agent i is E. i When agent i performs action a, the expected reward is... then agent i's final action a... i =argmax a Q i (o i E i Let A be the final joint action of all agents, where A = {a1, a2, ..., a}. i ,…,a N }, i = 1, ..., N, where A is the decision output of the agent at this step.

Images