Multi-agent collaboration method based on group decision and single relation set

Abstract

The invention relates to a neural network structure and a training method which are used for the technical field of intelligent control, in particular to a multi-agent collaboration method based on group decision and a single relation set. A distributed multi-agent control information transmission structure is generated, so that the problem of exponential growth of a state space is solved. Secondly, a group decision method is used in the scheme, opinions of a plurality of agents are summarized, a final aggregated guidance opinion is formed to guide behaviors of the agents, and through the scheme design, the technical scheme solves the problems that a traditional multi-agent model is poor in control effect under the condition that the number of the agents is large, and the control effect is poor. Therefore, the model basis of the large-scale intelligent agent cluster is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent collaborative control, and in particular to a multi-agent collaborative method based on group decision-making and single-item relationship sets. Background Art

[0002] The field of multi-agent collaborative control refers to how multiple interacting agents achieve common goals through collaborative behavior. In multi-agent control, each agent has the ability to perceive, make decisions, and execute autonomously, and interact and cooperate with other agents to optimize the overall system and improve its efficiency. Multi-agent systems are commonly used in unmanned system collaborative control, intelligent transportation systems, multi-robot collaboration, the Internet of Things, distributed energy systems, etc.

[0003] Currently, many common methods applied to multi-agent systems (MAS) generally focus on the problem that the state space and action space grow exponentially with the increase in the number of agents. This problem leads to high training cost and high convergence difficulty of the model when the number of agents is large, and poor control effect of the agents. Currently, there are many solutions that use various methods to solve the problem of exponential growth of state space and action space, but these solutions generally have poor control effects on the collaborative behavior of agents. To solve this problem, a multi-agent collaboration method based on group decision-making and single-item relationship sets was developed. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-agent collaboration method based on group decision-making and unidirectional relationship sets. The present invention further realizes effective collaboration among multi-agents while ensuring the linear size of the state space and the action space by using unidirectional collaboration sets and group decision-making (GDM) methods, thereby solving many difficult problems in the current field of multi-agent system control.

[0005] The present invention provides a multi-agent collaboration method based on group decision-making and single-item relationship sets, comprising:

[0006] Constructing a multi-agent system, establishing a unidirectional relationship set in the multi-agent system according to preset screening conditions, wherein the preset screening conditions include, for example, the distance between agents;

[0007] Each agent sends action guidance opinions to other agents according to the one-way relationship set established in step S101, thereby constructing a multi-agent system with information flow;

[0008] In the multi-agent system, one of the agents receives guidance opinions on its own actions from several agents based on the one-way relationship concentration, and then merges the guidance opinions on its own actions from several agents to generate a reciprocal relationship matrix.

[0009] According to the reciprocal relationship matrix, specific action information is generated. In the reciprocal relationship matrix, a weight is assigned to each piece of guidance information according to the value of the guidance. The weights are summarized and analyzed to obtain the maximum influence of the guidance information on the action selection of the intelligent agent.

[0010] Furthermore, a multi-agent system is constructed, and a unidirectional relationship set is established in the multi-agent system according to preset screening conditions, wherein the preset screening conditions include, for example, the distance between agents, including;

[0011] A preset network is constructed based on at least five intelligent agents, wherein the five intelligent agents include a first intelligent agent, a second intelligent agent, a third intelligent agent, a fourth intelligent agent and a fifth intelligent agent, and the five intelligent agents perform unidirectional interaction modeling to obtain a unidirectional relationship set.

[0012] The out-degree of each agent is set to two, while the in-degree is unlimited. The data flow of the unidirectional relationship set not only collects the information of the surrounding agents, but also contains the environment information of the agent. The unidirectional relationship set is set to C x , which includes the agent itself x and p agents within its communication range, this set of one-way relationships represents the upper limit of the perception and interaction effects of the multi-agent system.

[0013] Furthermore, one of the agents in the multi-agent system receives action guidance opinions from several agents based on the unidirectional relationship in the unidirectional relationship concentration, and then merges the action guidance opinions from several agents to generate a reciprocal relationship matrix, including;

[0014] For each agent x For example, he has The mutual relationship matrix The action instructions he generates for the agent can be Obtained in.

[0015] The following calculation formula can calculate the reciprocal relationship matrix for each action guide. This structure ensures the consistency of the reciprocal relationship matrix.

[0016] F=(f ij ) m×m .

[0017]

[0018] Furthermore, one of the agents in the multi-agent system receives action guidance opinions from several agents based on the unidirectional relationship in the unidirectional relationship concentration, and then merges the action guidance opinions from several agents to generate a reciprocal relationship matrix, which also includes:

[0019] In group decision making, all agents are involved. x The relationship set can be agent x Generate action guidance if u agents provide relational guidance According to the above group decision-making, it is transformed into an RPR matrix set {F1, F2, ..., F u}.

[0020] 5. The method for improving the collaborative effect of multiple intelligent agents based on group decision-making and single-item relationship sets as claimed in claim 1 is characterized in that specific action information is generated according to the reciprocal relationship matrix, a weight is assigned to each piece of guidance information in the reciprocal relationship matrix according to the value of the guidance, and the weights are summarized and analyzed to obtain the greatest influence of the guidance information on the action selection of the intelligent agent, including:

[0021] By assigning weights to each decision maker’s action guidance, the weights of u agents’ guidance opinions are expressed as the primitive operator WA = {V1, V2, ..., V u}. In the generalized V k After that, the guidance weight operation operator WA will be generated. According to this operator, the reciprocity relationship matrix F C In It can be expressed as

[0022] The beneficial effects of the present invention are as follows: The technical solution of the present application solves the problem that the state space and action space in the traditional multi-agent system will grow exponentially with the number of agents. First, the first aspect of the technical solution of the present application uses the method of one-way relationship sets to avoid the use of global state space, and generates a distributed multi-agent control information transmission structure, thereby solving the problem of exponential growth of the state space. Secondly, the scheme uses the group decision making (GDM) method to summarize the opinions of multiple agents and form a final aggregated guidance opinion to guide the behavior of the agent. This method avoids the problem of exponential growth of the action space while ensuring the effectiveness of multi-agent collaboration and the feasibility of control model training. Through the above scheme design, the technical solution of the present application solves the problem that the traditional multi-agent model has poor control effect and the model is difficult to converge when facing a large number of agents, and realizes the model foundation for large-scale agent clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1A flowchart of a multi-agent collaboration method based on group decision-making and single-item relationship sets is provided for an embodiment of the present invention.

[0025] Figure 2 The unidirectional relationship and the organizational form arrow provided in the embodiment of the present invention are schematic diagrams of the information transmission direction.

[0026] Figure 3 The one-way relationship provided in the embodiment of the present invention is a collection of information flows within a specific agent.

[0027] Figure 4 A detailed diagram of group decision making provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The technical solutions provided by the embodiments of the present invention are described in detail in conjunction with the drawings.

[0029] The present invention provides a multi-agent collaboration method based on group decision-making and single-item relationship sets, comprising:

[0030] In step S101, a multi-agent system is constructed, and a unidirectional relationship set is established in the multi-agent system according to preset screening conditions, wherein the preset screening conditions include, for example, the distance between agents;

[0031] In step S102, each agent sends action guidance opinions to other agents according to the unidirectional relationship set established in step S101, so as to construct a multi-agent system with information flow;

[0032] In step S103, one of the agents in the multi-agent system receives action guidance opinions from several agents based on the unidirectional relationship in the unidirectional relationship set, and then merges the action guidance opinions from several agents to generate a reciprocal relationship matrix;

[0033] In step S104, specific action information is generated according to the reciprocal relationship matrix, a weight is assigned to each piece of guidance information in the reciprocal relationship matrix according to the value of the guidance, the weights are summarized and analyzed, and the greatest influence of the guidance information on the action selection of the intelligent agent is obtained.

[0034] In the first aspect, the technical solution of this application is about the division of relationship sets and unidirectional relationship sets of multi-agent systems;

[0035] In the prior art, in traditional multi-agent systems (MASs), the state space S MAS It includes the multi-agent joint state space and the environment state space, represented by S MAS =S AG ×S Env , assuming that the agent x The number is n, then the multi-agent set is represented by AG = {agent 1 ,agent 2 ,...,agent n}. The multi-agent joint state space is represented as S AG =S 1 ×S 2 ×…×S n , the action space is represented by A AG =A 1 ×A 2 ×…×A n Among them, S x and A x Represents the agent x If all agents in the multi-agent system are completely consistent, the following equation is satisfied.

[0036] A=A 1 =A 2 =…=A n ,a∈A,A AG =(A) n

[0037] S=S 1 =S 2 =…=S n ,s∈S,S AG =(S) n

[0038] According to the above description, it can be seen that when the number of agents increases, the size of the state space and action space grows exponentially with the number of agents n. This feature will increase the difficulty of model training and weaken the control effect. This is also the problem of traditional multi-agent control methods that this solution aims to solve.

[0039] The technical solution of this application is to solve this problem. The present invention provides a multi-agent collaboration method based on group decision-making and unidirectional relationship sets, and adopts a unidirectional relationship set method to replace the traditional multi-agent relationship set division method. The unidirectional relationship set needs to select other agent members according to certain conditions (divided according to the distance between agents in the case of this application) as part of the information in its unidirectional relationship set.

[0040] Since the information of each agent may change at each step, each agent generates a unidirectional relationship set that is updated at each step and is very flexible. The present invention provides a multi-agent coordination method based on group decision-making and unidirectional relationship sets. Compared with the traditional multi-agent control method, it avoids the problem of exponential growth of the global state space with the number of agents while ensuring the circulation of state information among multiple agents.

[0041] like Figure 2 As shown in the figure, each circle represents an agent. The left figure uses the partitioning method of unidirectional relationship sets, in which the out-degree of each agent is stipulated to be two, while the in-degree is not restricted. This ensures the low dimension of the input state space while ensuring the complex impact process on the key agents. This solves the problem of exponential growth of the state space in traditional multi-agent control methods.

[0042] right Figure 2 For the distributed multi-agent system divided by unidirectional relationship sets in the middle left figure, the data flow of the unidirectional relationship set not only collects the information of the surrounding agents, but also includes the environment information of the agents. We set the unidirectional relationship set as C x , which includes the agent itself x and p agents within its communication range. This set of one-way relationships represents the upper limit of the perception and interaction effects of the multi-agent system.

[0043] Expressed as Where each agent is defined as s x , Then agent x The state space of

[0044] On the second aspect, the technical solution of this application is about the generation of aggregated decisions based on group decision making (GDM);

[0045] In order to realize complex multi-agent cooperative behavior in the smallest possible state space, this scheme combines some group decision making (GDM) theories. Each agent is regarded as a decision maker participating in the group decision-making process.

[0046] When an agent receives multiple action instructions generated by other agents, it needs to summarize all the action instructions. The summary summarizes the instructions of all agents in the range, so that each agent's instructions can affect the final actions taken by all agents to complete complex collaborative behaviors.

[0047] In the group decision making (GDM) mechanism, relevant intelligent agents need to express their opinions on the behavioral decision-making process and organize all opinions through the group decision-making mechanism. Therefore, the unidirectional relationship set partitioning method extracts the collaborative correlation between intelligent agents and provides an organizational basis for each intelligent agent that needs to collaborate to make group decisions.

[0048] The technical solution of the present application first generates a reciprocal preference relation (RPR) matrix from discrete action guidance opinions.

[0049] For each agent x For example, he has RPR matrix The action instructions he generates for the agent can be Obtained in.

[0050] The following calculation formula can calculate the RPR matrix for each action guide. This structure ensures the consistency of the RPR matrix.

[0051] F=(f ij ) m×m .

[0052]

[0053] In group decision making (GDM), all agents are involved. x The relationship set can be agent x Generate action guidance. If u agents provide relational guidance According to the above formula, GDM transforms it into a set of RPR matrices {F1, F2, ..., F u}.

[0054] In this scheme, by assigning weights to the action guidance of each decision maker, it is ensured that high-quality intelligent agents can play a more important role in decision-making. At the same time, other methods can also be used to determine the way to generate the aggregate RPR matrix. In this scheme, according to the principle of the weighted operation operator WA, the weight should be proportional to the average expected return V k Related.

[0055] The weights of the guidance opinions of u agents are expressed as the primitive operator WA = {V1, V2, ..., V u}. In the generalized V k After that, the guidance weight operator WA will be generated. According to this operator, the RPR matrix F C In It can be expressed as

[0056] Given an RPR matrix FC , the final aggregate action guide G C It can be obtained by the inverse operation of the following equation.

[0057]

[0058] The specific derivation process of this formula is as follows.

[0059] The following formula is known to hold;

[0060]

[0061] So, given F, solve for G x ;

[0062] According to the known established formula;

[0063] if

[0064] or but;

[0065]

[0066] sorted out;

[0067]

[0068] so;

[0069]

[0070] Established.

[0071] Each agent generates guidance for other agents within its relationship set at each step. x If u other relationship sets are selected at a time, then the relationship set Contains other u agents to the agent x All guidance and agents x Guidance for oneself, including G0 x The guidance for oneself is a pending one. x Aggregate all guidance via GDM Get aggregation guidance G C , so all agents can choose the actions they will perform based on the aggregated guidance.

[0072] After the policy model converges, this opinion aggregation mechanism collects the opinions of all relevant agents to make decisions on the actions of a single agent. This method not only reduces the size of the state space, but also avoids dependence on the global state, and aggregates all action guidance information through the GDM method, thereby ensuring the control effect of the policy on the collaboration of agents.

[0073] In the first aspect of the technical solution of the present application, there are various methods for demarcating the parts participating in the unidirectional relationship set. In addition to dividing the intelligent bodies participating in the unidirectional relationship set by distance, there are many other schemes. This part does not affect the overall effect achieved by this solution.

[0074] In the second aspect of the technical solution of the present application, in the process of generating a reciprocal relationship matrix (RPR) from action guidance, whether the action guidance is collected first and then the RPR matrix is ​​generated uniformly or each intelligent agent generates the RPR matrix first and then collects and aggregates it, the order will not affect the effect of the scheme implementation. The method of aggregation by weight operator used when aggregating multiple RPR matrices to generate an aggregated RPR matrix is ​​not fixed, and many other methods can also be used to determine the generation method of the aggregated RPR matrix. This aggregation method will not affect the effect achieved by this scheme.

[0075] Unidirectional relationship set part:

[0076] In a multi-agent system, a one-way relationship is established based on preset screening conditions (such as the distance between agents). In the one-way relationship, each agent sends action guidance to other agents that meet the screening conditions (the distance between agents), thereby forming a multi-agent system that can flow information.

[0077] Group decision making section:

[0078] For a specific intelligent agent, after receiving action guidance opinions from several intelligent agents based on the unidirectional relationship mentioned above, the intelligent agent needs a method to merge multiple guidance opinions and generate the final action. This method is group decision-making, which will combine the intelligent agent's own environmental state information and the received action guidance information to generate a reciprocal relationship matrix, and generate specific action information based on the matrix. In the reciprocal relationship matrix, our method assigns weights to each guidance information based on the value of the guidance, thereby ensuring that important guidance information has the greatest impact on the action selection of the intelligent agent.

[0079] See also Figure 2 , which illustrates a one-way relationship and the organizational form, the arrow is the direction of information transmission.

[0080] See also Figure 3 , is the information flow within a specific agent in a one-way relationship set, where the Guidance list is Figure 1There are one-way arrows between each agent in the GDM system, where the GDM system is the group decision-making part. Its specific internal details are shown in the figure below. In the above figure, Policy is the behavior decision-making method of the agent. This part is not the focus of this patent, but it is added to help understand the relationship between the one-way relationship set and the group decision. If the process needs to be introduced in the patent description, please add this figure.

[0081] See also Figure 4 , which is the detailed diagram of group decision-making, where RPR list is the reciprocal relationship matrix. It can be seen that it is formed by the action guidance information sent by multiple intelligent agents, and then a series of calculations such as weights are performed to finally form a specific action.

[0082] It should also be noted that this scheme does not limit the action generation decision method. Many algorithms such as DQN can achieve the scheme effect. The focus of this scheme is to use unidirectional relationship sets and group decision making (GDM) schemes to ensure the linear size of state space and action space, and realize the model's effective control of the complex collaborative behavior of multiple agents.

[0083] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments. The above-described embodiments of the present invention do not constitute a limitation on the scope of protection of the present invention.

Claims

1. A multi-agent collaboration method based on group decision making and single-item relationship sets, characterized in that: include; S101 constructs a multi-agent system, and establishes a unidirectional relationship set in the multi-agent system according to a preset screening condition, wherein the preset screening condition includes, for example, the distance between agents; S102: each agent sends action guidance opinions to other agents according to the one-way relationship set established in step S101, so as to build a multi-agent system with information flow; S103: one of the agents in the multi-agent system receives action guidance opinions from several agents based on the one-way relationship in the one-way relationship concentration, and then merges the action guidance opinions from several agents to generate a reciprocal relationship matrix; S104 generates specific action information according to the reciprocal relationship matrix, assigns weights to each piece of guidance information in the reciprocal relationship matrix according to the value of the guidance, summarizes and analyzes the weights, and obtains the influence of the guidance information on the action selection of the intelligent agent.

2. The method for improving the multi-agent collaborative effect based on group decision-making and single-item relationship sets as claimed in claim 1, characterized in that: Constructing a multi-agent system, establishing a unidirectional relationship set in the multi-agent system according to preset screening conditions, wherein the preset screening conditions include, for example, the distance between agents, including; A preset network is constructed based on at least five intelligent agents, wherein the five intelligent agents include a first intelligent agent, a second intelligent agent, a third intelligent agent, a fourth intelligent agent and a fifth intelligent agent, and the five intelligent agents perform unidirectional interaction modeling to obtain a unidirectional relationship set. The out-degree of each agent is set to two, while the in-degree is unlimited. The data flow of the unidirectional relationship set not only collects the information of the surrounding agents, but also contains the environment information of the agent. The unidirectional relationship set is set to C x , which includes the agent itself x and p agents within its communication range, this set of one-way relationships represents the upper limit of the perception and interaction effects of the multi-agent system.

3. The method for improving the multi-agent collaborative effect based on group decision-making and single-item relationship sets as claimed in claim 1, characterized in that: One of the agents in the multi-agent system receives action guidance opinions from several agents based on the unidirectional relationship of the unidirectional relationship concentration, and then merges the action guidance opinions of several agents to generate a reciprocal relationship matrix, including; For each agent x For example, he has The mutual relationship matrix The action instructions he generates for the agent can be Obtained in. The following calculation formula can calculate the reciprocal relationship matrix for each action guide. This structure ensures the consistency of the reciprocal relationship matrix. F=(f ij ) m×m .

4. The method for improving the multi-agent collaborative effect based on group decision-making and single-item relationship sets as claimed in claim 1, characterized in that: One of the agents in the multi-agent system receives action guidance opinions from several agents based on the one-way relationship of the one-way relationship concentration, and then merges the action guidance opinions from several agents to generate a reciprocal relationship matrix, which also includes; In group decision making, all agents are involved. x The relationship set can be agent x Generate action guidance if u agents provide relational guidance According to the above group decision-making, it is transformed into an RPR matrix set {F1, F2, ..., F u }.

5. The method for improving the multi-agent collaborative effect based on group decision-making and single-item relationship sets as claimed in claim 1, characterized in that: Generate specific action information based on the reciprocal relationship matrix, assign weights to each piece of guidance information in the reciprocal relationship matrix according to the value of the guidance, summarize and analyze the weights, and obtain the greatest influence of the guidance information on the action selection of the intelligent agent, including; By assigning weights to each decision maker’s action guidance, the weights of u agents’ guidance opinions are expressed as the primitive operator WA = {V1, V2, ..., V u }. In the generalized V k After that, the guidance weight operation operator WA will be generated. According to this operator, the reciprocal relationship matrix F C In It can be expressed as for i,j=1,...,m.

Images