Multi-agent system communication method for cooperative navigation

Abstract

The invention relates to a multi-agent system communication method for cooperative navigation, which utilizes a reinforcement learning technology to enhance communication among agents through a dynamic communication mechanism so as to improve the stability and efficiency of a multi-agent system in cooperative navigation tasks in the fields of automatic control, robot cooperation and intelligent transportation. The method is particularly concerned with cooperation between agents and scalability to a large-scale robot team. The method mainly comprises the following steps: expressing an intelligent agent system structure, creating a hierarchical structure of the intelligent agent through clustering and hierarchical relationship generation, and optimizing information exchange; observation coding and aggregation: the observation of the intelligent agent is coded into a communication message, and aggregation is carried out through a hierarchical structure; action intention communication: the intelligent agent encodes the action intention and communicates with other intelligent agents to promote cooperation and coordination; and action generation and reward calculation: based on the aggregated information after communication, generating an action strategy, executing an action and calculating a reward. The method is used for intelligent agent cooperation navigation tasks.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning in multi-agents, and in particular to a multi-agent system communication method for cooperative navigation. Background Art

[0002] With the development of artificial intelligence, multi-agent systems are increasingly used in the fields of automated control, robot collaboration, and intelligent transportation. As a task that is widely used in real-world scenarios, cooperative navigation tasks have also attracted the attention of researchers. Applying reinforcement learning methods to multi-agent tasks is one of the hottest research topics in this field.

[0003] Multi-agent reinforcement learning is very helpful for designing and learning strategies for multi-agent systems, and special attention is paid to the cooperation between agents and the scalability to larger robot teams during the learning process. The present invention is based on a reinforcement learning method based on dynamic communication, which enhances the communication between agents by introducing a communication mechanism, thereby improving the stability of learning.

[0004] Multi-agent reinforcement learning refers to the use of multiple agents to make sequential decisions. Tasks in a multi-agent environment are more challenging than single-agent RL because the state transitions and rewards of each agent are also affected by the behavior of other agents. Therefore, for large-scale agent systems, a communication mechanism between them is very necessary. Existing communication methods are based on gate mechanisms, communication graphs, and priorities. They face the following two problems: First, when the number of agents is large, they will encounter the problem of non-convergence; second, the robustness of the communication system when the communication cost is too expensive or the communication environment is relatively random. Summary of the invention

[0005] Based on reinforcement learning technology, the present invention proposes a multi-agent system communication method for cooperative navigation tasks to address the above problems, which aims to solve the problems of learning instability and high communication cost caused by the extension of reinforcement learning in multi-agent systems in the prior art.

[0006] To achieve the above object, the present invention provides the following technical solution: a multi-agent system communication method for cooperative navigation, comprising the following steps:

[0007] Step 1: Represent the structure of the agent system, design the clustering of agents, and then generate the hierarchical relationship of agents to obtain the agent system representation with a hierarchical structure;

[0008] Step 2: Encode the observations of the agents, generate their own communication messages, and aggregate the observation messages;

[0009] Step 3: Encode the aggregated message of the agent, obtain the action intention of the agent, and communicate the action intention between the agents;

[0010] Step 4: Based on the communication results, generate the agent's actions, input them into the environment and calculate the reward obtained by the agent.

[0011] Preferably, the step 1 comprises the following steps:

[0012] Step 1.1, group the agents in a distributed environment;

[0013] Step 1.2: According to the grouping result in step 1.1, select a collaboration node for each group for inter-group communication to complete the generation of the hierarchical relationship.

[0014] Preferably, the step 2 comprises the following steps:

[0015] Step 2.1: Obtain observations of the agent from the environment;

[0016] Step 2.2: Encode the observations of the intelligent agents and generate communication messages that can be used for communication between intelligent agents;

[0017] Step 2.3, using the communication message generated in step 2.2, perform agent-based communication according to the hierarchical structure generated in step 1;

[0018] Step 2.4: Each agent performs weighted aggregation operations on the information it receives.

[0019] Preferably, the step 3 comprises the following steps:

[0020] Step 3.1: Based on the aggregated message generated in step 2.4, the agent generates the encoding of action intention, which represents the expected action and future plan;

[0021] Step 3.2: Using the action intention generated in step 3.1, the communication between the object center agents is carried out according to the hierarchical structure generated in step 1;

[0022] Step 3.3: Each agent uses a recurrent neural network to aggregate the information it receives.

[0023] Preferably, the step 4 comprises the following steps:

[0024] Step 4.1, generate the action strategy of each agent based on the information aggregated by the agents after communication;

[0025] Step 4.2: Output the action to the environment and calculate the reward based on the joint impact of the action on the environment.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. A hierarchical structure representation of intelligent agents is proposed. By clustering and generating hierarchical relationships, the information exchange between intelligent agents is optimized. This hierarchical communication structure reduces the communication cost while improving the efficiency and collaboration ability of the system.

[0028] 2. A novel two-level communication mechanism is proposed, which not only includes the exchange of environmental observation information, but also the sharing and coordination of intention information between agents. It is very helpful when dealing with large-scale agent systems and complex environments in cooperative navigation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is the overall flow chart of the present invention;

[0030] Figure 2 A flow chart showing the structure of the intelligent agent of the present invention;

[0031] Figure 3 This is the observation information communication flow chart of the present invention;

[0032] Figure 4 It is the action intention communication flow chart of the present invention;

[0033] Figure 5 The model output environment feedback flow chart of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The present invention will be further described below in conjunction with the accompanying drawings:

[0036] like Figure 1As shown, it is the overall flow chart of the whole method. The present invention includes four major steps: agent structure representation, observation information communication, action intention communication and model output environment feedback. Each major step contains two small steps. The agent structure representation is to prepare for observation information communication and action intention communication; observation information communication is to encode the observation of the agent and communicate with other agents in the multi-agent system; action intention communication encodes the future action intention of the agent and communicates with other agents in the multi-agent system to promote collaboration and coordination between agents, and achieve more efficient teamwork and task execution by transmitting future action intentions and planning information; model output environment feedback refers to the interaction between the action of the agent and the environment and the calculation method of environmental feedback finally output by the present invention.

[0037] Embodiment 1:

[0038] The following is an example of an environment with an agent scale of 15. Figure 2 The following table shows the position coordinates of the 15 agents.

[0039]

[0040]

[0041] like Figure 2 As shown in the figure, in the representation of the agent structure, the agents in the environment are first grouped and clustered according to the distance between the agents to reduce the impact of excessive interference information on the agent training. Then, in order to balance the agent's consideration of global information, inter-group communication channels are further established based on one-dimensional grouping and clustering, that is, a hierarchical relationship structure is generated. The details are as follows:

[0042] 1.1 Agent clustering. Agents are distributedly clustered, that is, they are gathered in a distributed environment. While each agent can work independently, it is convenient for them to communicate and coordinate through the network.

[0043] 1.1.1 Calculate the communication cost. In scenarios where the size of the agents and the execution area are relatively large, the cost of communication needs to be considered. When there are a large number of agents, the bandwidth of full communication will be very large, and the computational complexity will also increase; and when the execution area is large, the long communication distance in actual scenarios will also increase the communication cost. For communication between any two agents, the greater the distance between the agents, the higher the communication cost. Therefore, the present invention calculates the communication cost according to the distance:

[0044]

[0045] where r o is the observation radius of the agent, which is set to 1 in this example. Represents the distance between agent i and agent j, calculated as Euclidean distance:

[0046]

[0047] The agents move in a two-dimensional plane, the coordinates of agent 1 are (x1, y1), and the coordinates of agent 2 are (x2, y2).

[0048] 1.1.2 Edges within the observation radius. The communication cost of agents within the observation radius is 0, so we first generate a connection graph between agents according to the observation radius. Each agent is modeled as a node in the graph, using v i represents the i-th node; when the distance between two agents is less than the observation radius, they are connected by an edge, represented by e i,j Indicates v i and v j There is an edge between them.

[0049] The edge connection operation is performed on Example 1 to obtain the edge set: E = {e 2,3 ,e 3,4 ,e 6,7 ,e 7,8 ,e 7,11 ,e 8,9 ,e 8 ,10 ,e 8,11 ,e 9,10 ,e 11,12 ,e 12,13 ,e 13,14 ,e 13,15 ,e 14,15}

[0050] 1.1.3 Maximum complete subgraph clustering. At this point, we can use G = {V, E} to represent the entire graph, and then cluster the agents. For the node set V1 in the graph, if |V1|>1, and:

[0051] (1) and Both have e ij ∈E;

[0052] (2) and Make

[0053] When the above two conditions are met, that is, the subgraph composed of the vertices in V1 and their related edges is a complete graph and is not part of another larger complete subgraph, then the nodes of V1 are clustered into a group, among which there are nodes that belong to multiple groups at the same time, which are called boundary nodes.

[0054] An isolated node is regarded as a cluster containing 1 node, and S is used to represent the set of clusters. For Example 1, the result after clustering is: S = {V1, V2, V3, V4, V5, V6, V7, V8, V9, V 10}, where V1 = {v 1},V2={v 2 ,v 4},V3={v 3 ,v 4},V4={v 5},V5={v 6 ,v 7},V6={v 7 ,v 8 ,v 11},V7={v 8 ,v 9 ,v 10},V8={v11,v 12},V9={v 12 ,v 13},V 10 = {v 13 ,v 14 ,v 15}.

[0055] 1.1.4 Boundary node clustering. After the maximum complete subgraph clustering, isolated nodes and edge nodes appear in the graph. Use V iso represents an isolated node set, V bound represents an edge node set, such as in Example 1, V iso = {v 1 ,v 5},V bound = {v 4 ,v 7 ,v 8 ,v 11 ,v 12 ,v 13}. Clustering is performed according to the following algorithm until all boundary nodes are divided into a unique group:

[0056]

[0057]

[0058] 1.2 Generation of agent hierarchical relationships. Agents will eventually be clustered according to a hierarchical structure to form a multi-level agent system. The goal of hierarchical clustering is to improve the efficiency and collaboration of the entire system through hierarchical communication while ensuring low communication costs.

[0059] 1.2.1 Initialize the collaboration weights. represents the initial weight of each node. After the maximum complete subgraph clustering in 1.1.3, if agent i is not a boundary node, let If agent i is a boundary node, then when performing step 1.1.4 boundary node clustering, initialize Each pair of v i Perform an operation to update

[0060] 1.2.2 Update the collaboration weight. In the following time steps, record whether agent i was elected as a collaboration node in the previous time step. If so, update the collaboration weight:

[0061]

[0062] 1.2.3 Elect collaborative nodes. represents the collaboration weights of all agents in cluster j at time step t. When selecting collaboration nodes in cluster j, Perform normalization operation:

[0063]

[0064] After normalization The i-th value in is regarded as the probability of selecting agent i as a collaborative node. That is, when there is only one node in a cluster, it is elected as a collaborative node; when the number of nodes in the cluster is greater than 1, it is selected according to The probability represented by randomly selecting a collaborative node.

[0065] After the cooperation node election is completed, full-connection communication can be carried out between the cooperation nodes. At this point, the two-layer communication architecture is completed.

[0066] like Figure 3 The figure is a flow chart of the observation information communication process of the present invention. The present invention encodes and generates messages based on the observation of the agent and then communicates with other agents in the multi-agent system. The key observation data of the agent includes the location of landmarks and obstacle detection results within the field of view. This information is crucial for the decision-making and planning process of the agent and helps to improve the system's perception ability and ability to cope with complex environments. The details are as follows:

[0067] 2.1 Observation Encoding Communication Messages. The intelligent agent uses existing knowledge or information to promote communication. First, it needs to encode historical observations into messages. The present invention uses a GRU network to encode observations and other information:

[0068]

[0069] in, That is, it represents the local observation obtained by agent i from the environment at time step t, and the position of agent i at time step t After concatenation, it is used as the input of the GRU network. In addition, the internal hidden state of the previous step of GRU is also used in the input

[0070] 2.2 Agent-based message propagation and aggregation. According to the hierarchical structure of the agents generated in step 1, the agent system and its communication links are modeled as a graph, and based on the GNN network, the message propagation and aggregation are carried out in the form of intra-group aggregation - inter-group communication - intra-group aggregation, so that each agent finally obtains the aggregated message.

[0071] The specific steps of information exchange are as follows:

[0072] 2.2.1 Intra-group aggregation. In the intra-group aggregation step, each collaboration node receives the information of the agents in its own group and aggregates it into a whole. The aggregation process that occurs on the collaboration node v is:

[0073]

[0074] in, Represents the communication message generated by the agent represented as node u in the graph at the tth time step, where u∈N(v), that is, the agent in the same communication group as node v, and N(v) represents the nodes adjacent to node v, that is, the points with communication interaction. AGG1(·) is the aggregation function. In this step, to simplify the calculation, this chapter uses the mean function Mean(·) as the aggregation function. In particular, to prevent the loss of the node's own information and to ensure that the length of the communication message and the aggregated message is fixed during the transmission process, node v also concatenates the communication message generated by itself at time step t. CONCAT(·) is the concatenation operation, which can aggregate the information of its own node with other nodes in the group, facilitating the subsequent inter-group communication operation of the collaborative node.

[0075] 2.2.2 Inter-group communication. Inter-group communication is carried out between collaborative nodes, and the subgraph generated by the collaborative nodes and their associated edges in the communication link graph is a complete graph. The GNN network is still used on this complete graph to exchange information between collaborative nodes. Collaborative nodes receive information from other nodes and aggregate them together. The aggregation process is as follows:

[0076]

[0077] in Represents the messages that node u has aggregated in the group, where u∈V coop And u≠ι, but V coop The collection of collaborative nodes, V cooprepresents the set of all collaborative nodes. AGG2(·) calculates the weighted sum of each piece of information according to the distance.

[0078] 2.2.3 Intra-group broadcasting: The cooperative node broadcasts the aggregated message within the group, so that each ordinary node receives the aggregated message, and the aggregated information of the agents in the same group is the same.

[0079] After the information exchange in step 2 is performed to obtain aggregated information, the present invention enters the step of action intention communication, such as Figure 4 As shown, it is divided into two steps: action intention generation and intention communication. The specific implementation of each step is as follows:

[0080] 3.1 Action intention generation. Compared with the previous observations of the agent, action intention is used as a kind of future knowledge for the communication of the agent. The steps of generating and encoding action intention are as follows:

[0081] 3.1.1 Output action intention. This step uses a neural network π1 to output action intention for the agent based on the observation information and the aggregated information obtained in step 2:

[0082]

[0083] 3.1.2 Communication message generation. After obtaining the action intention, a recurrent neural network g is used m Encode the agent’s observations and intentions as encoding functions:

[0084]

[0085] in, That is, it represents the observation encoding obtained by agent i from the environment at time step t, is the hidden state of the encoding function of agent i at the previous time step, Encode the action intention information of agent i at time step t obtained in step 3.1.1. As the communication content between agents, this chapter directly uses the hidden state of the recurrent neural network As a communication message:

[0086]

[0087] 3.2 Message propagation and aggregation without a central agent Due to the flexibility of the hierarchical communication structure, the present invention only communicates action intentions between agents within the group scope.

[0088] 3.2.1 Communication weight generation. The present invention uses the attention mechanism to calculate the query vector and key vector of the communication message of two agents to obtain the communication weight between each two agents:

[0089]

[0090] Among them, d is the dimension of the matrix vector, and the purpose of introducing d is to prevent the dot product from being too large. is the query vector of agent i, is the key vector of agent j:

[0091]

[0092] 3.2.2 Intra-group message aggregation. After obtaining all the communication weights, a normalization operation is performed, a threshold is set, and the agents with too low weights are filtered out. The communication messages of the remaining agents are input into the LSTM for aggregation. The LSTM network helps and guides the agents to make coordinated decisions when integrating the action intention messages of each agent.

[0093] like Figure 5 As shown, after the message aggregation in step 3.2.2 is completed, it is input into a neural network called a policy network together with the observation to generate the final policy. The joint policy will have an impact on the agent. The environment will feedback a global reward, and the current global state, reward and action of the environment will be input into the value evaluation network for value evaluation. According to the evaluation results, the gradient is updated by minimizing the objective function. The details are as follows:

[0094] 4.1 Joint action generation. Each agent shares the policy network π2 and calculates its own strategy by aggregating information from their own observations and communicated action intentions:

[0095]

[0096] where v i is the message aggregation vector finally obtained in step 3.2.2. The joint strategy is the concatenation of the strategies of each agent:

[0097] A=(a 1 , a 2 , ..., a n )

[0098] 4.2 Strategy execution and reward calculation. After executing the joint strategy, the environment returns a global reward to the agent after jumping to a new state. The reward structure consists of three parts: distance reward, collision reward, and movement reward.

[0099] 4.2.1 Distance reward calculation. The distance between two entities is calculated in the same way as in step 1.1.1. represents the position of the ith agent, and represents the position of the kth landmark.

[0100] First, for each agent i, find the landmark j closest to agent i and record the distance:

[0101]

[0102] For each landmark i, the distance to the nearest agent j is also calculated:

[0103]

[0104] Finally, the distance reward is the negative of the sum of the closest distances between all agents and all landmarks. Maximizing the distance reward is to minimize the sum of the closest distances, that is:

[0105]

[0106] 4.2.2 Collision Rewards. The method for determining whether agents collide is as follows: for each agent, calculate the distance between it and other agents. When the distance is less than the sum of the radii, a collision is considered to have occurred. Collisions are divided into collisions between agents and collisions between agents and obstacles. For each agent or obstacle, calculate the number of times it collides with other entities and use Count(Agent i ) represents the number of collisions of the ith agent, Count(Obstacle i ) represents the number of collisions with the ith obstacle. The collision reward is the negative of the total number of collisions in the environment, that is:

[0107]

[0108] Among them, n is the total number of agents and m is the total number of obstacles.

[0109] 4.2.3 Movement Reward. The moving speed of the agent is calculated based on the agent's strategy vector. For an agent that chooses to stay still, its speed is 0. The slow moving agent is penalized, which is called the slow penalty value. The slow penalty value is calculated as follows:

[0110]

[0111] That is, for agents with a moving speed greater than 0.6, it can be considered that they have moved a long distance in this time step and are not penalized; for agents with a moving speed less than 0.6, they are penalized, and the smaller the moving speed, the greater the penalty value.

[0112] At the same time, if an agent has reached a landmark, the penalty for its slow movement is canceled. When an agent is far away from a landmark in the environment, it needs to be encouraged to move, that is, the distance weight is set:

[0113]

[0114] Finally, the mobile bonus is calculated as follows:

[0115]

[0116] 4.2.4 Calculate the total reward. The total reward consists of three types of rewards:

[0117] R = αR d +βR c +γR m

[0118] Among them, α, β, and γ are hyperparameters, which are the contribution weights of distance reward, collision reward, and movement reward to the total reward respectively.

[0119] After calculating the reward, the loss function can be calculated based on the reward size and the modules in the method can be updated.

[0120] The multi-agent cooperative navigation system of the present invention significantly improves the navigation efficiency and stability of multi-agents in complex environments through the combination of dynamic communication and a double-layer strategy network, and has broad application prospects and practical value.

[0121] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0122] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-agent system communication method for cooperative navigation, characterized in that: The following steps are involved: Step 1: Represent the structure of the agent system, design the clustering of agents, and then generate the hierarchical relationship of agents to obtain the agent system representation with a hierarchical structure; Step 2: Encode the observations of the agents, generate their own communication messages, and aggregate the observation messages; Step 3: Encode the aggregated message of the agent, obtain the action intention of the agent, and communicate the action intention between the agents; Step 4: Based on the communication results, generate the agent's actions, input them into the environment and calculate the reward obtained by the agent.

2. A multi-agent system communication method for cooperative navigation according to claim 1, characterized in that: The step 1 comprises the following steps: Step 1.1, group the agents in a distributed environment; Step 1.2: According to the grouping result in step 1.1, select a collaboration node for each group for inter-group communication to complete the generation of the hierarchical relationship.

3. A multi-agent system communication method for cooperative navigation according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2.1: Obtain observations of the agent from the environment. Step 2.2: Encode the observations of the intelligent agents and generate communication messages that can be used for communication between intelligent agents; Step 2.3, using the communication message generated in step 2.2, perform agent-based communication according to the hierarchical structure generated in step 1; Step 2.4: Each agent performs weighted aggregation operations on the information it receives.

4. The method for cooperative navigation-oriented multi-agent system communication according to claim 1, characterized in that: The step 3 comprises the following steps: Step 3.1: Based on the aggregated message generated in step 2.4, the agent generates the encoding of action intention, which represents the expected action and future plan; Step 3.2: Using the action intention generated in step 3.1, the communication between the object center agents is carried out according to the hierarchical structure generated in step 1; Step 3.3: Each agent uses a recurrent neural network to aggregate the information it receives.

5. The method for cooperative navigation-oriented multi-agent system communication according to claim 1, characterized in that: The step 4 comprises the following steps: Step 4.1, generate the action strategy of each agent based on the information aggregated by the agents after communication; Step 4.2: Output the action to the environment and calculate the reward based on the joint impact of the action on the environment.

Images