A multi-agent communication learning method

Abstract

The invention discloses a multi-agent communication learning method, which comprises: CriticNet, ActorNet, PriorNet and EncoderNet, wherein the CriticNet is used to calculate the importance of communication in the training stage, and is used to train the corresponding three networks on the terminal device, namely the ActorNet, the PriorNet and the EncoderNet, wherein the ActorNet is used to select the corresponding action at the intelligent agent end, acts on the intelligent agent end, and works in both the training stage and the execution stage, wherein the ActorNet needs to learn the strategy π of the intelligent agent in the training stage, and then generates the corresponding action according to the local observation and the received message, namely, π is the message received by the intelligent agent i at the time t, namely, π is the importance of the message of the intelligent agent j to the intelligent agent i, wherein the PriorNet is used for the intelligent agent to select the communication object, and the PriorNet will evaluate the intelligent agent observed in the local observation and output an importance value, wherein the EncoderNet is used for the intelligent agent to encode its own information to reduce the size of the message body.

Description

Technical Field

[0001] The present invention relates to a communication learning method, and in particular to a multi-agent communication learning method. Background Art

[0002] In a collaborative multi-agent system, all collaborating agents have only one global reward function. However, the observation range of each agent is limited. Therefore, when collaborating, there is a lack of global information for perception or decision-making, which leads to mutually exclusive decisions between agents and makes it difficult to achieve the global optimum.

[0003] As an advanced artificial intelligence technology, Deep Reinforcement Learning (DRL) has achieved great success in many challenging real-world problems. It is widely deployed on different devices, such as smart cars, smart phones, wearable devices, smart cameras, and other smart objects in edge networks. Among them, collaborative multi-agent reinforcement learning is a more difficult and more practical paradigm in DRL. Since each agent only has local observations and lacks global information, the action space is very large and the calculation is complex. At the same time, since there is only one global reward, it is difficult to assign corresponding rewards to independent agents, making it difficult to train and ensure convergence.

[0004] To solve this difficulty, the current mainstream multi-agent algorithms all adopt the centralized training and distributed execution (CTDE) architecture, which has global information during training and only the observations of the agent itself during execution. This architecture has a critic network during training, which updates the critic and actor networks based on the state-action combination of all agents. During execution, each agent has only an independent actor network and makes decisions based on local observations. Typical architectures of this type, such as IQL and QMIX, all use global information during training, but each agent can only make decisions based on local information during execution. These methods model other agents as part of the environment, but only solve the problem of a single agent. Therefore, convergence cannot be guaranteed, and the agent can easily fall into endless exploration.

[0005] Therefore, many studies have begun to focus on communication-based multi-agent reinforcement learning methods. The most direct idea of ​​this method is to introduce information transmission into the collaboration of multiple agents. Through information transmission between agents, "local centralization" is achieved, thereby further overcoming the problem of environmental instability and promoting collaboration between agents. The current mainstream methods include CommNet, which uses a mean unit between the policy networks of multiple agents to receive local observations of all agents, and broadcasts messages to all agents after generating them (star-shaped communication framework); while TarMAC is a fully connected network architecture, and all agents broadcast messages. Both star-shaped and fully connected network architectures are designed to ensure that all messages generated by agents are not missed, and to ensure that local observation information can be propagated to all agents, so that they can have global information for decision-making.

[0006] Although existing communication learning methods ensure that all agents can obtain information from all other agents, they also bring huge amounts of redundant information. Due to the different correlations between agents, information transmission between unrelated agents is not only useless, but may even have a negative impact on the decision-making of the agents.

[0007] At the same time, redundant information transmission also poses a huge challenge to the edge network. Due to the complex structure of the edge network and limited communication bandwidth resources, traditional communication learning methods are often difficult to apply to the edge environment. The main application scenario of multi-agent reinforcement learning is in the edge network environment. Therefore, in order to solve the mismatch between network bandwidth and the resources required for communication learning, this paper analyzes the impact of other agents' messages on the current agent, proposes an indicator for characterizing the importance of messages, and groups agents accordingly. The idea of ​​layered transmission reduces network communication volume and realizes a communication learning method for deep reinforcement learning in edge networks. Summary of the invention

[0008] One advantage of the present invention is that it provides a multi-agent communication learning method, wherein the multi-agent communication learning method introduces message passing between multiple agents to transmit local observations so that the agents can fully consider the global situation when making decisions.

[0009] One advantage of the present invention is that it provides a multi-agent communication learning method, wherein the multi-agent communication learning method designs an importance ranking index and an efficient grouping algorithm to reduce the amount of messages transmitted, thereby realizing an efficient communication learning method to effectively reduce the communication bandwidth consumption caused by unnecessary messages.

[0010] One advantage of the present invention is that it provides a multi-agent communication learning method, wherein the multi-agent communication learning method can be used for all multi-agent reinforcement learning in edge networks for various applications, such as multi-agent intelligent driving, robot navigation, logistics scheduling, etc.

[0011] One advantage of the present invention is that it provides a multi-agent communication learning method, wherein the multi-agent communication learning method is applicable to scenarios requiring multi-scene fusion perception, such as multi-camera fusion and the like.

[0012] The technical solution proposed by the present invention for the above-mentioned technical problem is as follows:

[0013] The present invention provides a multi-agent communication learning method, comprising:

[0014] CriticNet, wherein the CriticNet is used to calculate the importance of communication during the training phase and to train the corresponding three networks on the end device, namely ActorNet, PriorNet and EncoderNet;

[0015] ActorNet, where the ActorNet is used to select the corresponding action on the agent side, acts on the agent side, and works in both the training phase and the execution phase. The ActorNet needs to learn the agent's strategy π in the training phase, and then generate the corresponding action based on the local observation and the received message That is in is the message received by agent i at time t, It expresses the local observation of agent i at each time t;

[0016] PriorNet, where the PriorNet is used by the agent to select the object of communication. PriorNet will evaluate the agent observed in the local observation and output an importance value and

[0017] EncoderNet, wherein the EncoderNet is used by the agent to encode its own information to reduce the size of the message body.

[0018] Preferably, the CriticNet runs in the cloud and works only in the training phase. By calculating the global reward and communication priority, the CriticNet will calculate the network loss and pass the gradient back to the rest of the network and update the rest of the network parameters.

[0019] Preferably, when the importance value exceeds a certain threshold, it means that the current agent i needs to obtain the message of agent j to make a decision.

[0020] Preferably, the agent encodes its previous actions together with observations for reference by other agents, thereby improving the stability of cooperation.

[0021] Preferably, a method for calculating importance is further included, the steps are as follows:

[0022] Step A: Remove the message of agent j and observe whether it will lead to the output action of ActorNet network;

[0023] Step B: Since the action output by ActorNet is the distribution of an action set, KL divergence is used to calculate the difference between the action distributions output by the agent's ActorNet. The specific formula is as follows:

[0024]

[0025] Step C: Where o {i} represents the set of messages observed by agent i from all other agents, o {{i}\j} represents the set of messages of the other observed agents except agent j. The difference calculated by this formula indicates whether the decision distribution of messages without agent j is consistent with the decision distribution of messages with agent j.

[0026] Step D: If the difference is large, it means that the message of agent j is very important to agent i, so its communication confidence is relatively high;

[0027] Step E: After calculating the confidence of all agents, a confidence matrix M between agents is obtained, and the agents are grouped according to this confidence matrix.

[0028] Preferably, it further includes a distributed grouping method, and the PriorNet network outputs two values, namely query and signature: the signature vector is the information fingerprint of the agent itself, which contains the encoding of the position and label of the agent itself; the query vector is the query information, which represents the encoding of the set of agents with which the agent needs to communicate.

[0029] Preferably, it further includes a communication mechanism, which includes a handshake phase, an election phase, a communication phase and a decision phase, wherein in the handshake phase, all agents will broadcast the query and signature to the agents in the observation, and after receiving the query and signature, all agents will restore the communication confidence matrix by multiplying the vectors, wherein in the election phase, after all agents calculate the confidence matrix, they calculate the adjacency graph and select the agent with the largest degree as the preset agent, that is, most of the agents hope to obtain the message of the preset agent to make a decision, and the preset agent serves as the leader node, wherein in the communication phase, all non-leader nodes send their messages to the leader node, the leader encodes the received message through the encoder network, and then communicates between leaders, and the leader passes the message to other leaders, wherein in the decision phase, the leader makes a decision based on the received messages from other leaders, and sends its own decision and message to other non-leader agents in the same group, and other agents make the next decision accordingly.

[0030] Compared with the current mainstream methods, such as star and full connection, the beneficial effects of the present invention are:

[0031] 1. Both the fully connected and star communication networks ignore the impact of the message itself on the decision-making of the agent. Receiving inappropriate messages may affect the convergence of the agent and further affect the maximization of the global reward. The present invention proposes to use KL divergence to measure the importance of messages, ensuring that only valid information is transmitted, avoiding redundant message transmission, and improving the convergence rate.

[0032] 2. Both fully connected and star communication networks require a large number of end-to-end connections. The present invention adopts a grouping and leader election method to communicate, which greatly reduces the communication links and reduces the consumption of communication bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 A network diagram of the multi-agent communication learning method provided by the present invention.

[0035] Figure 2 Schematic diagram of spectral clustering of the multi-agent communication learning method provided by the present invention.

[0036] Figure 3 The invention provides an intelligent agent group communication for the multi-agent communication learning method.

[0037] Figure 4 The global reward of the cooperating agents in the multi-agent communication learning method provided by the present invention is improved.

[0038] Figure 5 The communication volume between multiple agents in the multi-agent communication learning method provided by the present invention. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0040] The typical distributed edge computing architecture consists of multiple edge devices (represented by "Device"). Assume that there are N edge devices, and each device i can be regarded as an intelligent agent. The intelligent agents can be connected to each other through networks such as WIFI and 5G, and have limited computing power and bandwidth resources. Each intelligent agent has an action set A. At each time t, intelligent agent i has its own local observation The agent uses its own observations And action strategy to select the next action and execute it, that is At the same time, when all agents have made corresponding actions, all agents can obtain a global reward value r = env (a0, a1, ..., a n ).

[0041] The goal of a collaborative multi-agent system is to maximize the cumulative value of the global reward r. Therefore, all agents need to grasp the global information of interest through message passing and achieve collaborative decision-making.

[0042] The present invention follows the architecture of CTDE, maintains comprehensive information exchange during the training phase, and performs information encoding and communication object selection according to the trained communication network during the execution phase.

[0043] like Figure 1As shown, the multi-agent communication learning method of the present invention includes CriticNet, ActorNet, PriorNet and EncoderNet, wherein the CriticNet is used to calculate the importance of communication in the training phase, and is used to train the corresponding three networks on the terminal device, namely the ActorNet, the PriorNet and the EncoderNet. Furthermore, the CriticNet runs in the cloud and only works in the training phase. By calculating the global reward and communication priority, the CriticNet will calculate the network loss and pass the gradient back to the remaining networks, and update the remaining network parameters. The ActorNet is used to select the corresponding action on the agent side, act on the agent side, and work in both the training phase and the execution phase. The ActorNet needs to learn the agent's strategy π in the training phase, and then generate the corresponding action based on the local observation and the received message. That is in is the message received by agent i at time t, where the PriorNet is used by the agent to select the object of communication. PriorNet will evaluate the agent observed in the local observation and output an importance value That is, the importance of the message of agent j to agent i. When the importance value exceeds a certain threshold, it means that the current agent i needs to obtain the message of agent j to make decisions. The EncoderNet is used for the agent to encode its own information. Since the agent's observation of the environment is low-dimensional and sparse, it needs to be converted into a high-dimensional representation through an encoding network to reduce the size of the message body. In addition, in addition to the observation information, the agent also needs to encode its previous actions together with the observation for reference by other agents to improve the stability of cooperation.

[0044] For the policy network ActorNet and the corresponding reward loss, the multi-agent communication learning method of the present invention uses the cross entropy loss function as the error and uses the gradient descent method as a means of parameter updating.

[0045] Furthermore, the multi-agent communication learning method of the present invention is about how to select the communication object of the agent and the interactive mode of communication.

[0046] The method for calculating the importance of the present invention is how to weight the other agents observed by agent i and allocate the execution degree of communication. The steps are as follows:

[0047] Step A: Remove the message of agent j and observe whether it will lead to the output action of ActorNet network;

[0048] Step B: Since the action output by ActorNet is the distribution of an action set, KL divergence is used to calculate the difference between the action distributions output by the agent's ActorNet. The specific formula is as follows:

[0049]

[0050] Step C: Where o {i} represents the set of messages observed by agent i from all other agents, o {{i}\j} represents the set of messages of the other observed agents except agent j. The difference calculated by this formula indicates whether the decision distribution of messages without agent j is consistent with the decision distribution of messages with agent j.

[0051] Step D: If the difference is large, it means that the message of agent j is very important to agent i, so its communication confidence is relatively high;

[0052] It is worth noting that the network needs to calculate the output of ActorNet multiple times, so it can only be calculated during the training phase. At the same time, the calculation result will be used as the supervision signal of PriorNet to train the network, so that there is no need to repeat the calculation during the execution phase, and the communication confidence can be directly calculated through PriorNet.

[0053] Step E: After calculating the confidence of all agents, a confidence matrix M between agents is obtained, and the agents are grouped according to this confidence matrix.

[0054] As shown in the figure below, this is an example matrix M. The matrix M is relatively sparse, which means that most agents do not need to communicate with each other. The spectral clustering algorithm can be used to group the agents. Spectral clustering is an algorithm evolved from graph theory and has been widely used in clustering. Its main idea is to regard all data as points in space, which can be connected by edges. The edge weight value between two points that are far away is low, while the edge weight value between two points that are close is high. By cutting the graph composed of all data points, the sum of the edge weights between different subgraphs after cutting is as low as possible, while the sum of the edge weights within the subgraph is as high as possible, so as to achieve the purpose of clustering.

[0055]

[0056] like Figure 3 As shown, through this clustering algorithm, the communication within each group can be made denser, while the communication between groups can be made sparser.

[0057] Since in the execution phase, the agents communicate in a distributed manner and there is no central node to help with grouping, the present invention proposes a distributed grouping method. The present invention allows the PriorNet network to output two values, namely query and signature: the signature vector is the information fingerprint of the agent itself, which contains the encoding of the position and label of the agent itself; the query vector is the query information, which represents the encoding of the set of agents that the agent needs to communicate with.

[0058] Furthermore, the communication mechanism of the present invention includes a handshake phase, an election phase, a communication phase and a decision phase, wherein in the handshake phase, all agents broadcast the query and signature to the agents in the observation, and after receiving the query and signature, all agents can restore the communication confidence matrix by multiplying the vectors, wherein in the election phase, after calculating the confidence matrix, all agents calculate the adjacency graph and select the agent with the largest degree, that is, most of the agents hope to obtain the message of this agent to make a decision, so it can serve as a leader node, wherein in the communication phase, all non-leader nodes send their messages to the leader node, the leader encodes the received message through the encoder network, and then communicates between leaders, and the leader passes the message to other leaders, wherein in the decision phase, the leader makes a decision based on the received messages from other leaders, and sends its own decision and message to other non-leader agents in the same group, and other agents make the next decision accordingly.

[0059] Through the above-mentioned grouped communication mode, the present invention effectively reduces the cost of communication, reduces communication links, and realizes efficient communication learning of multi-agent reinforcement learning, thereby realizing the calculation and measurement of the importance of messages through KL divergence in multi-agents; the grouping of agents is realized by using the spectral clustering algorithm on the confidence matrix, thereby reducing communication links; the election within the group is carried out through the out-degree of the graph, and the leader node is selected from it to realize communication between groups and reduce the amount of communication.

[0060] like Figure 4 As shown, the feasibility of the present invention has been proven through sufficient experiments and verified in the open source multi-agent reinforcement learning environment of OpenAI. It can be found that the present invention can help improve the cooperation between multi-agents and maximize the global reward.

[0061] like Figure 5As shown, the communication volume of the present invention will gradually and steadily decrease with the training. At the beginning, the intelligent agents learn to improve cooperation through communication, so the communication volume increases rapidly. As the training continues, the grouping method starts to work and gradually reduces the communication volume.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-agent communication learning method, characterized in that: include: CriticNet, wherein the CriticNet is used to calculate the importance of communication during the training phase and to train the corresponding three networks on the end device, namely ActorNet, PriorNet and EncoderNet; ActorNet, where the ActorNet is used to select the corresponding action on the agent side, acts on the agent side, and works in both the training phase and the execution phase. The ActorNet needs to learn the agent's strategy π in the training phase, and then generate the corresponding action based on the local observation and the received message That is in is the message received by agent i at time t, It expresses the local observation of agent i at each time t; PriorNet, where the PriorNet is used by the agent to select the object of communication. PriorNet will evaluate the agent observed in the local observation and output an importance value EncoderNet, wherein the EncoderNet is used by the agent to encode its own information to reduce the size of the message body.

2. The multi-agent communication learning method according to claim 1, characterized in that: The CriticNet runs in the cloud and works only in the training phase. By calculating the global reward and communication priority, the CriticNet will calculate the network loss and pass the gradient back to the rest of the network and update the rest of the network parameters.

3. The multi-agent communication learning method according to claim 1, characterized in that: When the importance value exceeds a certain threshold, it means that the current agent i needs to obtain the information of agent j to make decisions.

4. The multi-agent communication learning method according to claim 1, characterized in that: The agent encodes its previous actions together with observations for reference by other agents, thereby improving the stability of cooperation.

5. The multi-agent communication learning method according to claim 1, characterized in that: Further including a method for calculating importance, the steps are as follows: Step A: Remove the message of agent j and observe whether it will lead to the output action of ActorNet network; Step B: Since the action output by ActorNet is the distribution of an action set, KL divergence is used to calculate the difference between the action distributions output by the agent's ActorNet. The specific formula is as follows: Step C: Where o {i} represents the set of messages observed by agent i from all other agents, o {{i}\j} represents the set of messages of the other observed agents except agent j. The difference calculated by this formula indicates whether the decision distribution of messages without agent j is consistent with the decision distribution of messages with agent j. Step D: If the difference is large, it means that the message of agent j is very important to agent i, so its communication confidence is relatively high; Step E: After calculating the confidence of all agents, a confidence matrix M between agents is obtained, and the agents are grouped according to this confidence matrix.

6. The multi-agent communication learning method according to claim 1, characterized in that: Further including a distributed grouping method, the PriorNet network outputs two values, namely query and signature: the signature vector is the information fingerprint of the agent itself, which contains the encoding of the position and label of the agent itself; the query vector is the query information, which represents the encoding of the set of agents with which the agent needs to communicate.

7. The multi-agent communication learning method according to claim 6, characterized in that: It further includes a communication mechanism, which includes a handshake phase, an election phase, a communication phase and a decision phase. In the handshake phase, all agents will broadcast the query and signature to the agents in the observation. After receiving the query and signature, all agents restore the communication confidence matrix by multiplying the vectors. In the election phase, after calculating the confidence matrix, all agents calculate the adjacency graph and select the agent with the largest degree as the preset agent, that is, most of the agents hope to obtain the message of the preset agent to make a decision. The preset agent is the leader node. In the communication phase, all non-leader nodes send their messages to the leader node. The leader encodes the received message through the encoder network, and then communicates between leaders. After the leader passes the message to other leaders, in the decision phase, the leader makes a decision based on the received messages from other leaders, and sends its own decision and message to other non-leader agents in the same group, and other agents make the next decision based on this.

Images