A multi-agent information interaction method fusing local target features and cooperative features

Abstract

The application discloses a kind of multi-agent information interaction methods of fusing local target feature and collaborative feature.The method includes: (1) initialize "multi-target collaborative navigation" task scene, wherein each agent includes an action generator and an action evaluator;(2) construct agent information interaction channel, which extracts local target feature and collaborative feature according to the local observation of each agent at time t, respectively encoded as two types of interaction information at time t, and sent to each interaction object;(3) for each agent, fuse the local observation and the received each interaction information according to the hierarchical importance weight, obtain the global observation input into the action generator, and output the action of the agent;(4) cycle steps 2-3, collect interaction trajectory samples;(5) use loss function La and Lc to train the action generator and the action evaluator of each agent to maximize the team reward and achieve the goal of completing the collaborative task.

Description

Technical Field

[0001] This invention relates to the field of multi-agent reinforcement learning, and specifically to a multi-agent information interaction method that integrates local target features and cooperative features. Background Technology

[0002] Multi-agent reinforcement learning (MAL) is a sub-topic within deep reinforcement learning that combines with multi-agent systems. It extends and expands upon single-agent reinforcement learning algorithms to multi-agent environments, primarily used to generate behavioral policies for collaborative tasks. The goal of MML is to maximize the cumulative reward of the group. However, in multi-agent collaborative environments, each agent, due to limitations in its perception range or sensor noise, can only obtain partial observations. Its understanding of the environmental and teammate states is incomplete, making it difficult to consider the impact of other agents on itself and the overall team when updating its own behavioral policies. Therefore, it struggles to generate behavioral policies with team collaboration capabilities.

[0003] To address these issues, multi-agent information interaction methods have emerged. These methods explicitly assume information sharing between agents. During collaborative strategy training, agents extract and transmit features from their own local observations, and receive and fuse local features from teammates. This supplements their understanding of the incompletely observable environment and the states of their teammates, thereby enhancing their collaborative capabilities and generating collaborative behavioral strategies that maximize team rewards.

[0004] Existing collaborative strategy generation methods based on information interaction can establish discrete or continuous communication channels between agents. For the generation of shared information, methods such as value Q-networks or message networks can be used, with network parameters updated in reverse through team rewards. The generated shared information is then broadcast among agents. For message fusion, mathematical operations such as summation and averaging can be used to fuse messages, and an attention mechanism is further introduced to autonomously learn the importance weights of each message. However, most existing information interaction methods follow a share-observation model, primarily embedding and sharing features from the agent's current local observations. This model lacks utilization of the topological features of the collaborative environment.

[0005] In summary, current information interaction methods among multiple agents mainly address the problem of incomplete perception of environmental and teammate states by agents through local feature representation and sharing of local features among agents, thereby improving the decision-making accuracy of collaborative strategies and the completion rate of collaborative tasks. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a multi-agent information interaction method that integrates local target features and collaborative features. In a multi-agent autonomous collaborative decision-making environment, based on graph neural networks and multi-layer weight allocation, local target information and collaborative information perceived by each agent within its limited perception range are shared with other agents. This enhances the agents' understanding of task execution in areas they cannot perceive, enabling them to consider more comprehensive target information and teammate status when making decisions. This fosters collaboration, facilitates rational division of labor within the team, avoids internal competition among multiple agents for the same target, and improves the team's task completion efficiency. This method can be applied to multi-target collaborative navigation scenarios in fields such as unmanned systems (unmanned surface vessels) swarms.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A multi-agent information interaction method that integrates local target features and cooperative features includes the following steps:

[0009] Step 1: Initialize the "Multi-Target Collaborative Navigation" task scenario. The task objective is for multiple agents to collaboratively reach all target points without collisions or conflicts. Each agent includes an action generator and an action evaluator. The process is as follows: Initialize the "Multi-Target Collaborative Navigation" task scenario. The task objective is for each agent to autonomously choose a suitable target and collaboratively reach its respective target point without collisions or internal target competition. This scenario is the foundation for agent interaction, providing observation information to the agents. The actions of the agents affect this scenario, changing its real-time state. The scenario updates the agents' action strategies based on the feedback reward information from the action execution effect. Initialize the number of agents as N and the number of target points as L. Initialize the local perception range, position, and velocity of the N agents and the positions of the L target points. The number of targets observed within the local perception range is l, and the number of teammate agents is j. Initialize the parameters of the action generator and action evaluator for the N agents.

[0010] Step 2: Construct an intelligent agent information interaction channel. This channel extracts local target features and cooperative features based on the local observations of each agent at time t, encodes them into two types of interaction information at time t, and sends them to each interaction object. The process is as follows:

[0011] (2-1) Obtaining Local Observation Information of Agent i: From the initial scene state in step 1, extract the local observations of agent i, including the agent's own position and velocity information on the x-axis and z-axis. i ={v x ,v z ,p x ,p z}; Location information of l target entities observed within the perception range of agent i. Position and velocity information of j cooperating teammates within the perception range

[0012] (2-2) Local Target Feature Extraction and Interaction Information Generation: The position information of l target entities observed within the perception range of agent i is input into a multilayer perceptron and encoded as a target feature vector. The l target features and the features of agent i itself are used as nodes to form a target relationship graph. A graph neural network is used to extract graph features as target interaction information E. t This includes the topological features between the target and the agent, which can reflect the agent's target selection preferences.

[0013] (2-3) Local Collaborative Feature Extraction and Interaction Information Generation: The position and velocity information of j cooperative teammates within the perception range of agent i are input into a multilayer perceptron and encoded as teammate feature vectors. The features of j teammates and the features of agent i itself are used as nodes to form a collaborative relationship graph. A graph neural network is used to extract graph features as collaborative interaction information T. t This includes the topological features between agents, which can reflect the local cooperative state between agents.

[0014] (2-4) Interactive Information Transmission: Transmitting the target interactive information E t With collaborative interaction information T t Simultaneously, the interaction information is passed to the interaction object, and each agent receives the following interaction information at time t:

[0015] Step 3: For each agent, its local observations and received interaction information are fused according to hierarchical importance weights to obtain global observations, which are then input into the action generator to output the agent's actions; the process is as follows:

[0016] (3-1) First-layer interactive information fusion: based on importance weight α N-1 and β N-1 Weighted interaction information of the same type is applied. Target interaction information is then fused into... Collaborative interactive information fusion

[0017] (3-2) Second-layer interactive information fusion: based on importance weight ω E and ω T Different types of interactive information are weighted and merged into a global observation. Through layered fusion, intelligent agents can flexibly choose to focus on target selection or cooperate at different stages of multi-target navigation.

[0018] (3-3) Action selection: The global observation obtained by fusing interaction information The input is fed into the action generator, which consists of a multi-layered sensing mechanism that maps global observations into actions to be performed and outputs them.

[0019] Step 4: Repeat steps 2-3 to collect interaction trajectory samples; the process is as follows: collect interaction trajectory samples, including environmental observations, action selections, and rewards for environmental feedback at each moment.

[0020] Step 5: Train the action generator and action evaluator for each agent using loss functions La and Lc to maximize team rewards and achieve the goal of completing the collaborative task. The process is as follows:

[0021] (5-1) Update the action evaluator using the loss function Lc, specifically as follows:

[0022]

[0023] Where φ represents the parameters of the motion evaluator. These are the parameters for the information exchange channel. r represents the expected value at time t. t The reward is the environmental feedback, and γ is a hyperparameter. The global observation obtained at time t after information exchange The action evaluation value is obtained as input.

[0024] (5-2) Update the action generator using the loss function La, specifically:

[0025]

[0026] Where θ represents the motion evaluator parameters. Here, c represents the parameters of the information exchange channel, and S represents the information entropy reward; π represents the hyperparameter. θ This represents the policy output of the action generator; This represents the agent's global observation at time t.

[0027]

[0028] in, π represents the ratio of the old action generator to the new action generator in a single iteration update. θ (a t |gcu t ) represents the new action generator π θ The probability of observing gcu at time t and outputting action a. Represents the old action generator The probability of observing gcu at time t and outputting action a is A. tdenoted as the advantage function, which is calculated from the reward value and the action value output by the action evaluator, and ∈ represents a preset constant used to control the magnitude of policy updates.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] This invention adds two types of topological structure information: local target features and collaborative features, to the extraction of topological structure information within the collaborative environment. This results in richer interactive information that is more conducive to collaboration. By introducing a graph neural network, the spatial structural relationships between target nodes and teammate agent nodes within the agent's local perception range are extracted. Then, the topological structure information and the agent's local observation information are combined to generate shared interactive information. Sharing this interactive information containing topological structure features allows agents in "multi-target collaborative navigation" tasks to consider not only global target information but also the state information of teammates when making decisions. This fosters collaborative cooperation and reasonable division of labor within the team, avoiding internal competition among multiple agents for the same target, thereby improving the team's task completion efficiency. Attached Figure Description

[0031] Figure 1 A flowchart of a multi-agent information interaction method that integrates local target features and collaborative features.

[0032] Figure 2 A flowchart illustrating the specific implementation of a multi-agent information interaction method that integrates local target features and collaborative features.

[0033] Figure 3 This is a simulation diagram of a "multi-target cooperative navigation" task scenario. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0035] This invention proposes a multi-agent information interaction method that integrates local target features and collaborative features. Building upon existing multi-agent information interaction methods that rely on sharing and observation, it adds shared topological structure information. To extract topological structure information from the collaborative environment and generate more favorable interaction information, a graph neural network is introduced. This network extracts the spatial structural relationships between the agent and various target nodes, as well as with teammate agent nodes, within the agent's local perception range. Then, it combines the topological structure information with the agent's local observation information to generate the shared interaction information. Sharing this interaction information, which incorporates topological features, allows agents to consider more comprehensive target information and teammate states during decision-making, enabling collaborative and reasonable division of labor within the team. This avoids internal competition among multiple agents for the same target, thereby improving the team's task completion efficiency.

[0036] The basic features of this invention are mainly as follows: First, it generates target interaction information E. t This includes the topological features between the target and the agent; sharing this type of interaction information can reflect the agent's target selection preferences. Secondly, it generates collaborative interaction information T. t This includes topological features between agents; sharing this type of interaction information can reflect the local collaborative state between agents. Thirdly, when fusing the two types of interaction information, a hierarchical importance weighting allocation is used: first, interaction information of the same type is weighted, and then interaction information of different types is fused. Through hierarchical fusion, agents can flexibly choose to focus on target selection or cooperate at different stages of multi-target navigation, forming more effective collaboration.

[0037] like Figure 1 As shown, a multi-agent information interaction method that integrates local target features and cooperative features is implemented in the following steps:

[0038] Step 1: Initialize the "Multi-Target Collaborative Navigation" task scenario. The task objective is for multiple agents to collaboratively reach all target points without collisions or conflicts. Each agent includes an action generator and an action evaluator. The specific process is as follows: Initialize the "Multi-Target Collaborative Navigation" task scenario. The task objective is for each agent to autonomously choose a suitable target and collaboratively reach its respective target point without collisions or internal target competition. This scenario is the foundation for agent interaction, providing observation information to the agents. The actions of the agents affect this scenario, changing its real-time state. The scenario updates the agents' action strategies based on the feedback reward information from the action execution effects. Initialize the number of agents as N and the number of target points as L. Initialize the local perception range, position, and velocity of the N agents and the positions of the L target points. The number of targets observed within the local perception range is l, and the number of teammate agents is j. Initialize the parameters of the action generator and action evaluator for the N agents.

[0039] Step 2: Construct an intelligent agent information interaction channel. This channel extracts local target features and cooperative features based on the local observations of each agent at time t, encodes them into two types of interaction information at time t, and sends them to each interaction object. The specific process is as follows:

[0040] (2-1) Obtaining Local Observation Information of Agent i: From the initial scene state in step 1, extract the local observations of agent i, including the agent's own position and velocity information on the x-axis and z-axis. i ={v x ,v z ,p x ,p z}; Location information of l target entities observed within the perception range of agent i. Position and velocity information of j cooperating teammates within the perception range

[0041] (2-2) Local Target Feature Extraction and Interaction Information Generation: The position information of l target entities observed within the perception range of agent i is input into a multilayer perceptron and encoded as a target feature vector. The l target features and the features of agent i itself are used as nodes to form a target relationship graph. A graph neural network is used to extract graph features as target interaction information E. t This includes the topological features between the target and the agent, which can reflect the agent's target selection preferences.

[0042] (2-3) Local Collaborative Feature Extraction and Interaction Information Generation: The position and velocity information of j cooperative teammates within the perception range of agent i are input into a multilayer perceptron and encoded as teammate feature vectors. The features of j teammates and the features of agent i itself are used as nodes to form a collaborative relationship graph. A graph neural network is used to extract graph features as collaborative interaction information T. t This includes the topological features between agents, which can reflect the local cooperative state between agents.

[0043] (2-4) Interactive Information Transmission: Transmitting the target interactive information E t With collaborative interaction information T t Simultaneously, the interaction information is passed to the interaction object, and each agent receives the following interaction information at time t:

[0044] Step 3: For each agent, its local observations and received interaction information are fused according to hierarchical importance weights to obtain global observations, which are then input into the action generator to output the agent's actions. The specific process is as follows:

[0045] (3-1) First-layer interactive information fusion: based on importance weight α N-1 and β N-1 Weighted interaction information of the same type is applied. Target interaction information is then fused into... Collaborative interactive information fusion

[0046] (3-2) Second-layer interactive information fusion: based on importance weight ω E and ω T Different types of interactive information are weighted and merged into a global observation. Through layered fusion, intelligent agents can flexibly choose to focus on target selection or cooperate at different stages of multi-target navigation.

[0047] (3-3) Action selection: The global observation obtained by fusing interaction information The input is fed into the action generator, which consists of a multi-layered sensing mechanism that maps global observations into actions to be performed and outputs them.

[0048] Step 4: Repeat steps 2-3 to collect interaction trajectory samples. The specific process is as follows: collect interaction trajectory samples, including environmental observations, action selections, and rewards for environmental feedback at each moment.

[0049] Step 5: Train the action generator and action evaluator for each agent using loss functions La and Lc to maximize team rewards and achieve the goal of completing the collaborative task. The specific process is as follows:

[0050] (5-1) Update the action evaluator using the loss function Lc, specifically as follows:

[0051]

[0052] Where φ represents the parameters of the motion evaluator. These are the parameters for the information exchange channel. r represents the expected value at time t. t The reward is the environmental feedback, and γ is a hyperparameter. The global observation obtained at time t after information exchange The action evaluation value is obtained as input.

[0053] (5-2) Update the action generator using the loss function La, specifically:

[0054]

[0055] Where θ represents the motion evaluator parameters. Here, c represents the parameters of the information exchange channel, and S represents the information entropy reward; π represents the hyperparameter. θ This represents the policy output of the action generator; This represents the agent's global observation at time t.

[0056]

[0057] in, π represents the ratio of the old action generator to the new action generator in a single iteration update. θ (a t |gcu t ) represents the new action generator π θ The probability of observing gcu at time t and outputting action a. Represents the old action generator The probability of observing gcu at time t and outputting action a is A.t denoted as the advantage function, which is calculated from the reward value and the action value output by the action evaluator, and ∈ represents a preset constant used to control the magnitude of policy updates.

[0058] Experimental Description and Results: The simulation environment is a collaborative task scenario of "multi-target cooperative navigation". Source: https: / / github.com / sumitsk / matrl.git The task objective is for multiple agents to collaboratively navigate to various target points without collisions. The more target points reached and the more accurate the target point coverage, the higher the task completion rate. The simulation environment is set as follows: a 2D scene size of 20 units * 20 units, an agent size of 1 unit, a target point size of 1 unit, 3 or 6 agents, and 3 or 6 target points. The local perception range of the agents is a unit circle with a radius of 5. Agent action settings include discrete action types and 5 action categories (forward, backward, left, right, stop). Agent state settings include agent speed + agent position coordinates + target point position coordinates within the perception range + teammate agent speed + position coordinates within the perception range. The reward setting is the sum of the negative distances between all target points and their nearest agents. Table 1 shows the comparison results between the method of this invention and existing benchmark methods.

[0059] Table 1. Experimental Results (Average results from 10,000 tests)

[0060]

[0061] Table 1 shows the comparison results between the method of this invention and various benchmark methods in the "multi-target cooperative navigation" task. We trained the method of this invention and the benchmark methods in two settings with 3 and 6 agents. During the training process, the initial positions of the agents and the target points were randomized. The training rounds were 7000, and the maximum number of steps to interact with the environment in each round was 50.

[0062] Three benchmarks were used for comparison: S% represents the task success rate; DIST represents the target coverage, i.e., the average distance between the target point and the nearest agent (the smaller the better); and AEL represents the average number of steps completed, i.e., the number of steps required to complete one round of the task (the smaller the better). Existing benchmark methods used for comparison include Q-MIX, COMA, MADDPG, and SAEG.

[0063] Among them, the Q-MIX and COMA methods lack information interaction mechanisms and cannot complete the "multi-objective cooperative navigation" task even when set to fully observable. A comparison with these two methods demonstrates the importance of feature sharing in completing incompletely observable cooperative tasks. The MADDPG method uses a centralized training framework with decentralized decision-making, exhibiting an implicit information interaction process and performing well with three agents. However, it lacks the addition of topological structure information; a comparison with this method demonstrates the importance of adding topological structure information. The SAEG method has an explicit information interaction process and uses graph neural networks to represent local features, but it does not consider the dual-interaction information sharing and fusion mechanism of local target features and cooperative features. A comparison with this method demonstrates the importance of fusing local target features and cooperative features.

[0064] It can be seen that the method of this invention improves upon existing methods in terms of success rate, target coverage, and average number of steps completed. In a task setting with 3 agents, the method of this invention has the highest task completion rate and requires the fewest steps to complete the task. In a task setting with 6 agents, the increased number of agents and the increased task difficulty mean that only SAEG and the method of this invention can complete the multi-target cooperative navigation task. The method of this invention also has the highest task completion rate and better target point coverage, demonstrating more effective information interaction and better cooperation between agents.

Claims

1. A multi-agent information interaction method that integrates local target features and cooperative features, characterized in that, Includes the following steps: Step 1: Initialize the "Multi-target Collaborative Navigation" task scenario. The task objective is for multiple agents to reach all target points collaboratively without collisions or conflicts. Each agent includes an action generator and an action evaluator. Step 2: Construct an intelligent agent information interaction channel. Based on the local observations of each intelligent agent at time t, this channel extracts local target features and cooperative features, encodes them into two types of interaction information at time t, and sends them to each interaction object. Step 3: For each agent, its local observations and the received interaction information are fused according to the hierarchical importance weights to obtain global observations, which are then input into the action generator to output the agent's actions. Step 4: Repeat steps 2-3 to collect interaction trajectory samples; Step 5: Train the action generator and action evaluator for each agent using loss functions La and Lc to maximize team rewards and complete the collaborative task. The process of step 2 is as follows: (2-1) Obtaining Local Observation Information of the Agent: From the initial scene state in step 1, extract the local observations of agent i, including the agent's own position and velocity information, and the observations within the perception range of agent i. l Location information of the target entity, within the sensing range j Location and speed information of each cooperating teammate; (2-2) Local target feature extraction and interaction information generation: This involves extracting local target features observed within the perception range of agent i. l The location information of each target entity is encoded into a target feature vector, and then... l The target features and the agent's own features are used as nodes to form a target relationship graph, and graph features are extracted as target interaction information. This includes the topological features between the target and the agent, which can reflect the agent's target selection preferences; (2-3) Local cooperative feature extraction and interaction information generation: Within the perception range of agent i j The position and velocity information of each cooperating teammate are encoded into a teammate feature vector, and then... j The features of each teammate and the features of agent i itself are used as nodes to form a collaborative relationship graph. The graph features are extracted as collaborative interaction information. This includes the topological features between agents, which can reflect the local cooperative state between agents; (2-4) Interactive Information Transmission: Two types of interactive information are transmitted to the interactive object, and each agent receives... .

2. The multi-agent information interaction method that integrates local target features and cooperative features according to claim 1, characterized in that, Step 1 is as follows: Initialize the "multi-target cooperative navigation" task scenario. The task objective is for each agent to autonomously select a suitable target and cooperate to reach its respective target point without collisions or internal target competition. Initialize the number of agents to N and the number of target points to L. Initialize the local perception range, position, and velocity of the N agents and the positions of the L target points, where the number of targets observed within the local perception range is... l The number of teammate agents is j Initialize the parameters of N agent action generators and action evaluators.

3. The multi-agent information interaction method that integrates local target features and cooperative features according to claim 1, characterized in that, The process of step 3 is as follows: (3-1) First-layer interactive information fusion: based on importance weight and Weighted interaction information of the same type is weighted, and the target interaction information is fused into Collaborative and interactive information fusion ; (3-2) Second-layer interactive information fusion: based on importance weight and Different types of interactive information are weighted and merged into a global observation. Through layered fusion, the intelligent agent can flexibly choose to focus on target selection or cooperate at different stages of multi-target navigation. (3-3) Action selection: global observations obtained by fusing interactive information The input is fed into the action generator, which consists of a multi-layered sensing mechanism that maps global observations into actions to be performed and outputs them.

4. The multi-agent information interaction method that integrates local target features and cooperative features according to claim 1, characterized in that, Step 4 involves collecting interaction trajectory samples, which include environmental observations, action selections, and rewards for environmental feedback at each moment.

5. The multi-agent information interaction method that integrates local target features and cooperative features according to claim 1, characterized in that, The process of step 5 is as follows: (5-1) Update the action evaluator using the loss function Lc, specifically as follows: (1); in, These are the parameters for the motion evaluator, and these are the parameters for the information interaction channel. This represents the expected value at time t. As a reward for environmental feedback, For hyperparameters, The global observation obtained at time t after information exchange The action evaluation value is obtained as input; (5-2) Update the action generator using the loss function La, specifically: (2); in, For motion evaluator parameters, These are the parameters for the information exchange channel. C S is a hyperparameter; S represents the information entropy reward. This represents the policy output of the action generator; This represents the agent's global observation at time t; (3); in, This represents the ratio of the old action generator to the new action generator in a single iteration. Indicates a new action generator At time t, the observation is as follows Output the probability of action 'a'. Represents the old action generator At time t, the observation is as follows Output the probability of action 'a'. The advantage function is calculated from the reward value and the action value output by the action evaluator. This represents a preset constant used to control the magnitude of policy updates.

Images