Decomposed value evaluation multi-agent cooperative navigation method based on attention mechanism

Abstract

The application discloses a decomposition type value evaluation multi-agent cooperative navigation method based on an attention mechanism, and comprises the following steps: modeling a cooperative navigation task of a multi-agent as a decentralized partially observable Markov decision process; designing a pre-encoding module and a local attention module with two-stage attention encoding to extract and encode environment feature information of the multi-agent, and the value network and the policy network share the local attention module; designing a multi-agent policy-value method under decomposition type value evaluation; constructing a static target navigation environment to guide a mobile agent to actively find an optimal position; constructing a cooperative hunting environment; constructing a multi-agent cooperative navigation model, and obtaining an optimal model through training and testing, so that the multi-agent cooperation with a confidence distribution problem is realized to achieve a navigation without explicit and complete perception. The cooperative navigation method has faster convergence, higher success rate, shorter time and less collision, and has good generalization ability and robustness.

Description

Technical Field

[0001] This invention relates to the field of cooperative control of multi-agent systems, and more specifically to a decompositional value assessment method for multi-agent cooperative navigation based on attention mechanisms. Background Technology

[0002] With the development of mobile intelligent agent technology and the increasing complexity of production task environments, multi-agent systems have become a research hotspot in the fields of intelligent agents and artificial intelligence, achieving significant development results. Compared with single-agent systems, they are more accurate and robust in environmental perception, and can decompose complex tasks into multiple relatively simple sub-tasks to improve work efficiency. They are widely used in logistics distribution, emergency rescue, and security inspection.

[0003] Currently, mainstream multi-agent cooperative control algorithms fall into two categories: centralized and distributed. Centralized algorithms utilize a central controller to aggregate all state information to make decisions and output joint actions. While this ensures optimal cooperative control decisions, it suffers from numerous problems, including high computational complexity, dimensionality explosion, and excessive redundant state information. Furthermore, obtaining accurate global state information is difficult in reality, and its robustness is also low; failure of the central controller can lead to system paralysis. In distributed algorithms, each agent makes decisions based on its own local observations, resulting in good computational efficiency and facilitating the expansion of multi-agent systems. However, due to the lack of global state information and a global perspective, it is prone to making suboptimal decisions. Some distributed algorithms also rely on explicit communication mechanisms or require perfect perception of the surrounding environment, which is difficult to achieve in real-world scenarios.

[0004] Regarding the information allocation problem in cooperative control of multi-agent systems, the following key challenges remain. First, communication limitations are a major challenge. In complex environments with signal obstruction and strong interference, communication quality easily degrades, leading to untimely and inaccurate information allocation. Simultaneously, limited communication bandwidth struggles to meet the demands of numerous agents simultaneously transmitting large amounts of data, necessitating research into efficient compression and transmission technologies. Second, when multiple agents perform tasks in the same area, a large amount of duplicate or conflicting information may be generated, increasing processing burden and reducing decision-making efficiency. Intelligent information filtering and fusion mechanisms need to be designed. Finally, when the environment changes, the agents' information allocation strategies need to be rapidly adjusted. Existing adaptive capabilities are insufficient to maintain optimal information allocation consistently in complex and ever-changing environments. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a decompositional value evaluation multi-agent cooperative navigation method, MAVFAC, based on an attention mechanism. This method solves the problem of reliability allocation in multi-agent scenarios caused by the fact that the traditional MLGA2C algorithm still learns a centralized Q-value, which makes it impossible to effectively measure the contribution of each agent to the overall cooperative task and makes it difficult to refine the reward function of individuals. The algorithm is also prone to generating false rewards and creating lazy agents.

[0006] To achieve the above-mentioned technical objectives, the present invention provides the following technical solution:

[0007] A decompositional value assessment multi-agent cooperative navigation method based on attention mechanisms includes the following steps:

[0008] S1. Model the multi-agent cooperative navigation task as a decentralized partially observable Markov decision process;

[0009] S2. Design a local attention module to extract and encode environmental feature information of multiple agents. The local attention module includes a pre-encoding network and an attention encoding module, and has a mechanism for the critic network and actor network to share the local attention module.

[0010] S3. Design a multi-agent actor-critic method under decompositional value assessment to solve the centralized Q-value problem;

[0011] S4. Construct a static target navigation environment to guide the mobile intelligent agent to actively find the best position, and limit the control details of the state space, action space and reward function involved in the static target navigation task.

[0012] S5. Construct a cooperative encirclement environment and limit the control details of the state space, action space, and reward function involved in the cooperative encirclement task;

[0013] S6. Construct a multi-agent cooperative navigation model following steps S1 to S5. Obtain the optimal model through training and testing to achieve explicit and fully perceptive navigation for multi-agent cooperative navigation with confidence allocation problems.

[0014] Furthermore, step S1 specifically includes:

[0015] The collaborative navigation task is modeled as a decentralized, partially objective Markov decision process, represented as:

[0016] U =<I,S,O,Ω,A,T,R> ;

[0017] Where U represents the decentralized partially observable Markov decision process; I represents the set of agents; S represents the global state information of the environment; O represents the set of states observed locally by each agent; Ω is the observation function; A is the joint action of the agents; T is the transition function; and R is the immediate reward.

[0018] After modeling, the agent selects actions based on the current state information and policy, and transitions to the next state according to the transition function, while receiving an immediate reward for each agent. The immediate reward then guides the agent to make better action decisions until the navigation task is completed.

[0019] Furthermore, the precoding network in the local attention module in step S2 is specifically as follows:

[0020] In the local attention module, each entity is equipped with an independent precoding network. Each precoding network consists of a fully connected embedding layer, a leaky ReLU activation function, and a LayerNorm. The precoding of each agent i is first obtained through the precoding network. The formula is expressed as:

[0021]

[0022] Among them, f self It is a self-intelligent agent precoding network. It is the information vector of the agent i itself, which includes its global position and global velocity;

[0023] Pre-encoding of other different categories of entities observed locally by agent i The formula is expressed as:

[0024]

[0025] Where, n m Let m represent other different categories of entities locally observed by agent i, where m = 1, 2, ..., M; these other different categories of entities include landmarks, other agents, obstacles, and target prey. K represents the information vectors of other different categories of entities. <N m N is the number of entity categories within the field of vision of agent i. m This represents the default number of entity categories that agent i can observe, when K <N m When using placeholder zeros to fill the vector space, Precoding networks representing other different categories of entities.

[0026] Furthermore, the attention encoding module in the local attention module in step S2 is specifically as follows:

[0027] The attention encoding module contains three types of attention matrices: query matrix and query matrix. Key matrix Sum matrix Furthermore, an independent key matrix is ​​assembled for each type of entity. Sum matrix However, only the agent itself possesses the query matrix Q. i The formula for the mapping relationship of each attention matrix is ​​expressed as follows:

[0028]

[0029]

[0030]

[0031] in, This represents the pre-encoding set of various entities locally observed by agent i; finally, the local attention encodings of various entities are obtained by weighted dot product. The formula is expressed as:

[0032]

[0033] in, Attention weights are used to measure other types of entity precoding. The degree of correlation with the self-agent agent; the local attention encoding of the self-agent agent is determined by its pre-encoding. It was pieced together.

[0034] Furthermore, step S3 specifically refers to:

[0035] S31. Construct a global Q-network consisting of a local critic network, an attention mix network, and a target network. The local critic network is a double-local Q-network, using a truncated double-Q technique to avoid overestimating the action-state value function. The target network is used for stable updates, obtaining the parameters of the local critic network and the attention mix network through a soft update method. The local critic network receives the spliced ​​local attention encoding of its own agent. and agent action a i Output local Q-value q i (o,a), the formula is expressed as:

[0036]

[0037]

[0038] in, Pre-encoding for the intelligent agent i, Encode local attention for various entities; and It is a two-layer fully connected network, where ψ1 and ψ2 are the parameters of a bilocal Q-network;

[0039] S32. Feed the local Q-values ​​and local attention encodings into the attention mix module to calculate the global Q-value; first, calculate the weight α of the local Q-values ​​to the global Q-values ​​using the dot product attention mechanism. global The formula is expressed as:

[0040]

[0041]

[0042] Among them, the global query matrix Q gloabl Through a hybrid query matrix The global key K is obtained through mapping. global Through the hybrid bond matrix Mapped to obtain, Transpose it; Generated through random initialization and optimized and updated during training using a loss function based on the global Q-value via backpropagation; cat A sequence of local attention codes for the agent itself, containing n local attention codes for the agent itself;

[0043] α global Then, the global Q value Q(o,a) is obtained by weighted summation, and the formula is expressed as:

[0044] Q(o,a)=α global ·q+b;

[0045] The local Q-value sequence q consists of the local Q-values ​​of n agents; the bias b is used to better fit the value function, and the bias b is directly equal to q.

[0046] S33. After obtaining the global Q value, the joint regression loss function L is used... Q (ψ) Update the global Q-network, expressed by the formula:

[0047]

[0048] Where (o,a,r,o')~D represents the current state o, current action a, immediate reward r, and next state o' sampled from the experience replay pool D; the target value y i We obtain this using the Bellman equation, i.e.

[0049]

[0050] Where, πθ For the strategy used, log(π) θ (a' i |o' i )) represents the introduced policy entropy; α is the temperature coefficient that balances maximizing reward and maximizing policy entropy; global Q-value estimation. The calculation method is similar to that of the global Q-value, except that it uses the state o' and action a' of the next time step; i For the current immediate reward, γ∈[0,1] is the discount factor that balances the short-term reward and long-term benefit of the agent;

[0051] S34, Distributed Policy Networks Encode with Local Attention As input, it outputs actions; the distributed policy network relies on gradients provided by the global Q-network for updates; gradients The expression is as follows:

[0052]

[0053] Here, θ is the parameter of the distributed policy network; random action resampling technique avoids the non-differentiability problem of the policy network.

[0054] Furthermore, step S4 specifically includes:

[0055] S41. Set the state space of each agent in the static target navigation environment. i for:

[0056]

[0057] Where, p i and v i These are the global position and global velocity of agent i, respectively; It is the relative position between the landmark and the intelligent agent i. Indicates the relative position of other agents to agent i;

[0058] S42. Set the action space of each agent to force F. i =[f x ,f y ];f x and f y This indicates the force components generated by the actuator along the x-axis and y-axis;

[0059] S43. Set the objective function and constraints for the static target navigation task and the execution process, including:

[0060] The objective function is to minimize the average time for the swarm of agents to reach all target points:

[0061]

[0062] in, For arrival time, π θ The strategy chosen for the agent;

[0063] Arrival constraint, constraining agent i to arrive at time Arrive at the target

[0064]

[0065] Collision constraints constrain agent i to avoid collisions during operation:

[0066]

[0067] in, Let d be the Euclidean distance between agents i and j. coll This is the collision distance threshold;

[0068] S44. The reward function for each agent i when performing a static target navigation task is designed as follows:

[0069]

[0070] The distance reward, ensuring a single agent can navigate to a landmark, is calculated by determining the distances from all landmarks to agent i, expressed by the formula:

[0071]

[0072] Among them, w g Represents the distance coefficient. It is the Euclidean distance between agent i and the nth landmark;

[0073] Penalty for time step; The collision penalty is expressed by the formula:

[0074]

[0075] Where C1 and C2 are constants;

[0076] Final reward The reward for multi-agent collaborative navigation task completion, i.e.

[0077] Furthermore, step S5 specifically includes:

[0078] S51. Set the state space o' for each agent i in the cooperative trapping task environment. i for:

[0079]

[0080] Where, p i and v i These are the global position and global velocity of agent i, respectively; It is the relative position between the landmark and the intelligent agent i. Indicates the relative position of other agents to agent i; and These are the relative positions between the target prey and agent i, and the speed of the target prey itself;

[0081] S52. Set the objective function and constraints for the cooperative encirclement task as follows:

[0082] The objective function of a cooperative encirclement task is to minimize the average time required to encircle all prey.

[0083]

[0084] in, In order to catch up with time, π θ The strategy chosen for the agent

[0085] Encirclement constraint, constraining agent i to be in pursuit time Successfully pursued the target

[0086] δ represents the distance threshold for successful capture;

[0087] S53. Design the total reward function R for executing a cooperative encirclement mission. t for:

[0088]

[0089] The formula for capturing rewards is as follows:

[0090]

[0091] The distance penalty is expressed by the formula:

[0092]

[0093] Among them, w dist Represents the distance coefficient. It is the Euclidean distance between agent i and the target prey's location.

[0094] Based on the above technical solution, the present invention has at least the following beneficial effects:

[0095] 1. The method proposed in this invention can work in scenarios with limited perception and no explicit communication; it alleviates the common confidence allocation problem in multi-agent cooperative control scenarios.

[0096] 2. This invention designs a phased and categorized encoding method that enables an agent to dynamically assess the importance of surrounding entities to its decision-making based on its own state; thereby focusing on extracting local feature information at the entity level within the agent, providing more accurate local encoding for subsequent cooperative control decisions, and achieving more refined feature extraction.

[0097] 3. This invention designs a decompositional value assessment multi-agent actor-critic method as a decision module for cooperative control; by mixing local Q-values ​​and local attention encodings of each agent through an attention hybridization module, each agent can directly output actions, and this process is completely independent of other agents, without the need for central controller coordination; it achieves effective differentiation of the global Q-value of each agent's contribution to the overall cooperative task, thereby guiding the agents' policy learning more accurately and efficiently. Attached Figure Description

[0098] Figure 1 This is a system flowchart of the multi-agent cooperative navigation method MAVFAC based on attention mechanism decomposition value evaluation proposed in this invention.

[0099] Figure 2 This is a network structure block diagram of the MAVFAC method proposed in this invention;

[0100] Figure 3 This is a diagram showing the overall training architecture of the MAVFAC method proposed in this invention;

[0101] Figure 4 This is a schematic diagram illustrating a scenario of multi-agent collaborative task execution in this invention; Figure 4 In the middle (a) and (b), the static target navigation task and the cooperative encirclement task are respectively represented;

[0102] Figure 5 This is the training reward curve for static target navigation in an intelligent agent environment in this invention; Figure 5 In the figure, (a), (b), and (c) are the training reward curves for 3, 6, and 9 agents cooperating, respectively.

[0103] Figure 6 This is the training reward curve for cooperative encirclement in a 3-agent environment in this invention; Figure 6 In the figure, (a), (b), and (c) are the training reward curves for 3, 6, and 9 agents cooperating, respectively.

[0104] Figure 7This is a schematic diagram comparing the reward curves of MAVFAC, baseline, and ablation in a 9-agent static target navigation scenario in this invention. Detailed Implementation

[0105] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the following description is provided in conjunction with the accompanying drawings. Figure 1-7 The present invention will be further described in detail below with reference to specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.

[0106] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] like Figure 1 As shown, this invention proposes a decompositional value evaluation multi-agent cooperative navigation method based on an attention mechanism, which specifically includes the following steps:

[0108] S1. Model the multi-agent cooperative navigation task as a decentralized partially observable Markov decision process;

[0109] In a preferred embodiment, step S1 specifically comprises:

[0110] The collaborative navigation task is modeled as a decentralized, partially objective Markov decision process, represented as:

[0111] U =<I,S,O,Ω,A,T,R> ;

[0112] Where U represents the decentralized partially observable Markov decision process; I represents the set of agents; S represents the global state information of the environment; O represents the set of states observed locally by each agent; Ω is the observation function; A is the joint action of the agents; T is the transition function; and R is the immediate reward.

[0113] After modeling, the agent selects actions based on the current state information and policy, and transitions to the next state according to the transition function, while receiving an immediate reward for each agent. The immediate reward then guides the agent to make better action decisions until the navigation task is completed.

[0114] S2. Design a local attention module to extract and encode environmental feature information of multiple agents. The local attention module includes a pre-encoding network and an attention encoding module, and has a mechanism for the critic network and actor network to share the local attention module.

[0115] In a preferred embodiment, the precoding network in the local attention module is specifically as follows:

[0116] In the local attention module, each entity is equipped with an independent precoding network. Each precoding network consists of a fully connected embedding layer, a leaky ReLU activation function, and a LayerNorm. The precoding of each agent i is first obtained through the precoding network, as expressed by the formula:

[0117]

[0118] Among them, f self It is a self-intelligent agent precoding network. It is the information vector of the intelligent agent i itself, which includes its global position and global velocity. It should be noted that, in this application, "intelligent agent i" refers to the main intelligent agent that is currently making decisions, and its observed information vector includes its own state (global position, velocity) and relative information of surrounding entities (landmarks, other intelligent agents, etc.).

[0119] The pre-encoding of other different categories of entities observed locally by agent i is expressed by the formula:

[0120]

[0121]

[0122] Where, n m Let m represent other different categories of entities locally observed by agent i, where m = 1, 2, ..., M; these other different categories of entities include landmarks, other agents, obstacles, and target prey. K represents the information vectors of other different categories of entities. <N m N is the number of entity categories within the field of vision of agent i. m This represents the default number of entity categories that agent i can observe, when K <N m When using placeholder zeros to fill the vector space, Precoding networks representing other different categories of entities;

[0123] The attention encoding module in the local attention module is specifically as follows:

[0124] The attention encoding module contains three types of attention matrices: query matrix and query matrix. Key matrix Sum matrix Furthermore, each type of entity is equipped with an independent key matrix and value matrix, but only the agent itself possesses a query matrix; the mapping relationship of each attention matrix is ​​expressed by the formula:

[0125]

[0126]

[0127]

[0128] in, This represents the pre-encoded set of various entities locally observed by agent i; finally, the local attention codes for each entity are obtained through dot product weighting. The formula is expressed as:

[0129]

[0130] in, Attention weights are used to measure other types of entity precoding. The degree of correlation with the self-agent agent; the local attention encoding of the self-agent agent is determined by its pre-encoding. and It was pieced together.

[0131] State is the core basis for decision-making; the state of an agent (such as position and velocity) directly affects its action choices. For example, in navigation tasks, an agent needs to determine its direction of movement based on its own position, and in encirclement tasks, it needs to adjust its pursuit strategy based on its own speed. This application's local attention module designs a two-stage encoding: first, it pre-encodes the state of the agent itself and other entities; then, it uses the pre-encoded state of the agent itself as the query and the pre-encoded states of other entities as the key and value to design local attention encoding. This design allows the agent to dynamically assess the importance of surrounding entities to its decision-making based on its own state. For example, if the agent needs to approach a target landmark, its own position will dominate the attention weight, filtering out landmark information related to the target. Different entities (such as landmarks, obstacles, and other agents) have different influence patterns on decision-making. For example, landmarks may represent target locations and require long-term attention; obstacles require collision avoidance and real-time response; the cooperative behavior of other agents requires dynamic coordination, etc.

[0132] Traditional methods typically encode all observation information in a mixed manner. However, the MAVFAC method proposed in this invention achieves more refined feature extraction through phased and categorized encoding. After the two-stage encoding in the aforementioned local attention module, the agent can better assess the importance of various entity information in local observations, gain a more accurate understanding of environmental information, and thus make better cooperative control decisions.

[0133] S3. Based on step S2, this application designs a multi-agent actor-critic method under decompositional value evaluation to alleviate the reliability allocation problem of the previously proposed MLGA2C algorithm when the cooperative control task is complex and cannot be subdivided into individual rewards due to the direct learning of centralized Q-values. The overall training paradigm is as follows: Figure 2 As shown;

[0134] As a preferred embodiment, such as Figure 3 As shown, step S3 specifically involves:

[0135] S31. Construct a global Q-network consisting of a local critic network, an attention mix network, and a target network. The local critic network is a double-local Q-network, using a truncated double-Q technique to avoid overestimating the action-state value function. The target network is used for stable updates, obtaining the parameters of the local critic network and the attention mix network through a soft update method. The local critic network receives the spliced ​​local attention encoding of its own agent. and agent action a i Output local Q-value q i (o,a), the formula is expressed as:

[0136]

[0137]

[0138] in, Pre-encoding for the intelligent agent i, Encode local attention for various entities; and It is a two-layer fully connected network, where ψ1 and ψ2 are the parameters of a bilocal Q-network;

[0139] S32. Feed the local Q-values ​​and local attention encodings into the attention mix module to calculate the global Q-value; first, calculate the weight α of the local Q-values ​​to the global Q-values ​​using the dot product attention mechanism. global The formula is expressed as:

[0140]

[0141]

[0142] Among them, the global query matrix Q gloabl Through a hybrid query matrix The global key K is obtained through mapping. global Through the hybrid bond matrix Mapped to obtain, Transpose it; Generated through random initialization and optimized and updated during training using a loss function based on the global Q-value via backpropagation; cat A sequence of local attention codes for the agent itself, containing n local attention codes for the agent itself;

[0143] α global Then, the global Q value Q(o,a) is obtained by weighted summation, and the formula is expressed as:

[0144] Q(o,a)=α global ·q+b;

[0145] The local Q-value sequence q consists of the local Q-values ​​of n agents; the bias b is used to better fit the value function, and the bias b is directly equal to q, which can reduce the number of network parameters and improve computational efficiency.

[0146] S33. After obtaining the global Q value, the joint regression loss function L is used... Q (ψ) Update the global Q-network, expressed by the formula:

[0147]

[0148] Where (o,a,r,o')~D represents the current state o, current action a, immediate reward r, and next state o' sampled from the experience replay pool D; the target value y i We obtain this using the Bellman equation, i.e.

[0149]

[0150] Where, π θ For the strategy used, log(π) θ (a' i |o' i )) represents the introduced policy entropy; α is the temperature coefficient that balances maximizing reward and maximizing policy entropy; global Q-value estimation. The calculation method is similar to that of the global Q-value, except that it uses the state o' and action a' of the next time step; i For the current immediate reward, γ∈[0,1] is the discount factor that balances the short-term reward and long-term benefit of the agent;

[0151] S34, Distributed Policy Networks Encode with Local Attention As input, it outputs actions; the distributed policy network relies on gradients provided by the global Q-network for updates; gradients The expression is as follows:

[0152]

[0153] Here, θ represents the parameters of the distributed policy network; random action resampling techniques are used to avoid the non-differentiability problem of the policy network; in this application, the input to the policy network is not the original observation or global state s, but a local attention encoding. This differs from existing methods (such as SAC) that directly use the original observation or global state. This application can filter redundant information and improve the input quality of the policy network through local attention encoding.

[0154] The value-policy network of this application is now complete. Next, to enable the agent to adapt to different environments and complete different types of tasks, this application designs different environments for different reinforcement learning tasks, such as... Figure 4 As shown, this includes both static target navigation and cooperative encirclement environments. The environmental parameters used are shown in Table 1 below:

[0155] Table 1 Environmental Parameters

[0156]

[0157] The specific process is detailed in steps S4 and S5 below.

[0158] S4. Construct a static target navigation environment to guide the mobile intelligent agent to actively find the best position, and limit the control details of the state space, action space and reward function involved in the static target navigation task.

[0159] Regarding the static target navigation environment, the task objective is for N mobile intelligent agents to collaboratively navigate to N static target points. As a preferred implementation, step S4 specifically involves:

[0160] S41. Set the state space of each agent in the static target navigation environment as follows:

[0161]

[0162] Where, p i and v i These are the global position and global velocity of agent i, respectively. It is the relative position between the landmark and the intelligent agent i. Indicates the relative position of other agents to agent i;

[0163] S42. Set the action space of each agent to force F. i =[f x ,fy ];f x and f y This indicates the force components generated by the actuator along the x-axis and y-axis;

[0164] S43. Set the objective function and constraints for the static target navigation task and the execution process, including:

[0165] The objective function is to minimize the average time for the swarm of agents to reach all target points:

[0166]

[0167] in, For arrival time, π θ The strategy chosen for the agent;

[0168] Arrival constraint, constraining agent i to arrive at time Arrive at the target

[0169]

[0170] Collision constraints constrain agent i to avoid collisions during operation:

[0171]

[0172] in, Let d be the Euclidean distance between agents i and j. coll This is the collision distance threshold;

[0173] S44. The reward function for each agent i when performing a static target navigation task is designed as follows:

[0174]

[0175] The distance reward, ensuring a single agent can navigate to a landmark, is calculated by determining the distances from all landmarks to agent i, expressed by the formula:

[0176]

[0177] Among them, w g Represents the distance coefficient. It is the Euclidean distance between agent i and the nth landmark;

[0178] Penalty for time step; The collision penalty is expressed by the formula:

[0179]

[0180] Where C1 and C2 are constants;

[0181] Final reward The reward for multi-agent collaborative navigation task completion, i.e.

[0182] In this embodiment, multiple sets of static target navigation experiments were conducted, with the number of agents in the experimental scenarios being 3, 6, and 9, respectively. Reward curves for different algorithms during training were plotted and compared, with the comparison referenced... Figure 5 As can be seen, compared with other algorithms, the MAVFAC algorithm has a faster convergence speed and enables the agent to obtain higher rewards.

[0183] In addition, in this embodiment, 200 sets of randomized experiments were conducted to test four performance metrics of the models trained by different algorithms. The four performance metrics, including success rate (an agent reaching all target points with a low average number of collisions during operation is considered a success), time step (the time it takes for the agent to reach all targets), number of collisions (the number of collisions per agent per episode), and coverage (the number of target points covered in one episode), were compared. The test results are shown in Table 2 below:

[0184] Table 2. Four performance metrics of models trained by different algorithms in a static target navigation environment.

[0185]

[0186] It can be seen that the MAVFAC algorithm proposed in this invention maintains a high success rate, short step size, comprehensive coverage, and low number of collisions in all static target navigation scenarios.

[0187] S5. Next is the cooperative encirclement task. We will again construct the cooperative encirclement environment, defining the control details of the state space, action space, and reward function involved in the cooperative encirclement task;

[0188] In cooperative encirclement missions, the agent is required to learn to encircle prey in multiple groups based on limited local observations and joint rewards; as a preferred implementation, step S5 specifically includes:

[0189] S51. Set the state space o' for each agent i in the cooperative trapping task environment. i for:

[0190]

[0191] Where, p i and v i These are the global position and global velocity of agent i, respectively; It is the relative position between the landmark and the intelligent agent i. Indicates the relative position of other agents to agent i; and These are the relative positions between the target prey and agent i, and the speed of the target prey itself;

[0192] S52. Set the objective function and constraints for the cooperative encirclement task as follows:

[0193] The objective function of a cooperative encirclement task is to minimize the average time required to encircle all prey.

[0194]

[0195] in, In order to catch up with time, π θ The strategy chosen for the agent

[0196] Encirclement constraint, constraining agent i to be in pursuit time Successfully pursued the target

[0197] δ represents the distance threshold for successful capture;

[0198] S53. Design the total reward function R for executing a cooperative encirclement mission. t for:

[0199]

[0200] The formula for capturing rewards is as follows:

[0201]

[0202] In this embodiment, C1 = 1, C2 = 5, and C3 = 10 are set respectively to encourage multi-agent cooperation;

[0203] The distance penalty is expressed by the formula:

[0204]

[0205] Among them, w dist Represents the distance coefficient. It is the Euclidean distance between agent i and the target prey's location.

[0206] Similarly, in this embodiment, multiple cooperative encirclement experiments are still conducted, with the number of participating agents being 3, 6, and 9, respectively. Reward curves for N different algorithms during training are plotted and compared, with the comparison referenced... Figure 6It can be seen that the MAVFAC algorithm maintains excellent performance in cooperative trapping environments with different numbers of agents and has higher rewards during training.

[0207] Similarly, 200 sets of randomized experiments were conducted to compare three performance metrics used in the cooperative encirclement experiment: success rate (capturing all prey is considered a success), time step (time taken to capture all prey), and number of captures (number of prey captured in one episode). The premise was that a single agent cannot capture multiple prey simultaneously; collaborative work (such as group encirclement) is necessary to efficiently complete the task. The test results are shown in Table 3 below:

[0208] Table 3. Three performance metrics of different algorithms in a cooperative encirclement environment.

[0209]

[0210] It can be seen that the MAVFAC algorithm has the highest success rate and the shortest time to successfully capture targets. Furthermore, its average number of targets captured is also higher than other baseline algorithms. This indicates that the MAVFAC algorithm has excellent performance in cooperative capture tasks with varying numbers of agents.

[0211] S6. Construct a multi-agent cooperative navigation model following steps S1 to S5. Obtain the optimal model through training and testing to achieve explicit and fully perceptive navigation for multi-agent cooperative navigation with confidence allocation problems.

[0212] The method proposed in this invention solves the reliability allocation problem in multi-agent cooperation through a collaborative design of decompositional value evaluation (dynamic contribution allocation), local attention encoding (precise feature extraction), and distributed policy execution (adaptive to local observations). This embodiment will then use ablation experiments and one-shot experiments to verify the technical effectiveness of the proposed method in reducing lazy agents and improving collaborative efficiency in complex tasks.

[0213] First, the ablation experiment, as follows: Figure 7 As shown, each time one module of the algorithm is removed, the reward curves are compared in a 9-agent static target navigation scenario. The modules removed are, in order: removing the attention Mix module (w / o MIX), removing the local attention policy network (w / o LA Policy), and removing the local attention value network (w / o LA Critic). Analyzing the experimental results, we can find that each component has a positive impact on the MAVFAC algorithm. All three components improve the algorithm's performance, and when they are all combined, a 1+1+1>3 effect occurs.

[0214] Finally, a one-shot inference experiment was conducted in this embodiment. Without performing two-stage training, based on a training model with a cooperative encirclement scenario of 9 agents, the MAVFAC algorithm performed generalized inference in scenarios with 6, 12, and 15 agents to evaluate the direct inference capability of the MAVFAC algorithm. The performance metrics obtained are shown in Table 4 below:

[0215] Table 4 Results of the One-Shot Reasoning Experiment

[0216]

[0217] Analysis of the experimental performance metrics reveals that the MAVFAC algorithm exhibits a high success rate, short time step, and a large number of captures for each one-shot inference experiment. This result demonstrates that the MAVFAC algorithm possesses strong direct inference capabilities, and the trained model can be directly applied to cooperative encirclement scenarios with varying numbers of agents without requiring additional training.

[0218] In summary, the method proposed in this invention performs well in tasks such as static target navigation and cooperative encirclement. Compared with traditional methods, it has a faster convergence speed, higher success rate, shorter arrival time and fewer collisions, demonstrating good generalization ability and robustness.

[0219] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0220] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0221] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A decompositional value assessment multi-agent cooperative navigation method based on attention mechanism, characterized in that, Specifically, the following steps are included: S1. Model the multi-agent cooperative navigation task as a decentralized partially observable Markov decision process; S2. Design a local attention module to extract and encode environmental feature information of multiple agents. The local attention module includes a pre-encoding network and an attention encoding module, and has a mechanism for the critic network and actor network to share the local attention module. S3. Design a multi-agent actor-critic method under decompositional value evaluation to solve the centralized Q-value problem; Step S3 specifically includes: S31. Construct a global Q-network consisting of a local critic network, an attention mix network, and a target network. The local critic network is a double-local Q-network, using a truncated double-Q technique to avoid overestimating the action-state value function. The target network is used for stable updates, obtaining the parameters of the local critic network and the attention mix network through a soft update method. The local critic network receives the spliced ​​local attention encoding of its own agent. and agent actions Output local Q value The formula is expressed as: ； ； in, Pre-encoding for the intelligent agent i, Encode local attention for various entities; and It is a two-layer fully connected network. and These are the parameters of a dual local Q-network; S32. Feed the local Q-values ​​and local attention encodings into the attention mix module to calculate the global Q-value; first, calculate the weights of the local Q-values ​​on the global Q-values ​​using the dot product attention mechanism. The formula is expressed as: ； ； Among them, the global query matrix Through a hybrid query matrix Mapped to global key Through the hybrid bond matrix Mapped to obtain, Transpose it; , It is generated through random initialization and optimized and updated during training by backpropagation based on the loss function of the global Q value; A sequence of local attention codes for the agent itself, containing n local attention codes for the agent itself; get Then, the global Q value is obtained by weighted summation. The formula is expressed as: ； Among them, the local Q-value sequence It consists of the local Q values ​​of n agents; the bias b is used to better fit the value function, and the bias b is directly equal to q; S33. After obtaining the global Q value, use the joint regression loss function. The global Q-network is updated using the following formula: ； in, This represents the current state sampled from the experience replay pool D. Current action Instant rewards and the state at the next moment Target value We obtain this using the Bellman equation, i.e. ； in, For the strategy used, For the introduction of policy entropy; It is the temperature coefficient that balances maximizing returns and maximizing policy entropy; global Q-value estimation. The calculation method is similar to that of the global Q-value, except that it uses the state at the next time step. and actions ; For the current immediate reward, γ∈[0,1] is the discount factor that balances the short-term reward and long-term benefit of the agent; S34, Distributed Policy Networks Encode with Local Attention As input, it outputs actions; the distributed policy network relies on gradients provided by the global Q-network for updates; gradients The expression is as follows: ； in, These are the parameters of the distributed policy network; random action resampling techniques are used to avoid the non-differentiability problem of the policy network. S4. Construct a static target navigation environment to guide the mobile intelligent agent to actively find the best position, and limit the control details of the state space, action space and reward function involved in the static target navigation task. S5. Construct a cooperative encirclement environment and limit the control details of the state space, action space, and reward function involved in the cooperative encirclement task; S6. Construct a multi-agent cooperative navigation model following steps S1 to S5. Obtain the optimal model through training and testing to achieve explicit and fully perceptive navigation for multi-agent cooperative navigation with confidence allocation problems.

2. The multi-agent cooperative navigation method for decompositional value evaluation based on attention mechanism according to claim 1, characterized in that, Step S1 is as follows: The collaborative navigation task is modeled as a decentralized, partially objective Markov decision process, represented as: U=<I,S,O,Ω,A,T,R> ; Where U represents the decentralized partially observable Markov decision process; I represents the set of agents; S represents the global state information of the environment; O represents the set of states observed locally by each agent; Ω is the observation function; A is the joint action of the agents; T is the transition function; and R is the immediate reward. After modeling, the agent selects actions based on the current state information and policy, and transitions to the next state according to the transition function, while receiving an immediate reward for each agent. The immediate reward then guides the agent to make better action decisions until the navigation task is completed.

3. The multi-agent cooperative navigation method for decompositional value evaluation based on attention mechanism according to claim 1, characterized in that, The precoding network in the local attention module in step S2 is specifically as follows: In the local attention module, each entity is equipped with an independent precoding network. Each precoding network consists of a fully connected embedding layer, a leaky ReLU activation function, and a LayerNorm. The precoding of each agent i is first obtained through the precoding network. The formula is expressed as: ； in, It is a self-intelligent agent precoding network. It is the information vector of the agent i itself, which includes its global position and global velocity; Pre-encoding of other different categories of entities observed locally by agent i The formula is expressed as: ； ； ... ； ； in, Let m represent other different categories of entities locally observed by agent i, where m = 1, 2, ..., M; these other different categories of entities include landmarks, other agents, obstacles, and target prey. For information vectors of other different categories of entities, It is the number of entity categories within the field of vision of the intelligent agent i. This represents the default number of entity categories that agent i can observe. When using placeholder zeros to fill the vector space, ,…, Precoding networks representing other different categories of entities.

4. The multi-agent cooperative navigation method for decompositional value evaluation based on attention mechanism according to claim 1, characterized in that, The attention encoding module in the local attention module in step S2 is specifically as follows: The attention encoding module contains three types of attention matrices: query matrix and query matrix. Key matrix Sum matrix Furthermore, each type of entity is equipped with an independent key matrix. Sum matrix However, only the intelligent agent itself possesses the query matrix. The formula for the mapping relationship of each attention matrix is ​​expressed as follows: ； in, This represents the pre-encoding set of various entities locally observed by agent i; finally, the local attention encodings of various entities are obtained by weighted dot product. The formula is expressed as: ； in, Attention weights are used to measure other types of entity precoding. The degree of correlation with the self-agent agent; the local attention encoding of the self-agent agent is determined by its pre-encoding. and It was pieced together.

5. The multi-agent cooperative navigation method for decompositional value evaluation based on attention mechanism according to claim 1, characterized in that, Step S4 specifically includes: S41. Set the state space for each agent in the static target navigation environment. for: ； in, and These are the global position and global velocity of agent i, respectively; It is the relative position between the landmark and the intelligent agent i. Indicates the relative position of other agents to agent i; S42. Set the action space of each agent to force. ; and This indicates the force components generated by the actuator along the x-axis and y-axis; S43. Set the objective function and constraints for the static target navigation task and the execution process, including: The objective function is to minimize the average time for the swarm of agents to reach all target points: ； in, For arrival time, The strategy chosen for the agent; Arrival constraint, constraining agent i to arrive at time Arrive at the target : ； Collision constraints constrain agent i to avoid collisions during operation: ； in, Let Euclidean distance be the distance between agents i and j. This is the collision distance threshold; S44. The reward function for each agent i when performing a static target navigation task is designed as follows: ； The distance reward, ensuring a single agent can navigate to a landmark, is calculated by determining the distances from all landmarks to agent i, expressed by the formula: ； in, Represents the distance coefficient. It is the Euclidean distance between agent i and the nth landmark; Penalty for time step; The collision penalty is expressed by the formula: ； in, It is a constant; Final reward The reward for multi-agent collaborative navigation task completion, i.e. .

6. The multi-agent cooperative navigation method for decompositional value evaluation based on attention mechanism according to claim 1, characterized in that, Step S5 is as follows: S51. Set the state space for each agent i in the cooperative trapping task environment. for: ； in, and These are the global position and global velocity of agent i, respectively; It is the relative position between the landmark and the intelligent agent i. Indicates the relative position of other agents to agent i; and These are the relative positions between the target prey and agent i, and the speed of the target prey itself; S52. Set the objective function and constraints for the cooperative encirclement task as follows: The objective function of a cooperative encirclement task is to minimize the average time required to encircle all prey. ； in, In order to keep the pursuit going, The strategy chosen for the agent Encirclement constraint, constraining agent i to be in pursuit time Successfully pursued the target : ; The distance threshold for successful pursuit; S53. Design the total reward function for executing cooperative encirclement missions. for: ； The formula for capturing rewards is as follows: ； The distance penalty is expressed by the formula: ； in, Represents the distance coefficient. It is the Euclidean distance between agent i and the target prey's location.

Images