Multi-agent training method based on heterogeneous dynamic graph attention mechanism

Abstract

The invention relates to a multi-agent training method based on a heterogeneous dynamic graph attention mechanism, and the method comprises the steps: initializing agents, building a scene and a heterogeneous relation graph, building a deep reinforcement learning model based on the heterogeneous dynamic graph attention mechanism, carrying out the training, generating the aggregation features of the agents, and carrying out the multi-agent collaborative decision making through the trained model. And multi-agent control is realized. According to the method, a graph attention mechanism is used for aggregating the state feature vectors of the intelligent agents, the state feature vectors keep low dimensionality and contain related information of other intelligent agents, and the calculation burden in the subsequent strategy optimization process is reduced; a graph attention mechanism is subdivided into node-level attention and semantic-level attention, rich semantic relations between heterogeneous agents are captured, and more efficient collaboration or competition between the agents is achieved; a dynamic attention mechanism is introduced, the model is allowed to dynamically adjust the weight of neighbor nodes according to the characteristics of query nodes, the complex relation between agents can be better expressed, and the expression ability of the model is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of calculation, inference or counting, and in particular to a multi-agent training method based on a heterogeneous dynamic graph attention mechanism. Background Art

[0002] Reinforcement learning is an important learning paradigm in artificial intelligence technology. Through interaction with the environment, the intelligent agent accumulates experience and adjusts strategies from the trial and error process, ultimately achieving task automation and optimization. It has been successfully applied to complex scenarios such as Go games, autonomous driving, and intelligent recommendations.

[0003] In multi-agent reinforcement learning, there are cooperative and / or competitive relationships between agents. This diverse relationship requires agents to fully consider the dynamic behaviors and intentions of other agents when making decisions, making strategy optimization more challenging. In addition, multi-agent systems usually contain heterogeneous agents, that is, agents differ in attributes, capabilities, goals, etc.; this heterogeneity provides flexibility for multi-agent systems in task division and role collaboration, but also further increases the difficulty of modeling and algorithm design.

[0004] In order to solve the problem of effective collaboration or competition among heterogeneous multi-agents, researchers have proposed a variety of technologies and methods, among which the common methods include the Centralized Training Decentralized Execution (CTDE) framework and the joint reward mechanism. The CTDE framework improves the efficiency of strategy optimization through centralized training and maintains decentralization in decision-making to adapt to the actual environment. However, this framework is susceptible to information asymmetry between training and decision-making in complex environments, which may lead to a decrease in strategy adaptability. The joint reward mechanism, by designing joint rewards, enables agents to focus on the overall goal while optimizing their own behavior, promotes the collaborative cooperation of the system, effectively improves the efficiency of collaboration, and avoids the problem of agents only focusing on individual interests in decision-making. However, the joint reward mechanism faces the problem of credit allocation in practice, that is, it is difficult to clarify the specific contribution of a single agent to team collaboration. Summary of the invention

[0005] In response to the problems existing in the prior art, the present invention provides a multi-agent training method based on a heterogeneous dynamic graph attention mechanism to improve the effects of collaboration and / or competition among heterogeneous multi-agents.

[0006] The technical solution adopted by the present invention is a multi-agent training method based on a heterogeneous dynamic graph attention mechanism, the method comprising the following steps:

[0007] S1 initializes all agents and establishes a scene and a heterogeneous relationship graph based on the multi-agent;

[0008] S2 builds a deep reinforcement learning model based on heterogeneous dynamic graph attention mechanism to generate aggregate features of intelligent agents;

[0009] S3 trains the deep reinforcement learning model;

[0010] S4 uses the trained model to make multi-agent collaborative decisions and realize multi-agent control.

[0011] Preferably, the agents are configured so that any agent has corresponding goals, behaviors, and attributes.

[0012] When the present invention is applied, there should be at least one agent among the multiple agents that has a goal and / or behavior and / or attribute that is different from one or more other agents.

[0013] Preferably, any agent is used as a heterogeneous node, and the relationship type between any two agents is used as a heterogeneous edge to construct a heterogeneous relationship graph.

[0014] Preferably, the deep reinforcement learning model includes an input layer arranged in sequence, a node-level attention module including one or more parallel node-level attention layers, a semantic-level attention layer, and a parallel strategy network and value network.

[0015] Preferably, the node-level attention layer includes a fully connected layer, a concatenation layer, an attention layer, a normalization layer, and a Dropout layer that are connected in sequence, and the semantic-level attention layer includes a fully connected layer and a normalization layer that are connected in sequence.

[0016] In the graph attention network, the most suitable meta-path is selected to process specific downstream tasks. The meta-path can be designed artificially. The heterogeneous relationship graph is subdivided into relationship sub-graphs according to the meta-path. The relationship between nodes in the relationship sub-graph is captured by node-level attention. Therefore, the relationship sub-graph corresponds to the node-level attention. The number of node-level attention layers is equal to the number of meta-paths.

[0017] Preferably, in S2, generating the aggregated features of the agent comprises the following steps:

[0018] S2.1 Calculate meta-path using dynamic attention mechanism Lower Agent Relative to the intelligent agent The attention weight , the meta-path is associated with a node-level attention layer;

[0019] S2.2 Based on , computational agents Meta-path based Node-level attention aggregation features ;

[0020] S2.3 Calculate the attention weight of semantic level attention;

[0021] S2.4 Based on the results of S2.2 and S2.3, calculate the agent Aggregating features based on attention mechanism for heterogeneous dynamic graphs.

[0022] Preferably, in S2.1, the attention weight is improved by linear normalization calculation .

[0023] Preferably, in S2.4, after obtaining the aggregated features, feature contrast enhancement processing is performed on the aggregated features.

[0024] Preferably, the aggregated features of each agent are compared with the global or other agent features to eliminate differences and improve the aggregated features to input the parallel policy network and value network.

[0025] In the present invention, the dynamic attention mechanism allows the model to dynamically adjust the weight ranking of neighbor nodes according to the characteristics of the query node. Compared with the traditional static attention, the attention weight ranking of the neighbor nodes is fixed and has nothing to do with the characteristics of the query node. Dynamic attention can significantly improve the expressiveness of the model, and the introduction of Min-Max normalization method calculation , which can prevent an agent’s attention weight from being too high or too low, thereby improving the numerical stability of the model.

[0026] Furthermore, feature contrast enhancement uses the difference between aggregated features and original features to highlight the importance of information, thereby reducing redundant information and allowing the reinforcement learning model to focus more on useful information.

[0027] Preferably, in S4, when training the deep reinforcement learning model, the overall objective function is positively correlated with the strategy loss and the entropy regularization term, and negatively correlated with the value loss.

[0028] The present invention relates to a multi-agent training method based on a heterogeneous dynamic graph attention mechanism, which includes initializing all agents, establishing a scene and a heterogeneous relationship graph based on the multi-agent, and establishing a deep reinforcement learning model based on the heterogeneous dynamic graph attention mechanism; training the deep reinforcement learning model to generate the aggregated features of the agents, and using the trained model to perform multi-agent collaborative decision-making to achieve multi-agent control.

[0029] The beneficial effects of the present invention are:

[0030] (1) Use the graph attention mechanism to aggregate the state feature vectors of the agents, so that the state feature vectors input to the reinforcement learning algorithm contain relevant information of other agents while maintaining low dimensions, reducing the computational burden in the subsequent strategy optimization process;

[0031] (2) The graph attention mechanism is subdivided into node-level attention and semantic-level attention to capture the rich semantic relationships between heterogeneous agents and achieve more efficient collaboration or competition between agents.

[0032] (3) In the process of calculating the attention score, a dynamic attention mechanism is introduced, which allows the model to dynamically adjust the weights of neighbor nodes according to the characteristics of the query node, which can better express the complex relationships between intelligent agents and enhance the expressive power of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the method of the present invention;

[0034] Figure 2 It is a structural schematic diagram of the deep reinforcement learning model based on the heterogeneous dynamic graph attention mechanism of the present invention;

[0035] Figure 3 It is a schematic diagram for comparing the training effects of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific implementation described herein is only used to explain the present invention and is not intended to limit the present invention.

[0037] The present invention relates to a multi-agent training method based on a heterogeneous dynamic graph attention mechanism. Specifically, in a heterogeneous multi-agent reinforcement learning environment, considering that traditional reinforcement learning algorithms are difficult to effectively capture the relationship between agents and limit the performance of multi-agents in collaborative or competitive tasks, node-level and semantic-level graph attention mechanisms are introduced on the basis of the existing reinforcement learning framework. The state characteristics of the agents are weightedly fused through the graph attention mechanism and then output to the reinforcement learning layer, thereby constructing a heterogeneous graph attention mechanism to achieve more efficient collaboration and / or competition between agents and improve task execution efficiency. Furthermore, a dynamic attention mechanism is also introduced. The dynamic attention mechanism allows the model to dynamically adjust the weight ranking of neighbor nodes according to the characteristics of the query node, so as to better capture the complex relationship between nodes, thereby significantly improving the expression ability of the model to meet the collaboration and / or competition needs of heterogeneous multi-agents.

[0038] The method comprises the following steps:

[0039] (1) Initializing all agents and establishing a multi-agent-based scenario and heterogeneous relationship graph;

[0040] (2) Establish a deep reinforcement learning model based on the attention mechanism of heterogeneous dynamic graphs to generate aggregate features of intelligent agents;

[0041] (3) training the deep reinforcement learning model;

[0042] (4) Use the trained model to make collaborative decisions among multiple agents and realize multi-agent control.

[0043] The method of the present invention is described in detail below.

[0044] (1) Initialize all intelligent agents and establish a scene and a heterogeneous relationship graph based on the multi-agent.

[0045] The agents are configured so that any agent has corresponding goals, behaviors, and attributes. Taking into account the multi-agent scenario in which the maximization method is applied, in the same multi-agent system, the multi-agents are defined as heterogeneous agents. Here, as long as there is an agent whose goals, behaviors, or attributes are different from those of other agents, it is regarded as a heterogeneous agent.

[0046] A heterogeneous relationship graph is constructed by taking any agent as a heterogeneous node and the relationship type between any two agents as a heterogeneous edge. To be precise, the heterogeneous nodes of the heterogeneous relationship graph are nodes of different types, and the heterogeneous edges are edges of different types. Each agent is added to the graph as a node, and different types of edges are used to represent different relationships between agents.

[0047] (2) Establish a deep reinforcement learning model based on the heterogeneous dynamic graph attention mechanism to generate the aggregated features of the intelligent agent.

[0048] The deep reinforcement learning model includes an input layer arranged in sequence, a node-level attention module including one or more parallel node-level attention layers, a semantic-level attention layer, and a parallel strategy network and value network; wherein the node-level attention and the semantic-level attention are respectively used to calculate the attention weight between the agent and its neighbors under a certain meta-path and the importance weight of the meta-path, the number of node-level attention layers is equal to the number of meta-paths in the heterogeneous relationship graph, the meta-path here refers to a path sequence composed of different node types and relationships, which is used to capture the multi-level relationship between nodes, and by designing different meta-paths, the heterogeneous relationship graph can be subdivided into multiple relationship subgraphs based on different meta-paths, and the neighbor nodes of the node under different meta-paths are different, so that the model can capture more detailed semantic relationships between nodes.

[0049] The node-level attention layer includes a fully connected layer, a concatenation layer, an attention layer, a normalization layer, and a Dropout layer connected in sequence, and the semantic-level attention layer includes a fully connected layer and a normalization layer connected in sequence; the state features and the adjacency matrix of the agent are passed through the node-level attention layer and the semantic-level attention layer in turn to capture the multidimensional semantic relationships between different types of agents, so that the generated agent aggregation features not only include the state features of the agent itself, but also incorporate the state features of other agents.

[0050] Generate the aggregate features of the agent, specifically, including the following steps:

[0051] (2-1) Using dynamic attention mechanism to calculate meta-path Lower Agent Relative to the intelligent agent The attention weight , and further, the attention weights are improved by linear normalization calculation ,get

[0052] α i j Φ p = a Φ p ⊤ L e a k y R e L U ( W Φ p ⋅ [ h i ∥ h j ] ) − m i n k ∈  i Φ p  a Φ p ⊤ L e a k y R e L U ( W Φ p ⋅ [ h i ∥ h k ] ) m a x k ∈  i Φ p  a Φ p ⊤ L e a k y R e L U ( W Φ p ⋅ [ h i ∥ h k ] )

[0053] in, Representing an Agent Meta-path based The neighbor set of Represents the feature vector and The splicing operation, represents the activation function, and Represents the meta path The learnable weight matrix and parameter vector in the next node-level attention, min and max represent the minimum and maximum operations respectively;

[0054] (2-2) Based on , computational agents Meta-path based Node-level attention aggregation features ,

[0055]

[0056] in, represents the activation function, Representing an Agent The characteristic vector of represents a learnable weight matrix used to map the concatenated feature vector to the attention space;

[0057] (2-3) Calculate the attention weight of semantic level attention;

[0058]

[0059] in, represents the number of meta-paths, Represents a meta path The attention score is used to reflect the importance of the meta-path, satisfying , here Represents a meta-path based The number of subgraph nodes, represents the semantic level attention vector, and Represents the learnable weight matrix and bias vector in semantic-level attention;

[0060] (2-4) Based on the results of (2-2) and (2-3), calculate the agent Aggregate features based on heterogeneous dynamic graph attention mechanism,

[0061]

[0062] in, For intelligent agents In the meta path The node-level attention aggregation feature under Metapath The semantic level attention weight of

[0063] Furthermore, after obtaining the aggregated features, feature contrast enhancement processing is performed on the aggregated features;

[0064] The aggregated features of each agent are compared with the global or other agent features to eliminate the differences and improve the aggregated features to input the parallel policy network and value network.

[0065] This is because the aggregated features of each agent usually contain a certain degree of noise or redundant information, and by comparing the aggregated features of each agent with the global or other agent features, their differences are eliminated, so that their uniqueness can be more clearly presented;

[0066] The feature contrast enhancement calculation formula is as follows:

[0067]

[0068] in, Representing an Agent The aggregation characteristics of Representing an Agent The state characteristics, Represents the number of agents.

[0069] In the dynamic attention, query nodes generate different attention score distributions according to different query contents, so that different query vectors can pay attention to different key vectors.

[0070] This aggregate feature Input into the policy network and value network of the reinforcement learning algorithm to improve the stability of policy learning and the accuracy of value estimation, thereby achieving better collaboration and competition effects.

[0071] (3) Training the deep reinforcement learning model.

[0072] When training the deep reinforcement learning model, based on the input and output of the policy network and the value network, the overall objective function is expressed as follows:

[0073]

[0074] in, is the policy loss of the Actor network, is the value loss of the Critic network, is the entropy regularization term, and is a weight hyperparameter used to control the weight of policy loss, value loss, and entropy regularization term in the objective function;

[0075] For the policy network, the optimized objective function

[0076] L A c t o r ( θ ) = E [ m i n ( r t ( θ ) ⋅ A ( s , a ) , c l i p ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) ⋅ A ( s , a ) ) ]

[0077] in, represents the expected function, is the probability ratio of the new strategy to the old strategy, It is a hyperparameter, usually 0.2, and the clip function limits the probability ratio exist [ 1 − ϵ ， 1 + ϵ  ] Within the range, is the advantage function, , Indicates in status Execute an action After the return, Indicates in status The average expected return under

[0078] For the value network, the optimization objective function

[0079] L C r i t i c ( ϕ ) = E [ ( R t − V ϕ ( s t ) ) 2 ]

[0080] Computational Value Network Estimated value and target value By optimizing this term, the value network can more accurately estimate the value of each state, so that the policy network can obtain more accurate reward feedback when updating;

[0081] Entropy regularization term It is used to increase the randomness of the strategy and encourage the strategy to maintain a certain degree of randomness when choosing actions, so as to better explore and satisfy the needs in the early stages of training.

[0082]

[0083] in, Indicates in status Next select action The probability of Represents logarithmic calculation.

[0084] (4) Use the trained model to make collaborative decisions among multiple agents and realize multi-agent control.

[0085] A specific embodiment of the present invention is given below, and its specific implementation scenario is not limited to this.

[0086] (1) Initialize the capture scene and environment (heterogeneous multi-agent scene), including three pursuer agents (hereinafter referred to as "pursuers") and one escaper agent (hereinafter referred to as "escaper"); set the action space and state space of each agent, configure the map, set its size and obstacle position; the goal (task) of the pursuer is to capture a escaper within a specified step length, and avoid obstacles and collisions between similar agents during the collaborative capture process; the goal of the escaper is not to be captured by the pursuer within a specified step length, and to avoid obstacles during the escape process. The capture is required to be successful if the distance between any pursuer and the escaper is less than the set capture distance ;

[0087] Defining Agents The state feature vector as follows:

[0088] h i = [ ( x i , y i ) , { ( x j − x i , y j − y i ) } j ≠ i , { d i j } j ≠ i , d m i n , ( x o b s , y o b s ) ]

[0089] in, Representing an Agent Location in the roundup map, Representing other agents Relative Agent location, Representing other agents With Agent The relative distance Representing an Agent The distance to the nearest agent, Indicates the location of the obstacle.

[0090] In the pursuit environment, the relational subgraph formed by the “pursuer-competition-escaper” meta-path can describe the competitive relationship between the pursuer and the escapee; the relational subgraph formed by the “pursuer-competition-escaper-competition-pursuer” meta-path can be used to simultaneously capture the collaborative relationship between the pursuers and the competitive relationship between the pursuers and the escapee.

[0091] (2) A deep reinforcement learning model based on the heterogeneous dynamic graph attention mechanism is established, including a sequentially arranged input layer, a node-level attention module including one or more parallel node-level attention layers, a semantic-level attention layer, and a parallel strategy network and value network; in the encirclement and capture scenario, the pursuer agent model contains two node-level attention layers, which are used to capture the collaborative relationship between the pursuers and the competitive relationship between the pursuers and the escapees, respectively; the escapee agent model contains one node-level attention layer, which is used to capture the competitive relationship between the escapee and the three pursuers.

[0092] by The aggregated features are subjected to feature contrast enhancement processing to obtain improved aggregated features that eliminate differences and more clearly display their uniqueness.

[0093] (3) training the deep reinforcement learning model;

[0094] (4) Use the trained model to make collaborative decisions among multiple agents and realize multi-agent control.

[0095] The pursuer agent (HAT-MAPPO) trained by the method of the present invention was compared with the pursuer agent (MAPPO) trained by the traditional reinforcement learning method. The opponent selected the same escaper agent. After 100 rounds of confrontation tests, the win and loss of both sides were recorded in detail, and the winning rate comparison of both sides was obtained. The specific data of the winning rate is shown in Table 1.

[0096] Table 1 Comparison of test success rates of the pursuer agent model of the present invention

[0097] Pursuer Agent Win Rate HAT-MAPPO 86% MAPPO 77%

[0098] The method of the present invention (HAT-MAPPO) and the traditional reinforcement learning method (MAPPO) are used to train the pursuer agent in the same environment. The comparison results are as follows: Figure 3 As shown, it can be seen that compared with the traditional method, the effect of the multi-agent training method based on the heterogeneous dynamic graph attention mechanism of the present invention is significantly improved.

[0099] The present invention also relates to a computer-readable storage medium, on which a multi-agent training program based on a heterogeneous dynamic graph attention mechanism is stored. When the program is executed by a processor, the multi-agent training method based on the heterogeneous dynamic graph attention mechanism is implemented.

[0100] The present invention also relates to a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the multi-agent training method based on the heterogeneous dynamic graph attention mechanism is implemented.

[0101] The present invention also relates to an application of the multi-agent training method based on the heterogeneous dynamic graph attention mechanism, which is applied to heterogeneous multi-agent collaborative decision-making scenarios.

[0102] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0104] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0106] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0107] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-agent training method based on heterogeneous dynamic graph attention mechanism, characterized by: The method comprises the following steps: S1 initializes all agents and establishes a scene and a heterogeneous relationship graph based on the multi-agent; S2 builds a deep reinforcement learning model based on heterogeneous dynamic graph attention mechanism to generate aggregate features of intelligent agents; S3 training the deep reinforcement learning model; S4 uses the trained model to make multi-agent collaborative decisions and realize multi-agent control.

2. A multi-agent training method based on heterogeneous dynamic graph attention mechanism according to claim 1, characterized in that: The agents are configured so that each agent has corresponding goals, behaviors, and attributes.

3. A multi-agent training method based on heterogeneous dynamic graph attention mechanism according to claim 2, characterized in that: Taking any agent as a heterogeneous node and the relationship type between any two agents as a heterogeneous edge, a heterogeneous relationship graph is constructed.

4. According to claim 1, a multi-agent training method based on heterogeneous dynamic graph attention mechanism is characterized by: The deep reinforcement learning model includes an input layer arranged in sequence, a node-level attention module including one or more parallel node-level attention layers, a semantic-level attention layer, and a parallel strategy network and value network.

5. A multi-agent training method based on heterogeneous dynamic graph attention mechanism according to claim 4, characterized in that: The node-level attention layer includes a fully connected layer, a concatenation layer, an attention layer, a normalization layer, and a Dropout layer connected in sequence, and the semantic-level attention layer includes a fully connected layer and a normalization layer connected in sequence.

6. A multi-agent training method based on heterogeneous dynamic graph attention mechanism according to claim 4, characterized in that: In S2, generating the aggregated features of the agent includes the following steps: S2.1 Using dynamic attention mechanism to calculate meta-path Lower Agent Relative to the intelligent agent The attention weight , the meta-path is associated with a node-level attention layer; S2.2 Based on , computational agents Meta-path based Node-level attention aggregation features ; S2.3 Calculate the attention weight of semantic level attention; S2.4 Based on the results of S2.2 and S2.3, calculate the agent Aggregating features based on attention mechanism for heterogeneous dynamic graphs.

7. A multi-agent training method based on heterogeneous dynamic graph attention mechanism according to claim 6, characterized in that: In S2.1, the attention weights are improved by linear normalization calculation .

8. The multi-agent training method based on heterogeneous dynamic graph attention mechanism according to claim 6 is characterized by: In S2.4, after the aggregated features are obtained, feature contrast enhancement processing is performed on the aggregated features.

9. A multi-agent training method based on heterogeneous dynamic graph attention mechanism according to claim 8, characterized in that: The aggregated features of each agent are compared with the global or other agent features to eliminate the differences and improve the aggregated features to input the parallel policy network and value network.

10. A multi-agent training method based on heterogeneous dynamic graph attention mechanism according to claim 4, characterized in that: In S4, when training the deep reinforcement learning model, the overall objective function is positively correlated with the strategy loss and the entropy regularization term, and negatively correlated with the value loss.

Images