Multi-agent collaborative decision-making system and method based on hierarchical cognitive map self-encoding

Abstract

The application discloses a multi-agent collaborative decision-making system and method based on layered cognitive graph self-encoding, wherein the method comprises the following steps: acquiring local observation, action and interaction message of each agent at each time step, constructing an initial communication graph by taking the interaction message as a graph node, and generating an adjacency matrix by calculating the correlation strength between nodes through an attention network; based on the adjacency matrix and node features, K-step message passing is performed by using a graph attention network to generate a latent graph encoding local interaction information; a three-layer graph pooling operation is performed on the latent graph, and an improved DiffPool algorithm is used for each layer of pooling to aggregate node features and adjacency relationships according to a clustering assignment matrix, and finally a 64-dimensional global cognitive representation is output; and the application aims to solve the problem that the decision-making performance is limited due to the rigidity of the cognitive mode, the low efficiency of information integration, the weakness of global collaboration and the insufficient adaptability to complex scenes in the multi-agent reinforcement learning technology in the prior art.

Description

Technical Field

[0001] This invention relates to the technical field of information technology, specifically to a multi-agent collaborative decision-making system and method based on hierarchical cognitive graph autoencoding. Background Technology

[0002] Reinforcement learning (RL), as one of the core technologies in the field of artificial intelligence, has made breakthrough progress in many complex real-world scenarios, demonstrating powerful autonomous learning and decision-making capabilities. However, when reinforcement learning technology is extended to multi-agent systems to form multi-agent reinforcement learning (MARL), it faces unique and severe challenges. The most critical issue is partial observability: each agent can only acquire local environmental information based on its own sensors, unable to directly perceive the complete state of the global environment, and also struggling to fully grasp the behavioral intentions and state information of other agents. This information limitation necessitates that agents integrate fragmented local observations and interaction information to construct a coherent understanding of the dynamic environment, thereby achieving effective collaborative decision-making. This has become a fundamental cognitive challenge that urgently needs to be addressed in the MARL field.

[0003] Communication mechanisms among multiple agents are widely recognized as a key approach to overcoming the aforementioned bottlenecks. Information sharing among agents can enrich the collective perception dimension and enhance the overall level of environmental awareness. While existing MARL methods have made some progress in communication mechanism research, most of these studies focus on optimizing the sending end of communication, such as how to generate semantically valuable message content, how to dynamically select communication timing to save bandwidth resources, and how to accurately match communication targets to improve interaction efficiency. Specifically, current mainstream MARL methods rely on predefined and rigid cognitive patterns, meaning that in each decision-making step, fixed rules are used to integrate all available information, regardless of how dynamically the environmental state changes or how complex the interaction relationships between agents evolve; the information aggregation strategy remains unchanged. This "one-size-fits-all" approach has limited ability to adapt to the dynamic evolution of the environment and is insufficient in modeling the subtle and variable interaction dependencies between agents, potentially limiting the robustness and adaptability of the model in complex multi-agent collaborative tasks.

[0004] To address the aforementioned issues, this invention designs an adaptive cognitive framework capable of dynamically capturing environmental changes and agent interaction dependencies, and achieving efficient information integration. This framework is of great significance in promoting the practical application of multi-agent technology in complex real-world scenarios. Summary of the Invention

[0005] This invention provides a multi-agent collaborative decision-making system and method based on hierarchical cognitive graph autoencoder, aiming to solve the problem of limited decision-making performance in existing multi-agent reinforcement learning techniques due to rigid cognitive patterns, inefficient information integration, weak global collaboration, and insufficient adaptability to complex scenarios.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A multi-agent collaborative decision-making method based on hierarchical cognitive graph autoencoding includes: acquiring local observations, actions, and interaction messages of each agent at each time step; constructing an initial communication graph using the interaction messages as graph nodes; generating an adjacency matrix by calculating the association strength between nodes through an attention network; performing K-step message passing using a graph attention network based on the adjacency matrix and node features to generate a latent graph encoding local interaction information; performing a three-layer graph pooling operation on the latent graph, with each pooling layer employing an improved DiffPool algorithm to aggregate node features and adjacency relationships according to a clustering allocation matrix, ultimately outputting a 64-dimensional global cognitive representation; extracting local features through a two-layer MLP based on the global cognitive representation and fusing them using Hadamard product to generate personalized features; and inputting the personalized features, local observations, and actions into the three-layer MLP. The Q-function calculates the Q-value of each agent and sums them to obtain the global Q-value. During training, the temporal difference loss, regularization loss, and reconstruction loss are calculated and weighted to form the total loss function. The parameters are updated through backpropagation using the Adam optimizer. The target Q-network parameters are synchronized from the main network every 1000 steps. If the number of training steps reaches 10 million or the model converges, training is terminated and the model weights are saved.

[0008] In one aspect of this disclosure, the step of acquiring local observations, actions, and interaction messages of each agent at each time step, constructing an initial communication graph using the interaction messages as graph nodes, and generating an adjacency matrix by calculating the association strength between nodes through an attention network includes:

[0009] Receive 64-dimensional interaction messages sent by N agents at time step t, and construct a node feature matrix. ;

[0010] Concatenate the messages of any two agents i and j into a 128-dimensional vector, input it into an attention network consisting of two fully connected layers and a ReLU activation function, and output a scalar value in the interval [0,1]. ;

[0011] Repeat the above calculation for all i,j∈{1,…,N} to form an N×N adjacency matrix A, where ;

[0012] Combine X and A to construct the initial communication graph. .

[0013] In one aspect of this disclosure, the step of generating a latent graph encoding local interaction information by using a graph attention network for K-step message passing based on the adjacency matrix and node features includes:

[0014] Will Input the first layer of the graph attention network, which contains 8 attention heads. Each head independently calculates the weights of the neighboring nodes and aggregates the features in a weighted manner, and outputs the representation of the intermediate nodes.

[0015] The intermediate node representations and the original adjacency matrix A are then input into the second-layer graph attention network, which also employs an 8-head attention mechanism. The output is the node feature matrix after two-step propagation. ;

[0016] Will Forming a latent map with A This serves as the input for subsequent hierarchical aggregation.

[0017] In one aspect of this disclosure, a three-layer graph pooling operation is performed on the latent graph, with each pooling layer employing an improved DiffPool algorithm. This algorithm aggregates node features and adjacency relationships based on a clustering allocation matrix, ultimately outputting a 64-dimensional global cognitive representation, including:

[0018] In the l-th layer (l=1,2,3) pooling, the current graph... Input a dedicated GNN layer, output a clustering assignment matrix ,in ;

[0019] Embedded by embedding dedicated GNN layer computing nodes Execute feature aggregation With structural aggregation ;

[0020] After three layers of pooling, the number of nodes is reduced to 12.5% ​​of the initial number. The final node features X^{(3)} are input into a fully connected network and output a 64-dimensional global cognitive representation c.

[0021] In one aspect of this disclosure, the step of extracting local features through a two-layer MLP based on the global cognitive representation and the local observations of each agent, and then using Hadamard product fusion to generate personalized features, includes:

[0022] The 64-dimensional local observation o_i of agent i is input into a two-layer MLP with a hidden layer dimension of 128 and ReLU activation, and the output is a 64-dimensional local feature. ;

[0023] Calculate personalized features , where ⊙ represents the Hadamard volume;

[0024] Repeat the above operation for all agents to generate N personalized feature vectors.

[0025] In one aspect of this disclosure, the step of using a three-layer MLP Q-function to calculate the Q-value of each agent and summing them to obtain the global Q-value includes:

[0026] The local observation o_i, candidate action a_i, and personalized feature p_i of agent i are concatenated into a 192-dimensional vector, which is input into a three-layer MLP with a hidden layer dimension of 256, and the output is a scalar. ;

[0027] The global Q-value is obtained by summing the Q-values ​​of all agents. ;

[0028] During the training phase, an ε-greedy strategy is used to select actions, with ε decreasing linearly from 0.9 to 0.05; during the inference phase, the action that maximizes Q_i is selected as the action to be executed.

[0029] In one aspect of this disclosure, during training, the temporal difference loss, regularization loss, and reconstruction loss are calculated, weighted and summed to form a total loss function, and the parameters are updated through backpropagation using the Adam optimizer. The target Q-network parameters are synchronized from the main network every 1000 steps, including:

[0030] Calculate time difference loss Where r is the global reward of the environment, and γ is the discount factor. Output for the target Q network;

[0031] The original node embedding is reconstructed from c using graph inverse pooling. And reconstruct the global state of the environment from c. Calculate the regularization loss With reconstruction loss ;

[0032] Constructing total loss ;

[0033] The training data is grouped into batches of 256, each containing 20 consecutive time step sequences, and then input into the model to calculate L.

[0034] The Adam optimizer is used with an initial learning rate of 1×10⁻⁶. -4 After training steps exceed 5 million, the rate decreases to 1×10⁻ 5 Weight decay coefficient 1×10⁻ 5 Gradient clipping threshold 1.0;

[0035] The parameters of the master Q network are copied to the target Q network every 1000 steps.

[0036] In one aspect of this disclosure, the step of terminating training and saving model weights if the number of training steps reaches 10 million or the model converges includes:

[0037] If the moving average of the global reward on the monitoring validation set does not show a significant improvement for 500,000 consecutive steps, the model is considered to have converged.

[0038] When the number of training steps reaches 10 million or the convergence condition is met, all trainable parameters of the cognitive module, decision-making module and training module are written to a disk file.

[0039] End the current training process.

[0040] In another aspect, this disclosure also relates to a multi-agent collaborative decision-making system based on hierarchical cognitive graph autoencoders, comprising:

[0041] The communication graph construction module is used to acquire local observations, actions, and interaction messages of each agent at each time step, use the interaction messages as graph nodes to construct the initial communication graph, and generate an adjacency matrix by calculating the association strength between nodes through an attention network.

[0042] The message passing module is used to perform K-step message passing based on the adjacency matrix and node features using a graph attention network to generate a latent graph that encodes local interaction information.

[0043] The hierarchical aggregation module is used to perform a three-layer graph pooling operation on the latent graph. Each pooling layer adopts an improved DiffPool algorithm, which aggregates node features and adjacency relationships according to the clustering allocation matrix, and finally outputs a 64-dimensional global cognitive representation.

[0044] The feature fusion module is used to extract local features based on the global cognitive representation and the local observations of each agent through a two-layer MLP, and to generate personalized features by fusion using Hadamard product;

[0045] The Q-value calculation module is configured to use the three-layer MLP Q function of personalized features, local observations and action inputs to calculate the Q-value of each agent and sum them to obtain the global Q-value.

[0046] The loss optimization module is used to calculate the temporal difference loss, regularization loss, and reconstruction loss during training, and then sum them up to form the total loss function. The parameters are updated through backpropagation of the Adam optimizer, and the target Q network parameters are synchronized from the main network every 1000 steps.

[0047] The training termination module is used to determine whether the number of training steps has reached 10 million or the model has converged. If so, the training is terminated and the model weights are saved.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] This invention dynamically captures the interaction dependencies between agents through adaptive communication graph modeling, and utilizes a hierarchical graph pooling mechanism to achieve efficient information extraction from local messages to global cognition, significantly improving the collaborative decision-making ability of multiple agents in partially observable environments. Its multi-objective training framework, which combines regularization and reconstruction loss, enhances the stability and generalization performance of the model in complex scenarios. Furthermore, the personalized feature fusion mechanism based on Hadamard product enables each agent to maintain decision-making differences while sharing global semantics, thereby significantly improving task completion efficiency and overall system adaptability while reducing communication and computational overhead. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0051] Figure 1 This is a flowchart of a multi-agent collaborative decision-making method based on hierarchical cognitive graph autoencoder according to the present invention.

[0052] Figure 2 This is a framework diagram of a multi-agent collaborative decision-making system based on hierarchical cognitive graph autoencoder according to the present invention. Detailed Implementation

[0053] The present invention will be further described below with reference to embodiments. These embodiments are merely some, not all, of the embodiments of the present invention. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the protection scope of the present invention.

[0054] Please see Figure 1 as well as Figure 2 As shown in the figure, this embodiment discloses a multi-agent collaborative decision-making method and system based on hierarchical cognitive graph autoencoder, and its specific implementation is described below.

[0055] At each time step, local observations, actions, and interaction messages of each agent are acquired. These interaction messages are used as graph nodes to construct an initial communication graph. An adjacency matrix is ​​generated by calculating the association strength between nodes using an attention network. Based on the adjacency matrix and node features, a graph attention network is used for K-step message passing to generate a latent graph encoding local interaction information. A three-layer graph pooling operation is performed on the latent graph, with each pooling layer employing an improved DiffPool algorithm. Node features and adjacency relationships are aggregated according to a clustering allocation matrix, ultimately outputting a 64-dimensional global cognitive representation. Based on this global cognitive representation and the local observations of each agent, local features are extracted using a two-layer MLP and fused using Hadamard product to generate personalized features. The personalized features, local observations, and actions are then input into the three-layer MLP. The Q-function calculates the Q-value of each agent and sums them to obtain the global Q-value. During training, the temporal difference loss, regularization loss, and reconstruction loss are calculated and weighted to form the total loss function. The parameters are updated through backpropagation using the Adam optimizer. The target Q-network parameters are synchronized from the main network every 1000 steps. If the number of training steps reaches 10 million or the model converges, training is terminated and the model weights are saved.

[0056] In one aspect of this disclosure, the step of acquiring local observations, actions, and interaction messages of each agent at each time step, constructing an initial communication graph using the interaction messages as graph nodes, and generating an adjacency matrix by calculating the association strength between nodes through an attention network includes:

[0057] Receive 64-dimensional interaction messages sent by N agents at time step t, and construct a node feature matrix. ;

[0058] Concatenate the messages of any two agents i and j into a 128-dimensional vector, input it into an attention network consisting of two fully connected layers and a ReLU activation function, and output a scalar value in the interval [0,1]. ;

[0059] Repeat the above calculation for all i,j∈{1,…,N} to form an N×N adjacency matrix A, where ;

[0060] Combine X and A to construct the initial communication graph. .

[0061] In some embodiments, the step of generating a latent graph encoding local interaction information by using a graph attention network for K-step message passing based on the adjacency matrix and node features includes:

[0062] Will Input the first layer of the graph attention network, which contains 8 attention heads. Each head independently calculates the weights of the neighboring nodes and aggregates the features in a weighted manner, and outputs the representation of the intermediate nodes.

[0063] The intermediate node representations and the original adjacency matrix A are then input into the second-layer graph attention network, which also employs an 8-head attention mechanism. The output is the node feature matrix after two-step propagation. ;

[0064] Will Forming a latent map with A This serves as the input for subsequent hierarchical aggregation.

[0065] In one aspect of this disclosure, a three-layer graph pooling operation is performed on the latent graph, with each pooling layer employing an improved DiffPool algorithm. This algorithm aggregates node features and adjacency relationships based on a clustering allocation matrix, ultimately outputting a 64-dimensional global cognitive representation, including:

[0066] In the l-th layer (l=1,2,3) pooling, the current graph... Input a dedicated GNN layer, output a clustering assignment matrix ,in ;

[0067] in, and These represent the number of graph nodes before and after the l-th pooling layer, respectively.

[0068] This represents the floor function;

[0069] Embedded by embedding dedicated GNN layer computing nodes Execute feature aggregation With structural aggregation ;

[0070] After three layers of pooling, the number of nodes is reduced to 12.5% ​​of the initial number. The final node features X^{(3)} are input into a fully connected network and output a 64-dimensional global cognitive representation c.

[0071] In one aspect of this disclosure, the step of extracting local features through a two-layer MLP based on the global cognitive representation and the local observations of each agent, and then using Hadamard product fusion to generate personalized features, includes:

[0072] The 64-dimensional local observation o_i of agent i is input into a two-layer MLP with a hidden layer dimension of 128 and ReLU activation, and the output is a 64-dimensional local feature. ;

[0073] Calculate personalized features , where ⊙ represents the Hadamard volume;

[0074] Repeat the above operation for all agents to generate N personalized feature vectors.

[0075] In one aspect of this disclosure, the step of using a three-layer MLP Q-function to calculate the Q-value of each agent and summing them to obtain the global Q-value includes:

[0076] The local observation o_i, candidate action a_i, and personalized feature p_i of agent i are concatenated into a 192-dimensional vector, which is input into a three-layer MLP with a hidden layer dimension of 256, and the output is a scalar. ;

[0077] The global Q-value is obtained by summing the Q-values ​​of all agents. ;

[0078] During the training phase, an ε-greedy strategy is used to select actions, with ε decreasing linearly from 0.9 to 0.05; during the inference phase, the action that maximizes Q_i is selected as the action to be executed.

[0079] In one aspect of this disclosure, during training, the temporal difference loss, regularization loss, and reconstruction loss are calculated, weighted and summed to form a total loss function, and the parameters are updated through backpropagation using the Adam optimizer. The target Q-network parameters are synchronized from the main network every 1000 steps, including:

[0080] Calculate time difference loss Where r is the global reward of the environment, and γ is the discount factor. Output for the target Q network;

[0081] The original node embedding is reconstructed from c using graph inverse pooling. And reconstruct the global state of the environment from c. Calculate the regularization loss With reconstruction loss ;

[0082] in, This is the cosine similarity function.

[0083] For global forms of feedback obtained from the environment to assist training (such as global rewards, task completion flags, or privileged state information in the simulation environment);

[0084] The corresponding feedback is reconstructed from the global cognitive representation;

[0085] To calculate the expected value of the training batch data.

[0086] Constructing total loss ;

[0087] The training data is grouped into batches of 256, each containing 20 consecutive time step sequences, and then input into the model to calculate L.

[0088] The Adam optimizer is used with an initial learning rate of 1×10⁻⁶. -4 After training steps exceed 5 million, the rate decreases to 1×10⁻ 5 Weight decay coefficient 1×10⁻ 5 Gradient clipping threshold 1.0;

[0089] The parameters of the master Q network are copied to the target Q network every 1000 steps.

[0090] In one aspect of this disclosure, the step of terminating training and saving model weights if the number of training steps reaches 10 million or the model converges includes:

[0091] If the moving average of the global reward on the monitoring validation set does not show a significant improvement for 500,000 consecutive steps, the model is considered to have converged.

[0092] When the number of training steps reaches 10 million or the convergence condition is met, all trainable parameters of the cognitive module, decision-making module and training module are written to a disk file.

[0093] End the current training process.

[0094] In another aspect, this disclosure also relates to a multi-agent collaborative decision-making system based on hierarchical cognitive graph autoencoders, comprising:

[0095] The communication graph construction module is used to acquire local observations, actions, and interaction messages of each agent at each time step, use the interaction messages as graph nodes to construct the initial communication graph, and generate an adjacency matrix by calculating the association strength between nodes through an attention network.

[0096] The message passing module is used to perform K-step message passing based on the adjacency matrix and node features using a graph attention network to generate a latent graph that encodes local interaction information.

[0097] The hierarchical aggregation module is used to perform a three-layer graph pooling operation on the latent graph. Each pooling layer adopts an improved DiffPool algorithm, which aggregates node features and adjacency relationships according to the clustering allocation matrix, and finally outputs a 64-dimensional global cognitive representation.

[0098] The feature fusion module is used to extract local features based on the global cognitive representation and the local observations of each agent through a two-layer MLP, and to generate personalized features by fusion using Hadamard product;

[0099] The Q-value calculation module is configured to use the three-layer MLP Q function of personalized features, local observations and action inputs to calculate the Q-value of each agent and sum them to obtain the global Q-value.

[0100] The loss optimization module is used to calculate the temporal difference loss, regularization loss, and reconstruction loss during training, and then sum them up to form the total loss function. The parameters are updated through backpropagation of the Adam optimizer, and the target Q network parameters are synchronized from the main network every 1000 steps.

[0101] The training termination module is used to determine whether the number of training steps has reached 10 million or the model has converged. If so, the training is terminated and the model weights are saved.

[0102] In the specific implementation process, the system first receives input data from N agents at each time step t, including the local observations of each agent. Execution of actions ( (for the action space dimension) and interaction messages These interaction messages are generated by the internal communication modules of each agent and serve as the initial features of the graph nodes. All interaction messages are stacked to form a node feature matrix. Subsequently, the system constructs an initial communication graph. The adjacency matrix It is calculated through an attention network. Specifically, for any two agents i and j, their interaction messages are... and Concatenate into a 128-dimensional vector The input is an attention network consisting of two fully connected layers, each followed by a ReLU activation function, and the output is a scalar value. This value represents the association strength between agents i and j. This operation is repeated for all i,j∈{1,…,N} to form the complete adjacency matrix A, where... Therefore, the initial communication diagram The node features are X, and the edge weights are A.

[0103] Next, the system inputs G_0 into the message passing module, which contains a two-layer graph attention network (GAT). The first layer of GAT contains eight independent attention heads, each of which performs a weighted aggregation of features from its neighboring nodes: for node i, the output of its k-th attention head is... ,in For learnable weight matrix, Let N(i) be the normalized attention coefficient, and N(i) be the set of neighbors of node i (including itself). The outputs of the 8 heads are concatenated to form a 64-dimensional intermediate node representation. The second layer of GAT also adopts an 8-head structure, using the intermediate node representation output from the first layer as input features, and continues to use the original adjacency matrix A for message passing, finally outputting the node feature matrix after two-step propagation. At this point, the system generates a latent map. This graph encodes local interaction information, which serves as input for subsequent hierarchical aggregation.

[0104] Subsequently, the system enters the hierarchical aggregation module, performing a three-layer graph pooling operation. Each pooling layer employs an improved DiffPool algorithm. In the l-th layer (l=1,2,3), the current graph... The input consists of two dedicated graph neural networks (GNNs): one for generating the clustering assignment matrix. Another one used to generate node embeddings .in, , For example, if the initial number of agents N=64, then the number of nodes after the first pooling layer is n_1=32, the second layer is n_2=16, and the third layer is n_3=8. Clustering assignment matrix. Each row represents the probability distribution of a node in the original graph being assigned to a cluster in the new graph. Node embedding Through GNN and This is obtained after message passing. Subsequently, the system performs feature aggregation: and structural aggregation: After three layers of pooling, the final node features are... It is fed into a fully connected network (output dimension 64) to generate a 64-dimensional global cognitive representation. .

[0105] After obtaining the global cognitive representation c, the system enters the feature fusion module. For each agent i, its 64-dimensional local observations... The input is a two-layer MLP. The first layer has 128 hidden units and uses ReLU activation. The second layer outputs 64-dimensional local features. Subsequently, the system calculates personalized features. , where ⊙ represents the Hadamard product. This operation fuses global contextual information with individual perceptual features element-wise to generate a personalized representation for agent i. This process is executed in parallel for all N agents, resulting in N personalized feature vectors. .

[0106] Next, the system inputs personalized features, local observations, and candidate actions into the Q-value calculation module. For agent i, its local observations... Candidate actions (Assuming the action space is discrete or embedded as a vector) and personalized features Concatenated into a 192-dimensional vector The input is a three-layer MLP Q-function. The hidden layer dimension of this MLP is 256, with no special activation function restrictions (typically the last layer has no activation), and the output is a scalar. The Q-values ​​of all agents are summed to obtain the global Q-value. During the training phase, the system uses an ε-greedy strategy to select actions: ε is initially set to 0.9 and linearly decays to 0.05 over the first 9 million steps; during the inference phase, each agent selects the action that maximizes Q_i as the action to be executed.

[0107] During training, the system calculates the total loss and updates parameters using the loss optimization module. First, the temporal difference loss is calculated. , where r is the global reward returned by the environment, and γ is the discount factor (usually set to 0.99). Let c' be the Q-value output by the target Q-network at the next time step t+1, c' be the global cognitive representation at time t+1, and a' be the action selected at time t+1. Next, the system performs graph depooling: starting from the final global cognitive representation c, the original node embeddings are gradually reconstructed through learnable depooling layers. At the same time, the system also reconstructs the global state of the environment from c. (Assuming the environment provides global state) (For supervision). Based on this, the regularization loss is calculated. Where i≠j, Cosine similarity is used to constrain the discriminativeness of reconstructed embeddings by different agents; reconstruction loss. This is used to ensure the fidelity of the self-encoder.

[0108] Total loss is defined as ,

[0109] in, This represents the regularization loss weight coefficient, used to adjust the regularization loss. The degree of contribution to the total loss controls the strength of the discriminative constraints on the reconstruction embedding of different agents.

[0110] This represents the reconstruction loss weighting coefficient, used to adjust the reconstruction loss. The weighting of the impact on the total loss balances the fidelity constraints of the autoencoder on the original node embedding and the global state of the environment.

[0111] In this embodiment, Take 0.1, Take 0.05.

[0112] It should be noted that although the agent can only obtain local observations when making decisions, However, during the training phase, modern reinforcement learning simulation platforms (such as StarCraft II and Multi-Agent Particle Environment) can typically provide global information. Used to calculate auxiliary loss, in order to accelerate training and improve model performance. Reconstruction loss This approach leverages this privileged information, forcing the global cognitive representation to encode task-relevant global semantics without compromising its effectiveness in partially observable environments. During model deployment and inference, only local observations are required, without any global information.

[0113] Training data is input in batches, with each batch containing 256 samples. Each sample is a time-step sequence of length 20 (i.e., 20 consecutive time steps). (Tuples). The system uses the Adam optimizer with an initial learning rate of 1×10⁻⁶. -4 When the number of training steps exceeds 5 million, the learning rate decays to 1×10. -5 The weight decay coefficient is 1×10. -5 The gradient clipping threshold is set to 1.0 to prevent gradient explosion. The parameters of the target Q-network are copied from the main Q-network every 1000 training steps to stabilize the training process.

[0114] Training termination is controlled by the training termination module. The system continuously monitors the global reward moving average on the validation set (window size of 100,000 steps). If this metric improves by less than a preset threshold (e.g., 0.5%) within 500,000 consecutive steps, the model is considered to have converged. Furthermore, if the training steps reach 10 million, training is terminated regardless of convergence. After training, the system saves all trainable parameters (including attention network weights, GAT parameters, DiffPool allocation and embedding of GNN, MLP weights, unpooling layer parameters, etc.) from the communication graph construction module, message passing module, hierarchical aggregation module, feature fusion module, Q-value calculation module, and loss optimization module to a disk file for subsequent deployment or inference.

[0115] In practical applications, such as multi-UAV collaborative search missions, each UAV acts as an intelligent agent. Its local observations include its own position, speed, and target information within the sensor's detection range. Interaction messages are periodically broadcast by the onboard communication module. The global environmental reward is a weighted sum of the number of successfully detected targets and energy consumption. The system executes the above process in each control cycle (e.g., 100ms): constructing a communication graph, transmitting messages, hierarchically aggregating to generate global cognition, fusing individual features, calculating the Q-value, and selecting an action (e.g., adjusting heading or speed). The training phase takes place in a simulation environment, using the aforementioned loss function and optimization strategy. The deployment phase loads the saved model weights, performs only forward inference, and does not calculate the loss.

[0116] The modules of the entire system are logically connected sequentially: the output G_0 of the communication graph construction module serves as the input of the message passing module; the output of the message passing module... As input to the hierarchical aggregation module; the output c of the hierarchical aggregation module and each agent Common input feature fusion module; output of feature fusion module and , Together, they constitute the input of the Q-value calculation module; the output of the Q-value calculation module is used for loss calculation, while the gradient backpropagation of the loss optimization module updates the parameters of all modules. The modules are connected through tensor data streams, without explicit hardware interfaces, but can be implemented as functions or class objects in distributed computing frameworks (such as PyTorch or TensorFlow), and their internal parameters are automatically managed through shared memory or computation graphs.

[0117] Regarding parameter configuration, all MLP weights are initialized using Xavier initialization, GNN layers use a normalized adjacency matrix for message passing, the output of the attention network is constrained to the [0,1] interval by a Sigmoid function, and the clustering assignment matrix of the DiffPool is used. Softmax normalization is used to ensure probabilistic interpretability. The unpooling operation maps low-dimensional cluster features back to the high-dimensional node space using a learned mapping matrix; its parameters are jointly trained with the pooling layer. The entire system can achieve real-time inference with GPU acceleration, and the latency for a single forward propagation is less than 10ms (when N≤64).

[0118] The above implementation fully describes the entire process from data input, graph construction, message passing, hierarchical aggregation, feature fusion, Q-value calculation to loss optimization and training termination, covering all necessary technical details, including data dimensions, network structure, hyperparameter settings, training strategies, and data flow between modules, ensuring that those skilled in the art can implement the technical solution of this invention based on this specification.

[0119] To enable those skilled in the art to fully understand and implement this invention, the specific implementation principles of this invention are further supplemented below with a specific application scenario.

[0120] In multi-UAV cooperative search and target acquisition missions, each UAV acts as an independent intelligent agent. Its onboard sensors can only acquire local environmental information, including its own latitude and longitude coordinates, flight speed vector, altitude, and the positions of nearby targets or obstacles detected by radar or visual sensors, forming a 64-dimensional local observation vector. Each UAV periodically broadcasts its own state summary via its wireless communication module, forming a 64-dimensional interactive message. This message is generated by the internal encoder based on current observations and historical actions, rather than being directly transmitted as raw data, thus reducing bandwidth consumption. In each control cycle (e.g., 100ms), the system first processes all interaction messages from the N UAVs. Stacked as a node feature matrix And construct the initial communication graph. The adjacency matrix A is not a pre-defined fixed topology, but is dynamically generated through a learnable attention network: the messages of any two drones i and j are concatenated and input into a two-layer fully connected network, outputting the association strength a. ij ∈[0,1], this value reflects the necessity of cooperation between the two in the current task phase (such as target approach, formation adjustment, or obstacle avoidance). For example, when two UAVs simultaneously detect the same high-value target, their a ij A significant increase in the value indicates a need to strengthen information interaction; while drone pairs that are far away or unrelated to the mission are assigned low weights, thereby achieving adaptive allocation of communication resources and avoiding redundant computation caused by full connectivity. Subsequently, the system will... A two-layer Graph Attention Network (GAT) is used for message passing. The first layer of the GAT has eight attention heads that aggregate neighbor information from different semantic subspaces; for example, some heads may focus on velocity consistency, while others focus on relative orientation, ultimately concatenating these to form an intermediate representation. The second layer of the GAT further integrates higher-order interactions, making the output feature X of each node... (2) It not only contains information about direct neighbors, but also implicitly encodes the group dynamics within a two-hop range. The resulting latent graph... The original communication structure was retained, but the node features already contained local collaborative semantics, laying the foundation for subsequent hierarchical abstraction.

[0121] Entering the hierarchical aggregation phase, the system adopts a three-layer improved DiffPool operation pair. Coarsening is performed. In each pooling layer, the clustering assignment network dynamically determines which drones should be grouped into the same functional cluster based on the current graph structure—for example, in a target encirclement scenario, drones closer to the target might be clustered into the "encirclement cluster," while peripheral drones might be clustered into the "vigilance cluster." Clustering assignment matrix S (l) Softmax normalization ensures probabilistic interpretability, while the embedded network generates representative features within the cluster. After three layers of pooling, the 64 original nodes are compressed into 8 high-level cluster nodes, whose features are mapped to a 64-dimensional global cognitive representation c via a fully connected layer. This representation is not a simple average or max pooling result, but a task-level semantic summary extracted through a differentiable graph pooling mechanism, which can reflect the overall situation (such as whether the target is surrounded, whether there are communication blind spots, and whether the energy consumption distribution is balanced), thereby solving the problem of fragmentation in global value estimation in existing technologies.

[0122] During the feature fusion phase, each UAV will perform its local observations 0 i Individual features f are encoded through two layers of MLP. i Then, perform a Hadamard product operation with the global cognition c to obtain personalized features. The principle behind this operation is that the Hadamard product implements a gating mechanism, where high activation dimensions in the global context c enhance f. i The response of the corresponding channel in the middle is high, while low activation dimensions suppress irrelevant features. For example, if the global cognition indicates "energy consumption should be reduced first", then the energy consumption-related dimension value in c is high, and P will be low. i This mechanism strengthens the correlation between speed, altitude, and energy consumption in local observations, guiding the Q function to favor energy-saving actions. Conversely, if global cognition emphasizes "rapid encirclement," then speed and direction-related features are highlighted. This dynamic modulation mechanism allows agents to retain individual decision-making differences while sharing global understanding, effectively avoiding policy homogenization.

[0123] In Q-value calculation, After concatenation, the input is a three-layer MLP, and the output is the action value Q. i Because of P i Having incorporated global collaborative intent, the Q-function can evaluate the long-term benefits of actions at the group level, rather than solely based on local rewards. For example, a drone might temporarily sacrifice its own detection efficiency (decreased local reward), but its course adjustment helps form an encirclement (increased global reward). This action benefits from P... i Those containing global cooperative signals receive a positive evaluation in their Q-value and are thus selected for execution. All Q... i The summation forms the global Q-value, which is used for time-difference updates to ensure that the policy optimization direction is consistent with the group objective.

[0124] During training, the system introduces a graph depooling reconstruction mechanism: starting from the global cognition c, it reconstructs the original node embeddings by upsampling layer by layer through learnable depooling layers. Regularization loss Force the reconstructed embeddings of different agents to maintain discriminativeness and prevent representation collapse; reconstruction loss. This constrains the fidelity of the autoencoder, ensuring that c can fully reconstruct the original interaction graph and the global state of the environment. This dual reconstruction mechanism not only improves the robustness of the model but also provides additional supervision signals for training, alleviating the gradient sparsity problem in some observable environments and accelerating convergence.

[0125] In summary, in multi-UAV collaborative missions, this invention achieves adaptive modeling of environmental changes and agent dependencies by dynamically constructing a communication graph, hierarchically abstracting group interactions, and personalized fusion of global and local information, supplemented by self-encoding regularization constraints. Compared to fixed communication topologies or linear information integration methods, this scheme achieves significant improvements in target capture success rate, communication overhead, and policy convergence speed, with its advantages becoming more pronounced when the number of agents increases or the environmental dynamics intensify.

[0126] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-agent collaborative decision-making method based on hierarchical cognitive graph autoencoder, characterized in that, include: At each time step, local observations, actions, and interaction messages of each agent are acquired. The interaction messages are used as graph nodes to construct an initial communication graph. An adjacency matrix is ​​generated by calculating the association strength between nodes through an attention network. Based on the adjacency matrix and node features, a graph attention network is used to perform K-step message passing to generate a latent graph that encodes local interaction information. A three-layer graph pooling operation is performed on the latent graph. Each pooling layer uses an improved DiffPool algorithm to aggregate node features and adjacency relationships based on the clustering allocation matrix, and finally outputs a 64-dimensional global cognitive representation. Based on the global cognitive representation and the local observations of each agent, local features are extracted through a two-layer MLP and personalized features are generated by fusion using Hadamard product. The personalized features, local observations, and actions are input into a three-layer MLP Q function to calculate the Q value of each agent and sum them to obtain the global Q value. During training, the temporal difference loss, regularization loss, and reconstruction loss are calculated, weighted and summed to form the total loss function. The parameters are updated through backpropagation using the Adam optimizer, and the target Q network parameters are synchronized from the main network every 1000 steps. If the number of training steps reaches 10 million or the model converges, training is terminated and the model weights are saved.

2. The multi-agent collaborative decision-making method based on hierarchical cognitive graph autoencoder according to claim 1, characterized in that, The process of acquiring local observations, actions, and interaction messages of each agent at each time step, using the interaction messages as graph nodes to construct an initial communication graph, and generating an adjacency matrix by calculating the association strength between nodes through an attention network includes: Receive 64-dimensional interaction messages sent by N agents at time step t, and construct a node feature matrix. ; Concatenate the messages of any two agents i and j into a 128-dimensional vector, input it into an attention network consisting of two fully connected layers and a ReLU activation function, and output a scalar value in the interval [0,1]. ; Repeat the above calculation for all i,j∈{1,…,N} to form an N×N adjacency matrix A, where ; Combine X and A to construct the initial communication graph. .

3. The multi-agent collaborative decision-making method based on hierarchical cognitive graph autoencoder according to claim 1, characterized in that: The step of generating a latent graph encoding local interaction information by using a graph attention network for K-step message passing based on the adjacency matrix and node features includes: Will Input the first layer of the graph attention network, which contains 8 attention heads. Each head independently calculates the weights of the neighboring nodes and aggregates the features in a weighted manner, and outputs the representation of the intermediate nodes. The intermediate node representations and the original adjacency matrix A are then input into the second-layer graph attention network, which also employs an 8-head attention mechanism. The output is the node feature matrix after two-step propagation. ; Will Forming a latent map with A This serves as the input for subsequent hierarchical aggregation.

4. The multi-agent collaborative decision-making method based on hierarchical cognitive graph autoencoder according to claim 1, characterized in that: The process involves performing a three-layer graph pooling operation on the latent graph. Each pooling layer employs an improved DiffPool algorithm, aggregating node features and adjacency relationships based on a clustering allocation matrix. The final output is a 64-dimensional global cognitive representation, including: In the l-th layer (l=1,2,3) pooling, the current graph... Input a dedicated GNN layer, output a clustering assignment matrix ,in ; Embedded by embedding dedicated GNN layer computing nodes Execute feature aggregation With structural aggregation ; After three layers of pooling, the number of nodes is reduced to 12.5% ​​of the initial number. The final node features X^{(3)} are input into a fully connected network and output a 64-dimensional global cognitive representation c.

5. The multi-agent collaborative decision-making method based on hierarchical cognitive graph autoencoder according to claim 1, characterized in that: The process, based on the global cognitive representation and the local observations of each agent, extracts local features through a two-layer MLP and generates personalized features using Hadamard product fusion, including: The 64-dimensional local observation o_i of agent i is input into a two-layer MLP with a hidden layer dimension of 128 and ReLU activation, and the output is a 64-dimensional local feature. ; Calculate personalized features , where ⊙ represents the Hadamard volume; Repeat the above operation for all agents to generate N personalized feature vectors.

6. The multi-agent collaborative decision-making method based on hierarchical cognitive graph autoencoder according to claim 1, characterized in that: The process of using a three-layer MLP Q function to input personalized features, local observations, and actions, calculating the Q-value of each agent, and summing them to obtain the global Q-value includes: The local observation o_i, candidate action a_i, and personalized feature p_i of agent i are concatenated into a 192-dimensional vector, which is input into a three-layer MLP with a hidden layer dimension of 256, and the output is a scalar. ; The global Q-value is obtained by summing the Q-values ​​of all agents. ; During the training phase, an ε-greedy strategy is used to select actions, with ε decreasing linearly from 0.9 to 0.05; during the inference phase, the action that maximizes Q_i is selected as the action to be executed.

7. The multi-agent collaborative decision-making method based on hierarchical cognitive graph autoencoder according to claim 1, characterized in that: During training, the temporal difference loss, regularization loss, and reconstruction loss are calculated, weighted, and summed to form the total loss function. The parameters are then updated via backpropagation using the Adam optimizer. The target Q-network parameters are synchronized from the main network every 1000 steps, including: Calculate time difference loss Where r is the global reward of the environment, and γ is the discount factor. Output for the target Q network; The original node embedding is reconstructed from c using graph inverse pooling. And reconstruct the global state of the environment from c. Calculate the regularization loss With reconstruction loss ; Constructing total loss ; The training data is grouped into batches of 256, each containing 20 consecutive time step sequences, and then input into the model to calculate L. The Adam optimizer is used with an initial learning rate of 1×10⁻⁶. -4 After training steps exceed 5 million, the rate decreases to 1×10⁻ 5 Weight decay coefficient 1×10⁻ 5 Gradient clipping threshold 1.0; The parameters of the master Q network are copied to the target Q network every 1000 steps.

8. The multi-agent collaborative decision-making method based on hierarchical cognitive graph autoencoder according to claim 1, characterized in that: If the number of training steps reaches 10 million or the model converges, then training is terminated and the model weights are saved, including: If the moving average of the global reward on the monitoring validation set does not show a significant improvement for 500,000 consecutive steps, the model is considered to have converged. When the number of training steps reaches 10 million or the convergence condition is met, all trainable parameters of the cognitive module, decision-making module and training module are written to a disk file. End the current training process.

9. A multi-agent collaborative decision-making system based on hierarchical cognitive graph autoencoder, characterized in that, include: The communication graph construction module is used to acquire local observations, actions, and interaction messages of each agent at each time step, use the interaction messages as graph nodes to construct the initial communication graph, and generate an adjacency matrix by calculating the association strength between nodes through an attention network. The message passing module is used to perform K-step message passing based on the adjacency matrix and node features using a graph attention network to generate a latent graph that encodes local interaction information. The hierarchical aggregation module is used to perform a three-layer graph pooling operation on the latent graph. Each pooling layer adopts an improved DiffPool algorithm, which aggregates node features and adjacency relationships according to the clustering allocation matrix, and finally outputs a 64-dimensional global cognitive representation. The feature fusion module is used to extract local features based on the global cognitive representation and the local observations of each agent through a two-layer MLP, and to generate personalized features by fusion using Hadamard product. The Q-value calculation module is used to input the personalized features, local observations and actions into a three-layer MLP Q function, calculate the Q-value of each agent, and sum them to obtain the global Q-value. The loss optimization module is used to calculate the temporal difference loss, regularization loss, and reconstruction loss during training, and then sum them up to form the total loss function. The parameters are updated through backpropagation of the Adam optimizer, and the target Q network parameters are synchronized from the main network every 1000 steps. The training termination module is used to determine whether the number of training steps has reached 10 million or the model has converged. If so, the training is terminated and the model weights are saved.

Images