GNN enhanced multi-agent optimization method for agent communication network
By using graph neural networks to enhance multi-agent optimization, and combining graph convolutional networks and dual-Q networks, the traditional MASAC algorithm addresses the issues of missing structural priors and insufficient topological robustness in large-scale, heterogeneous, and topologically dynamic networks. This improves scalability and training stability while reducing computational complexity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-01
- Publication Date
- 2026-04-14
AI Technical Summary
Traditional MASAC algorithms suffer from problems such as lack of structural priors, insufficient topological robustness, limited scalability, and low computational efficiency in multi-agent cooperative optimization, making them particularly difficult to apply effectively in large-scale, heterogeneous, and dynamically dynamic network environments.
A graph neural network (GNN) is used to enhance the multi-agent optimization method. Node embedding vectors are obtained through graph convolutional networks, action distribution parameters are output using GCN-Actor networks, sampled actions are obtained through reparameterized sampling, and temporal difference target values are calculated through permutation symmetric pooling and double Q networks to update GCN-Critic and GCN-Actor networks.
It improves the scalability and training stability of large-scale, heterogeneous, and topologically dynamic networks, solves the problems of missing structural priors and insufficient topological robustness, reduces computational complexity, and improves training efficiency.
Smart Images
Figure CN121859955A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of cross-application technology of multi-agent deep reinforcement learning (MADRL) and graph neural networks (GNN), and in particular to a GNN-enhanced multi-agent optimization method for agent communication networks. Background Technology
[0002] With the rapid development of artificial intelligence technology, multi-agent deep reinforcement learning (MADRL) has become an important technical approach for solving complex distributed decision-making problems. In fields such as wireless communication, smart grids, autonomous driving, and multi-robot collaboration, multi-agent systems need to perform real-time coordination and resource optimization in dynamic environments, which places higher demands on the scalability, stability, and adaptability of algorithms.
[0003] The Multi-Agent Soft Actor-Critic (MASAC) algorithm, an important branch of MADRL, demonstrates good performance in continuous control tasks by introducing the maximum entropy principle to balance exploration and exploitation. However, traditional MASAC typically uses a multilayer perceptron (MLP) as a neural network approximator for policy and value functions, which faces challenges when dealing with large-scale, heterogeneous, and topologically dynamic multi-agent networks.
[0004] Graph Neural Networks (GNNs), as an emerging deep learning architecture, are specifically designed for processing graph-structured data. They possess inherent advantages such as local message passing and permutation invariance, providing a new technical approach for modeling multi-agent systems. Combining GNNs with MADRLs holds promise for fundamentally improving the performance of algorithms in complex network environments.
[0005] Among related technologies, the MASAC algorithm based on MLP has the following key technical pain points in multi-agent collaborative optimization: (1) lack of structural priors. Traditional MLP ignores the topological relationship between agents, cannot effectively capture local neighborhood interaction patterns, and is difficult to model complex multi-agent collaborative structures; (2) insufficient topological robustness. It is sensitive to agent number permutation and network topology changes, has poor deployment adaptability in dynamic environments, and the system stability is affected; (3) limited scalability. As the number of agents increases, the state-action space grows exponentially, resulting in low training sample efficiency and difficulty in applying it to large-scale systems; (4) poor training stability. In high-dimensional heterogeneous scenarios, the variance of centralized commentators is large, the gradient update noise is high, and training oscillation or non-convergence problems are easy to occur; (5) low computational efficiency. It lacks a parameter sharing mechanism. The network parameters grow linearly with the number of agents, the computational complexity is high, and the real-time performance is difficult to guarantee, which needs to be solved urgently. Summary of the Invention
[0006] This application provides a GNN-enhanced multi-agent optimization method for agent communication networks to address the problems of missing structural priors, insufficient topological robustness, and low computational efficiency in related technologies, thereby improving scalability and training stability in large-scale, heterogeneous, and topologically dynamic networks.
[0007] To achieve the above objectives, the first aspect of this application proposes a GNN-enhanced multi-agent optimization method for agent communication networks, comprising the following steps: Obtain the observation graph of the system to be optimized at the target time, and use a preset graph convolutional network to perform message passing on the observation graph to obtain the node embedding vector of each agent. The observation graph includes multiple agents and graph-level global vectors. The node embedding vector of each agent is input into a preset GCN-Actor network to output action distribution parameters for each agent through the preset GCN-Actor network, and the sampled action of each agent is obtained by reparameterizing and sampling the output action distribution parameters of each agent. The node embedding vector of each agent is permuted and symmetric pooled to obtain the graph-level embedding vector of each agent. A joint action vector is obtained based on the sampled actions of each agent. The graph-level embedding vector, the graph-level global vector, and the joint action vector of each agent are concatenated to form a fusion feature. The fusion feature is input into the dual-Q network of the preset GCN-Critic network to obtain a first value and a second value. The temporal difference target value is calculated based on the first value and the second value. The preset GCN-Critic network and the preset GCN-Actor network are updated according to the temporal difference target value.
[0008] According to one embodiment of this application, updating the preset GCN-Critic network and the preset GCN-Actor network based on the temporal difference target value includes: The preset dual-Q network of the GCN-Critic network is updated according to the temporal difference target value to obtain the updated dual-Q network; The policy gradient is calculated based on the parameters of the updated dual-Q network, and the preset GCN-Actor network is updated based on the policy gradient. The target Q network of the preset GCN-Critic network is updated based on a preset soft update mechanism.
[0009] According to one embodiment of this application, the timing difference target value is: ; in, The time-series difference target value is... For instant rewards, As a discount factor, To terminate the indicator variable, For Actor parameters, For intelligent agents Local observations I The number of agents.
[0010] According to one embodiment of this application, the dual-Q network is: ; in, For state-action value function, For mathematical expectation, For time ( t + k Instant rewards Current moment t state, Current moment t The action performed.
[0011] According to one embodiment of this application, the first The sampling action of each agent is: ; in, For the first i Each agent samples actions; GCNActor is the intelligent agent. i Output motion distribution parameters, Let be the mean of the action distribution. The standard deviation of the action distribution; Standard Gaussian noise, Let represent a multivariate standard normal distribution with a mean of zero vector and a covariance of identity matrix.
[0012] The GNN-enhanced multi-agent optimization method for agent communication networks proposed in this application obtains an observation graph, obtains node embedding vectors through graph convolutional network message passing, inputs the action distribution parameters of the GCN-Actor network, obtains sampled actions through reparameterized sampling, obtains graph-level embedding vectors through permutational symmetric pooling of node embeddings, concatenates the graph-level embeddings, graph-level global vectors, and joint action vectors to form a fusion feature, inputs it into a dual-Q network to obtain the first and second values, calculates the temporal difference target value, and updates the GCN-Critic and GCN-Actor networks. This solves the problems of missing structural priors, insufficient topological robustness, and low computational efficiency in related technologies, and improves scalability and training stability in large-scale, heterogeneous, and topologically dynamic networks.
[0013] To achieve the above objectives, a second aspect of this application proposes a GNN-enhanced multi-agent optimization device for agent communication networks, comprising: The acquisition module acquires the observation map of the system to be optimized at the target time, and uses a preset graph convolutional network to perform message passing on the observation map to obtain the node embedding vector of each agent. The observation map includes multiple agents and graph-level global vectors. The sampling module inputs the node embedding vector of each agent into a preset GCN-Actor network, so as to output action distribution parameters for each agent through the preset GCN-Actor network, and performs reparameterized sampling on the output action distribution parameters of each agent to obtain the sampled action of each agent. The input module performs permutation symmetric pooling on the node embedding vector of each agent to obtain the graph-level embedding vector of each agent, obtains a joint action vector based on the sampled actions of each agent, and concatenates the graph-level embedding vector, the graph-level global vector, and the joint action vector to form a fusion feature. The fusion feature is then input into the dual-Q network of the preset GCN-Critic network to obtain a first value and a second value. The temporal difference target value is calculated based on the first value and the second value. The optimization module updates the preset GCN-Critic network and the preset GCN-Actor network according to the time-series difference target value.
[0014] According to one embodiment of this application, the optimization module includes: The preset dual-Q network of the GCN-Critic network is updated according to the temporal difference target value to obtain the updated dual-Q network; The policy gradient is calculated based on the parameters of the updated dual-Q network, and the preset GCN-Actor network is updated based on the policy gradient. The target Q network of the preset GCN-Critic network is updated based on a preset soft update mechanism.
[0015] According to one embodiment of this application, the timing difference target value is: ; in, The time-series difference target value is... For instant rewards, As a discount factor, To terminate the indicator variable, For Actor parameters, For intelligent agents Local observations I The number of agents.
[0016] According to one embodiment of this application, the dual-Q network is: ; in, For state-action value function, For mathematical expectation, For time ( t + k Instant rewards Current moment t state, Current moment t The action performed.
[0017] According to one embodiment of this application, the first The sampling action of each agent is: ; in, For the first i Each agent samples actions; GCNActor is the intelligent agent. i Output motion distribution parameters, Let be the mean of the action distribution. The standard deviation of the action distribution; Standard Gaussian noise, Let represent a multivariate standard normal distribution with a mean of zero vector and a covariance of identity matrix.
[0018] The GNN-enhanced multi-agent optimization device for agent communication networks proposed in this application acquires an observation graph, obtains node embedding vectors through graph convolutional network message passing, inputs the action distribution parameters of the GCN-Actor output, reparameterizes and samples the actions, obtains graph-level embedding vectors through permutational symmetric pooling of node embeddings, concatenates the graph-level embeddings, graph-level global vectors, and joint action vectors to form a fusion feature, inputs it into a dual-Q network to obtain the first and second values, calculates the temporal difference target value, and updates the GCN-Critic and GCN-Actor networks. This solves the problems of missing structural priors, insufficient topological robustness, and low computational efficiency in related technologies, and improves scalability and training stability in large-scale, heterogeneous, and topologically dynamic networks.
[0019] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the GNN-enhanced multi-agent optimization method for agent communication networks as described in the above embodiments.
[0020] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the GNN-enhanced multi-agent optimization method for agent communication networks as described in the above embodiments.
[0021] To achieve the above objectives, a fifth aspect of this application provides a computer program product, which, when executed by a processor, implements the GNN-enhanced multi-agent optimization method for agent communication networks as described in the above embodiments.
[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a GNN-enhanced multi-agent optimization method for agent communication networks provided in an embodiment of this application; Figure 2 This is a schematic diagram of the GCN-Actor and GCN-Critic network structure according to an embodiment of this application; Figure 3 This is a schematic diagram of the training data flow of the multi-agent GCN-Actor / Critic framework according to an embodiment of this application; Figure 4 This is a block diagram of a GNN-enhanced multi-agent optimization device for agent communication networks provided according to an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation
[0024] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0025] The following describes, with reference to the accompanying drawings, a GNN-enhanced multi-agent optimization method for agent communication networks proposed according to an embodiment of this application. First, the GNN-enhanced multi-agent optimization method for agent communication networks proposed according to an embodiment of this application will be described with reference to the accompanying drawings.
[0026] Figure 1 This is a flowchart of a GNN-enhanced multi-agent optimization method for agent communication networks according to an embodiment of this application.
[0027] like Figure 1 As shown, the GNN-enhanced multi-agent optimization method for agent communication networks includes the following steps: In step S101, the observation graph of the system to be optimized at the target time is obtained, and a preset graph convolutional network is used to perform message passing on the observation graph to obtain the node embedding vector of each agent. The observation graph includes multiple agents and graph-level global vectors.
[0028] In this context, the system to be optimized refers to the whole composed of multiple agents, the interaction environment, and related constraints in a multi-agent deep reinforcement learning scenario. The target time point refers to the specific time node used to acquire system observations, trigger agent decisions, and perform network computations. The observation graph is a graph-structured representation of the system state at the target time. An agent is an entity capable of autonomously perceiving its environment, making dynamic decisions, and executing actions. The graph-level global vector is a quantitative representation of the global state in the observation graph of multi-agent reinforcement learning, independent of agents and edges.
[0029] Specifically, the system to be optimized (such as a physical system / communication system) is abstracted as a graph that evolves over time, i.e., an observation graph: ; in, For a moment t The system diagram (i.e., the observation diagram); V is the set of nodes; For a momentt The set of edges; For node feature set; It is a graph-level global vector.
[0030] Specifically, nodes represent agents and coordinating nodes, edges represent communication / interaction relationships, and edge weights can encode channel or interaction strength. The node set V contains one agent and one coordinating / base station node (optional), totaling ( N = I +1) nodes. Node characteristics It is the core information carrier in the graph structure that describes the local state and attributes of an agent, in which agent nodes i ( i The agent index contains local observations and normalized attributes, such as normalized distance to the coordinating node, normalized computing power, and historical load. The coordinating node contains normalized representations of system-level global parameters (such as bandwidth, downlink power, noise density, and number of participating agents). The edge set... The graph-level global vector connects the coordinating node and each agent via bidirectional edges, or connects agents to agents when direct cooperation exists. The edge weights can be set as channel gain / link quality or cooperation strength. Global auxiliary features used for centralized evaluation (such as system-level resource limits).
[0031] Furthermore, by employing the adjacency matrix with self-loops and normalization, the formula is as follows: ; in, For the normalized adjacency matrix, This is the degree matrix. This is the original adjacency matrix. It is an identity matrix.
[0032] This operation can balance the impact of node degree differences on information transmission. Edge weights are set based on channel, link quality, or cooperation strength, making the graph structure fit the actual interaction scenario. At the same time, features are normalized and scale constraints are applied to improve the numerical stability of the training process.
[0033] Observation map As a pre-defined graph convolutional network with structured input, it is processed... L Layer graph convolution extracts node embedding vectors, where L The number of convolutional layers in the graph.
[0034] Under the conditions of bounded spectral norm and bounded graph degree, the dependency of node embeddings on initial features satisfies the inequality: ; in, This is the upper bound of the product of the spectral norms of the GCN layer weights. This is the upper bound of the spectral norm of the normalized adjacency matrix. For nodes and The shortest path length, For nodes of Jump to the neighboring region.
[0035] The above inequalities show that the influence between node embeddings decreases exponentially with distance in the graph, highlighting the locality of GCN, that is, node embeddings are mainly influenced by their distance in the graph. L The influence of node characteristics within the skip neighborhood provides a theoretical guarantee for GCN to capture local interaction information in multi-intelligent scenarios.
[0036] In step S102, the node embedding vector of each agent is input into a preset GCN-Actor network so that the preset GCN-Actor network outputs action distribution parameters for each agent, and the output action distribution parameters of each agent are reparameterized and sampled to obtain the sampled action of each agent.
[0037] Optionally, in some embodiments, the first The sampling action of each agent is: ; in, For the first i Each agent samples actions; GCNActor is the intelligent agent. i Output motion distribution parameters, Let be the mean of the action distribution. The standard deviation of the action distribution; Standard Gaussian noise, Let represent a multivariate standard normal distribution with a mean of zero vector and a covariance of identity matrix.
[0038] GCN-Actor (Graph Convolutional Network - Actor Network) refers to a distributed policy network built on graph convolutional networks, belonging to the Actor component of the Actor-Critic framework. Action distribution parameters refer to the set of key parameters used to quantify and characterize the probability distribution of agent actions. Reparameterized sampling is a technique proposed to address the problem of non-differentiability in sampling operations and to achieve efficient backpropagation of policy gradients.
[0039] Specifically, the pre-defined GCN-Actor network... Perform message passing and embedding computation to extract node embedding vectors. , For the first L The node embedding vector of agent i after the GCN layer is used to output action distribution parameters for each agent. The sampling action for each agent is obtained through reparameterized sampling. The policy loss is: ; in, It refers to the mathematical expectation of the distribution of states, actions, noise, etc. The entropy temperature coefficient For the first k The state-action value function (Q function) of each Critic. To minimize the value of the updated dual-Q network, For this Critic parameter, s 'a' represents the global state, and 'a' represents the joint action.
[0040] In this context, parameter sharing means that all agents share the same set of Actor network parameters. That is, for any intelligent agent All use strategies Parameters do not change Change, solely due to input Different outputs result in different outputs. These are the independent parameters for each agent in the non-shared scheme (for comparison). The purpose of parameter sharing is to decouple the parameter size from the number of agents, ensure invariance to number permutation, and improve cross-agent generalization ability; during training, gradients are jointly calculated and updated for all agent samples. This enables experience sharing.
[0041] In step S103, the node embedding vector of each agent is permuted and symmetric pooled to obtain the graph-level embedding vector of each agent. A joint action vector is obtained based on the sampling action of each agent. The graph-level embedding vector, graph-level global vector and joint action vector of each agent are concatenated to form a fusion feature. The fusion feature is input into the dual-Q network of the preset GCN-Critic network to obtain the first value and the second value. The temporal difference target value is calculated based on the first value and the second value.
[0042] Optionally, in some embodiments, the dual-Q network is: ; in, For state-action value function, For mathematical expectation, For time ( t + k Instant rewards Current moment t state, Current moment t The action performed.
[0043] Optionally, in some embodiments, the timing difference target value is: ; in, The target value for time-series difference is... For instant rewards, As a discount factor, This is a termination indicator variable (1 for termination, 0 otherwise). For Actor parameters, For intelligent agents Local observations I The number of agents.
[0044] In this context, permutational symmetric pooling refers to aggregating the node embedding vectors of multiple agents into a graph-level embedding vector through a symmetric aggregation method, ensuring that the aggregation result is unaffected by the agent's number / order. The joint action vector is a globally integrated vector representing the sampled actions executed by all agents at the same target time, used to quantify the overall action state of the system. The dual-Q network refers to using two structurally identical but parameter-independent Q-networks to implement action selection and action value evaluation, respectively. GCN-Critic (Graph Convolutional Network Critic) is the core component for value evaluation, combining the structure-awareness of graph convolutional networks with the stability design of dual-Q networks to provide global state-action pair value estimates for collaborative agents. The first value and second value refer to the state-action value estimates output by the two independent Q-networks, respectively. The temporal difference target value is the target signal used to supervise the optimization of the Critic network's value estimation.
[0045] Specifically, the pre-defined GCN-Critic network employs a dual-Q structure and a soft-object network. The input consists of three parts: a graph-level embedding vector obtained through GCN, used to encode the dynamic interactions of multiple agents; the original global vector, i.e., the graph-level global vector. Preserves the direct representation of physical constraints; joint action vector The three vectors mentioned above are fused through a concatenation layer and then fed into a dual-Q network. The readout operation uses permutation symmetric pooling methods such as mean or sum, and the final output is the state-action value function. The concatenation layer is a layer that concatenates graph-level embedding vectors, graph-level global vectors, and joint action vectors. Permutation-symmetric readout is a readout operator that performs symmetric aggregation on node / graph embeddings. According to... The time-series difference target value was calculated. .
[0046] Among them, the isomorphic permutation orbital Orbit ( G After performing symmetrical readings and averaging, the variance of the value estimate satisfies: ; in, For the image Replacement orbit, For the sample size, This represents the variance of the single-sample noise.
[0047] Compared to MLP estimation without permutation invariance (variance is...) The variance under this mechanism is lower, which can alleviate the estimation fluctuations caused by subtle changes in agent numbering and topology, and enhance the robustness of value estimation in dynamic multi-agent scenarios.
[0048] In step S104, the preset GCN-Critic network and the preset GCN-Actor network are updated according to the temporal difference target value.
[0049] Furthermore, in some embodiments, updating the preset GCN-Critic network and the preset GCN-Actor network according to the temporal difference target value includes: updating the dual-Q network of the preset GCN-Critic network according to the temporal difference target value to obtain the updated dual-Q network; calculating the policy gradient according to the parameters of the updated dual-Q network, and updating the preset GCN-Actor network according to the policy gradient; and updating the target Q network of the preset GCN-Critic network based on a preset soft update mechanism.
[0050] In reinforcement learning, the policy gradient is the core gradient signal used to optimize the policy network parameters. Essentially, it is the gradient of the policy parameters with respect to the expected cumulative reward, and its direction points towards parameter updates that improve the long-term return of the policy. The soft update mechanism refers to the gradual fusion of current network parameters with the target network parameters to prevent drastic changes in the target value due to fluctuations in the current network, thereby ensuring the stability of the value function learning.
[0051] Specifically, the pre-defined GCN-Critic network update is achieved by minimizing the Temporal Difference Error (TD), updating the current dual-Q network through gradient descent to make the predicted state-action value as close as possible to the target value. In a multi-agent setting, This is a centralized Critic's value estimate of joint state s for joint action a, used to guide the update strategy of the pre-defined GCN-Actor network.
[0052] The default soft update mechanism can be the Polyak soft update mechanism: ; in, For the target Q network, For the current Q network, This is the soft update coefficient.
[0053] Specifically, the target Q-network is used to calculate the temporal difference target value. Its parameters are not directly copied from the current Q-network parameters, but are updated gradually according to the scale. ) represents the soft update coefficient, indicating the update weight of the current network parameters on the target network. In the GCN-Critic network, the soft update mechanism ensures that the target Q-network's value assessment of the joint state-joint action is more stable, providing a reliable target signal for the dual Q-value optimization of the Critic network and the policy gradient calculation of the Actor network, supporting the stable convergence of the multi-agent cooperative policy.
[0054] The update strategy of the GCN-Actor network aims to maximize long-term value while exploring and utilizing entropy regularization balance. The core logic is to leverage the minimum value of the updated dual-Q network (consistent value overestimation) and construct the policy loss based on the entropy of the action distribution. ; in, It refers to the mathematical expectation of the distribution of states, actions, noise, etc. The entropy temperature coefficient For the first k The state-action value function (Q function) of each Critic. To minimize the value of the updated dual-Q network, For this Critic parameter, s 'a' represents the global state, and 'a' represents the joint action.
[0055] Specifically, the Actor parameters are achieved by minimizing this loss through gradient descent. θ The update process employs a paradigm of joint optimization of policy (Actor) and value (Critic), while introducing a maximum entropy regularization term to ensure exploration capability while optimizing the policy and preventing premature policy convergence. A soft update mechanism for the target network alleviates overfitting and drift issues in value estimation, making training more stable. A mini-batch parallel sampling method is used to fully utilize data and improve sample efficiency, while enhancing the model's scalability in large-scale multi-agent scenarios.
[0056] Under the common Lipschitz Continuity and boundedness assumptions, when the mini-batch size is m, the variance of the policy gradient satisfies: ; in, For variance operators, For small batch production, To match the total number of intelligent agents Approximately independent constants.
[0057] Therefore, as the size m of the mini-batch sampling increases, the variance of the policy gradient will gradually decrease, supporting large-scale parallel training, effectively reducing update noise, and improving the stability and convergence reliability of the training process.
[0058] To facilitate further understanding of the graph neural network-enhanced multi-agent SAC optimization method of this application's embodiments, the following is combined with... Figure 2 and Figure 3 Further explanation is needed.
[0059] like Figure 2 As shown, Figure 2 This is a schematic diagram of the GCN-Actor and GCN-Critic network structures provided according to an embodiment of this application. The GCN-Actor network, through a hierarchical message passing mechanism, enables nodes to aggregate information from themselves and their neighbors, thereby capturing local and neighborhood interaction features. It employs a parameter-sharing and identity encoding compatible design, reducing the number of parameters while distinguishing different agents, improving the model's scalability in scenarios with varying numbers of agents, and outputting node-level action distributions, simultaneously supporting decision-making needs in both continuous and discrete action spaces. The GCN-Critic network, with structured centralized input as its core, integrates graph-level embedding vectors generated by GCN, original global vectors, and joint action vectors. It uses a permutation-symmetric readout operator to reduce the interference of agent numbering on value estimation and reduce irrelevant variance. Combined with the design of a dual-Q network and a target network, the dual-Q network suppresses value overestimation, while the target network maintains the stability of value estimation through soft updates, jointly improving the reliability of value assessment.
[0060] Figure 3 This is a schematic diagram of the training data flow of the multi-agent GCN-Actor / Critic framework according to an embodiment of this application. A self-iterative closed loop is formed through data flow. The GCN-Actor generates the actions of each agent according to the current policy, interacts with the environment to generate samples and stores them in the experience replay buffer. Batch sampling of samples drives the GCN-Critic to update, and then optimizes the GCN-Actor based on the updated Critic output. At the same time, the target Q network is kept stable through soft updates. The updated Actor interacts with the environment again to generate new samples to supplement the replay pool. The interaction-sampling-update process is repeated so that the policy gradually converges to the global optimal solution of multi-agent cooperation in the accumulation of data.
[0061] Therefore, by explicitly encoding the topological relationships and neighborhood interaction patterns between agents through graph convolutional networks, the problem of missing structural priors is effectively solved; the graph neural network design based on permutation invariance is insensitive to agent numbering and topological changes, significantly improving topological robustness; the local message passing and parameter sharing mechanism makes the algorithm complexity approximately linear with the number of agents, effectively alleviating the problem of limited scalability; the permutation symmetric readout and double-Q network design significantly reduce the variance of value estimation, and the policy gradient noise decays with batch size, solving the problem of poor training stability; through parameter sharing and graph structured computation, the number of network parameters and computational complexity are greatly reduced, solving the problem of low computational efficiency.
[0062] The GNN-enhanced multi-agent optimization method for agent communication networks proposed in this application obtains an observation graph, obtains node embedding vectors through graph convolutional network message passing, inputs the action distribution parameters of the GCN-Actor network, obtains sampled actions through reparameterized sampling, obtains graph-level embedding vectors through permutational symmetric pooling of node embeddings, concatenates the graph-level embeddings, graph-level global vectors, and joint action vectors to form a fusion feature, inputs it into a dual-Q network to obtain the first and second values, calculates the temporal difference target value, and updates the GCN-Critic and GCN-Actor networks. This solves the problems of missing structural priors, insufficient topological robustness, and low computational efficiency in related technologies, and improves scalability and training stability in large-scale, heterogeneous, and topologically dynamic networks.
[0063] Next, referring to the accompanying drawings, a GNN-enhanced multi-agent optimization device for agent communication networks proposed according to an embodiment of this application is described.
[0064] Figure 4 This is a block diagram of an embodiment of the GNN-enhanced multi-agent optimization method for agent communication networks according to this application.
[0065] like Figure 4 As shown, the GNN-enhanced multi-agent optimization device 10 for intelligent agent communication networks includes: an acquisition module 100, a sampling module 200, an input module 300, and an optimization module 400.
[0066] The acquisition module 100 acquires the observation map of the system to be optimized at the target time, and uses a preset graph convolutional network to perform message passing on the observation map to obtain the node embedding vector of each agent. The observation map includes multiple agents and graph-level global vectors. The sampling module 200 inputs the node embedding vector of each agent into a preset GCN-Actor network, so as to output action distribution parameters for each agent through the preset GCN-Actor network, and performs reparameterized sampling on the output action distribution parameters of each agent to obtain the sampled action of each agent. The input module 300 performs permutation symmetric pooling on the node embedding vector of each agent to obtain the graph-level embedding vector of each agent, and obtains the joint action vector according to the sampled actions of each agent. The graph-level embedding vector, graph-level global vector and joint action vector of each agent are concatenated to form a fusion feature. The fusion feature is input into the dual-Q network of the preset GCN-Critic network to obtain the first value and the second value. The temporal difference target value is calculated based on the first value and the second value.
[0067] The optimization module 400 updates the preset GCN-Critic network and the preset GCN-Actor network according to the temporal difference target value.
[0068] According to one embodiment of this application, the optimization module 400 includes: The pre-defined dual-Q network of the GCN-Critic network is updated based on the temporal difference target value to obtain the updated dual-Q network; The policy gradient is calculated based on the updated parameters of the dual-Q network, and the preset GCN-Actor network is updated based on the policy gradient. The target Q network of the preset GCN-Critic network is updated based on a preset soft update mechanism.
[0069] According to one embodiment of this application, the timing difference target value is: ; in, The target value for time-series difference is... For instant rewards, As a discount factor, To terminate the indicator variable, For Actor parameters, For intelligent agents Local observations I The number of agents.
[0070] According to one embodiment of this application, the dual-Q network is: ; in, For state-action value function, For mathematical expectation, For time ( t + k Instant rewards Current moment t state, Current moment t The action performed.
[0071] According to one embodiment of this application, the first The sampling action of each agent is: ; in, For the first i Each agent samples actions; GCNActor is the intelligent agent. i Output motion distribution parameters, Let be the mean of the action distribution. The standard deviation of the action distribution; Standard Gaussian noise, Let represent a multivariate standard normal distribution with a mean of zero vector and a covariance of identity matrix.
[0072] It should be noted that the foregoing explanation of the embodiment of the multi-agent SAC optimization method based on graph neural network enhancement also applies to the multi-agent SAC optimization device based on graph neural network enhancement in this embodiment, and will not be repeated here.
[0073] The GNN-enhanced multi-agent optimization device for agent communication networks proposed in this application acquires an observation graph, obtains node embedding vectors through graph convolutional network message passing, inputs the action distribution parameters of the GCN-Actor output, reparameterizes and samples the actions, obtains graph-level embedding vectors through permutational symmetric pooling of node embeddings, concatenates the graph-level embeddings, graph-level global vectors, and joint action vectors to form a fusion feature, inputs it into a dual-Q network to obtain the first and second values, calculates the temporal difference target value, and updates the GCN-Critic and GCN-Actor networks. This solves the problems of missing structural priors, insufficient topological robustness, and low computational efficiency in related technologies, and improves scalability and training stability in large-scale, heterogeneous, and topologically dynamic networks.
[0074] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.
[0075] When the processor 502 executes the program, it implements the GNN-enhanced multi-agent optimization method for agent communication networks provided in the above embodiments.
[0076] Furthermore, electronic devices also include: Communication interface 503 is used for communication between memory 501 and processor 502.
[0077] The memory 501 is used to store computer programs that can run on the processor 502.
[0078] The memory 501 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.
[0079] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0080] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.
[0081] Processor 502 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention.
[0082] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described GNN-enhanced multi-agent optimization method for agent communication networks.
[0083] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the embodiments of the GNN-enhanced multi-agent optimization method for intelligent agent communication networks described above.
[0084] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0085] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0086] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.
Claims
1. A GNN-enhanced multi-agent optimization method for an agent-oriented communication network, characterized in that, Includes the following steps: Obtain the observation graph of the system to be optimized at the target time, and use a preset graph convolutional network to perform message passing on the observation graph to obtain the node embedding vector of each agent. The observation graph includes multiple agents and graph-level global vectors. The node embedding vector of each agent is input into a preset GCN-Actor network to output action distribution parameters for each agent through the preset GCN-Actor network, and the sampled action of each agent is obtained by reparameterizing and sampling the output action distribution parameters of each agent. The node embedding vector of each agent is permuted and symmetric pooled to obtain the graph-level embedding vector of each agent. A joint action vector is obtained based on the sampled actions of each agent. The graph-level embedding vector, the graph-level global vector, and the joint action vector of each agent are concatenated to form a fusion feature. The fusion feature is input into the dual-Q network of the preset GCN-Critic network to obtain a first value and a second value. The temporal difference target value is calculated based on the first value and the second value. The preset GCN-Critic network and the preset GCN-Actor network are updated according to the temporal difference target value.
2. The method according to claim 1, characterized in that, The step of updating the preset GCN-Critic network and the preset GCN-Actor network according to the temporal difference target value includes: The preset dual-Q network of the GCN-Critic network is updated according to the temporal difference target value to obtain the updated dual-Q network; The policy gradient is calculated based on the parameters of the updated dual-Q network, and the preset GCN-Actor network is updated based on the policy gradient. The target Q network of the preset GCN-Critic network is updated based on a preset soft update mechanism.
3. The method according to claim 1, characterized in that, The target value for the time-series difference is: ； in, The time-series difference target value is... For instant rewards, As a discount factor, To terminate the indicator variable, For Actor parameters, For intelligent agents Local observations I The number of agents.
4. The method according to claim 2, wherein the dual-Q network is: ； in, For state-action value function, For mathematical expectation, For time ( t + k Instant rewards Current moment t state, Current moment t The action performed.
5. The method according to claim 1, characterized in that, No. The sampling action of each agent is: ； in, For the first i Each agent samples actions; GCNActor is the intelligent agent. i Output motion distribution parameters, Let be the mean of the action distribution. The standard deviation of the action distribution; Standard Gaussian noise, Let represent a multivariate standard normal distribution with a mean of zero vector and a covariance of identity matrix.
6. A GNN-enhanced multi-agent optimization device for agent communication networks, characterized in that, include: The acquisition module acquires the observation map of the system to be optimized at the target time, and uses a preset graph convolutional network to perform message passing on the observation map to obtain the node embedding vector of each agent. The observation map includes multiple agents and graph-level global vectors. The sampling module inputs the node embedding vector of each agent into a preset GCN-Actor network, so as to output action distribution parameters for each agent through the preset GCN-Actor network, and performs reparameterized sampling on the output action distribution parameters of each agent to obtain the sampled action of each agent. The input module performs permutation symmetric pooling on the node embedding vector of each agent to obtain the graph-level embedding vector of each agent, obtains a joint action vector based on the sampled actions of each agent, and concatenates the graph-level embedding vector, the graph-level global vector, and the joint action vector to form a fusion feature. The fusion feature is then input into the dual-Q network of the preset GCN-Critic network to obtain a first value and a second value. The temporal difference target value is calculated based on the first value and the second value. The optimization module updates the preset GCN-Critic network and the preset GCN-Actor network according to the time-series difference target value.
7. The apparatus according to claim 6, wherein the optimization module comprises: The preset dual-Q network of the GCN-Critic network is updated according to the temporal difference target value to obtain the updated dual-Q network; The policy gradient is calculated based on the parameters of the updated dual-Q network, and the preset GCN-Actor network is updated based on the policy gradient. The target Q network of the preset GCN-Critic network is updated based on a preset soft update mechanism.
8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the GNN-enhanced multi-agent optimization method for agent communication networks as described in any one of claims 1-5.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the GNN-enhanced multi-agent optimization method for agent communication networks as described in any one of claims 1-5.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the GNN-enhanced multi-agent optimization method for agent communication networks as described in any one of claims 1-5.