Multi-agent collaborative task state embedding method based on multi-scale hypergraph and multi-dimensional aggregation

Abstract

The invention relates to the field of deep learning, and discloses a multi-agent collaborative task state embedding method based on a multi-scale hypergraph and multi-dimensional aggregation. In order to solve the problems that an existing graph neural network is difficult to capture a high-order interaction relation, poor in dynamic environment adaptability and limited in communication, the method comprises the steps of constructing a state observation graph; calculating an adjacent matrix according to explicit states such as position and speed; generating a latent layer feature adjacency matrix through nonlinear conversion and similarity calculation of a graph convolutional network, and fusing the latent layer feature adjacency matrix with the Hadamard product of the interactive graph; constructing a multi-scale hypergraph (containing S scales) according to the latent layer matrix, and searching a high-density sub-matrix to form hyperedges; building a two-stage information aggregation model: integrating multi-dimensional features and calculating association degree and interaction types in a hyperedge aggregation stage, and updating node features by using a graph attention network GAT in a node aggregation stage; a multi-agent soft behavior-commentator algorithm MASAC is fused, a behavior and reward function is designed, and an MHGNN-MASAC model is formed; the method is applied to cooperative control task decision.

Description

Technical Field

[0001] The patent of this invention belongs to the field of multi-agent reinforcement learning and deep learning, and in particular relates to a multi-agent collaborative task state embedding method based on multi-scale hypergraph and multi-dimensional aggregation. Background Art

[0002] With the rapid development of artificial intelligence (AI), multi-agent systems have been widely used in numerous fields, such as drone swarms, autonomous vehicle fleets, and intelligent transportation systems. In these systems, multiple agents need to collaborate to complete complex tasks, and accurate and efficient state representation is key to achieving collaborative control of these agents.

[0003] In multi-agent systems, the interactions between agents are complex and ever-changing, and the operating environment is often fraught with challenges such as uncertainty and incomplete information. Traditional state embedding methods, mostly based on simple graph structures or adjacency relationships, struggle to effectively capture the complex, high-level interactions between agents. For example, methods based on physical distance or simple fully connected networks are prone to information redundancy or insufficient representation when dealing with complex systems. These methods fail to fully reflect the deep-level collaborative and competitive relationships between agents, thereby limiting the agents' ability to perceive the environment and their decision-making efficiency.

[0004] In recent years, graph neural networks (GNNs) have been introduced to model the interaction relationships between intelligent agents, learning the local interaction patterns of intelligent agents through information propagation between nodes and edges. A search revealed that Chinese patent CN119996045A discloses an APT attack detection and tracing method based on graph neural networks and reinforcement learning. This method applies graph neural networks to reinforcement learning, uses AGNN to generate more challenging samples, then uses graph structure modeling and graph neural networks for feature extraction, and uses Monte Carlo trees combined with reinforcement learning to gradually trace attack events in the system. Chinese patent CN119987301A discloses an intelligent agent-based ultra-thin glass tempering control method based on graph neural networks and reinforcement learning. This method uses graph neural networks for feature extraction and embedding of initial data, and a temporal convolutional network model for information aggregation. Finally, it optimizes control through reinforcement learning to achieve temperature, pressure, and stress control of ultra-thin tempered glass.

[0005] However, the information transfer mechanism of GNNs mainly targets binary relationships between pairs of nodes, and its ability to model high-order interactions between multiple agents is limited. In addition, in dynamic environments, the interactions between agents change over time, and traditional methods have difficulty adapting to such dynamic changes, resulting in insufficient model stability and generalization capabilities.

[0006] In practical applications, multi-agent systems also face challenges with communication limitations, such as communication delays, interruptions, and insufficient bandwidth. These issues prevent agents from timely acquiring information about the states of other agents, impacting the accuracy of their decisions and the overall performance of the system. Therefore, achieving efficient multi-agent state representation and decision optimization in complex environments with limited communication remains a key challenge in current multi-agent system research.

[0007] In summary, existing multi-agent state embedding methods have obvious limitations in dealing with complex interactive relationships, adaptability to dynamic environments, and communication constraints. There is an urgent need for a new method that can effectively overcome these challenges in order to improve the collaborative control capabilities and task execution efficiency of multi-agent systems in complex environments. Summary of the Invention

[0008] The purpose of this invention is to solve the problems of inaccurate state embedding and low collaborative control efficiency in multi-agent systems caused by the difficulty of traditional graph structures in capturing high-order interactive relationships, poor adaptability to dynamic environments, and limited communication. To achieve the above objectives, the present invention adopts the following technical solutions:

[0009] The present invention provides a multi-agent collaborative task state embedding method based on multi-scale hypergraph and multi-dimensional aggregation, comprising the following steps:

[0010] Step 1: Construct a state observation graph based on the state information of each agent in the multi-agent system. The state observation graph is an undirected graph containing a node set V and an edge set E.

[0011] Step 2: Based on the state observation graph, calculate the strength of the association between agents according to the explicit state, and generate the adjacency matrix of the multi-agent system as the interaction graph;

[0012] Step 3: Generate the latent feature adjacency matrix through nonlinear transformation and similarity calculation, and fuse it with the interaction graph to serve as the final adjacency matrix of the multi-agent system, i.e., the final interaction graph;

[0013] Step 4: Based on the latent feature adjacency matrix, search and construct multi-scale hyperedges and hypergraphs;

[0014] Step 5: Build a two-stage multi-dimensional information aggregation model, including the hyperedge information aggregation stage and the node information aggregation stage;

[0015] Step 6: Build the multi-agent reinforcement learning algorithm model MASAC and integrate the multi-scale hypergraph and two-stage multi-dimensional information aggregation into it to form a new model MHGNN-MASAC;

[0016] Step 7: Use the MHGNN-MASAC model to make decisions on the collaborative control tasks of the multi-agent system.

[0017] In the above scheme, step 1 includes:

[0018] A state observation graph is constructed based on the state information of each agent in the multi-agent system. The state observation graph is an undirected graph, which includes a node set V = {v i |i=1,...,N} and the edge set E={e ij |v i , v j ∈V,v i ≠v j};

[0019] V is a node set, representing all agents in the multi-agent system;

[0020] E is an edge set, which represents the interaction relationship between agents. Each edge e ij ∈E represents node v i and node v j The connection relationship between them.

[0021] In the above scheme, step 2 includes:

[0022] According to the state observation graph, the strength of the association between agents is calculated according to the explicit state, and the adjacency matrix of the multi-agent system is generated as the interaction graph, where:

[0023] The explicit state includes the position information, speed information and distance information of the intelligent agent;

[0024] The strength of the association is determined by the Euclidean distance between the agents, which is defined as:

[0025]

[0026] Among them, d ij represents the Euclidean distance between agent i and agent j, (x i ,y i ) and (x j ,y j ) are the two-dimensional coordinates of agent i and agent j respectively;

[0027] According to the Euclidean distance and the agent exploration distance threshold d thread Comparison of the two agents determines whether there is a connection between them: If

[0028] d ij ≤d thread Then there is a connection between agents i and j;

[0029] Adjacency matrix A o is an N×N matrix, where N is the number of agents and element A ij Defined as:

[0030]

[0031] Optionally, the element A ij , which is equal to 1, indicates that there is a connection between agents i and j.

[0032] In the above scheme, step 3 includes:

[0033] Non-linear transformation: A graph convolutional network is used to perform non-linear transformation on the position and velocity information of agents in a multi-agent system to obtain high-dimensional feature information:

[0034]

[0035] Where X=(x1, x2, ..., x n ) is the information matrix composed of the state information of all agents, H is the output feature matrix of GCNs, that is, high-dimensional features, W is the learnable weight matrix, σ is the activation function, is a non-negative symmetric matrix, a normalized adjacency matrix, defined as follows:

[0036]

[0037] in, λ is the self-loop weight coefficient, I N is the N×N identity matrix, is the degree matrix, used for normalization;

[0038] Similarity calculation: Calculate the similarity between agents based on the high-dimensional feature information and generate a latent feature adjacency matrix:

[0039]

[0040] h i and h j are the high-dimensional feature vectors of agent i and agent j respectively;

[0041] Fusion: The latent feature adjacency matrix is ​​combined with the adjacency matrix A of the interaction graph o Through Hadamard product fusion, the final adjacency matrix is ​​obtained:

[0042]

[0043] in To normalize the latent feature adjacency matrix, it is calculated as follows:

[0044]

[0045] ⊙ represents the Hadamard product.

[0046] In the above solution, in step 4, a multi-scale hyperedge and hypergraph are constructed based on the latent feature adjacency matrix, where:

[0047] Multi-scale hypergraph definition: The constructed multi-scale hypergraph is represented as G = {G (0) , G (1) ,...,G (S)}, which includes hypergraphs of S scales. For any hypergraph of scale s, G (s) =(V, E (s) ) has a hyperedge set To express group relationships, M s represents the number of hyperedges of scale s, G (0) =(V, E (0) ) is a specific hypergraph whose edge model is the finest pairwise agent connections.

[0048] Hyperedge construction method: Search for high-density submatrices from the latent feature adjacency matrix to construct hyperedges, and solve the following optimization problem:

[0049]

[0050] Among them, A Ω represents the elements in the submatrix A′ related to the node group Ω, ||A Ω || 1,1 is the L1 norm of the matrix, which represents the sum of the absolute values ​​of all elements. The goal is to find a node group Ω so that the feature similarity between the nodes in the group is maximized, thereby forming a strongly connected hyperedge and obtaining a dimension K. (s) ×K (s) The sub-matrix K (s) is the number constraint of hyperedge nodes of scale s, v i It is a node that must be included;

[0051] Optimization solution:

[0052] When the total number of agents is small, an enumeration algorithm is used to solve the problem;

[0053] When the total number of agents is large, a greedy algorithm is used to approximate the solution: select the starting node v i , iteratively add with v i The new node with the largest relevance weight;

[0054] Hyperedge set: The hyperedge set of each scale s is Where M represents the number of hyperedges of this scale, and each hyperedge Connect a group of highly related agents.

[0055] In the above solution, step 5 builds a two-stage multi-dimensional information aggregation model, which includes the following stages:

[0056]

[0057] in, It is a super edge The aggregation representation of is the correlation, c i，l is one of the preset types of interaction relationships, indicating the probability of the lth type in all interaction relationships; F l (·) represents the lth type of interaction function, which is implemented by the multi-layer perceptron MLP, h j is the eigenvector of node j in the hyperedge;

[0058] Calculation of association and interaction types: embedding z through joint state i calculate:

[0059]

[0060] Where: w j is the weight of node j on the hyperedge, F w It is implemented by a multi-layer perceptron, where g is a vector sampled from the Gumbel (0, 1) distribution and τ is a parameter that controls the smoothness of the distribution.

[0061] Node information aggregation phase: Aggregate information from the node perspective and update node features:

[0062]

[0063] in, Represents node v i The set of neighbor nodes, V k It is a superedge e k The node set in , GAT(·) is a graph attention network used to perform weighted aggregation of information of neighbor nodes and hyperedges, where is the feature vector of node j in the sth layer.

[0064] In the above solution, step 6 includes:

[0065] The MASAC algorithm refers to a multi-agent soft actor-critic algorithm. Based on this algorithm, behavior and reward functions are designed for multi-agent collaborative control tasks. The collaborative control task is a multi-agent leader-follower task in a random control noise environment. The multi-agent cluster system is assumed to include n agents, all of which move on a two-dimensional plane and satisfy the particle dynamics model, which is expressed as a second-order integral model:

[0066]

[0067] Where n is the number of agents, is the agent’s position, is the agent speed, is the agent acceleration, also known as the control input of the system;

[0068] The behavioral design refers to the control quantity u i As the behavior of follower i, the behavior of the follower is expressed as:

[0069]

[0070] The follower's speed information is updated as follows:

[0071]

[0072] The reward function is mainly based on the loss function L sys , defined as:

[0073] r=max(-L sys , r min {}}

[0074] Among them, r min is the minimum reward and punishment value, and satisfies r min The strategy optimization goal is to minimize the system error. When r approaches 0, the system error approaches 0 and the multi-agent system reaches a stable state.

[0075] Loss function L sys It refers to the loss function of the leader-follower collaborative control system and is expressed as:

[0076]

[0077] in, and are the position and velocity of follower i, and are the position and velocity of the leader, φ l () is a bounded convex function with a minimum value of 0, and φ l (0)=0,φ f () is a bounded convex function with a minimum value of 0, and φ f (0)=0, represents the position coefficient of the leader, represents the position coefficient of the follower, represents the leader's speed coefficient, The speed coefficient of the follower is shown in Figure 2. The above four coefficients are obtained from experiments under different environments and scales. They are generally 0.015, 0.2, 1.0, and 0.8 respectively. represents the exploration distance threshold between the leader and the follower, represents the mutual exploration distance threshold between followers, represents the neighboring agents of follower i.

[0078] In the above scheme, step 7 includes:

[0079] Step 7.1: Collect the two-dimensional initial position and velocity information of the agents in the multi-agent system as the initial state information of the agents;

[0080] Step 7.2: The agent is regarded as a node, and the initial state information of the agent is regarded as the node attribute information, and a graph structure is constructed as a state observation graph;

[0081] Step 7.3: Calculate the initial Euclidean distance between nodes based on the node attribute information, and compare the Euclidean distance with the agent exploration distance to obtain the correlation between agents as the initial adjacency matrix;

[0082] Step 7.4: Input the initial interaction graph and the initial state information of the agent into the actor network of the MHGNN-MASAC model. After the GCNs nonlinear transformation under the multi-scale hypergraph structure, the initial high-dimensional features are obtained.

[0083] Step 7.5: Calculate the initial high-dimensional features as similarities to form the initial latent feature adjacency matrix;

[0084] Step 7.6: Fuse the initial latent feature matrix and the initial adjacency matrix through the Hadamard product to update the latent feature adjacency matrix;

[0085] Step 7.7: The high-dimensional features obtained in step 7.4 are integrated and updated through a two-stage multi-dimensional information aggregation model;

[0086] Step 7.8: The integrated overall features are inferred by a multi-layer perceptron to obtain the agent's behavior, i.e., acceleration;

[0087] Step 7.9: The actor network outputs the adjacency matrix of the agent's actions and latent features;

[0088] Step 7.10: Update the agent's speed and position information based on the inferred action.

[0089] Step 7.11: Input the updated agent state information, the latent feature adjacency matrix output by the actor, and the agent behavior actions inferred by the actor into the critic network of the MHGNN-MASAC model to evaluate the actor's inference results;

[0090] Step 7.12: Update the parameters of the actor network and the critic network through soft update technology;

[0091] Step 7.13: Repeat steps 7.1 to 7.12 to train the entire model and obtain the optimal multi-agent reinforcement learning model.

[0092] Step 7.14: Close the training and use the optimal model parameters to test the multi-agent system collaborative control task.

[0093] Because the present invention adopts the above technical solution, it has the following beneficial effects:

[0094] 1. By introducing a multi-scale hypergraph structure, the present invention can effectively capture the complex and multi-level interactive relationships in multi-agent systems. Unlike traditional graph neural networks that can only process binary relationships between pairs of nodes, the hypergraph structure of the present invention connects multiple nodes through hyperedges, thereby achieving modeling of higher-order interactive relationships. In group tasks involving intelligent agents, hyperedges can represent collaborative relationships between multiple intelligent agents, not just relationships between two agents. This ability to model high-order interactive relationships enables the model to more accurately describe complex collaborative patterns between intelligent agents, thereby improving the adaptability of intelligent agents to the environment and their decision-making efficiency.

[0095] Second, the two-stage multidimensional information aggregation method proposed in this paper aggregates information from both the hyperedge and node perspectives, significantly improving the efficiency and accuracy of information transmission. In the first stage, through hyperedge aggregation, the information of all agents in a hyperedge is aggregated from multiple dimensions into a global representation of the hyperedge, improving the interpretability of the agent features. In the second stage, information is aggregated from the node perspective, considering each node's neighbor nodes and hyperedge information, further enhancing the node feature representation. This allows for node updates to be combined with richer information, thereby obtaining more accurate overall features.

[0096] Third, the multi-scale hypergraph structure of the present invention exhibits excellent dynamic adaptability, enabling dynamic adjustment of hyperedge construction to adapt to environmental changes. In dynamic environments, the interactions between agents change over time. The multi-scale hypergraph structure can capture these changes by dynamically adjusting hyperedges, thereby maintaining the adaptability and stability of the model.

[0097] Fourth, this invention significantly improves the efficiency of policy learning and decision-making quality in multi-agent systems through a multi-scale hypergraph structure and a two-stage multidimensional information aggregation method. This model more efficiently captures the complex interactions between agents, accelerating the policy learning process. Furthermore, by optimizing information transmission mechanisms, reducing redundant information, and increasing the effective utilization of environmental features by agents, the quality of decisions is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 Flow chart of the method of the present invention.

[0099] Figure 2 This is the structural diagram of the MHGNN-MASAC model of the present invention. DETAILED DESCRIPTION

[0100] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0101] like Figure 1 As shown in FIG, a multi-agent collaborative task state embedding method based on multi-scale hypergraph and multi-dimensional aggregation includes the following steps:

[0102] A. Constructing a state observation diagram for a multi-agent system:

[0103] The state observation diagram of the multi-agent system is constructed by presenting the elements and their states contained in the current multi-agent system in the form of a diagram.

[0104] Specifically, the elements of the multi-agent system are the homogeneous agents in the system, whose states include speed information and position information. Constructing an observation graph means forming an undirected graph with the agents as points and the connections between agents as edge weights:

[0105] V={v i |i=1,...,N}

[0106] E={e ij |v i , v j ∈V,v i ≠v j}

[0107] Among them, V is the node set, which represents all agents in the multi-agent system, and E is the edge set, which represents the interaction relationship between agents. Each edge e ij ∈E represents node v i and node v j The connection relationship between them.

[0108] B. Calculate the adjacency matrix of the multi-agent system as the interaction graph based on the explicit state:

[0109] Specifically, the explicit state includes position information, speed information, and distance information. The strength of the association refers to the Euclidean distance between agents, and whether there is a connection is determined based on the distance. The Euclidean distance is defined as follows:

[0110]

[0111] Where x and y are the two-dimensional coordinate information of the agents, and d is the Euclidean distance between agents 1 and 2. Judging connections based on distance means comparing the Euclidean distance between agents with the agent exploration distance threshold. If d ij ≤d thread , then it means that there is a connection between two agents i and j, where d ij represents the Euclidean distance between agents i and j, d thread Represents the exploration distance threshold of the agent.

[0112] Specifically, the adjacency matrix refers to an N×N matrix A o , where N is the number of agents in the multi-agent system, and its element A ij The definition is as follows:

[0113]

[0114] If element A ij If it is equal to 1, it means that there is a connection between agents i and j.

[0115] C. Calculate the latent feature adjacency matrix and fuse it with the interaction graph:

[0116] Specifically, it includes two steps: nonlinear transformation and similarity calculation. The nonlinear transformation uses graph convolutional networks (GCNs) to perform nonlinear transformation on the position and velocity information of the agents in the multi-agent system to obtain high-dimensional feature information:

[0117]

[0118] Where X=(x1, x2, ..., x n ) is the information matrix composed of the state information of all agents, H is the output feature matrix of GCNs, that is, high-dimensional features, is a non-negative symmetric matrix, a normalized adjacency matrix, defined as follows:

[0119]

[0120] in, The adjacency matrix introduces self-loops; It is the degree matrix used for normalization. Using the degree matrix to differentiate its own weight is equivalent to normalizing the rows and columns separately.

[0121] The similarity calculation is defined as follows:

[0122]

[0123] Among them, A ijrepresents the similarity between agent i and agent j, that is, the weight in the latent feature adjacency matrix, h i and h j are the high-dimensional feature vectors of agent i and agent j, respectively. The larger the weight value, the stronger the task correlation between the two agents, and vice versa. The latent feature adjacency matrix and the adjacency matrix are then fused into one adjacency matrix through the Hadamard product:

[0124] in To normalize the latent feature adjacency matrix, it is calculated as follows:

[0125]

[0126] D. Constructing hyperedges and hypergraphs at multiple scales:

[0127] Specifically, the multi-scale hypergraph is defined as:

[0128] G={G (0) , G (1) ,...,G (S)}

[0129] Among them, there are S scale hypergraphs. For any scale s hypergraph, G(s) = (V, E(s)) has a hyperedge set E(s) = To express group relationships, there are M s hyperedges, each of which connects multiple agents to represent a common relationship.

[0130] Among them, a larger s indicates a larger size of the intelligent group. Among them, G (0) =(V, E (0) ) is a specific hypergraph whose edge model is the finest pairwise agent connections. Each G (s) The topological structure of To express, if the i-th node is included in the j-th hyperedge, then otherwise

[0131] The construction of multi-scale hyperedges based on the latent feature adjacency matrix refers to finding a high-density submatrix from the latent feature adjacency matrix. The goal is to find a group of agents with strong correlations between them, thus forming a hyperedge. Construct a hyperedge set E for the sth scale (s) , which contains groups of multiple agents, and searches for these groups by optimizing high-density submatrices:

[0132]

[0133] Among them, AΩ represents the elements in the submatrix A′ related to the node group Ω, ||A Ω || 1,1 is the L1 norm of the matrix, which represents the sum of the absolute values ​​of all elements. The goal is to find a node group Ω such that the feature similarity between nodes in the group is maximized, thereby forming a strongly connected hyperedge and obtaining a dimension K. (s) ×K (s) submatrix of .

[0134] The first constraint of this optimization problem is to limit the number of nodes per hyperedge to K (s) , that is, each hyperedge contains at most K (s) The second constraint is to ensure that each hyperedge contains at least the i-th node, ensuring that each node participates in at least one hyperedge.

[0135] The method to solve this optimization problem is: when the total number of agents is small, a simpler enumeration algorithm is used to find the optimal solution; when the multi-agent system is large, a greedy algorithm is used to approximate v first. i , and then add v in each move i The new node with the largest correlation weight value is solved to obtain the hyperedge set E at each scale (s) , where each hyperedge consists of a group of highly related agents. At the s-th level, the hyperedge is constructed as follows:

[0136]

[0137] Where M represents the number of hyperedges at the sth level. Each hyperedge Connecting multiple agents indicates that there is a strong task relevance and interaction strength between them. These hyperedges will form a hypergraph of the sth level.

[0138] E. Build a two-stage multi-dimensional information aggregation model:

[0139] Specifically, it includes two stages: the hyperedge information aggregation stage and the node information aggregation stage.

[0140] The hyperedge information aggregation phase aggregates information from the perspective of the hyperedge. By aggregating the node information within each hyperedge, a global representation of the hyperedge is obtained. This hyperedge representation is a fusion of the feature information of all nodes within the hyperedge, reflecting the task collaboration or information exchange relationship represented by the hyperedge.

[0141] This stage adopts a multi-dimensional interaction embedding model. Through the mechanism of autonomous learning, the interaction relationships between agents are divided into different categories, and different interaction functions and interaction strengths are set for each category of interaction relationship. For the i-th hyperedge in the hypergraph with scale s, The interaction embedding of is defined as:

[0142]

[0143] in, It is a super edge The aggregation representation of is the correlation, c i，l is one of the preset interaction relationship types, indicating the probability of the lth type in all interaction relationships; for each interaction relationship type, a learnable interaction function F is designed l (·), which is implemented by a multi-layer perceptron (MLP).

[0144] The degree of association and the type of interaction can be obtained through a joint state embedding, which is a latent state that reflects the overall information of the agents in the group. For a hypergraph of any scale s, the hyperedge The joint embedding of is the weighted sum of the embeddings of the agent states associated with it:

[0145]

[0146] Among them, w j is the weight, which reflects the proportion of the j-th node in the i-th hyperedge. The calculation formula is:

[0147]

[0148] Among them, F w (·) is implemented by an MLP. Then the inferred correlation is obtained and interaction type c i The method is:

[0149]

[0150] c i =softmax(F c (z i )+g) / τ

[0151] Among them, the softmax function is used to constrain the strength value; g is a vector sampled from the Gumbel (0, 1) distribution, and τ is used to control the smoothness of the distribution; F c (·) Modeled by MLP.

[0152] The node information aggregation stage aggregates information from the perspective of the node, and updates its own features according to the node's neighbor nodes and the involved hyperedges. i , if with hyperedge e k Associated, node v iThe feature representation of can be updated as follows:

[0153]

[0154] in, Represents node v i The set of neighbor nodes, V k It is a superedge e k The node set in , GAT(·) is a graph attention network used to perform weighted aggregation of information from neighboring nodes and hyperedges.

[0155] F. Integrating multi-scale hypergraph and two-stage multi-dimensional information aggregation into MASAC to form MHGNN-MASAC:

[0156] The MASAC algorithm refers to the multi-agent soft actor-critic algorithm, based on which behavior and reward functions are designed for multi-agent collaborative control tasks.

[0157] Specifically, the collaborative control task refers to the random control of a multi-agent leader-follower task in a noisy environment. The multi-agent cluster system is assumed to include n agents, all of which move on a two-dimensional plane and satisfy the particle dynamics model, which is expressed as a second-order integral model:

[0158]

[0159] Where n is the number of agents, is the agent’s position, is the agent speed, is the agent acceleration, also known as the control input of the system.

[0160] The behavioral design refers to the control quantity u i As the behavior of follower i, the behavior of the follower is expressed as:

[0161]

[0162] The follower's speed information is updated as follows:

[0163]

[0164] The reward function is mainly based on the loss function L sys , defined as:

[0165] r=max{-L sys ,r min {}}

[0166] Among them, r min is the minimum reward and punishment value, and satisfies r minThe strategy optimization goal is to minimize the system error. When r approaches 0, the system error approaches 0 and the multi-agent system reaches a stable state.

[0167] Loss function L sys It refers to the loss function of the leader-follower collaborative control system and is expressed as:

[0168]

[0169] in, and are the position and velocity of follower i, and are the position and velocity of the leader respectively. l () is a bounded convex function with a minimum value of 0, and φ l (0)=0,φ f () is a bounded convex function with a minimum value of 0, and φ f (0) = 0. c are weight coefficients.

[0170] System error E r Expressed as:

[0171]

[0172] Where e(t) is the system control error of the multi-agent system at iteration time t, which is defined as:

[0173] e(t)=e lf (t)+e ff (t)

[0174] Among them, e lf (t) is the speed consistency control error at iteration time t designed based on the relative position and relative speed between the leader and the follower:

[0175]

[0176] e ff (t) is the collision avoidance control error designed based on the collision situation caused by the position and velocity between followers:

[0177]

[0178] The integration of multi-scale hypergraph and two-stage multi-dimensional information aggregation into MASAC refers to replacing the feature extraction and state embedding module GCNs in MASAC with a multi-scale hypergraph structure, and using a two-stage multi-dimensional information aggregation model to integrate and update graph features, and fusing them into a new model MHGNN-MASAC.

[0179] G. Use the MHGNN-MASAC model to make decisions on the collaborative control tasks of the multi-agent system:

[0180] Specifically, the steps include:

[0181] (1) Collecting the two-dimensional initial position and velocity information of the agents in the multi-agent system as the initial state information of the agents;

[0182] (2) The agent is regarded as a node, and the initial state information of the agent is the node attribute information, forming a graph structure as a state observation graph;

[0183] (3) Calculate the initial Euclidean distance between nodes based on the node attribute information, and compare the Euclidean distance with the agent exploration distance to obtain the correlation between agents as the initial adjacency matrix;

[0184] (4) The initial interaction graph and the initial state information of the agent are input into the actor network of the MHGNN-MASAC model, and the initial high-dimensional features are obtained after nonlinear transformation of GCNs under the multi-scale hypergraph structure;

[0185] (5) Calculate the initial high-dimensional features as similarities to form the initial latent feature adjacency matrix;

[0186] (6) Fusing the initial latent feature matrix and the initial adjacency matrix through the Hadamard product to update the latent feature adjacency matrix;

[0187] (7) The high-dimensional features obtained in (4) are integrated and updated through a two-stage multi-dimensional information aggregation model;

[0188] (8) The integrated overall features are inferred by a multi-layer perceptron to obtain the agent’s behavior, i.e., acceleration;

[0189] (9) The actor network outputs the adjacency matrix of the agent’s behavior and latent features;

[0190] (10) Update the speed and position information of the intelligent agent based on the inferred behavior;

[0191] (11) The updated agent state information, the latent feature adjacency matrix output by the actor, and the agent behavior actions inferred by the actor are input into the critic network of the MHGNN-MASAC model to evaluate the inference results of the actor;

[0192] (12) Update the parameters of the actor network and critic network through soft update technology;

[0193] (13) Repeat operations (1) to (12) to train the entire model and obtain the optimal multi-agent reinforcement learning model;

[0194] (14) Close the training and use the optimal model parameters to test the multi-agent system collaborative control task.

[0195] The use of the optimal model parameters to test the collaborative control task of the multi-agent system means that the network is no longer trained and updated, and the optimal MHGNN-MASAC model is always used to reason about the state information of the multi-agent, make reasonable decisions, and control the behavior of the system.

[0196] This invention combines deep learning technology and reinforcement learning technology, and adopts hypergraph neural network and multi-dimensional feature aggregation method to realize efficient interaction relationship modeling and state embedding of multi-agent system. Compared with current technology, it captures interaction relationships from a higher-dimensional and higher-order perspective, and aggregates features from more dimensions, which improves the ability to capture complex interaction relationships between agents and the state embedding ability, thereby greatly improving the collaborative control capability of multi-agent system.

[0197] In order to facilitate those skilled in the art to better understand the technical concept of the present invention and the contribution of the present invention compared with the prior art, the relationship between the technical problem solved by the present invention, the technical solution, and the technical effects is further explained:

[0198] 1. The present invention solves the technical problem that traditional graph neural networks (GNNs) can only model binary interaction relationships and are difficult to capture high-order collaboration patterns of multiple agents through the technical means of constructing multi-scale hyperedges and hypergraphs based on the latent feature adjacency matrix in step 4. Traditional methods rely on edge connections between node pairs and cannot effectively represent group-level interactions (such as collaborative tasks of multiple agents), resulting in state embedding ignoring complex dependencies. The hypergraph structure of the present invention connects multiple nodes (such as optimization problems) through hyperedges. Solving high-density submatrices) allows for modeling collaborative relationships among groups of agents at a higher level. For example, in drone swarming tasks, a single hyperedge can represent the formation collaboration of multiple drones, rather than a pairwise distance relationship. This approach dynamically adapts to varying task granularities through multi-scale design (scale s controls the size of the swarm), significantly improving the ability to capture high-order interactions. This allows state embeddings to more comprehensively reflect the essence of system collaboration, ultimately enhancing the accuracy of collaborative decision-making.

[0199] 2. The present invention solves the technical problems of redundant information transmission and insufficient feature representation in dynamic environments by building a two-stage multidimensional information aggregation model in step 5. Traditional single-stage aggregation is prone to losing multidimensional features and has poor interpretability. The two-stage aggregation of the present invention firstly performs the hyperedge information aggregation stage (Formula ) Integrate the multi-dimensional features of nodes within the hyperedge, through the correlation and interaction type c i，1 Distinguish the intensity and type of collaboration (such as competition or cooperation), and approximate the real interaction relationship model through a trainable interaction type function; secondly, in the node information aggregation stage (Formula ) fuses neighboring node and hyperedge information, performing weighted aggregation using a graph attention network (GAT). This approach decomposes the information flow from both the hyperedge and node perspectives, reducing redundancy while retaining task-relevant features. For example, in an autonomous driving convoy scenario, hyperedge aggregation captures the overall trajectory pattern of the convoy, while node aggregation refines individual vehicle decisions, thereby improving feature interpretability and environmental adaptability, ultimately optimizing collaborative control efficiency.

[0200] 3. The present invention solves the technical problems of state embedding distortion and slow strategy learning in communication-constrained environments by fusing the latent feature adjacency matrix with the reinforcement learning model (MHGNN-MASAC) in steps 3 and 6. Explicit states (such as positions) in dynamic environments are susceptible to noise interference, and traditional reinforcement learning models are difficult to utilize deep features. In step 3, the present invention uses nonlinear transformation (GCNs generates high-dimensional features H) and similarity calculation to solve the technical problems of state embedding distortion and slow strategy learning in communication-constrained environments. Extract latent features and then interact with the explicit graph (adjacency matrix A o ,) Hadamard product fusion to enhance robustness; In step 6, the multi-scale hypergraph and aggregation model are integrated into the multi-agent soft actor-critic algorithm (MASAC) to design a task-driven reward function (such as r = max{-L sys , r min This approach combines deep learning with reinforcement learning, enabling state embedding to dynamically adapt to environmental changes (such as noise interference) and guiding policy optimization through a reward function to minimize system error. For example, in leader-follower tasks, the model quickly converges to a stable state, significantly improving the real-time and reliability of collaborative control.

[0201] In summary, the present invention systematically solves the bottleneck of state embedding in multi-agent collaborative tasks through multi-scale hypergraph modeling of high-order interaction, multi-dimensional information aggregation to optimize feature transfer, and deep learning-reinforcement learning fusion to enhance dynamic adaptability, achieving a breakthrough improvement in collaborative control capabilities.

Claims

1. A multi-agent collaborative task state embedding method based on multi-scale hypergraph and multi-dimensional aggregation, characterized by: The following steps are involved: Step 1: Construct a state observation graph based on the state information of each agent in the multi-agent system. The state observation graph is an undirected graph containing a node set V and an edge set E. Step 2: Based on the state observation graph, calculate the strength of the association between agents according to the explicit state, and generate the adjacency matrix of the multi-agent system as the interaction graph; Step 3: Generate the latent feature adjacency matrix through nonlinear transformation and similarity calculation, and fuse it with the interaction graph to serve as the final adjacency matrix of the multi-agent system, i.e., the final interaction graph; Step 4: Based on the latent feature adjacency matrix, search and construct multi-scale hyperedges and hypergraphs; Step 5: Build a two-stage multi-dimensional information aggregation model, including the hyperedge information aggregation stage and the node information aggregation stage; Step 6: Build the multi-agent reinforcement learning algorithm model MASAC and integrate the multi-scale hypergraph and two-stage multi-dimensional information aggregation into it to form a new model MHGNN-MASAC; Step 7: Use the MHGNN-MASAC model to make decisions on the collaborative control tasks of the multi-agent system.

2. The method according to claim 1, characterized in that Step 1 includes: A state observation graph is constructed based on the state information of each agent in the multi-agent system. The state observation graph is an undirected graph, which includes a node set V = {v i |i=1,...,N} and edge set E=(e ij |v i , v j ∈V,v i ≠v j }; V is a node set, representing all agents in the multi-agent system; E is an edge set, which represents the interaction relationship between agents. Each edge e ij ∈E represents node v i and node v j The connection relationship between them.

3. The method according to claim 1, characterized in that Step 2 includes: According to the state observation graph, the strength of the association between agents is calculated according to the explicit state, and the adjacency matrix of the multi-agent system is generated as the interaction graph, where: The explicit state includes the position information, speed information and distance information of the intelligent agent; The strength of the association is determined by the Euclidean distance between the agents, which is defined as: Among them, d ij represents the Euclidean distance between agent i and agent j, (x i ,y i ) and (x j ,y j ) are the two-dimensional coordinates of agent i and agent j respectively; According to the Euclidean distance and the agent exploration distance threshold d thread Comparison of the agents to determine whether there is a connection: If d ij ≤d thread Then there is a connection between agents i and j; Adjacency matrix A o is an N×N matrix, where N is the number of agents and element A ij Defined as: Optionally, the element A ij , which is equal to 1, indicates that there is a connection between agents i and j.

4. The method according to claim 1, wherein Step 3 includes: Non-linear transformation: A graph convolutional network is used to perform non-linear transformation on the position and velocity information of agents in a multi-agent system to obtain high-dimensional feature information: Where X=(x1, x2, ..., x n ) is the information matrix composed of the state information of all agents, H is the output feature matrix of GCNs, that is, high-dimensional features, W is the learnable weight matrix, σ is the activation function, is a non-negative symmetric matrix, a normalized adjacency matrix, and is defined as follows: in, λ is the self-loop weight coefficient, I N is the N×N identity matrix, is the degree matrix, used for normalization; Similarity calculation: Calculate the similarity between agents based on the high-dimensional feature information and generate a latent feature adjacency matrix: h i and h j are the high-dimensional feature vectors of agent i and agent j respectively; Fusion: The latent feature adjacency matrix is ​​combined with the adjacency matrix A of the interaction graph o Through Hadamard product fusion, the final adjacency matrix is ​​obtained: in To normalize the latent feature adjacency matrix, it is calculated as follows: ⊙ represents the Hadamard product.

5. The method according to claim 1, wherein In step 4, a multi-scale hyperedge and hypergraph are constructed based on the latent feature adjacency matrix, where: Multi-scale hypergraph definition: The constructed multi-scale hypergraph is represented as G = {G (0) , G (1) ,...,G (S) }, which includes hypergraphs of S scales. For any hypergraph of scale s, G (s) =(V, E (s) ) has a hyperedge set To express group relationships, M s represents the number of hyperedges of scale s, G (0) =(V, E (0) ) is a specific hypergraph whose edge model is the finest pairwise agent connections. Hyperedge construction method: Search for high-density submatrices from the latent feature adjacency matrix to construct hyperedges, and solve the following optimization problem: Among them, A Ω represents the elements in the submatrix A′ related to the node group Ω, ||A Ω || 1,1 is the L1 norm of the matrix, which represents the sum of the absolute values ​​of all elements. The goal is to find a node group Ω so that the feature similarity between the nodes in the group is maximized, thereby forming a strongly connected hyperedge and obtaining a dimension K. (s) ×K (s) The sub-matrix K (s) is the number constraint of hyperedge nodes of scale s, v i It is a node that must be included; Optimization solution: When the total number of agents is small, an enumeration algorithm is used to solve the problem; When the total number of agents is large, a greedy algorithm is used to approximate the solution: select the starting node v i , iteratively add with v i The new node with the largest relevance weight; Hyperedge set: The hyperedge set of each scale s is Where M represents the number of hyperedges of this scale, and each hyperedge Connect a group of highly related agents.

6. The method according to claim 1, characterized in that Step 5 builds a two-stage multi-dimensional information aggregation model, which includes the following stages: in, It is a super edge The aggregation representation of is the correlation, c i，l is one of the preset types of interaction relationships, indicating the probability of the lth type in all interaction relationships; F l (·) represents the lth type of interaction function, which is implemented by the multi-layer perceptron MLP, h j is the eigenvector of node j in the hyperedge; Calculation of association and interaction types: embedding z through joint state i calculate: c i =softmax(F c (z i )+g) / τ Where: w j is the weight of node j on the hyperedge, F w It is implemented by a multi-layer perceptron, where g is a vector sampled from the Gumbel (0, 1) distribution and τ is a parameter that controls the smoothness of the distribution. Node information aggregation phase: Aggregate information from the node perspective and update node features: in, Represents node v i The set of neighbor nodes, V k It is a superedge e k The node set in , GAT(·) is a graph attention network used to perform weighted aggregation of information of neighbor nodes and hyperedges, where is the feature vector of node j in the sth layer.

7. The method according to claim 1, characterized in that The step 6 comprises: The MASAC algorithm refers to a multi-agent soft actor-critic algorithm. Based on this algorithm, behavior and reward functions are designed for multi-agent collaborative control tasks. The collaborative control task is a multi-agent leader-follower task in a random control noise environment. The multi-agent cluster system is assumed to include n agents, all of which move on a two-dimensional plane and satisfy the particle dynamics model, which is expressed as a second-order integral model: Where n is the number of agents, is the agent’s position, is the agent speed, is the agent acceleration, also known as the control input of the system; The behavioral design refers to the control quantity u i As the behavior of follower i, the behavior of the follower is expressed as: The follower's speed information is updated as follows: The reward function is mainly based on the loss function L sys , defined as: r=max{-L sys ,r min {}} Among them, r min is the minimum reward and punishment value, and satisfies r min The strategy optimization goal is to minimize the system error. When r approaches 0, the system error approaches 0 and the multi-agent system reaches a stable state. Loss function L sys It refers to the loss function of the leader-follower collaborative control system and is expressed as: in, and are the position and velocity of follower i, and are the position and velocity of the leader, φ l () is a bounded convex function with a minimum value of 0, and φ l (0)=0,φ f () is a bounded convex function with a minimum value of 0, and φ f (0)=0, represents the position coefficient of the leader, represents the position coefficient of the follower, represents the leader's speed coefficient, The speed coefficient of the follower is shown in Figure 2. The above four coefficients are obtained from experiments under different environments and scales. They are generally 0.015, 0.2, 1.0, and 0.8 respectively. represents the exploration distance threshold between the leader and the follower, represents the mutual exploration distance threshold between followers, represents the neighboring agents of follower i.

8. The method according to claim 1, characterized in that The step 7 comprises: Step 7.1: Collect the two-dimensional initial position and velocity information of the agents in the multi-agent system as the initial state information of the agents; Step 7.2: The agent is regarded as a node, and the initial state information of the agent is regarded as the node attribute information, and a graph structure is constructed as a state observation graph; Step 7.3: Calculate the initial Euclidean distance between nodes based on the node attribute information, and compare the Euclidean distance with the agent exploration distance to obtain the correlation between agents as the initial adjacency matrix; Step 7.4: Input the initial interaction graph and the initial state information of the agent into the actor network of the MHGNN-MASAC model. After the GCNs nonlinear transformation under the multi-scale hypergraph structure, the initial high-dimensional features are obtained. Step 7.5: Calculate the initial high-dimensional features as similarities to form the initial latent feature adjacency matrix; Step 7.6: Fuse the initial latent feature matrix and the initial adjacency matrix through the Hadamard product to update the latent feature adjacency matrix; Step 7.7: The high-dimensional features obtained in step 7.4 are integrated and updated through a two-stage multi-dimensional information aggregation model; Step 7.8: The integrated overall features are inferred by a multi-layer perceptron to obtain the agent's behavior, i.e., acceleration; Step 7.9: The actor network outputs the adjacency matrix of the agent's actions and latent features; Step 7.10: Update the agent's speed and position information based on the inferred action. Step 7.11: Input the updated agent state information, the latent feature adjacency matrix output by the actor, and the agent behavior actions inferred by the actor into the critic network of the MHGNN-MASAC model to evaluate the actor's inference results; Step 7.12: Update the parameters of the actor network and the critic network through soft update technology; Step 7.13: Repeat steps 7.1 to 7.12 to train the entire model and obtain the optimal multi-agent reinforcement learning model. Step 7.14: Close the training and use the optimal model parameters to test the multi-agent system collaborative control task.

Images