Multi-agent combat state characterization and collaborative decision-making method based on structural information

Abstract

The invention discloses a multi-agent combat state characterization and collaborative decision-making method based on structural information, and the method comprises the steps: S10, collecting data, and carrying out the preprocessing of the data; s20, performing state action representation learning on the data: based on the preprocessed interaction data, learning state action embedding dynamically related to a combat environment by adopting a structural mutual information guided variational auto-encoder framework; s30, performing single agent exploration: constructing a hierarchical structure of a state action space based on value estimation, and calculating a structure entropy as an internal reward; and S40, carrying out multi-agent cooperation: constructing an interaction graph based on dynamic prediction of a battlefield task situation, and fusing two types of internal rewards of prediction and novelty to form a cooperative combat decision. According to the method, a unified framework is constructed through a structural information theory, and three core problems of'novelty-predictability 'balance, single / multi-agent scene adaptation and low sample efficiency in multi-agent combat state representation are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-agent combat and intelligent decision-making, and particularly relates to a multi-agent combat state representation and cooperative decision-making method based on structural information. BACKGROUND

[0002] The existing multi-agent reinforcement learning method often faces two types of bottlenecks in a complex combat / countermeasure scene: firstly, the state representation lacks explicit modeling of interaction structure and causal relationship (such as formation cooperation, command and control chain, perception-disturbance-attack link, etc.), resulting in insufficient description of cooperative / countermeasure mode and non-stationary strategy distribution; secondly, the exploration mechanism mainly relies on external rewards or heuristic noise, which is difficult to maintain stable and efficient learning in a sparse reward and multi-objective, strong constraint (safety risk, limited communication, energy budget, rule constraint) environment.

[0003] In view of the above problems, the academia and industry begin to introduce structural information theory and intrinsic motivation mechanism, and drive the agent to form a combat state representation with discriminability and stability by means of structural mutual information, predictability and novelty. However, the existing schemes mainly focus on single agent or static relationship, and lack consideration of dynamic evolution of multi-agent interaction graph, strategy dependence and uncertainty propagation, which is difficult to simultaneously consider sample efficiency, cross-task generalization ability and explainability, and is also difficult to meet the real-time cooperation and robust decision-making requirements of complex battlefield. SUMMARY

[0004] In order to solve the above problems, the application proposes a multi-agent combat state representation and cooperative decision-making method based on structural information, taking structural information theory as a unified framework, and giving a systematic engineering solution around the three core problems of "novelty-predictability" balance in multi-agent combat state representation, single / multi-agent scene adaptation and low sample efficiency.

[0005] To achieve the above purpose, the technical scheme adopted by the application is: a multi-agent combat state representation and cooperative decision-making method based on structural information, comprising the following steps:

[0006] S10, collecting data and pre-processing the data;

[0007] S20, state-action representation learning of the data: based on the pre-processed interaction data, a structural mutual information guided variational autoencoder framework is used to learn state-action embedding related to the dynamic of the combat environment;

[0008] S30, single-agent exploration: based on the value estimation, a hierarchical structure of the state-action space is constructed, and the structural entropy is calculated as an intrinsic reward;

[0009] S40, multi-agent collaboration: based on the dynamic prediction of battlefield task situation, an interaction graph is constructed, and the internal rewards of prediction and novelty are fused to form collaborative combat decision.

[0010] Further, the data is preprocessed, including:

[0011] Anomaly value processing is performed by linear interpolation correction;

[0012] The unified sparse reward magnitude is completed to calibrate the reward signal; for single-agent tasks, the target achievement reward is normalized to the interval [10, 20]; and for multi-agent team rewards, the number of agents is weighted and then normalized;

[0013] Based on the global clock, the communication perception delay sequence is completed by copying the previous valid data to strictly align all agent time steps.

[0014] Further, the data is preprocessed, including:

[0015] S201, state coding: using a convolutional neural network or a multilayer perceptron as a state encoder The preprocessed observation data O t is mapped to a compact state representation At the same time, the next time observation O t+1 is encoded to obtain the next time compact state representation

[0016] S202, joint embedding construction: construct a joint embedding Z t of the state S t and the action A t ;

[0017] S203, structure mutual information optimization: first, construct a bipartite graph G zs =(Z,S,E zs ) of the joint embedding Z t and the current state S t , depicting the joint distribution of Z t and S t , E zs is the edge set of the bipartite graph G zs ; the vertex set is the sample Z t of Z n and the sample S t of s n ; the edge weight e zs (z i ,s j ) = p(z i ,sj ), i.e., the joint probability of Z t t satisfies∑ i,j p(z i ,s j ) = 1; for Z t and the next state S t+1 , construct a bipartite graph G zs' = (Z, S', E zs' ), E zs' is the edge set of the bipartite graph G zs' , and the edge weight is p(z i ,s′ j ), i.e., the joint probability of Z t and S t+1 ;

[0018] Secondly, based on the two-layer approximate binary tree, the bipartite graph G zs and the bipartite graph G zs' are calculated respectively to obtain the structured mutual information I SI (Z t ; S t ) and I SI (Z t ; S t+1 ); the structured mutual information I SI (Z t ; S t ) measures the structural redundancy of Z t and the current state S t ; the structured mutual information I SI (Z t ; S t+1 ) measures the structural prediction ability of Z t and the future state S t+1 ;

[0019] Finally, the learning of the dynamic relevant embedding is realized by optimizing the loss function L smi ;

[0020] By minimizing L smi , the goals of reducing I SI (Z t ; S t ) and improving I SI (Z t ; S t+1 ) are realized at the same time, so that Z t eliminates the redundant information related to the current state while retaining the dynamic information critical for predicting the future state, and finally obtains the dynamic relevant embedding Z t with compactness and dynamic relevance, which provides high-quality basic representation for the subsequent exploration and decision-making of the agent and the multi-agent cooperation; ​

[0021] The structured mutual information is computed by a two-layered approximate binary tree, and the loss function L is optimized smi : minimizing Z t The structured mutual information between Z t and S t , maximizing Z t The structured mutual information between Z t+1 and S t+1 , finally obtaining the embedding Z t rich in dynamic laws.

[0022] Further, the training objective of the state encoder is achieved by minimizing the following reconstruction loss:

[0023]

[0024] where, denotes the expectation under the t-th and t+1-th observation, denotes the state decoder.

[0025] Further, when constructing the joint latent embedding Z t that fuses the state S t and the action A t , a variational autoencoder framework is adopted, and the encoder maps the state-action pair to the posterior distribution The embedding is regularized by aligning the posterior distribution with the prior distribution p(Z t ), and this process is achieved by minimizing the KL divergence:

[0026]

[0027] where, denotes the expectation under the joint embedding Z t , D KL denotes the KL divergence, denotes the posterior distribution of the joint embedding encoder, p(Z t ) denotes the prior distribution, and θ z denotes the parameters of the joint embedding encoder;

[0028] Finally, the joint embedding Z t is sampled from the posterior distribution by the reparameterization trick.

[0029] Further, the expression of the loss function L smi is:

[0030] L smi = L up + L z|s + L s|z ;

[0031] where, Lup By approximating Z t marginal distribution q m (Z t ), control Z t With S t Shannon mutual information upper bound, indirectly minimizing I SI (Z t S t );L z|s Through the conditional probability model q z|s Quantization Z t For S t The conditional entropy is used to further filter redundant associations; L s|z By predicting model q s|z Maximize Z t For S t+1 The predictive likelihood indirectly increases I SI (Z t S r+1 ).

[0032] Furthermore, single-agent exploration includes the following steps:

[0033] S301, Hierarchical community division: (1) Constructing a complete state-action graph G sa To characterize the basic structure of state-action relationships; the complete state-action graph G sa Vertex set V sa It consists of all the unique state-action pairs in the environment. Edge weights are used to quantify the value difference between state-action pairs, and are defined based on the estimation results of the value function Q(s,a):

[0034]

[0035] Where i and j are two vertices. For the long-term value estimate of vertex i, ||·||2 denotes the L2 norm; w ij The smaller the weight, the more similar the values ​​of the two state-action pairs;

[0036] (2) The structural framework of the K-layer coding tree T: from the root node to the deep sub-community; root node λ: corresponds to the full set of vertices V λ =V sa , representing the undivided complete state-action space; intermediate layer nodes: the node α in the h-th layer corresponds to the sub-community. Sub-communities are mutually exclusive and completely cover the parent community; deep nodes, layer K: correspond to the smallest sub-community, containing state-action pairs with highly similar value estimates;

[0037] (3) Optimal coding tree generation based on minimizing K-dimensional structural entropy: Optimal coding tree The core is to calculate the structural entropy of each node in the coding tree and optimize the total entropy by minimizing the K-dimensional structural entropy.

[0038] S302, distribution graph construction and structural entropy calculation:(1) Construct the distribution graph G' sa , which depicts the state-action pair exploration coverage, and the vertex set is consistent with the state-action complete graph G sa , that is, V' sa = V sa , but the vertex degree is redefined as the access probability of the state-action pair to quantify the frequency of exploration by the agent; the degree d' u of vertex u is equal to the access probability p(u).

[0039] (2) Based on the coding tree , calculate the lower bound of the structural entropy of G' sa :

[0040] The lower bound of the structural entropy is a quantitative indicator of exploration coverage under the value condition, and is calculated based on the hierarchical structure of the optimal coding tree:

[0041]

[0042] where H(S, A) represents the overall exploration entropy of the state-action space, estimated by k-nearest neighbors:

[0043]

[0044] where is the embedding vector of the state-action pair, is twice the distance to the kth nearest neighbor, reflecting the diversity of global exploration, K represents the number of layers of the coding tree, η h represents the weight coefficient of the hth layer, H(u h ) represents the average entropy of all sub-communities at the hth layer, d z represents the dimension of the state-action pair embedding, n represents the total number of samples, n h represents the total number of samples in the hth layer sub-community, is the embedding vector of the hth layer sub-community state-action pair, C represents a constant offset, and i is the sample index of the sub-community.

[0045] S303, intrinsic reward generation:

[0046] Define the lower bound of the structural entropy as the single-agent intrinsic reward and the external reward to define the total reward:

[0047]

[0048] ​The agent is guided to preferentially explore high-value and insufficiently covered areas;

[0049] where β>0 is a hyper-parameter balancing exploration and exploitation, and the extrinsic reward is directly provided by the environment.

[0050] Further, for any non-root node α in the coding tree, the structured entropy is:

[0051]

[0052] where g α is the sum of edge weights connecting the internal and external vertices of the subcommunity V α , vol(G sa ) is the volume of the graph G sa , and vol α is the volume of the subcommunity V α . is the corresponding subcommunity volume of the parent node α - .

[0053] The total structured entropy of the coding tree is the sum of the structured entropies of all non-root nodes:

[0054] H T (G sa )=∑ α∈T,α≠λ H T (G sa ;α);

[0055] where T is the set of all non-root nodes in the coding tree, and λ is the root node of the coding tree.

[0056] The K-dimensional structured entropy is the minimum of the total structured entropy in all coding trees with height less than or equal to K:

[0057]

[0058] The corresponding coding tree is the optimal K-layer coding tree The generation process uses hierarchical spectral clustering: starting from the root node, each layer recursively divides the subcommunity according to value similarity until K layers are reached, finally making the deep subcommunity V h naturally gather value-estimated similar state-action pairs.

[0059] Further, multi-agent collaboration includes the steps of:

[0060] S401, dynamic model training: in multi-agent collaboration, each agent n i trains a dynamic model for predicting the future state evolution of neighboring agents n j within its receptive field. The input is the current state. With action The output is the next state prediction.

[0061] The goal of model training is to minimize the predicted value. Compared with the true value The mean square error;

[0062] S402, Interaction Graph Construction and Adjustment: Constructing a Multi-Agent Interaction Graph G m = (N, E, W), where N is the set of agents, E is the set of directed edges, W is the set of edge weights, and the vertices are agents; for agent n i and its neighbors n within its perceptual domain j There exists a directed edge e ij =(n i ,n j ), representing n i For n j The predicted relationship, edge weight w i,j Quantization n i For n j State prediction capability:

[0063]

[0064] weight w i,j The larger the value of n, the greater the value of n. i For n j The more accurate the prediction;

[0065] Adjusting the diagram includes the following steps:

[0066] Normalization: Make the sum of the outgoing edge weights of each agent equal to 1, ensuring that the weights can be interpreted as transition probabilities;

[0067] Strong connectivity: By adding edges with small probabilities, it is guaranteed that there is a path between any two agents, satisfying the uniqueness of the steady-state distribution of the Markov chain;

[0068] Adjusted weight w′ i,j The calculation formula is:

[0069]

[0070] Where ∈ is a small constant, the first term preserves the relative importance of the original prediction relation, the second term ensures global connectivity, |N| represents the number of agents, and the adjusted graph notation is G'. m =(N,E,W');

[0071] S403, Predictive Intrinsic Reward: Adjusted Interaction Graph G' mIts adjacency matrix can be regarded as the transition probability matrix of a Markov chain;

[0072] Solve for the steady-state distribution of this Markov chain:

[0073] in, In the long run, agent n j The probability of being relied upon by other agents quantifies its global prediction reliability; the higher the value, the more reliable the prediction. j The more trusted the predictions are throughout the system;

[0074] For agent n i The predictive intrinsic reward is:

[0075]

[0076] Where, |N i | represents the intelligent agent n i The number of neighbors, Represents agent n j The probability in the steady-state distribution of a Markov chain. Represents agent n i The dynamic model;

[0077] S404, Novelty Intrinsic Reward, includes the following steps:

[0078] For agent n i Extract subgraphs within its receptive domain Contains n i and all its neighbors N i , used to depict local collaborative relationships;

[0079] To quantify G' m,i The structural diversity within it, constructing its coding tree T m,i And calculate the K-dimensional directed structure entropy:

[0080]

[0081] Here, min represents the operation of finding the minimum value, i.e., from all subgraphs G' m,i Constructed coding tree T m,i In the process, find the one that maximizes the total structured entropy of the coding tree. "The minimum result, T" m,i For subgraph G' m,i The constructed encoding tree, the total structured entropy of the encoding tree The sum of the structured entropy of all non-root nodes:

[0082]

[0083] Wherein, the entropy of each non-root node α Defined as:

[0084]

[0085] Among them, g α To connect sub-communities V α The sum of edge weights of internal and external vertices, vol(G' m,i ) is the graph G' m,i volume, vol α For Sub-Community V α volume, For parent node α - The corresponding sub-community volume;

[0086] intelligent agent n i The intrinsic reward for novelty is directly derived from the K-dimensional directed structural entropy of its local subgraph:

[0087]

[0088] S405, Total Reward Fusion: In a multi-agent system, the total reward for each agent is a weighted combination of extrinsic team rewards and two types of intrinsic rewards.

[0089]

[0090] in, α represents external rewards, provided directly by the environment; α, γ > 0 represents balance coefficients; by dynamically adjusting α and γ, an adaptive balance of exploration and utilization is achieved, ultimately enabling efficient learning of collaborative strategies in sparse reward scenarios.

[0091] Furthermore, a three-layer perceptron with ReLU nonlinear activation is used as the dynamic model.

[0092] The beneficial effects of adopting this technical solution are:

[0093] This invention first optimizes the state-action representation to ensure dynamic correlation in time sequence. Then, it constructs a graph structure for the interaction relationships between state-action pairs or agents, using tactical value differences or prediction errors to characterize global correlations. It innovatively introduces structural entropy, generating intrinsic rewards through graph-based hierarchical partitioning and entropy value calculation, used to quantify the "value orientation of exploration" and the "balance of cooperation." In multi-agent combat / adversarial simulations and collaborative combat scenarios (such as mission planning, adversarial simulations, and formation coordination), structural entropy can guide the exploration of "high-value and safe policy / state-action feature intervals," avoiding dangerous or low-value areas. Combined with single / multi-agent adaptation modules, it improves exploration and cooperation efficiency from both tactical value estimation and dynamic prediction perspectives (single agents focus on key policies and state-action features, while multi-agents focus on accurate credit allocation). This method can capture global dependencies without the need for additional manual design of external rewards, and strengthen the learning of high-value regions and collaborative behaviors. It not only improves performance in sparse reward scenarios, but also meets the actual needs of multi-agent combat missions, and solves key problems such as high-dimensional representation, security exploration and collaborative combat.

[0094] To balance the value of single-agent exploration and avoid wasting exploration resources in low-value areas, this invention adopts hierarchical state-action structure modeling: a fully connected graph is constructed based on the tactical value function, and a multi-layer coding tree is generated by minimizing structural entropy. This quantifies value differences and forms intrinsic incentives, guiding the agent to prioritize the exploration of areas with high tactical value but not fully covered, thereby significantly improving exploration efficiency and safety.

[0095] To balance the "cooperation-diversity" of multi-agent systems and resolve the contradiction between cooperative convergence and strategy diversity, this invention constructs an interaction graph based on dynamic prediction, uses prediction error to measure the dependencies between agents, and uses structural entropy to evaluate cooperative diversity: while ensuring the predictability of behavior, it retains the necessary novelty, avoids strategy homogenization, and takes into account both efficient cooperation and flexible response to battlefield changes (such as formation reconfiguration and ad hoc target switching).

[0096] This invention integrates single / multi-agent modules within a unified structural information framework: structural mutual information constrains learning, encoding trees characterize hierarchical structures, and structural entropy is used to uniformly measure exploration value and collaborative diversity, forming a "novel exploration under predictability constraints." Specifically, single-agent modules focus on mining value structures through encoding trees, while multi-agent modules focus on quantifying interaction and correlation through mutual information and structural entropy, thereby adapting to different combat mission types and distribution conditions and enhancing the method's versatility and transferability.

[0097] To improve sample efficiency and generalization ability, this invention employs dual-path feature extraction combined with structural mutual information constraints to highlight task-related dynamic laws and suppress redundant information. Furthermore, it designs intrinsic rewards using structural entropy to reduce dependence on sparse extrinsic rewards, enabling the strategy to learn key laws more quickly and possess stronger cross-task adaptability (such as the migration from reconnaissance-strike to escort-blockade).

[0098] End-to-end unsupervised optimization: Under unsupervised settings, this invention automatically mines battlefield environment and interaction patterns through structural information, connects "representation learning - exploration / collaboration - strategy optimization" in a closed loop, and reconstructs and verifies the consistency between representation and strategy through dual paths, which helps to achieve end-to-end self-driven learning in complex real-world scenarios.

[0099] This invention employs dynamic modeling and global credit allocation. It trains a dynamic model for each agent to minimize the mean squared error between predicted and actual values, thus characterizing the nonlinear dynamic influence between agents. The adjusted adjacency matrix of the interaction graph is treated as a Markov chain transition matrix, and its steady-state distribution directly quantifies the global prediction reliability of each agent. The resulting predictive intrinsic reward guides agents to prioritize adapting to the needs of highly reliable neighbors, alleviating the credit ambiguity problem of "global reward diluting individual contributions" and avoiding suboptimal strategies due to local objectives.

[0100] To ensure the controllability of collaborative balance, this invention integrates two intrinsic rewards—predictive and novel—to dynamically balance collaboration: predictive rewards guarantee the stability of global cooperation, while novelty rewards (based on the dimensional directed structural entropy of local subgraphs) incentivize the exploration of new collaborative patterns. Furthermore, by dynamically adjusting the balance coefficient (e.g., increasing it early to promote exploration and increasing it later to enhance stability), it aligns with the optimization rhythm of "exploration first, then convergence," thereby breaking through the traditional dilemma of "rigid collaboration or disordered exploration."

[0101] Adaptability to environmental changes: When the battlefield situation distribution, targets and constraints change dynamically, the dynamic model and interaction graph weights can be updated adaptively over time to continuously characterize the dependency structure and influence strength between agents; Under the conditions of communication restrictions / interference or rule constraints (restricted areas, combat rules) changes, it can adapt to task switching and distribution drift without complete retraining, significantly improving the robustness and transferability of the strategy. Attached Figure Description

[0102] Figure 1 This is a schematic diagram of a multi-agent combat state representation and collaborative decision-making method based on structural information according to the present invention. Detailed Implementation

[0103] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings.

[0104] In this embodiment, see Figure 1 As shown, this invention proposes a multi-agent combat state representation and cooperative decision-making method based on structural information, including the following steps:

[0105] S10, Collect data and preprocess the data;

[0106] S20, learn state-action representations from data: based on preprocessed interactive data, a variational autoencoder framework guided by structural mutual information is used to learn state-action embeddings that are dynamically related to the combat environment.

[0107] S30, Single agent exploration: Construct a hierarchical structure of the state-action space based on value estimation, and calculate the structural entropy as the intrinsic reward;

[0108] S40 enables multi-agent collaboration: it constructs an interaction graph based on dynamic prediction of battlefield mission situation, and integrates two types of intrinsic rewards—predictive and novel—to form collaborative combat decisions.

[0109] For data collection: Collection sources include: interaction logs between simulation / wargaming platforms and adversarial training platforms, sensor / event streams (radar / electro-optical / communication / positioning, etc.) of real combat systems, historical replay data, etc., covering key states, actions and team / individual rewards of multiple agents.

[0110] Scene selection principles: matching task step size with round length; consistency between observation dimensions and available sensor signals; alignment of action space with controllable decision variables; retention of key exploration nodes and collaborative interaction data under sparse reward conditions.

[0111] Data integrity verification: Remove sequences with unreasonable jumps, missing actions, and abnormal reward labels to ensure physical / logical consistency and temporal continuity (including records of handling communication packet loss and delays).

[0112] As an optimization of the above embodiments, data preprocessing includes:

[0113] (1) Linear interpolation is used for correction and outlier handling;

[0114] Abnormal transition points in the state are repaired using linear interpolation; out-of-bounds actions are replaced with training set statistics (such as mean / truncation) for that dimension to ensure executability and safety.

[0115] (2) Unify the sparse reward magnitude and complete the reward signal calibration; for single agent tasks, normalize the goal achievement reward to the interval [10,20]; while for multi-agent team rewards, weight them according to the number of agents and then normalize them to the interval [20,50], so as to ensure that the reward signals of different tasks are comparable.

[0116] (3) Clock alignment:

[0117] Based on the global tactical clock, the system copies the preceding valid data to compensate for the delay in asynchronous observation / action flows, thereby achieving strict alignment of time steps for all agents and ensuring the timing accuracy of collaborative relationship modeling and causal inference.

[0118] As an optimization of the above embodiments, the state-action representation learning of the data includes the following steps:

[0119] S201, State Encoding: Using a convolutional neural network or a multilayer perceptron as the state encoder. The preprocessed observation data O t Mapping to compact state representation Simultaneously encode the next time-time observation O t+1 Obtain the compact state representation of the next time step.

[0120] State encoder The training objective is achieved by minimizing the following reconstruction loss:

[0121]

[0122] in, This represents the expectation under the observations at step t and step t+1. This indicates the state decoder.

[0123] The loss constraint S t ,S t+1 It contains critical information sufficient to reconstruct current / next moment operational observations, ensuring the reconstructibility and discriminability of situational evolution.

[0124] S202, Joint Embedding Construction: Constructing the fusion state S t With action A t The joint embedding of Z t .

[0125] A variational autoencoder framework is adopted, and the encoder... Map state-action pairs to posterior distributions By making the posterior distribution similar to the prior distribution p(Z) t Alignment regularizes the embeddings, a process achieved by minimizing the KL divergence:

[0126]

[0127] in, Indicates the joint embedding of Z t The expectation of D KL Denotes KL divergence, Let p(Z) represent the posterior distribution of the joint embedding encoder. t ) represents the prior distribution, θ z The parameters of the joint embedded encoder are represented; KL divergence loss ensures the compactness and smoothness of the latent space, thereby improving generalization ability and the stability of downstream prediction.

[0128] Ultimately, the joint embedding of Z t It is obtained by sampling from the posterior distribution through reparameterization techniques.

[0129] This regularization promotes a compact and smooth potential space, which facilitates cross-scenario generalization and predictive stability in combat missions.

[0130] S203, Structural Mutual Information Optimization: First, for the joint embedding Z... t With the current state S t Construct a bipartite graph G zs =(Z,S,E) zs ), depicting Z t With S t Joint distribution, E zs For bipartite graph G zs The edge set; the vertex set is Z. t The sample Z = {z1, z2, ..., z} n} and S t The sample S = {s1, s2, ..., s} n}; Edge weight e zs (z i ,s j )=p(z i ,s j ), that is, Z t With S t The joint probability satisfies ∑ i,j p(z i ,s j ) = 1; for Z t With the next state S t+1 Construct a bipartite graph G zs' =(Z,S',E zs' E zs' For bipartite graph G zs' The edge set with edge weight p(z) i ,s′ j ), that is, Z t With S t+1 The joint probability;

[0131] Secondly, based on two layers of approximate binary trees, the bipartite graph G is processed respectively. zs And bipartite graph G zs' Calculations are performed to obtain the structured mutual information I. SI (Zt S t ) and I SI (Z t S r+1 Structured mutual information I SI (Z t S t ), measuring Z t With the current state S t Structural redundancy; structured mutual information I SI (Z t S t+1 ), measuring Z t With future state S t+1 Structural prediction capability;

[0132] Structured mutual information I SI It is an indicator for measuring structural dependencies between variables, and its calculation depends on the hierarchical division of the bipartite graph.

[0133] Two-layer approximate binary tree T 2 The structural design is as follows: the root node corresponds to all vertices of the bipartite graph, containing Z and all samples of the state set; the intermediate layer consists of the child nodes of the root node, and each intermediate node contains one Z. t The tree consists of one sample and one state sample; leaf nodes are children of intermediate nodes, and each leaf node corresponds to a single sample; the tree structure is optimized using the stretching operator of the HCSE algorithm to obtain the optimal tree that minimizes the structured entropy of the bipartite graph. Based on optimal tree Its l-transform, structured mutual information is defined as:

[0134]

[0135] Among them, H 1 (X) and H 1 (Y) represents the one-dimensional structured entropy, calculated using the following formula: (Since the total edge weight sum of a bipartite graph is 1, the volume vol(G) xy ) = 2), (The joint entropy of the l-transform tree).

[0136] Finally, by optimizing the loss function L smi To achieve learning of dynamic related embeddings.

[0137] The loss function L smi The expression is:

[0138] L smi =L up +L z|s +L s|z ;

[0139] Among them, Lup By approximating Z t marginal distribution q m (Z t ), control Z t With S t Shannon mutual information upper bound, indirectly minimizing I SI (Z t S t );L z|s Through the conditional probability model q z|s Quantization Z t For S t The conditional entropy is used to further filter redundant associations; L s|z By predicting model q s|z Maximize Z t For S t+1 The predictive likelihood indirectly increases I SI (Z t S t+1 ).

[0140] By minimizing L smi At the same time, it reduces I SI (Z t S t ) and improve I SI (Z t S t+1 The goal of Z is to make Z t By removing redundant information related to the current state while retaining key dynamic information for predicting future states, a dynamically correlated embedding Zi is obtained that combines compactness with dynamic relevance. t This provides high-quality foundational representations for subsequent agent exploration, decision-making, and multi-agent collaboration. It achieves dynamically relevant embeddings that are "more relevant to the future and less redundant to the present," which can be used for subsequent tactical exploration and collaborative decision-making.

[0141] As an optimization of the above embodiments, single-agent exploration includes the following steps:

[0142] S301, Hierarchical community division: (1) Constructing a complete state-action graph G sa To characterize the basic structure of state-action relationships; the complete state-action graph G sa Vertex set V sa It consists of all the unique state-action pairs in the environment. Edge weights are used to quantify the value difference between state-action pairs, and are defined based on the estimation results of the value function Q(s,a):

[0143]

[0144] Where i and j are two vertices. For the long-term value estimate of vertex i, ||·||2 denotes the L2 norm; w ij The smaller the weight, the more similar the values ​​of the two state-action pairs; for example, "different movement actions to approach the target" may have similar Q values ​​and edge weights close to 0.

[0145] (2) The structural framework of the K-layer coding tree T: from the root node to the deep sub-community; root node λ: corresponds to the full set of vertices V λ =V sa , representing the undivided complete state-action space; intermediate layer nodes: the node α in the h-th layer corresponds to the sub-community. Sub-communities are mutually exclusive and completely cover the parent community; deep nodes, layer K: correspond to the smallest sub-community, containing state-action pairs with highly similar value estimates (such as "all valid actions within the target area"). The size of the sub-community is usually 50 to 200 vertices to avoid overly fine division.

[0146] (3) Optimal coding tree generation based on minimizing K-dimensional structural entropy: Optimal coding tree It is obtained by minimizing the K-dimensional structural entropy. The core is to calculate the structural entropy of each node in the coding tree and optimize the total entropy.

[0147] For any non-root node α in the coding tree, its structured entropy is:

[0148]

[0149] Among them, g α To connect sub-communities V α The sum of edge weights between internal and external vertices, vol(G) sa ) is graph G sa volume, vol α For Sub-Community V α volume, For parent node α - The corresponding sub-community volume;

[0150] The total structured entropy of the coding tree is the sum of the structured entropies of all non-root nodes:

[0151] H T (G sa )=Σ α∈T,α≠λ H T (G sa ;α);

[0152] Where T is the set of all non-root nodes in the coding tree, and λ is the root node of the coding tree;

[0153] The K-dimensional structural entropy is the minimum total structured entropy among all coding trees with a height less than or equal to K:

[0154]

[0155] The corresponding coding tree is the optimal K-layer coding tree. The generation process employs hierarchical spectral clustering: starting from the root node, each layer recursively divides into sub-communities based on value similarity, until layer K is reached, ultimately resulting in a deep sub-community V. h Naturally clustered value estimation of similar state-action pairs provides a structured basis for guiding subsequent exploration.

[0156] S302, Constructing the distribution map and calculating the structural entropy: (1) Constructing the distribution map G′ sa Characterizes the exploration coverage of state and action, vertex set and state-action complete graph G sa Consistency, i.e., V′ sa =V sa However, vertex degree is redefined as the probability of accessing a state-action pair to quantify how frequently the agent explores that pair; the degree d′ of vertex u u It is equal to the access probability p(u), which is obtained by the k-nearest neighbor (k-NN) estimation algorithm.

[0157] (2) Based on coding tree Calculate G' sa Lower bound of structural entropy:

[0158] The lower bound of structural entropy is a quantitative indicator of exploration coverage under value conditions, based on the optimal coding tree. Hierarchical structure calculation:

[0159]

[0160] Where H(S,A) represents the overall exploration entropy of the state-action space, estimated by the k nearest neighbors:

[0161]

[0162] in, This is the embedding vector for the state-action pair. It is twice the distance to its k-th nearest neighbor, reflecting the diversity of global exploration, where K represents the number of layers in the coding tree, and η h H(u) represents the weight coefficient of the h-th layer. h ) represents the average entropy of all sub-communities in layer h, and d z This represents the dimension of the state-action pair embedding, where n represents the total number of samples. h This represents the total number of samples in the h-th sub-community. Let be the embedding vector of the state-action pair of the h-th sub-community, where C represents the constant offset and i is the sample index of the sub-community;

[0163] Distribution map G′ saThe lower bound of structural entropy combines the "value importance" and "exploration sufficiency" of state-action pairs, providing a quantitative basis for the generation of subsequent intrinsic rewards and ensuring that agents prioritize exploring high-value but under-covered areas.

[0164] S303, Intrinsic Reward Generation:

[0165] Define the lower bound of structural entropy as the intrinsic reward within a single agent. External rewards Fusion is defined as the total reward:

[0166]

[0167] Guide intelligent agents to prioritize exploring high-value but under-covered areas.

[0168] This design combines "tactical value importance" with "exploration adequacy" to guide priority exploration of areas with high tactical value but insufficient coverage.

[0169] As an optimization of the above embodiments, multi-agent cooperation includes the following steps:

[0170] S401, Dynamic Model Training: In multi-agent collaboration, each agent n i Each combat unit trains a dynamic model. Used to predict neighboring agents n within its receptive field j The future state evolution; The input is the current state. With action The output is the next state prediction.

[0171]

[0172] The goal of model training is to minimize the predicted value. Compared with the true value The mean square error;

[0173] S402, Interaction Graph Construction and Adjustment: Constructing a Multi-Agent Interaction Graph G m = (N, E, W), where N is the set of agents, E is the set of directed edges, W is the set of edge weights, and the vertices are agents; for agent n i and its neighbors n within its perceptual domain j There exists a directed edge e ij =(n i ,n j ), representing n i For n j The predicted relationship, edge weight w i,j Quantization n i For n jState prediction capability:

[0174]

[0175] weight w i,j The larger the value of n, the greater the value of n. i For n j The more accurate the prediction;

[0176] The original graph may contain isolated nodes or local disconnections, causing steady-state distribution calculations to fail. Therefore, graph adjustment is performed, including the following steps:

[0177] Normalization: Make the sum of the outgoing edge weights of each agent equal to 1, ensuring that the weights can be interpreted as transition probabilities;

[0178] Strong connectivity: By adding edges with small probabilities, it is guaranteed that there is a path between any two agents, satisfying the uniqueness of the steady-state distribution of the Markov chain;

[0179] Adjusted weight w′ i,j The calculation formula is:

[0180]

[0181] Where ∈ is a small constant (e.g., 0.01), the first term preserves the relative importance of the original prediction relationship, the second term ensures global connectivity, |N| represents the number of agents, and the adjusted graph notation is G'. m =(N,E,W'), used for stable modeling of cross-unit collaborative dependencies.

[0182] S403, Predictive Intrinsic Reward: Adjusted Interaction Graph G' m Its adjacency matrix can be regarded as the transition probability matrix of a Markov chain;

[0183] Solve for the steady-state distribution of this Markov chain:

[0184] in, In the long run, agent n j The probability of being relied upon by other agents quantifies its global prediction reliability; the higher the value, the more reliable the prediction. j The more trusted the predictions are throughout the system;

[0185] For agent n i The predictive intrinsic reward is:

[0186]

[0187] Where, |N i | represents the intelligent agent n i The number of neighbors, Represents agent nj The probability in the steady-state distribution of a Markov chain. Represents agent n i The dynamic model;

[0188] Reward guidance n i The behavior is adjusted in the direction of "making highly reliable neighbors more predictable", thereby enhancing the stability of collaboration.

[0189] S404, Novelty Intrinsic Reward, aims to incentivize agents to explore underutilized collaborative patterns and avoid behavioral rigidity. It includes the following steps:

[0190] For agent n i Extract subgraphs within its receptive domain Contains n i and all its neighbors N i , used to depict local collaborative relationships;

[0191] To quantify G' m,i The structural diversity within it, constructing its coding tree T m,i And calculate the K-dimensional directed structure entropy:

[0192]

[0193] Here, min represents the operation of finding the minimum value, i.e., from all subgraphs G' m,i Constructed coding tree T m,i In the process, find the one that maximizes the total structured entropy of the coding tree. "The minimum result, T" m,i For subgraph G' m,i The constructed encoding tree, the total structured entropy of the encoding tree The sum of the structured entropy of all non-root nodes:

[0194]

[0195] Wherein, the entropy of each non-root node α Defined as:

[0196]

[0197] Among them, g α To connect sub-communities V α The sum of edge weights of internal and external vertices, vol(G' m,i ) is the graph G' m,i volume, vol α For Sub-Community V α volume, For parent node α - The corresponding sub-community volume;

[0198] intelligent agent n i The intrinsic reward for novelty is directly derived from the K-dimensional directed structural entropy of its local subgraph:

[0199]

[0200] This reward motivates n i Try new neighbor interaction patterns (such as connecting with neighbors you don't cooperate with frequently) to increase the diversity of collaboration strategies.

[0201] S405, Total Reward Fusion: In a multi-agent system, the total reward for each agent is a weighted combination of extrinsic team rewards and two types of intrinsic rewards.

[0202]

[0203] in, α represents external rewards, provided directly by the environment; α, γ>0 represents balance coefficients; by dynamically adjusting α and γ (e.g., increasing γ in the early stage of training for exploration, and increasing α in the later stage for convergence), an adaptive balance of exploration utilization is achieved, ultimately enabling efficient learning of collaborative strategies in sparse reward scenarios.

[0204] Preferably, a three-layer perceptron with ReLU nonlinear activation is used as the dynamic model.

[0205] This invention is based on dynamic relevance representation learning using structured mutual information. In traditional reinforcement learning, state and action representations often contain redundant information, which affects the accuracy of dynamic prediction. The innovation of this invention lies in integrating structured mutual information into the framework of a variational autoencoder. It uses a bipartite graph to depict the distribution relationship between the joint embedding and the current and future states, and calculates the structured mutual information based on two layers of approximate binary trees. By optimizing a specific loss, the structured mutual information between the joint embedding and the future state is maximized to improve dynamic prediction capabilities. Simultaneously, the structured mutual information between the joint embedding and the current state is minimized, filtering out redundant information. The result is a compact embedding rich in dynamic information, providing a high-quality representation foundation for subsequent exploration and collaboration.

[0206] This invention is based on hierarchical state-action structure mining of coding trees: Traditional maximum entropy exploration methods tend to overemphasize low-value states, resulting in low exploration efficiency. This invention constructs a complete graph of state-action pairs based on the single agent value function, treats value differences as edge weights, and obtains multi-level coding trees by minimizing K-dimensional structural entropy to capture the hierarchical relationship of the state-action space. Combined with the structural entropy calculation of the distribution graph, a sub-community entropy correction term is introduced to ensure that the intrinsic reward guarantees global exploration coverage while preventing overexploration in low-value sub-regions, thus achieving efficient exploration based on value perception.

[0207] This invention presents a multi-agent intrinsic reward mechanism that integrates predictive reliability and structural diversity. Traditional multi-agent exploration processes struggle to balance predictability and behavioral diversity. This invention begins by constructing a directed interaction graph, using the exponential function of prediction error as edge weights to characterize the predictive capabilities between agents. An adjustment algorithm yields a strongly connected graph, and its steady-state distribution defines predictive intrinsic rewards to encourage agents to rely on highly reliable neighbors for predictions. Local subgraphs are extracted, and a coding tree is constructed to calculate K-dimensional directed structural entropy, which is used as a novel intrinsic reward to guide the exploration of diverse collaborative patterns. By weighted fusion of these two types of rewards, an adaptive balance between predictability and diversity in multi-agent collaboration is achieved.

[0208] To highlight military application scenarios, this invention targets mission environments such as joint operations, wargaming, and unmanned swarm collaboration, proposing an intrinsic reward mechanism that integrates "predictive reliability and structural diversity." First, in the operational interaction graph, the prediction errors between agents characterize "who is more predictable to whom," and normalization and strong connectivity enhancement ensure information reachability even under conditions of limited communication, electromagnetic countermeasures, and complex terrain obscuring. Based on this, predictive intrinsic rewards are extracted from the steady-state influence distribution, encouraging operational units to form predictable cooperation with highly reliable neighbors (such as formation maintenance, fire relay, and sensor coordination). This improves collaborative stability and C2 execution; simultaneously, it constructs a coding tree on the local subgraph of each unit and measures the directed structural entropy as an intrinsic reward for novelty, guiding the formation of diverse and non-homogeneous collaborative modes (such as ad hoc formation reconstruction, cross-domain reinforcement, and agile flanking maneuvers), enhancing adaptability to battlefield emergencies and changes in ROE / restricted zones; ultimately, it achieves a "explore first, converge later" training rhythm through adaptive weighting of the two types of rewards: in the early stage, it focuses on diversified maneuvers to open up high-value action channels, and in the later stage, it focuses on predictable collaboration to stably grasp the collaborative strike window and overall situational advantage.

[0209] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for multi-agent combat state representation and collaborative decision-making based on structural information, characterized in that, Including the following steps: S10, Collect data and preprocess the data; S20, learn state-action representations from data: based on preprocessed interactive data, a variational autoencoder framework guided by structural mutual information is used to learn state-action embeddings that are dynamically related to the combat environment. S30, Single agent exploration: Construct a hierarchical structure of the state-action space based on value estimation, and calculate the structural entropy as the intrinsic reward; S40 enables multi-agent collaboration: it constructs an interaction graph based on dynamic prediction of battlefield mission situation, and integrates two types of intrinsic rewards—predictive and novel—to form collaborative combat decisions.

2. The method for multi-agent combat state representation and cooperative decision-making based on structural information according to claim 1, characterized in that, Data preprocessing includes: Linear interpolation is used for correction and outlier handling; Unify the sparse reward magnitude and complete the reward signal calibration; for single agent tasks, normalize the goal achievement reward to the interval [10,20]; while for multi-agent team rewards, normalize them after weighting by the number of agents. Using a global clock as a reference, the communication sensing delay sequence is completed by copying previous valid data, so that the time steps of all agents are strictly aligned.

3. The method for multi-agent combat state representation and collaborative decision-making based on structural information according to claim 1, characterized in that, Learning state-action representations from data includes the following steps: S201, State Encoding: Using a convolutional neural network or a multilayer perceptron as the state encoder. The preprocessed observation data O t Mapping to compact state representation Simultaneously encode the next time-time observation O t+1 Obtain the compact state representation of the next time step. S202, Joint Embedding Construction: Constructing the fusion state S t With action A t The joint embedding of Z t ; S203, Structural Mutual Information Optimization: First, for the joint embedding Z... t With the current state S t Construct a bipartite graph G zs =(Z,S,E) zs ), depicting Z t With S r Joint distribution, E zs For bipartite graph G zs The edge set; the vertex set is Z. t The sample Z = {z1, z2, ..., z} n } and S t The sample S = {s1, s2, ..., s} n }; Edge weight e zs (z i ,s j )=p(z i ,Sj ) Z t With S t The joint probability satisfies ∑ i,j p(z i ,s j ) = 1; for Z t With the next state S t+1 Construct a bipartite graph G zs' =(Z,S',E zs' E zs' For bipartite graph G zs' The edge set with edge weight p(z) i ,s j '), that is, Z t With S t+1 The joint probability; Secondly, based on two layers of approximate binary trees, the bipartite graph G is processed respectively. zs And bipartite graph G zs' Calculations are performed to obtain the structured mutual information I. SI (Z t S t ) and I SI (Z t S t+1 Structured mutual information I SI (Z t S t ), measuring Z t With the current state S t Structural redundancy; structured mutual information I SI (Z t S t+1 ), measuring Z t With future state S t+1 Structural prediction capability; Finally, by optimizing the loss function L smi To achieve learning of dynamic related embeddings; By minimizing L smi At the same time, it reduces I SI (Z t S t ) and improve I SI (Z t S t+1 The goal of Z is to make Z t By removing redundant information related to the current state while retaining key dynamic information for predicting future states, a dynamically correlated embedding Zi is obtained that combines compactness with dynamic relevance. t This provides a high-quality basic representation for subsequent exploration and decision-making by intelligent agents and multi-agent collaboration.

4. The method for multi-agent combat state representation and collaborative decision-making based on structural information according to claim 3, characterized in that, State encoder The training objective is achieved by minimizing the following reconstruction loss: in, This represents the expectation under the observations at step t and step t+1. Represents the state decoder, O t For operational situation observation, S t This is the corresponding compact state representation.

5. The method for multi-agent combat state representation and cooperative decision-making based on structural information according to claim 3, characterized in that, Constructing the fusion state S t With action A t Joint potential embedding Z t At that time, a variational autoencoder framework was adopted, and the encoder... Map state-action pairs to posterior distributions By making the posterior distribution similar to the prior distribution p(Z) t Alignment regularizes the embeddings, a process achieved by minimizing the KL divergence: in, Indicates the joint embedding of Z t The expectation of D KL Denotes KL divergence, Let p(Z) represent the posterior distribution of the joint embedding encoder. t ) represents the prior distribution, θ z Indicates the parameters of the joint embedded encoder; Ultimately, the joint embedding of Z t It is obtained by sampling from the posterior distribution through reparameterization techniques.

6. The method for multi-agent combat state representation and cooperative decision-making based on structural information according to claim 3, characterized in that, The loss function L smi The expression is: L smi =L up +L z|s +L s|z ; Among them, L up By approximating Z t marginal distribution q m (Z t ), control Z t With S t Shannon mutual information upper bound, indirectly minimizing I SI (Z t S t );L z|s Through the conditional probability model q z|s Quantization Z t For S t The conditional entropy is used to further filter redundant associations; L s|z By predicting model q s|z Maximize Z t For S t+1 The predictive likelihood indirectly increases I SI (Z t S t+1 ).

7. The method for multi-agent combat state representation and cooperative decision-making based on structural information according to claim 1, characterized in that, Single-agent exploration includes the following steps: S301, Hierarchical community division: (1) Constructing a complete state-action graph G sa To characterize the basic structure of state-action relationships; the complete state-action graph G sa Vertex set V sa It consists of all the unique state-action pairs in the environment. Edge weights are used to quantify the value difference between state-action pairs, and are defined based on the estimation results of the value function Q(s,a): Where i and j are two vertices. For the long-term value estimate of vertex i, ||·||2 denotes the L2 norm; w ij The smaller the weight, the more similar the values ​​of the two state-action pairs; (2) The structural framework of the K-layer coding tree T: from the root node to the deep sub-community; root node λ: corresponds to the full set of vertices V λ =V sa , representing the undivided complete state-action space; intermediate layer nodes: the node α in the h-th layer corresponds to the sub-community. Sub-communities are mutually exclusive and completely cover the parent community; deep nodes, layer K: correspond to the smallest sub-community, containing state-action pairs with highly similar value estimates; (3) Optimal coding tree generation based on minimizing K-dimensional structural entropy: Optimal coding tree It is obtained by minimizing the K-dimensional structural entropy. The core is to calculate the structural entropy of each node in the coding tree and optimize the total entropy. S302, Constructing the distribution map and calculating the structural entropy: (1) Constructing the distribution map G′ sa Characterizes the exploration coverage of state and action, vertex set and state-action complete graph G sa Consistency, i.e., V′ sa =V sa However, the vertex degree is redefined as the access probability of a state-action pair to quantify how often the agent explores the pair; the degree d' of vertex u is equal to the access probability p(u); (2) Based on coding tree Calculate G' sa Lower bound of structural entropy: The lower bound of structural entropy is a quantitative indicator of exploration coverage under value conditions, based on the optimal coding tree. Hierarchical structure calculation: Where H(S,A) represents the overall exploration entropy of the state-action space, estimated by the k nearest neighbors: in, This is the embedding vector for the state-action pair. It is twice the distance to its k-th nearest neighbor, reflecting the diversity of global exploration, where K represents the number of layers in the coding tree, and η h H(u) represents the weight coefficients of the h-th layer. h ) represents the average entropy of all sub-communities in layer h, and d z This represents the dimension of the state-action pair embedding, where n represents the total number of samples. h This represents the total number of samples in the h-th sub-community. Let be the embedding vector of the state-action pair of the h-th sub-community, where C represents the constant offset and i is the sample index of the sub-community; S303, Intrinsic Reward Generation: Define the lower bound of structural entropy as the intrinsic reward within a single agent. External rewards Fusion is defined as the total reward: Guide intelligent agents to prioritize exploring high-value but under-covered areas; Here, β>0 is a hyperparameter balancing exploration and exploitation, representing extrinsic rewards. Provided directly by the environment.

8. The method for multi-agent combat state representation and cooperative decision-making based on structural information according to claim 7, characterized in that, For any non-root node α in the coding tree, its structured entropy is: Among them, g α To connect sub-communities V α The sum of edge weights between internal and external vertices, vol(G) sa ) is graph G sa volume, vol α For Sub-Community V α volume, vol α - For parent node α - The corresponding sub-community volume; The total structured entropy of the coding tree is the sum of the structured entropies of all non-root nodes: H T (G sa )=∑ α∈T,α≠λ HT ( G sa (α); Where T is the set of all non-root nodes in the coding tree, and λ is the root node of the coding tree; The K-dimensional structural entropy is the minimum total structured entropy among all coding trees with a height less than or equal to K: The corresponding coding tree is the optimal K-layer coding tree. The generation process employs hierarchical spectral clustering: starting from the root node, each layer recursively divides into sub-communities based on value similarity, until layer K is reached, ultimately resulting in a deep sub-community V. h Natural clustering value estimates similar state-action pairs.

9. The method for multi-agent combat state representation and collaborative decision-making based on structural information according to claim 1, characterized in that, Multi-agent collaboration includes the following steps: S401, Dynamic Model Training: In multi-agent collaboration, each agent n i Each trains a dynamic model Used to predict neighboring agents n within its receptive field j The future state evolution; The input is the current state. With action The output is the next state prediction. The goal of model training is to minimize the predicted value. Compared with the true value The mean square error; S402, Interaction Graph Construction and Adjustment: Constructing a Multi-Agent Interaction Graph G m = (N, E, W), where N is the set of agents, E is the set of directed edges, W is the set of edge weights, and the vertices are agents; for agent n i and its neighbors n within its perceptual domain j There exists a directed edge e ij =(n i ,n j ), representing n i For n j The predicted relationship, edge weight w i,j Quantization n i For n j State prediction capability: weight w i,j The larger the value of n, the greater the value of n. i For n j The more accurate the prediction; Adjusting the diagram includes the following steps: Normalization: Make the sum of the outgoing edge weights of each agent equal to 1, ensuring that the weights can be interpreted as transition probabilities; Strong connectivity: By adding edges with small probabilities, it is guaranteed that there is a path between any two agents, satisfying the uniqueness of the steady-state distribution of the Markov chain; Adjusted weight w′ i,j The calculation formula is: Where ∈ is a small constant, the first term preserves the relative importance of the original prediction relation, the second term ensures global connectivity, |N| represents the number of agents, and the adjusted graph notation is G'. m =(N,E,W'); S403, Predictive Intrinsic Reward: Adjusted Interaction Graph G' m Its adjacency matrix can be regarded as the transition probability matrix of a Markov chain; Solve for the steady-state distribution of this Markov chain: in, In the long run, agent n j The probability of being relied upon by other agents quantifies its global prediction reliability; the higher the value, the more reliable the prediction. j The more trusted the predictions are throughout the system; For agent n i The predictive intrinsic reward is: Where, |N i | represents the intelligent agent n i The number of neighbors, Represents agent n j The probability in the steady-state distribution of a Markov chain. Represents agent n i The dynamic model; S404, Novelty Intrinsic Reward, includes the following steps: For agent n i Extract subgraphs within its receptive domain Contains n i and all its neighbors N i , used to depict local collaborative relationships; To quantify G' m,i The structural diversity in the data is used to construct its coding tree T. m,i And calculate the K-dimensional directed structure entropy: Here, min represents the operation of finding the minimum value, i.e., from all subgraphs G' m,i Constructed coding tree T m,i In the process, find the one that maximizes the total structured entropy of the coding tree. The minimum result, T m,i For subgraph G' m,i The constructed encoding tree, the total structured entropy of the encoding tree The sum of the structured entropy of all non-root nodes: Wherein, the entropy of each non-root node α Defined as: Among them, g α To connect sub-communities V α The sum of edge weights of internal and external vertices, vol(G' m,i ) is the graph G' m,i volume, vol α For Sub-Community V α volume, vol α- For parent node α - The corresponding sub-community volume; intelligent agent n i The intrinsic reward for novelty is directly derived from the K-dimensional directed structural entropy of its local subgraph: S405, Total Reward Fusion: In a multi-agent system, the total reward for each agent is a weighted combination of extrinsic team rewards and two types of intrinsic rewards. in, α represents external rewards, provided directly by the environment; α, γ > 0 represents balance coefficients; by dynamically adjusting α and γ, an adaptive balance of exploration and utilization is achieved, ultimately enabling efficient learning of collaborative strategies in sparse reward scenarios.

10. A method for multi-agent combat state representation and cooperative decision-making based on structural information according to claim 9, characterized in that, A three-layer perceptron with ReLU nonlinear activation is used as the dynamic model.

Images