A multi-agent deep reinforcement learning-based cluster intention sharing and distributed cooperative game system

Abstract

The application provides a cluster intention sharing and distributed cooperative game system based on multi-agent deep reinforcement learning, and relates to the field of electric digital data processing. Each intelligent node in the system runs a double-headed Actor strategy network, simultaneously outputs a trajectory probability distribution matrix and a task target preference vector, and splices the two into a structured intention tensor with clear physical semantics. After compression and encoding, the intention tensor is broadcast to neighbor nodes through a low-bandwidth channel. After receiving the neighbor intention tensor, each node can directly trigger the construction of a local non-cooperative game model without decoding. The game payoff function is obtained by weighting and accumulating the probability projection value of the neighbor trajectory probability distribution on the corresponding direction sector of the node's candidate action and the neighbor task preference component. The Nash equilibrium approximate solver uses the first prediction step probability distribution of the neighbor intention tensor as the initial belief to accelerate the iterative convergence, and outputs the optimal response action. The system realizes implicit cooperation and dynamic obstacle avoidance of the cluster under the condition of no central scheduling.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing, specifically to a cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning. Background Technology

[0002] Multi-agent cooperative control is a core technical challenge in unmanned swarm systems. Existing technologies mainly fall into two categories: centralized scheduling and distributed coordination. Centralized scheduling relies on a central node to gather global information and issue control commands. However, in highly dynamic and unknown environments, it suffers from inherent drawbacks such as large communication latency, high risk of single-point failure, and poor scalability, making it difficult to meet the real-time coordination requirements of large-scale swarms.

[0003] In distributed collaborative schemes, multi-agent deep reinforcement learning has been widely studied due to its advantages of not requiring global information and being able to adapt to complex environments through training. Existing multi-agent deep reinforcement learning communication schemes typically involve each agent broadcasting the hidden state vector of the intermediate layer of the policy network to neighboring nodes. The receiver then uses a dedicated decoding network to restore the hidden state vector into an intermediate representation that can be used for decision-making. However, the hidden state vector lacks explicit physical semantics, the decoding process relies on the strong coupling between the network structures of the receiver and the sender, and any change in the network structure of any node may cause the global decoding mechanism to fail. At the same time, the decoding operation introduces additional computational overhead, which limits the real-time deployment capability of the system on computationally limited nodes.

[0004] Game theory offers another theoretical framework for distributed multi-agent cooperative decision-making. By constructing a local non-cooperative game model and solving for Nash equilibrium, each agent can autonomously make its optimal response under conditions without central scheduling. However, existing game theory methods typically use deterministic position predictions or simple state estimates as game inputs, failing to effectively utilize the probability distribution information of agent intentions. In highly dynamic and unknown environments where neighbor intentions are uncertain, payoff estimations based on deterministic predictions accumulate prediction errors, causing the equilibrium point to deviate from the actual optimal cooperative strategy, resulting in a significant decrease in cooperative effectiveness. Furthermore, existing schemes often use uniform priors as initial beliefs, ignoring the accelerating effect of inter-agent communication information on equilibrium solution, leading to limited iterative convergence speed under real-time constraints.

[0005] In the prior art, CN114662692A discloses a multi-agent interaction method and system based on swarm game theory. This scheme models the regular game at each stage of multi-agent reinforcement learning as a swarm game, using the Q-function values ​​of each agent as inputs to the game payoff function. It utilizes an inheritance-based BNN swarm game dynamic algorithm to obtain an approximate equilibrium policy by solving ordinary differential equations, and updates the Q-table of each agent with the expected reward corresponding to the equilibrium policy. This scheme reduces the computational complexity of equilibrium solution to some extent, but it has the following technical drawbacks.

[0006] First, the game payoff function of this scheme takes the value of the agent's local Q-table as the only input. The Q value is a scalar value and does not carry any structured semantic information about the agent's future movement intentions or task goal preferences. The game construction is entirely based on the accumulation of local historical experience and cannot reflect the real-time intention state of neighboring agents at the current moment. In a highly dynamic and unknown environment where the behavior of neighbors changes rapidly, there is a significant timeliness deviation between the payoff estimation based on historical Q values ​​and the actual collaborative needs.

[0007] Secondly, the scheme does not include any communication mechanism for intention information between agents. When making game decisions, each agent cannot obtain the current movement planning information of its neighbors. The estimation of each other's behavioral intentions by the game participants depends entirely on historical interaction experience. When the dynamics of the environment exceed the coverage of historical experience, the accuracy of the equilibrium solution cannot be guaranteed.

[0008] Third, the scheme uses uniform priors as the initial belief for solving the game equilibrium, without utilizing any prior information about the distribution of neighbor intentions to accelerate iterative convergence. In cluster collaboration scenarios with strict real-time constraints, the redundancy in the number of iteration steps caused by uniform initialization restricts the system's real-time decision-making ability.

[0009] The root cause of the above defects is that existing solutions lack a technical mechanism that can transmit the agent's real-time movement intentions between nodes in a structured semantic form and directly use it to drive the construction of local game payoffs, resulting in a structural disconnect between game decision-making and real-time intention perception.

[0010] Therefore, how to design a structured intention expression method that enables agents to broadcast intention information with clear physical semantics to neighboring nodes under low bandwidth constraints, and directly map this information to the payoff metric of local games, thereby realizing distributed collaborative decision-making based on real-time intention perception under decentralized scheduling conditions, is a technical problem that urgently needs to be solved in the field of multi-agent cluster collaborative control. Summary of the Invention

[0011] The purpose of this invention is to address the shortcomings by proposing a cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning.

[0012] The present invention adopts the following technical solution:

[0013] A cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning includes a dual-head Actor policy network, intent tensor encoder, low-bandwidth broadcast module, neighborhood intent receiving and weighted aggregation module, local game construction module and Nash equilibrium approximation solver integrated in the onboard computing unit of each intelligent node;

[0014] The dual-head Actor strategy network takes the node's local observation vector as input, and its output is connected to both the trajectory probability distribution output head and the task target preference output head. The trajectory probability distribution output head outputs the trajectory probability distribution matrix Π, where each element of the matrix represents the probability that a node moves to the corresponding sector in the corresponding prediction time step. The task target preference output head outputs the task target preference vector Θ, where each component corresponds to the preference intensity of different task targets.

[0015] The intent tensor encoder concatenates the trajectory probability distribution matrix Π with the task target preference vector Θ to form a structured intent tensor Φ. Each element of the structured intent tensor Φ has a clear physical semantics. After compression encoding, it is broadcast to each node in the neighborhood through a low-bandwidth broadcast module. The receiving node can directly substitute the elements of the structured intent tensor Φ into the game payoff function for calculation without decoding.

[0016] The neighborhood intent receiving and weighted aggregation module receives the structured intent tensor broadcast by neighboring nodes, decompresses and restores it, and constructs a weighted neighborhood intent set by weighting it according to the inverse of distance.

[0017] The local game construction module constructs a local non-cooperative game model with a weighted neighborhood intention set as the trigger input. The task collaboration payoff in the game payoff function is obtained by weighted summation of the probability projection value of the neighbor node trajectory probability distribution matrix Π on the sector corresponding to the candidate action of the current node and the corresponding component of the neighbor node task objective preference vector Θ.

[0018] The Nash equilibrium approximation solver directly uses the probability distribution of the first predicted step trajectory in the structured intention tensor of each neighbor node as the initial belief for the action selection of the neighbor node, and solves the Nash equilibrium using the hypothetical game iterative method and outputs the optimal response action.

[0019] Furthermore, the backbone output of the dual-head Actor policy network is connected to two independent gating units. Each gating unit consists of a single-layer linear transformation and sigmoid activation. The two gating units apply soft selection to the feature dimensions related to motion direction prediction and task state evaluation in the backbone latent vector, respectively, and then send them to the trajectory probability distribution output head and the task target preference output head. The parameters of the two gating units are independent, realizing soft decoupling of the two types of output information under the condition of sharing backbone features.

[0020] Furthermore, the compression encoding of the intent tensor encoder adopts differentiated quantization bit width. The probability values ​​in the trajectory probability distribution matrix Π are quantized with a higher bit width, and the preference values ​​in the task target preference vector Θ are quantized with a lower bit width. The quantization bit width is determined according to the numerical sensitivity of each sub-tensor element in the game payoff calculation, so as to ensure that the accuracy of the decompression and restoration after compression meets the numerical stability requirements of the game payoff calculation.

[0021] Furthermore, the neighborhood intent receiving and weighted aggregation module also includes an intent integrity verification unit and a node validity management unit. The node validity management unit maintains a continuous missing period counter for each neighbor node. When the number of consecutive missing periods of a neighbor node does not exceed a preset upper limit, it is replaced with a cached frame and weight decay is applied. When it exceeds the preset upper limit, the node is removed from the game participant set of the current decision period and a conservative obstacle avoidance flag of the corresponding direction is sent to the action parsing execution unit.

[0022] Furthermore, the local game construction module also includes a kinematic prediction submodule, which takes the node's current pose, velocity, and each alternative action as input, and calculates the predicted trajectory of the node in each prediction step under each alternative action step by step according to the uniform acceleration kinematic model of the mass, for use by the collision risk penalty calculation subunit and the formation maintenance benefit calculation subunit.

[0023] Furthermore, the game payoff function of the local game construction module also includes two sub-items: collision risk penalty and formation maintenance payoff. The collision risk penalty is obtained by weighted summation of the occupancy probability of each prediction step in the corresponding spatial sector of the neighbor node trajectory probability distribution matrix Π and the spatial distance estimate of the predicted trajectory of the current node, combined with the time step discount factor. The formation maintenance payoff is determined by the distance between the predicted position of the current node's alternative action and the centroid of the neighbor node's position, and the formation maintenance tendency component in the current node's task objective preference vector Θ. The weighting coefficients of the three payoff sub-items are determined and fixed after being determined by the multi-objective weight adaptive mechanism during the training phase.

[0024] Furthermore, the Nash equilibrium approximation solver is equipped with a maximum iteration count limit and a local timeout counter. When the iteration count reaches the limit and convergence is still not achieved, the current optimal response action is output and a non-convergence flag is sent to the action parsing execution unit. The action parsing execution unit applies a conservative scaling factor to the output speed command. When the solver triggers a timeout in multiple consecutive cycles, it adaptively increases the local maximum iteration count limit. The adjustment process is performed entirely locally at the node.

[0025] Furthermore, the system also includes a centralized training module that runs only during the training phase. The centralized training module consists of a global Critic network, an experience replay pool, and a parameter synchronization interface. During the training phase, a centralized training distributed execution framework is adopted. After training is completed, the centralized training module is taken offline, and each node only runs the local inference link to complete the distributed deployment.

[0026] Furthermore, the reward function of the centralized training module includes an intent validity reward component, using the probability value of the trajectory probability distribution matrix Π broadcast by the node in the sector corresponding to the actual executed action as the intent validity reward. Figure 1 The consistency metric is used to reward positive actions when the value is higher and penalize negative actions when the value is lower. The intention effectiveness reward is backpropagated to the trajectory probability distribution output head by the REINFORCE policy gradient method with the state value function estimated by the output of the global Critic network as the baseline, so that the trajectory probability distribution expressed by the intention tensor is highly consistent with the actual action.

[0027] Furthermore, the centralized training module adopts a phased training strategy. In the first phase, the weighted coefficients of the reward function are fixed, and multi-objective adaptive adjustment is performed only on the weighted coefficients of the three terms of the game payoff function. After convergence, the coefficients are solidified and written into each node. In the second phase, based on the solidified weighted coefficients of the payoff function, multi-objective adaptive adjustment is performed on the weighted coefficients of the three terms of the reward function. The two phases are executed sequentially to ensure that only one set of coefficients is in a dynamic adjustment state each time.

[0028] The beneficial effects achieved by this invention are:

[0029] This invention proposes a communication content design scheme for structured intent tensors. The trajectory probability distribution matrix of the agent and the task objective preference vector are concatenated into an intent tensor in a structured form with clear physical semantics for broadcasting. The receiver can directly substitute the elements of the intent tensor into the game payoff function without any decoding operation. This eliminates the decoding coupling layer between the communication module and the decision module in the existing hidden state sharing scheme, reduces the coupling dependency of the network structure between nodes, and improves the adaptability and robustness of the system in heterogeneous node deployment scenarios.

[0030] This invention proposes a game payoff construction mechanism based on intent tensor probability projection. The product of the projection probability of the neighbor node trajectory probability distribution on the sector corresponding to the candidate action of the current node and the neighbor's task objective preference component is used as the basis for measuring the task collaboration payoff. The uncertainty of the neighbor's intent is embedded in the payoff estimation in the form of a probability distribution. Compared with the existing schemes that construct payoff based on deterministic position prediction, it can provide a more robust compatibility estimate in high-dynamic scenarios where the neighbor's motion intent has a multi-peak distribution or high variance. It effectively avoids the problem of the game equilibrium point deviating from the optimal collaboration strategy due to the accumulation of prediction errors, and improves the collaboration stability of the cluster in a highly dynamic and unknown environment.

[0031] This invention proposes a hypothetical game iterative equilibrium solution mechanism that uses the probability distribution of the intent tensor as the initial belief. It directly uses the probability distribution of the first prediction step of the neighbor node intent tensor as the initial belief for choosing neighbor actions, replacing the uniform prior initialization method in the standard hypothetical game. This allows the equilibrium iteration process to start from a point closer to the real equilibrium, and obtain a more accurate equilibrium approximation under the same number of iterations. In the case of a unimodal concentration of the neighbor intent distribution, it can converge in a very small number of iterations, significantly reducing the computational latency of real-time decision-making and improving the real-time deployment capability of the system on computing-constrained nodes.

[0032] To further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the overall structural framework of the present invention;

[0034] Figure 2 This is a schematic diagram of the structured intent tensor composition of the present invention;

[0035] Figure 3 This is a schematic diagram illustrating the tensor broadcasting and neighborhood aggregation techniques of this invention.

[0036] Figure 4 This is a schematic diagram of the local game payoff construction mechanism of the present invention;

[0037] Figure 5 This is a schematic diagram of the centralized training and distributed execution framework of the present invention. Detailed Implementation

[0038] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of the present invention. Furthermore, the accompanying drawings of the present invention are for simple illustrative purposes only and are not depictions of actual dimensions; this is stated beforehand. The following embodiments will further describe the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of protection of the present invention.

[0039] Example 1.

[0040] This embodiment provides a cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning, combined with... Figure 1 It includes a local perception unit, a dual-head Actor policy network, an intent tensor encoder, a low-bandwidth broadcast module, a neighborhood intent reception and weighted aggregation module, a local game construction module, a Nash equilibrium approximation solver, and an action parsing execution unit. Each module is integrated into the onboard computing unit of each intelligent node. All nodes run the same module architecture, and the nodes are interconnected through a wireless ad hoc network.

[0041] The local sensing unit is connected to the input of the dual-head Actor policy network, and shares the location and velocity data of neighboring nodes with the neighborhood intent receiving and weighted aggregation module and the local game construction module.

[0042] The local sensing unit consists of an airborne inertial measurement unit, a ranging sensor, and a pose solver, and outputs a local observation vector. .

[0043] Local observation vector It is composed of three components pieced together:

[0044] The first group consists of the state components of node i itself, including the three-axis position coordinates, the three-axis axial velocities, the three-axis angular velocities, and the current remaining task quantity;

[0045] The second group consists of relative state components of neighboring nodes. For each neighboring node j in the neighborhood, its relative position vector and relative velocity vector are extracted, arranged from nearest to farthest distance, and then concatenated. If the number of neighbors is less than the preset maximum number of neighbors, zero vectors are used to fill the gap.

[0046] The third group is the environmental perception component, which arranges the distance values ​​measured by the ranging sensor in K sectors on the horizontal plane in sequence to reflect the proximity of obstacles in each direction.

[0047] The three components are concatenated to form a local observation vector of fixed dimensions. The data is fed into a dual-headed Actor policy network, where the local sensing unit outputs the absolute position coordinates of each neighbor node in the second component as a set of neighbor position vectors. The current velocity vector of each neighbor node is output separately as a group of neighbor velocity vectors. Both are updated uniformly at the beginning of each decision cycle, serving as a unified data source for the distance reciprocal weighted aggregation unit in the neighborhood intent reception and weighted aggregation module, the formation maintenance benefit calculation subunit and the collision risk penalty calculation subunit in the local game construction module, ensuring that the neighbor position and velocity data used by each module remain consistent within the same decision cycle.

[0048] Let i be the set of communication neighborhoods of node i;

[0049] Two-headed Actor policy networks use local observation vectors The backbone consists of a multi-layer fully connected network as input. The latent vector output by the backbone is fed into the trajectory probability distribution output head and the task target preference output head after passing through two independent gating units.

[0050] The gating unit consists of a single-layer linear transformation and sigmoid activation. It applies soft selection to the feature dimensions in the latent vector that are related to their respective output tasks. This makes the trajectory probability distribution output head focus on extracting the feature components in the latent vector that are related to motion direction prediction, while the task target preference output head focuses on extracting the feature components that are related to the current task state evaluation. The parameters of the two gating units are independent and are optimized separately through backpropagation during training. This achieves soft decoupling of the two types of output information while sharing the backbone features, and avoids mutual interference between the two output heads during gradient updates, which could lead to a decrease in the quality of one type of output.

[0051] The trajectory probability distribution output header consists of a linear layer and a row-wise softmax activation layer, outputting an H×K dimensional matrix. H represents the number of prediction time-domain steps, K represents the number of motion direction sectors uniformly divided on the horizontal plane, and matrix elements... This represents the probability that node i will move to sector k in the h-th step in the future, and the sum of each row is 1.

[0052] The task objective preference output head consists of a linear layer and a sigmoid activation layer, outputting an M-dimensional vector. Each component corresponds to the reconnaissance coverage weight, target tracking priority, formation maintenance tendency, and obstacle avoidance stress intensity, respectively. The value range of each component is [0,1], and they are independent of each other.

[0053] Combination Figure 2 The intent tensor encoder is connected to the two output heads of the dual-head Actor policy network and consists of a splicing layer and a structured quantization compression layer.

[0054] splicing layer will and Concatenate along the feature dimensions to form a structured intent tensor The structured intent tensor differs fundamentally from the hidden state vectors transmitted in existing multi-agent communication schemes in terms of information organization: the hidden state vector is a semantically non-semantic continuous feature of the intermediate layer of the policy network. The receiving node must use an additional decoding network to map it into an intermediate representation that can be used for decision-making. The decoding process not only introduces additional computational overhead, but also requires strong coupling between the network structure of the receiving and sending parties. Any change in the network structure of any node will cause the global decoding mechanism to fail.

[0055] Structured Intent Tensor Two subtensors and All have clear physical semantics. Each element directly corresponds to the occupancy probability of a specific time step and a specific direction of motion. Each component directly corresponds to the preference strength of a specific task objective, and the receiver node can obtain it without any decoding operation. The elements are directly substituted into the corresponding calculation positions of the game payoff function to realize the direct mapping between communication content and game decision input. This eliminates the decoding coupling layer between the communication module and the decision module in the hidden state sharing scheme, so that the node can immediately trigger game construction without additional reasoning steps after receiving the intent tensor.

[0056] The quantization bit width of the structured quantization compression layer is determined based on the numerical sensitivity of each sub-tensor element in the calculation of game payoffs.

[0057] right The probability value in the calculation directly participates in the multiplication and accumulation operations in the task collaboration benefit calculation subunit and the collision risk penalty calculation subunit. The probability difference between adjacent sectors is the key information for distinguishing the direction of node movement intention. Excessive quantization error will cause the probability projection values ​​of sectors corresponding to different candidate actions to tend to be the same, causing the benefit function to lose its ability to distinguish the direction of action. High bit-width fixed-point quantization is used to preserve fine-grained differences in the probability distribution.

[0058] right The preference value in the calculation is used as a task weight coefficient in the multiplication. Given that the preference value itself ranges from [0,1], the quantization error introduced by lower-width fixed-point quantization has a controllable impact on the ranking result of the payoff function and will not change the payoff relationship between different alternative actions. Low-bit-width fixed-point quantization is used.

[0059] The specific values ​​of the two bit widths are determined through offline simulation verification before system deployment. The verification method is to run game solving under both full-precision floating-point and quantized fixed-point conditions, compare the consistency rate of the optimal response action, and determine that the current bit width configuration meets the numerical stability requirements when the consistency rate is higher than the preset threshold.

[0060] The low-bandwidth broadcast module is connected to the output of the intent tensor encoder and consists of a frame encapsulation unit and a wireless RF transceiver. The frame encapsulation unit compresses the intent tensor. Encapsulated as a standard wireless data frame, the wireless RF transceiver broadcasts the data frame to all nodes in the communication neighborhood at the beginning of each decision cycle, while simultaneously receiving data frames sent by other nodes in the neighborhood.

[0061] The neighborhood intent receiving and weighted aggregation module is connected to the receiving end of the low-bandwidth broadcast module, and consists of an intent integrity verification unit, a decompression and restoration unit, a node validity management unit, and a distance reciprocal weighted aggregation unit.

[0062] The intent integrity verification unit performs frame verification on each received compressed intent tensor to determine whether the data frame is complete, and outputs the verification result to the node validity management unit.

[0063] The node validity management unit maintains a continuous missing period counter for each neighboring node j. The counter is incremented when the current frame verification of node j fails and reset to zero when the verification passes.

[0064] When the consecutive missing period count of node j does not exceed the preset upper limit, the intent tensor of the most recently successfully received and cached frame is used to replace the current frame for subsequent aggregation, and the corresponding weight coefficients are adjusted accordingly. Apply decay based on the number of consecutive missing periods;

[0065] When the count of consecutive missing cycles exceeds the preset limit, the node validity management unit sends the removal instruction of node j to the participant set determination unit after the neighborhood intent reception and weighted aggregation process of the current decision cycle is completed. The participant set determination unit can start the participant set determination operation only after receiving the complete removal instruction, so as to ensure that the removal instruction is fully effective before the game construction begins. At the same time, it sends the conservative obstacle avoidance flag of the location of node j to the action parsing execution unit.

[0066] The decompression and restoration unit confirms with the node validity management unit that the intent tensors of each neighboring node participating in the aggregation are restored to their original precision. .

[0067] Combination Figure 3 The neighbor location vector set output by the inverse distance weighted aggregation unit from the local sensing unit. Read the current position of each neighbor node j. Calculate the Euclidean distance between node i and node j. The weighting coefficient is calculated using the inverse of the distance. And normalize, and convert each tensor of the restoration intention to a normalized state. With corresponding weights Combine and output a weighted neighborhood intent set. ,in This is the set of valid neighbors after removing invalid nodes.

[0068] The local game construction module is connected to the output of the neighborhood intent receiving and weighted aggregation module, and consists of a kinematic prediction submodule, a participant set determination unit, a joint action space generation unit, and a payoff function calculation unit.

[0069] The kinematic prediction submodule employs a uniformly accelerated kinematic model for a point mass. It takes the current three-axis position and velocity of node i, along with the target direction and velocity magnitude corresponding to the candidate actions, as input. Acceleration is calculated by dividing the difference between the current velocity and the target velocity of the candidate action by a single control time step. The predicted position sequence of node i in the next H steps under each candidate action is calculated recursively at fixed time steps, forming a set of predicted trajectories. It is used jointly by the collision risk penalty calculation subunit and the formation maintenance benefit calculation subunit.

[0070] The participant set determination unit takes the current neighborhood set Nᵢ as its input and simultaneously receives removal instructions from the node validity management unit. After removing nodes with consecutive missing values ​​that exceed the limit, the set of valid participants is determined. .

[0071] The joint action space generation unit uses K directional sectors as the set of optional actions for each participant, and generates the Cartesian product of the action sets of each participant as the joint action space. .

[0072] After the set of participants and the joint action space are determined, the payoff function calculation unit calculates each candidate action. The task collaboration benefit calculation subunit, collision risk penalty calculation subunit, and formation maintenance benefit calculation subunit are called sequentially. After the three subunits are executed in sequence, their outputs are weighted and summed to obtain the complete benefit function value. After the payoff function is calculated, the payoff vector covering all candidate actions, along with the set of participants and the joint action space, are packaged together into a game model. The output is sent to the Nash equilibrium approximation solver. The solver can only start belief initialization after receiving the complete game model, ensuring that the payoff function calculation is completed before the belief initialization, thus eliminating the risk of data competition between the two.

[0073] Weighting coefficients of the three components of the payoff function , , In the first phase of intensive training, a multi-objective weight adaptive mechanism is used to dynamically adjust the weights. Based on the ratio of the historical gradient norms of the reward signals corresponding to each sub-item to the Actor network parameter updates, when the gradient norm of a certain sub-item is significantly smaller than that of other sub-items, the weight coefficient of that sub-item is appropriately increased. The final determination is made after the ratio of the gradient norms of the three sub-items tends to stabilize. , , The convergence and solidification processes are then written into the profit function calculation unit of each node.

[0074] Combination Figure 4 The task collaboration benefit calculation subunit uses the neighborhood intent set. The probability distribution of the trajectories of each neighbor node j With task objective preference vector For each alternative action at node i, take it as input. ,extract The first predicted walking vector is related to the action. Corresponding sector number probability value at With this probability value and The product of the reconnaissance coverage weight components Weighted summation, we get , For task collaboration benefits, this calculation method uses the projection probability of the neighbor node trajectory probability distribution onto the sector corresponding to the candidate action of the user as the measure of intent compatibility, rather than directly using the deterministic position prediction value of the neighbor node in the benefit calculation.

[0075] When the intentions of neighboring nodes are uncertain, the payoff calculation based on deterministic location prediction will result in a systematic bias in payoff estimation due to the accumulation of prediction errors, which will cause the game equilibrium point to deviate from the actual optimal cooperative strategy.

[0076] The probability projection-based calculation method embeds the uncertainty of the neighbor's intentions into the payoff estimation in the form of a probability distribution, making the payoff function value itself an expected measure of the uncertainty of the neighbor's true intentions. Even when the neighbor's intentions have a multi-peak distribution or high variance, it can still give a robust compatibility estimate, ensuring the stability of the game equilibrium point in a highly dynamic and unknown environment.

[0077] The introduction of the reconnaissance coverage weight component allows the compatibility metric to consider the intensity of the subjective preference of neighboring nodes for the current task objective, avoiding the spurious collaboration problem caused by judging collaboration solely based on the overlap of movement directions and ignoring task heterogeneity.

[0078] The collision risk penalty calculation sub-unit uses the probability distribution of all H-step trajectories of each neighbor node j. From the neighbor velocity vector group The current speed of node j read from the middle and the predicted trajectory of node i, the candidate action, output by the kinematic prediction submodule. For input.

[0079] For each prediction step h, with the current position of node j With speed The calculated position of node j at step h is obtained by recursion using the kinematic model of the particle. ,by Using the centroid coordinates of the corresponding sector spatial regions of K sectors as the center, we take the representative spatial position of that sector in step h. Calculate the predicted position of node i to represent the occupancy probability of that position. The Euclidean distance between any representative position and each representative position, when any representative position is equal to... When the distance is less than the safety spacing threshold, the corresponding occupancy probability is used. Multiplying the collision risk contribution by the time step discount factor and adding it to the penalty term, the output collision risk penalty is calculated. ;

[0080] This subunit is The probability-weighted approach estimates collision risk rather than just judging whether the highest probability sector is in conflict, so that low-probability but high-risk side approach paths can also be included in the penalty calculation, improving the comprehensiveness of obstacle avoidance decisions.

[0081] The group of neighbor position vectors output by the formation preservation benefit calculation subunit from the local sensing unit Read the current position of each neighbor node j. Calculate the centroids of neighboring nodes, and then predict trajectories based on the alternative actions output by the kinematic prediction submodule. The first step is to predict the Euclidean distance between the location and the centroid. Combine the preference vector of node i with the grouping to maintain the preference component. Output formation maintains benefits This component directly adjusts the actual weight of formation maintenance payoff in the overall payoff, allowing the strength of formation maintenance behavior to adaptively change with the node's own task state, rather than forcing all nodes to execute the same strength of formation constraints using a globally fixed weight. The local game construction module will integrate the complete game model. Output to the Nash equilibrium approximation solver.

[0082] The Nash equilibrium approximation solver is connected to the output of the local game construction module, upon receiving the complete game model. Then, the intent-aware belief initialization unit, the iterative optimal response calculation unit, and the belief update unit are started sequentially. Directly used as the initial belief for the selection of action j at node. Instead of using the standard fictional game initialization method that assigns uniform priors to all actions.

[0083] This initialization method utilizes the structured semantic information carried by the intent tensor to provide the solver with an initial belief that closely matches the actual intent distribution. This allows the iteration process to start from a point closer to the true equilibrium and obtain a more accurate equilibrium approximation than uniform initialization under the same number of iterations. In the case of a single-peak concentration of intent among neighboring nodes, it can converge within a very small number of iterations, significantly reducing the computational latency of real-time decision-making.

[0084] The iterative optimal response calculation unit calculates the expected return for each alternative action of node i under the current belief, and selects the action with the largest expected return as the optimal response for the current iteration round.

[0085] The belief update unit, according to the average hypothetical game rule, mixes the deterministic distribution corresponding to the optimal response into the beliefs of each neighbor and updates them. The above three units execute cyclically until the convergence condition is met, and then output the final optimal response action. .

[0086] The Nash equilibrium approximation solver uses the similarity between the optimal response actions in two consecutive iterations as a convergence criterion. Specifically, the optimal response actions output in the l-th iteration are identical to those in the (l-1)-th iteration. and When the results are consistent, the iteration is considered converged, the loop is terminated, and the result is output. As the final optimal response action .

[0087] The solver has a maximum limit on the number of iterations. If the solver fails to converge after reaching this limit, a timeout mechanism is triggered: the solver outputs the optimal response for the current iteration. As an approximate result, a non-convergence flag is simultaneously sent to the action parsing and execution unit.

[0088] The solver maintains a local timeout counter. When timeouts are triggered for multiple consecutive cycles, the solver adaptively increases the maximum number of local iterations until timeouts no longer occur consecutively or the hardware computing power constraint boundary is reached. This adaptive adjustment process is executed entirely locally on the node and does not depend on any external modules.

[0089] The action analysis execution unit is connected to the output of the Nash equilibrium approximation solver and consists of a flag priority arbitration subunit, a heading analysis subunit, and a flight control interface.

[0090] The priority arbitration subunit simultaneously receives the conservative obstacle avoidance flag sent by the node validity management unit and the non-convergence flag sent by the Nash equilibrium approximation solver. The two types of constraints are executed independently and superimposed.

[0091] The non-convergence flag triggers the global velocity scaling factor, uniformly scaling the output velocity command in all directions;

[0092] The conservative obstacle avoidance sign triggers a azimuth margin constraint, which only imposes additional restrictions on the velocity component in the azimuth direction specified by the sign. The two types of constraints do not overlap and both take effect when triggered simultaneously.

[0093] The heading analysis subunit will determine the optimal response action. The corresponding directional sector center angle is combined with the current speed of node i and parsed into speed command and yaw angle command. After being constrained by the flag priority arbitration sub-unit, the command is sent from the flight control interface to the underlying flight controller driver node to execute the motion control of the current cycle.

[0094] The system also includes a centralized training module, which runs only during the training phase and consists of a global Critic network, an experience replay pool, and a parameter synchronization interface.

[0095] The global Critic network takes the concatenation of the joint observation vectors and joint actions of all nodes as input, outputs the global state-action value function Q, updates the Critic parameters through temporal difference error, and calculates the policy gradient of the Actor network of each node using the Critic output.

[0096] Combination Figure 5 The centralized training module adopts a phased training strategy.

[0097] Phase 1 Fixed Global Reward Weighting Coefficient , , As preset initial values, only the weighting coefficients of the return function are applied. , , Perform multi-objective adaptive adjustment, dynamically adjusting the three coefficients based on the ratio of the historical gradient norm of the reward signal corresponding to each sub-item to the Actor network parameter update. The final determination is made after the gradient norm ratio tends to stabilize. , , The convergence and solidification processes are then written into the profit function calculation unit of each node.

[0098] The second stage is based on the cured product. , , Based on, and then to , , Multi-objective adaptive adjustment is performed, dynamically adjusting based on the ratio of the historical gradient norms of the three reward signals to the global Critic network parameter updates. The final determination is made after the gradient norm ratios stabilize. , , Converge and solidify.

[0099] The two-stage sequential execution ensures that only one set of coefficients is in a dynamic adjustment state at any given time, eliminating the risk of training instability caused by the mutual influence of the two sets of coefficients.

[0100] The reward function of the centralized training module consists of three components: task coverage reward, collision penalty, and intent effectiveness reward. These three reward signals are calculated once in each decision cycle and then weighted and summed to form the global reward. .

[0101] Task Coverage Rewards The calculation is based on the number of reconnaissance grids actually newly covered by the cluster in the current period. No reward is given for repeated coverage of already reconnaissance areas, and a reward multiplication coefficient is applied to cover high-priority target areas, so that the cluster spontaneously forms a task allocation strategy that prioritizes the coverage of high-value areas during the training process.

[0102] Collision Penalty Triggered when the actual distance between any two nodes is less than the safe distance threshold, a fixed negative value is taken, and a penalty is applied to both nodes that collide, so that both Actor networks are subject to gradient update constraints, avoiding the attribution bias of obstacle avoidance responsibility caused by unilateral penalty.

[0103] Intention effectiveness reward The communication quality of the intent tensor is used to explicitly optimize the intent tensor. During each decision epoch in the training phase, the intent tensor of node i... Generated and broadcast at the start of the cycle, the Nash equilibrium approximation solver outputs the optimal response action after completing the game solution within the cycle. The experience replay pool will recycle all data from the same period at the end of the cycle. and As part of the pairing data, the reward settlement module reads the pairings simultaneously when sampling from the experience replay pool. and ,extract In the first predicted walking vector Probability value of corresponding sector As a meaning Figure 1 Consistency measure; a higher value gives a positive reward, and a lower value gives a penalty.

[0104] The gradient propagation of the intention effectiveness reward on the trajectory probability distribution output head of the two-headed Actor policy network adopts the REINFORCE policy gradient method. The logarithm of the intention effectiveness reward is multiplied by the difference between the state value function estimate of the global Critic network output as the weighting coefficient of the policy gradient. This bypasses the non-differentiability problem in the Nash equilibrium solution process, enabling the intention effectiveness reward to effectively guide the trajectory probability distribution output head to improve the prediction probability of the actual action and prevent the intention tensor from degenerating into redundant communication load that is unrelated to the actual action.

[0105] The two output heads of the dual-head Actor policy network are jointly updated end-to-end through policy gradients. The generation quality of the intent tensor is directly optimized by the backpropagation of task reward, collision penalty and intent effectiveness reward, so that the trajectory probability distribution and task preference vector expressed by the intent tensor gradually converge into a cooperative signal that is truly effective for decision-making of neighboring nodes during the training process.

[0106] After training is completed, the centralized training module is taken offline, and each node only runs the complete inference link from the local perception unit to the action parsing and execution unit, completing distributed deployment and real-time collaborative decision-making without the intervention of any central scheduling node.

[0107] The content disclosed above is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent technical changes made based on the content of the present invention specification and drawings are included within the scope of protection of the present invention. Furthermore, the elements therein can be updated as technology develops.

Claims

1. A cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning, characterized in that, It includes a dual-head Actor policy network, an intent tensor encoder, a low-bandwidth broadcast module, a neighborhood intent reception and weighted aggregation module, a local game construction module, and a Nash equilibrium approximation solver, all integrated into the onboard computing unit of each intelligent node. The dual-head Actor strategy network takes the node's local observation vector as input, and its output is connected to both the trajectory probability distribution output head and the task target preference output head. The trajectory probability distribution output head outputs the trajectory probability distribution matrix Π, where each element of the matrix represents the probability that a node moves to the corresponding sector in the corresponding prediction time step. The task target preference output head outputs the task target preference vector Θ, where each component corresponds to the preference intensity of different task targets. The intent tensor encoder concatenates the trajectory probability distribution matrix Π with the task target preference vector Θ to form a structured intent tensor Φ. Each element of the structured intent tensor Φ has a clear physical semantics. After compression encoding, it is broadcast to each node in the neighborhood through a low-bandwidth broadcast module. The receiving node can directly substitute the elements of the structured intent tensor Φ into the game payoff function for calculation without decoding. The neighborhood intent receiving and weighted aggregation module receives the structured intent tensor broadcast by neighboring nodes, decompresses and restores it, and constructs a weighted neighborhood intent set by weighting it according to the inverse of distance. The local game construction module constructs a local non-cooperative game model with a weighted neighborhood intention set as the trigger input. The task collaboration payoff in the game payoff function is obtained by weighted summation of the probability projection value of the neighbor node trajectory probability distribution matrix Π on the sector corresponding to the candidate action of the current node and the corresponding component of the neighbor node task objective preference vector Θ. The Nash equilibrium approximation solver directly uses the probability distribution of the first predicted step trajectory in the structured intention tensor of each neighbor node as the initial belief for the action selection of the neighbor node, and solves the Nash equilibrium using the hypothetical game iterative method and outputs the optimal response action.

2. The cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning as described in claim 1, characterized in that, The backbone output of the dual-head Actor policy network is connected to two independent gating units. Each gating unit consists of a single-layer linear transformation and sigmoid activation. The two gating units apply soft selection to the feature dimensions related to motion direction prediction and task state evaluation in the backbone latent vector, respectively, and then send them to the trajectory probability distribution output head and the task target preference output head. The parameters of the two gating units are independent, realizing soft decoupling of the two types of output information under the condition of sharing backbone features.

3. The cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning as described in claim 2, characterized in that, The intent tensor encoder employs differentiated quantization bit widths for compression encoding. It uses higher bit width fixed-point quantization for the probability values ​​in the trajectory probability distribution matrix Π and lower bit width fixed-point quantization for the preference values ​​in the task target preference vector Θ. The quantization bit width is determined based on the numerical sensitivity of each sub-tensor element in the game payoff calculation, so as to ensure that the accuracy of the decompression and restoration after compression meets the numerical stability requirements of the game payoff calculation.

4. The cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning as described in claim 3, characterized in that, The neighborhood intent receiving and weighted aggregation module also includes an intent integrity verification unit and a node validity management unit. The node validity management unit maintains a counter for the consecutive missing periods of each neighbor node. When the number of consecutive missing periods of a neighbor node does not exceed the preset upper limit, it is replaced with a cached frame and weight decay is applied. When it exceeds the preset upper limit, the node is removed from the game participant set of the current decision period and a conservative obstacle avoidance flag in the corresponding direction is sent to the action parsing and execution unit.

5. The cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning as described in claim 4, characterized in that, The local game construction module also includes a kinematic prediction submodule, which takes the node's current pose, velocity, and each alternative action as input, and calculates the predicted trajectory of the node in each prediction step under each alternative action step by step according to the uniformly accelerated kinematic model of the mass. This is used by the collision risk penalty calculation subunit and the formation maintenance benefit calculation subunit.

6. The cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning as described in claim 5, characterized in that, The game payoff function of the local game construction module also includes two sub-items: collision risk penalty and formation maintenance payoff. The collision risk penalty is obtained by weighted summation of the occupancy probability of each prediction step in the corresponding spatial sector of the neighbor node trajectory probability distribution matrix Π and the spatial distance estimate of the predicted trajectory of the current node, combined with the time step discount factor. The formation maintenance payoff is determined by the distance between the predicted position of the current node's alternative action and the centroid of the neighbor node's position, and the formation maintenance tendency component in the current node's task objective preference vector Θ. The weighting coefficients of the three payoff sub-items are determined and fixed after being determined by the multi-objective weight adaptive mechanism during the training phase.

7. A cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning as described in claim 6, characterized in that, The Nash equilibrium approximation solver has a maximum iteration count limit and a local timeout counter. When the iteration count reaches the limit and convergence is still not achieved, the current optimal response action is output and a non-convergence flag is sent to the action parsing execution unit. The action parsing execution unit applies a conservative scaling factor to the output speed command. When the solver triggers a timeout in multiple consecutive cycles, it adaptively increases the local maximum iteration count limit. The adjustment process is performed entirely locally at the node.

8. A cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning as described in claim 7, characterized in that, The system also includes a centralized training module that runs only during the training phase. The centralized training module consists of a global Critic network, an experience replay pool, and a parameter synchronization interface. During the training phase, a centralized training distributed execution framework is adopted. After training is completed, the centralized training module is taken offline, and each node only runs the local inference link to complete the distributed deployment.

9. A cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning as described in claim 8, characterized in that, The reward function of the centralized training module includes an intent validity reward component. The probability value of the trajectory probability distribution matrix Π broadcast by the node in the sector corresponding to the actual action is used as a measure of intent consistency. The higher the value, the positive reward is given, and the lower the value, the penalty is given. The intent validity reward is backpropagated to the trajectory probability distribution output head through the REINFORCE policy gradient method with the state value function estimated by the output of the global Critic network as the baseline, so that the trajectory probability distribution expressed by the intent tensor is highly consistent with the actual action.

10. A cluster intent sharing and distributed collaborative game system based on multi-agent deep reinforcement learning as described in claim 9, characterized in that, The centralized training module adopts a phased training strategy. In the first phase, the weighted coefficients of the reward function are fixed, and multi-objective adaptive adjustment is performed only on the weighted coefficients of the three terms of the game payoff function. After convergence, the coefficients are solidified and written into each node. In the second phase, based on the solidified weighted coefficients of the payoff function, multi-objective adaptive adjustment is performed on the weighted coefficients of the three terms of the reward function. The two phases are executed sequentially to ensure that only one set of coefficients is in a dynamic adjustment state each time.

Images