An untrusted multi-agent implicit collaboration method and system based on space-time reasoning and adversarial reconstruction

Abstract

This invention discloses an implicit multi-agent cooperative method and system based on spatiotemporal reasoning and adversarial reconstruction. First, at each time step, a graph attention network is used to encode the instantaneous spatial relationships between the agent and its neighbors, generating instantaneous spatial relationship embeddings. Second, the sequence of instantaneous spatial relationship embeddings is input into a GTrXL network, utilizing its recurrent memory units to capture long-term temporal dependencies and infer the agent's latent intentions. Next, adversarial self-supervised reconstruction training is performed, reconstructing randomly masked neighbor state information through a decoder, forcing the model to learn standardized behavioral patterns. Finally, a joint loss function is constructed based on a multi-agent proximal policy optimization algorithm to jointly optimize the network parameters. This invention employs multi-agent proximal policy optimization for training, achieving efficient training and distributed execution through Critic and Actor networks, enabling robust implicit cooperation without explicit communication.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an implicit collaborative method and system for untrusted multi-agent systems based on spatiotemporal reasoning and adversarial reconstruction. Background Technology

[0002] With the development of unmanned aerial vehicle (UAV) technology, UAV crowdsensing has become an efficient data collection paradigm. In UAV crowdsensing tasks, multiple UAVs need to collaborate to collect data on points of interest (POIs) on the ground. Multi-agent reinforcement learning (MARL) is a commonly used technique for solving such collaborative optimization problems.

[0003] However, most existing MARL frameworks are based on a core assumption: that the environment is fully cooperative, or that agents can communicate explicitly with each other over high bandwidth via reliable channels (e.g., broadcasting their respective intentions or plans, i.e., “cheap talk”). This assumption often does not hold true in real-world, large-scale commercial deployments.

[0004] In untrusted environments, agents may be controlled by different, selfish entities. This leads to two core challenges:

[0005] 1. Selfish and deceptive behavior: Selfish agents may employ deceptive strategies to maximize their own interests (such as seizing high-value tasks), such as executing false trajectories to deceive competitors or hiding discovered task information.

[0006] 2. Unreliable communication: In real-world environments, communication between UAVs may be limited by bandwidth, latency, or interference, making it impossible to support high-frequency, complex intent exchanges.

[0007] In this environment, the MARL method, which relies on cheap conversation, will collapse due to receiving false information. Therefore, there is an urgent need for an "implicit collaboration" method that does not rely on explicit communication but infers the intentions of others by observing their behavior (trajectories, actions).

[0008] Existing implicit cooperation methods often only consider spatial or temporal information in a one-sided manner. For example, some graph neural network-based methods can model the current spatial relationships between agents well, but cannot capture their historical behavioral patterns, and therefore cannot distinguish whether an action is an immediate reaction or part of long-term planning. Other methods based on recurrent neural networks can capture temporal information, but struggle to handle dynamically changing, unstructured neighbor relationships.

[0009] Therefore, the present invention aims to solve how to integrate spatiotemporal information in an untrusted environment to achieve robust implicit collaboration and resist interference from deceptive behavior. Summary of the Invention

[0010] To overcome the reliance of existing MARL frameworks on assumptions of reliable communication and cooperation, this invention provides a method and system for untrusted multi-agent cooperation based on spatiotemporal reasoning and adversarial reconstruction. This invention enables agents to infer their underlying intentions simply by observing the behavioral trajectories of their neighbors, without the need for inexpensive conversations, and maintains efficient and robust collaborative decision-making capabilities in untrusted environments where deceptive behavior exists.

[0011] In a first aspect, the present invention provides an untrusted multi-agent collaborative method based on spatiotemporal reasoning and adversarial reconstruction, comprising the following steps:

[0012] Spatial Relationship Reasoning: At each time step, each agent acquires local observation information, constructs a neighborhood graph centered on itself, aggregates neighbor node information using a graph attention network, and combines edge feature encoding to generate the agent's instantaneous spatial relationship embedding.

[0013] Temporal intent reasoning: The instantaneous spatial relationship is embedded into the GTrXL network, and the time series is modeled through a recurrent memory mechanism to generate the final intent embedding;

[0014] Strategy and Value Generation: The final intent is embedded into the Actor network to generate the execution strategy of each agent; and the final intents of each agent are embedded and concatenated to form a global state representation, which is then input into the Critic network to generate a centralized value estimate of the current state.

[0015] Adversarial Reconstruction Training: During the training phase, adversarial self-supervised reconstruction training is completed through initial observation, masking, encoding, state reconstruction, and loss calculation to enhance robustness against anomalous behavior;

[0016] Joint optimization: Based on the multi-agent proximal policy optimization algorithm, a joint loss function is constructed to jointly optimize the network parameters.

[0017] Secondly, the present invention also provides an untrusted multi-agent cooperative system based on spatiotemporal reasoning and adversarial reconstruction, comprising:

[0018] The spatial relationship reasoning module is used to acquire local observation information at each time step, construct a neighborhood graph with itself as the center, aggregate neighbor node information using a graph attention network, and combine edge feature encoding to generate the instantaneous spatial relationship embedding of the agent.

[0019] The temporal intent reasoning module is used to embed the instantaneous spatial relationship into the input GTrXL network, model the time series through a recurrent memory mechanism, and generate the final intent embedding.

[0020] The strategy and value generation module is used to embed the final intent into the input Actor network to generate the execution strategy of each agent; and to splice the embedded final intents of each agent to form a global state representation input into the Critic network to generate a centralized value estimate of the current state.

[0021] The adversarial reconstruction training module is used to complete adversarial self-supervised reconstruction training during the training phase through initial observation, masking, encoding, state reconstruction and loss calculation, thereby enhancing robustness to anomalous behavior.

[0022] The model optimization module is used to construct a joint loss function based on the multi-agent proximal policy optimization algorithm to jointly optimize network parameters.

[0023] Thirdly, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described implicit cooperative method for untrusted multi-agent agents based on spatiotemporal reasoning and adversarial reconstruction.

[0024] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described implicit cooperative method for untrusted multi-agent systems based on spatiotemporal reasoning and adversarial reconstruction.

[0025] The beneficial effects of this invention are:

[0026] High robustness: By introducing an adversarial self-supervised reconstruction task, this invention forces the model to learn the canonical spatiotemporal patterns of the environment, enabling it to see through or ignore the deceptive behavior of its neighbors and maintain a high task completion rate in untrusted environments.

[0027] Implicit collaboration: This invention does not rely on explicit communication between agents at all, but rather infers intent from observed historical trajectories through the GAT-GTrXL architecture, thus achieving true implicit collaboration.

[0028] Spatiotemporal Fusion: This invention innovatively links space and time together. GAT processes who is nearby now, while GTrXL processes past behavioral patterns. The combination of the two enables the agent to perform high-order reasoning about the future intentions of its neighbors.

[0029] High efficiency and scalability: This invention employs a centralized training and decentralized execution architecture, ensuring training stability, convergence speed, and scalability during deployment. GTrXL's serialized batch processing design enables it to efficiently process long sequences, making it suitable for complex long-term tasks. Attached Figure Description

[0030] Figure 1A schematic diagram of the overall architecture of the untrusted multi-agent cooperative system provided by the present invention;

[0031] Figure 2 The core structure diagram of the STARNet (GAT-GTrXL) Actor-Critic provided by this invention;

[0032] Figure 3 A flowchart illustrating the adversarial self-supervised reconstruction task provided by this invention;

[0033] Figure 4 The flowchart for MAPPO training and serialization update provided by this invention. Detailed Implementation

[0034] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0035] This invention proposes an implicit collaborative method for untrusted multi-agent systems based on spatiotemporal reasoning and adversarial reconstruction. It employs an end-to-end MARL architecture (STARNet) that integrates spatiotemporal reasoning, deployed on each agent. The method comprises the following steps:

[0036] S1: Spatial Relationship Reasoning

[0037] At each time step, each agent acquires local observation information and constructs a neighborhood graph centered on itself. It then aggregates neighbor node information through a graph attention network (GAT) and combines it with edge feature encoding to generate the agent's instantaneous spatial relationship embedding. Unlike traditional graph neural networks, this embodiment introduces a dedicated edge feature encoder into the GAT, explicitly mapping dynamic interaction features such as the relative position of neighbors and the agent itself, and remaining battery power, to a high-dimensional space. Utilizing a multi-head attention mechanism, the model can dynamically assign higher weights to spatially more relevant or critically important neighbors (e.g., those with low battery power requiring collaborative updates) based on current task requirements, thereby accurately capturing instantaneous interaction features in complex local environments.

[0038] Furthermore, GAT employs a query-key-value (QKV) multi-head attention mechanism. GAT's attention input features include: a query (Q) vector generated from the agent's own node features, key (K) and value (V) vectors generated from neighboring node features, and edge embeddings encoded by edge features (EdgeEncoder). The edge features include at least: the relative position of neighboring agents to the current agent, and the remaining battery power of neighboring agents.

[0039] Furthermore, the spatial reasoning module receives the encoded node features. and edge embedding The spatial reasoning module first... and Perform QKV transformation; then perform the query vector Key vector With edge embedding The attention weights are obtained through concatenation and multi-head attention calculation. Finally, the value vector of the neighbors is weighted and aggregated. Output instantaneous spatial relationship embedding ,

[0040] S2: Temporal Intent Reasoning

[0041] Furthermore, in this embodiment of the invention, the instantaneous spatial relationship embedding is fed into a gated Transformer-XL (GTrXL) network. The GTrXL network utilizes its recurrent memory module to model the temporal series of the spatial relationship embedding, generating a final intent embedding that incorporates long-term temporal dependencies. Addressing the highly dynamic nature of multi-agent environments, this embodiment of the invention uses GTrXL instead of a traditional RNN, combining the parallel computing power of the Transformer with the long-range modeling capabilities of the recurrent memory unit. By capturing the historical trajectories of neighboring behaviors, the agent can perform high-order reasoning about the "motivation" of its neighbors (e.g., determining whether their trajectory is heading towards a task point or a deceptive inducement). Therefore, the gated unit effectively alleviates the gradient vanishing problem in deep networks when processing long sequences, ensuring the stable fusion of temporal features.

[0042] Furthermore, in the temporal intent reasoning step, the GTrXL network receives the recurrent memory state of the previous time step while processing the input of the current time step; after the self-attention calculation, the GTrXL network additionally includes a gating unit to dynamically adjust the information flow and fuse the output with the original input through residual connections and layer normalization.

[0043] Furthermore, the GTrXL network supports serialized batch processing. During training, continuous time-series batches are sampled from the experience replay pool, and the GTrXL network processes the entire sequence in parallel, achieving efficient backpropagation BPTT training.

[0044] S3: Strategy and Value Generation

[0045] Furthermore, S3 achieves collaboration through decentralized execution and centralized evaluation. First, the final intent is embedded and input into a decentralized Actor network to generate the execution policies (i.e., action probability distributions) of each agent. Then, the embedded final intents of all agents are concatenated to form a global state representation, which is input into a centralized Critic network to generate a centralized value estimate of the current state. This "centralized training, decentralized execution" architecture ensures that agents can make decisions based solely on local observations during the deployment phase, solving the problem of independent execution under unreliable communication. Simultaneously, accurate credit allocation through global information during the training phase improves the stability of policy convergence.

[0046] S4: Adversarial Self-Supervised Reconstruction

[0047] Furthermore, in the training phase, S4 introduces an auxiliary self-supervised reconstruction task to enhance the system's robustness to anomalous behavior, specifically including:

[0048] (a) Initial observation: agent Observing neighbors and The complete state;

[0049] (b) Random masking: Random masking is applied to the neighbor state features (such as relative position and power) in the local observation information of the agent;

[0050] (c) Encoding: The masked local observation information is input into the GAT network and the GTrXL network to generate an intent embedding containing missing information;

[0051] (d) State reconstruction: The intent containing missing information is embedded into a decoder network to reconstruct the masked neighbor state features;

[0052] (e) Loss calculation: Calculate the mean square error reconstruction loss between the reconstructed value and the true value.

[0053] Specifically, the adversarial self-supervised reconstruction is an adversarial defense mechanism that forces the model to recover neighbor states from incomplete information. The model must learn the canonical physical constraints of the environment and the inherent cooperative patterns among agents. This enables the model to "recognize" deceptive trajectories that do not conform to the canonical patterns it has learned, thus providing strong resistance to interference when facing untrusted neighbors.

[0054] S5: Joint Optimization

[0055] Furthermore, this embodiment of the invention constructs and optimizes a joint loss function based on the Multi-Agent Proximity Policy Optimization (MAPPO) algorithm. The following steps are performed when training the MAPPO algorithm:

[0056] Data collection: The agent interacts with the environment for rollout_len steps and stores observations, actions, rewards, completion states, value estimates, action log probabilities, and recurrent memory states in the experience replay pool;

[0057] GAE Calculation: After collecting rollout_len steps of data, the generalized advantage estimate (GAE) and returns are calculated based on the value estimate of the centralized Critic.

[0058] Serialization sampling: To adapt to the BPTT requirements of GTrXL, continuous, fixed-length trajectory sequences are sampled from the empirical playback pool, instead of randomly sampling single frames;

[0059] PPO Epoch and Forward Propagation: Perform ppo_epochs iterations. In each iteration: In reconstruction mode, the model performs forward propagation on the sequence batches, obtaining the action log probabilities, state value estimates, reconstructed neighbor edge features, and original neighbor edge features for the entire sequence at once.

[0060] Joint loss calculation: Constructing the joint loss function The joint loss function includes the PPO loss (including policy loss) calculated based on the Actor network and the Critic network. Value loss and entropy reward ), and the reconstruction loss The formula is as follows:

[0061]

[0062] Backpropagation and Update: Based on Joint Loss Perform backpropagation and use the gradient descent optimizer to update the parameters of all modules end-to-end;

[0063] Clear cache: Clear the experience replay pool and proceed to the next round of data collection.

[0064] Furthermore, this embodiment of the invention performs end-to-end joint optimization of all parameters of GAT, GTrXL, Actor, Critic, and Decoder using gradient descent. This multi-task learning paradigm unifies "cooperative effectiveness" and "representation robustness" in the same update process, ensuring that the model maintains accurate and robust representation of environmental information while pursuing high task rewards.

[0065] To clearly illustrate the present invention, the following detailed description of the implicit collaborative mechanism of UAV crowdsourcing based on STARNet, in conjunction with specific embodiments, is provided.

[0066] This embodiment is implemented in a UAV crowdsourcing environment, which contains N agents (UAVs) and K points of interest (POIs). The agents need to collaborate to cover all POIs as quickly as possible and with minimal energy consumption. The environment is untrusted; some agents may be configured as deceitful actors, executing false trajectories.

[0067] Furthermore, referring to Figure 1 The system architecture in this embodiment consists of STARNet models deployed on N agents and a centralized trainer in the cloud. During the execution phase, each agent relies solely on its local observations (such as its own state, sensor views, and observable neighbors) to input into its local STARNet model and output actions to the environment. During the training phase, the trainer collects data from the experience replay pools of all agents, centrally calculates gradients, and updates the model parameters of all agents.

[0068] Furthermore, referring to Figure 2 This is the core model of STARNet in this embodiment. The model of each agent consists of the following parts:

[0069] 1. Observation input:

[0070] Furthermore, intelligent agents exist Acquire local observation information at all times It is divided into: own_feats (location, battery level), neighbor_feats (relative location, battery level), task_feats, local grid, and legal action mask.

[0071] 2. Feature encoding:

[0072] Furthermore, in the feature encoding module, the original local observation information Node features are obtained through a linear encoder. The neighbor relationship information in ally_feats is fed into the edge feature encoder, which outputs the edge embedding. .

[0073] 3. Spatial Reasoning:

[0074] Furthermore, and It is sent to the Spatial Reasoning (MultiHeadGAT) module. This module first... and Perform QKV transformation; then perform the query vector Key vector With edge embedding The attention weights are obtained through concatenation and multi-head attention calculation. Finally, the value vector of the neighbors is weighted and aggregated. Output instantaneous spatial relationship embedding The formula is as follows:

[0075]

[0076] 4. Temporal Intent Reasoning

[0077] Furthermore, in the temporal intent reasoning module, It is fed into the GTrXL module. This module receives... and the memory of the previous moment Its GTrXL internally performs self-attention computation based on long-range memory and updates it through gating units, ultimately outputting the final intent embedding. and new memory states The formula is as follows:

[0078]

[0079] 5. Output head:

[0080] Furthermore, the output head includes:

[0081] Actor: Input into actor_head and output the action probability. ;

[0082] Critic: During training, all agents Collected and pieced together . It is fed into critic_head, which outputs the centralized value. ;

[0083] Decoder: In refactoring mode The information is fed into the decoder network and outputs the reconstructed neighbor information. .

[0084] Furthermore, referring to Figure 3 This embodiment employs an adversarial self-supervised reconstruction mechanism, which includes:

[0085] Initial observation: agent Observing neighbors and The complete state;

[0086] Masking: During training, the system randomly masks its neighbors with a probability of reconstruction_prob. Information;

[0087] Encoding: The STARNet core processes this corrupted observation and generates an intent embedding. ;

[0088] Reconstruction: Decoder Receives Its goal is to output the masked neighbors. Reconstruction of the original state ;

[0089] Loss Calculation: Calculate the reconstruction loss .

[0090] Specifically, in order to minimize The STARNet core must learn from context (i.e., the agent) and Inferring from the behavior and physical laws of the environment This is the proper state. When a real fraudster performs an abnormal action that is inconsistent with the context, this mechanism allows the model to better identify and filter such anomalous information, thereby maintaining the robustness of its own decision-making.

[0091] Furthermore, referring to Figure 4 The MAPPO training process according to an embodiment of the present invention specifically includes the following steps:

[0092] Data collection: The agent interacts with the environment for rollout_len steps and stores observations, actions, rewards, completion states, value estimates, action log probabilities, and recurrent memory states in the experience replay pool;

[0093] GAE Calculation: After collecting rollout_len steps of data, the generalized advantage estimate (GAE) and returns are calculated based on the value estimate of the centralized Critic.

[0094] Serialization sampling: To adapt to the BPTT requirements of GTrXL, continuous, fixed-length trajectory sequences are sampled from the empirical playback pool, instead of randomly sampling single frames;

[0095] PPO Epoch and Forward Propagation: Perform ppo_epochs iterations. In each iteration: In reconstruction mode, the model performs forward propagation on the sequence batches, obtaining the action log probabilities, state value estimates, reconstructed neighbor edge features, and original neighbor edge features for the entire sequence at once.

[0096] Joint loss calculation: PPO loss. : PPO trimming target loss. Centralized value With returns MSE loss. Strategy entropy. : Reconstruction loss. ;

[0097] Backpropagation and Update: Based on Joint Loss Perform backpropagation. Update the parameters of all modules end-to-end using the gradient descent optimizer;

[0098] Clear cache: Clear the experience replay pool and proceed to the next round of data collection.

[0099] Through the above steps, the method and system of the present invention can achieve efficient and robust multi-agent implicit collaboration in an untrusted environment without explicit communication by fusing spatial information through GAT, reasoning temporal intent through GTrXL, and resisting deceptive behavior through adversarial reconstruction tasks.

[0100] In one exemplary embodiment, an untrusted multi-agent implicit cooperative system based on spatiotemporal reasoning and adversarial reconstruction is also provided, comprising:

[0101] The spatial relationship reasoning module is used to acquire local observation information at each time step, construct a neighborhood graph with itself as the center, aggregate neighbor node information using a graph attention network, and combine edge feature encoding to generate the instantaneous spatial relationship embedding of the agent.

[0102] The temporal intent reasoning module is used to embed the instantaneous spatial relationship into the input GTrXL network, model the time series through a recurrent memory mechanism, and generate the final intent embedding.

[0103] The strategy and value generation module is used to embed the final intent into the input Actor network to generate the execution strategy of each agent; and to splice the embedded final intents of each agent to form a global state representation input into the Critic network to generate a centralized value estimate of the current state.

[0104] The adversarial reconstruction training module is used to complete adversarial self-supervised reconstruction training during the training phase through initial observation, masking, encoding, state reconstruction and loss calculation, thereby enhancing robustness to anomalous behavior.

[0105] The model optimization module is used to construct a joint loss function based on the multi-agent proximal policy optimization algorithm to jointly optimize network parameters.

[0106] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described implicit cooperative method for untrusted multi-agent systems based on spatiotemporal reasoning and adversarial reconstruction.

[0107] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described implicit cooperative method for untrusted multi-agent systems based on spatiotemporal reasoning and adversarial reconstruction.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for implicit cooperation among untrusted multi-agent agents based on spatiotemporal reasoning and adversarial reconstruction, characterized in that, Includes the following steps: Spatial Relationship Reasoning: At each time step, each agent acquires local observation information, constructs a neighborhood graph centered on itself, aggregates neighbor node information using a graph attention network, and combines edge feature encoding to generate the agent's instantaneous spatial relationship embedding. Temporal intent reasoning: The instantaneous spatial relationship is embedded into the GTrXL network, and the time series is modeled through a recurrent memory mechanism to generate the final intent embedding; Strategy and value generation: The final intent is embedded into the input Actor network to generate the execution strategy for each agent; The final intentions of each agent are embedded and spliced ​​together to form a global state representation, which is then input into the Critic network to generate a centralized value estimate of the current state. Adversarial Reconstruction Training: During the training phase, adversarial self-supervised reconstruction training is completed through initial observation, masking, encoding, state reconstruction, and loss calculation to enhance robustness against anomalous behavior; Joint optimization: Based on the multi-agent proximal policy optimization algorithm, a joint loss function is constructed to jointly optimize the network parameters.

2. The method according to claim 1, characterized in that, The graph attention network adopts a query-key-value based multi-head attention mechanism, and its input features include: query vectors generated by the agent's own node features, key and value vectors generated by the features of neighboring nodes, and edge embeddings generated by edge feature encoding.

3. The method according to claim 2, characterized in that, When embedding the node features and edges into the input graph attention network, the following steps are performed for spatial relationship reasoning: First, QKV transformation is performed on node features and edge embeddings; Secondly, the query vector, key vector, and edge embedding are concatenated; Next, attention weights are calculated using multi-head attention; Finally, the value vectors of the neighbors are weighted and aggregated to output the instantaneous spatial relationship embedding.

4. The method according to claim 1 or 3, characterized in that, In the temporal intent reasoning, the GTrXL network processes the input of the current time step through a self-attention mechanism, while simultaneously introducing the recurrent memory state of the previous time step, and dynamically updates the information flow using gating units, fusing the output with the original input through residual connections and layer normalization.

5. The method according to claim 4, characterized in that, The GTrXL network supports serialized batch processing, sampling continuous time series batches from the experience replay pool during training and processing the entire sequence in parallel to achieve backpropagation training.

6. The method according to claim 5, characterized in that, The adversarial self-supervised reconstruction training is completed through initial observation, masking, encoding, state reconstruction, and loss calculation, wherein: Initial observation: agent Observing neighbors and The complete state; Masking: Randomly masking the neighbor state features in the local observation information of the agent; Encoding: The masked local observation information is input into the GAT network and the GTrXL network to generate the intent embedding; State reconstruction: The decoder receives the intent embedding containing missing information and reconstructs the masked neighbor state features; Loss calculation: Calculate the mean square error between the reconstructed value and the true value, which is the reconstruction loss.

7. The method according to claim 1, characterized in that, The following steps are performed when training the multi-agent proximal policy optimization algorithm: Data collection: After the agent interacts with the environment for rollout_len steps, it stores the observations, actions, rewards, completion states, value estimates, action log probabilities, and recurrent memory states in the experience replay pool; GAE calculation: After collecting rollout_len steps of data, calculate the generalized advantage estimate and return based on the value estimate of the centralized Critic; Serialization sampling: To adapt to the backpropagation requirements of the GTrXL network, a continuous fixed-length trajectory sequence is sampled from the experience replay pool; PPO Epoch and Forward Propagation: Perform ppo_epochs iterations. In reconstruction mode, perform forward propagation on the sequence batch to obtain the action log probability, state value estimate, reconstructed neighbor edge features and original neighbor edge features of the sequence. Backpropagation and Update: Backpropagation is performed based on the joint loss, and the network parameters are updated end-to-end using the gradient descent optimizer; Clear cache: Clear the experience replay pool and proceed to the next round of data collection.

8. An untrusted multi-agent implicit cooperative system based on spatiotemporal reasoning and adversarial reconstruction, characterized in that, include: The spatial relationship reasoning module is used to acquire local observation information at each time step, construct a neighborhood graph with itself as the center, aggregate neighbor node information using a graph attention network, and combine edge feature encoding to generate the instantaneous spatial relationship embedding of the agent. The temporal intent reasoning module is used to embed the instantaneous spatial relationship into the input GTrXL network, model the time series through a recurrent memory mechanism, and generate the final intent embedding. The strategy and value generation module is used to embed the final intention into the input Actor network and generate the execution strategy for each agent. The final intentions of each agent are embedded and spliced ​​together to form a global state representation, which is then input into the Critic network to generate a centralized value estimate of the current state. The adversarial reconstruction training module is used to complete adversarial self-supervised reconstruction training during the training phase through initial observation, masking, encoding, state reconstruction and loss calculation, thereby enhancing robustness to anomalous behavior. The model optimization module is used to construct a joint loss function based on the multi-agent proximal policy optimization algorithm to jointly optimize network parameters.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images