Multi-agent simulation inference and game learning methods and devices

Abstract

This invention provides a multi-agent simulation inference and game theory learning method and apparatus, relating to the fields of artificial intelligence and automatic control technology. The method includes: receiving local observation data from multiple distributed agents in a simulation environment and constructing global situation features; inputting the global situation features into a game solver for multi-objective optimization to generate scheduling instructions; inputting the global situation features into a global value function network to output long-term expected payoff values ​​and generate policy update gradients; and distributing the policy update gradients and scheduling instructions to the distributed agents through multi-level communication channels. The method and apparatus provided by this invention, by generating global situation features and based on these features, generating scheduling instructions and policy update gradients, construct a hierarchical control structure and a hierarchical reinforcement learning mechanism, solving problems such as decision conflict, low communication efficiency, slow learning convergence, and insufficient collaborative effectiveness in traditional multi-agent systems in complex tasks.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and automatic control technology, and in particular to a multi-agent simulation deduction and game learning method and apparatus. Background Technology

[0002] With the deepening application of artificial intelligence technology in fields such as military simulation, intelligent transportation, and unmanned systems, multi-agent systems have been widely used to solve complex collaborative decision-making problems. In mission scenarios with clear command hierarchies and complex adversarial relationships (such as drone swarm collaboration and robot cooperation), the system urgently needs to perform efficient mission planning and multi-objective collaborative coordination based on the highly dynamic environment in order to maximize overall benefits.

[0003] To meet the aforementioned collaborative decision-making requirements, existing mainstream multi-agent reinforcement learning methods (such as Multi-Agent Deep Deterministic Policy Gradient (MADDPG) and Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (QMIX)) typically employ an egalitarian agent architecture or a simple centralized training mode for task execution and control. In an egalitarian architecture, each agent has equal decision-making power, making it difficult to reflect the command and control relationships present in actual tasks. In a simple centralized architecture, although a central node exists, it is usually only responsible for information collection and policy distribution, lacking true hierarchical task decomposition and global collaborative capabilities. These architectural shortcomings lead to problems such as decision-making conflicts, task overlap, and unreasonable resource allocation when the system faces complex and ever-changing adversarial tasks.

[0004] Therefore, how to improve the collaborative decision-making ability and learning efficiency of multi-agent systems in complex adversarial environments has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0005] This invention provides a multi-agent simulation inference and game learning method and apparatus to solve the defects of existing multi-agent systems in complex adversarial environments, such as decision-making conflicts caused by the flattened control structure, communication storms caused by the lack of guidance in the communication mechanism, and slow learning convergence caused by the lack of hierarchical knowledge transfer paths. It realizes the improvement of the collaborative decision-making ability and learning efficiency of multi-agent systems in complex adversarial environments.

[0006] This invention provides a multi-agent simulation inference and game learning method, applied to a central server, comprising: Receive local observation data from multiple distributed intelligent agents in a simulation environment, and construct global situation features based on the local observation data; The global situation features are input into the game solver for multi-objective optimization to generate scheduling instructions; the game solver is used to model the task allocation problem of the distributed agent as an incomplete information game. The global situation features are input into a global value function network, which outputs a long-term expected return value that characterizes the collaborative interaction state among the multiple distributed agents. Based on the long-term expected return value, a policy update gradient is generated. The policy update gradient and the scheduling instructions are distributed to the corresponding distributed agents through multi-level communication channels.

[0007] In some embodiments, the step of inputting the global situation features into the game solver for multi-objective optimization and generating scheduling instructions includes: Obtain the interaction history data of the adversarial target in the simulation environment; The interaction history data and the local observation data are input into a pre-trained adversary behavior modeling network, which outputs a long-term strategy intent prediction result for the adversary target; the long-term strategy intent prediction result is updated based on new observation evidence. The global situation characteristics and the long-term strategy intention prediction results are input into the game solver for multi-objective optimization to generate the scheduling instructions.

[0008] In some embodiments, generating the policy update gradient based on the long-term expected return value includes: Based on the long-term expected return value, value decomposition technique is used to analyze the individual contribution. Based on the individual contribution, the policy update gradient is generated for different distributed agents.

[0009] This invention provides a multi-agent simulation inference and game theory learning method, applied to distributed agents, comprising: Acquire local observation data and upload the local observation data to the central server; Receive scheduling instructions and policy update gradients issued by the central server; Based on the policy update gradient, update the network parameters of the pre-deployed local policy network of the distributed agent; The local observation data and the scheduling instructions are input into the local policy network to output personalized action policies. The personalized action strategy is executed in a simulation environment.

[0010] In some embodiments, the method further includes: When a communication failure occurs between the distributed intelligent agent and the central server, a direct collaborative link is established with other distributed intelligent agents within a preset proximity range through a local communication protocol. The local observation data is exchanged with other distributed agents within a preset proximity range based on the direct collaborative link.

[0011] In some embodiments, inputting the local observation data and the scheduling instruction into the local policy network and outputting a personalized action policy includes: Acquire historical interaction data of adversarial targets in a simulated environment; The interaction history data and the local observation data are input into a pre-trained adversary prediction model, which outputs the short-term action prediction results of the adversary target; the short-term action prediction results are updated based on new observation evidence. The short-term action prediction results, the local observation data, and the scheduling instructions are input into the local policy network to output a personalized action policy.

[0012] This invention provides a multi-agent simulation inference and game learning device, applied to a central server, comprising: The perception module is used to receive local observation data from multiple distributed agents in a simulation environment and construct global situation features based on the local observation data. The scheduling module is used to input the global situation features into the game solver for multi-objective optimization and generate scheduling instructions; the game solver is used to model the task allocation problem of the distributed agent as an incomplete information game. The strategy module is used to input the global situation features into the global value function network, output the long-term expected reward value to characterize the collaborative interaction state among the multiple distributed agents, and generate a strategy update gradient based on the long-term expected reward value. The distribution module is used to distribute the policy update gradient and the scheduling instruction to the corresponding distributed agents through a multi-level communication channel.

[0013] This invention provides a multi-agent simulation inference and game learning device, applied to distributed agents, comprising: The observation module is used to acquire local observation data and upload the local observation data to the central server. The receiving module is used to receive scheduling instructions and policy update gradients issued by the central server; The update module is used to update the network parameters of the pre-deployed local policy network of the distributed agent based on the policy update gradient. The action generation module is used to input the local observation data and the scheduling instructions into the local policy network and output personalized action policies. The execution module is used to execute the personalized action strategy in the simulation environment.

[0014] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the multi-agent simulation inference and game learning method as described above.

[0015] The present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multi-agent simulation inference and game learning method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-agent simulation inference and game learning method described above.

[0017] The multi-agent simulation inference and game learning method and apparatus provided by this invention, by receiving local observation data from multiple distributed agents in a simulation environment and constructing global situation features, breaks through the "information island" dilemma caused by the limited local vision of a single agent, effectively aggregating fragmented information at the underlying level and providing a macroscopic and comprehensive global perception foundation for the system. By inputting the global situation features into a game solver for multi-objective optimization, scheduling instructions are generated, and the task allocation problem is modeled as an incomplete information game, which highly matches the information asymmetry and uncertainty in real complex adversarial environments. Under complex constraints, scientific task decomposition and optimal resource scheduling are achieved, effectively avoiding decision conflicts and task overlaps among distributed individuals. By inputting the global situation features into a global value function network, the long-term expected payoff value representing the collaborative interaction state is output, and a strategy update is generated based on this. The gradient mechanism constructs a hierarchical reinforcement learning mechanism of "global evaluation-local update" to solve the typical "credit allocation" problem in multi-agent reinforcement learning from a macro-level collaborative perspective. This enables each agent to optimize its policy in the direction of maximizing global benefits, thereby significantly accelerating the convergence speed of model training. By distributing policy update gradients and scheduling instructions to the corresponding distributed agents through multi-level communication channels, a hierarchical and guided information transmission channel is established. This ensures that macro-level command intentions and cross-level learned knowledge can be accurately and efficiently delivered to the bottom-level nodes, effectively alleviating the communication storm problem caused by massive node interactions and improving the overall system collaboration and communication efficiency. By establishing a hierarchical control structure and an efficient global communication mechanism, the mechanism solves the problems of decision conflict, low communication efficiency, slow learning convergence, and insufficient collaborative effectiveness in traditional multi-agent systems in complex tasks. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts of the multi-agent simulation inference and game learning method provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the overall architecture of the multi-agent simulation inference and game learning method provided by the present invention.

[0022] Figure 3 This is a flowchart of the agent behavior prediction module of the multi-agent simulation inference and game learning method provided by the present invention.

[0023] Figure 4 This is the second flowchart of the multi-agent simulation inference and game learning method provided by the present invention.

[0024] Figure 5 This is a hierarchical learning architecture diagram of the multi-agent simulation inference and game learning method provided by the present invention.

[0025] Figure 6 This is a schematic diagram of the communication mechanism of the multi-agent simulation inference and game learning method provided by the present invention.

[0026] Figure 7 This is a schematic diagram of the system deployment of the multi-agent simulation inference and game learning method provided by the present invention in a typical application scenario.

[0027] Figure 8 This is one of the structural schematic diagrams of the multi-agent simulation inference and game learning device provided by the present invention.

[0028] Figure 9 This is the second schematic diagram of the structure of the multi-agent simulation inference and game learning device provided by the present invention.

[0029] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps, units, or modules is not necessarily limited to those explicitly listed, but may include other steps, units, or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0032] Figure 1 This is one of the flowcharts illustrating the multi-agent simulation inference and game learning method provided by this invention, such as... Figure 1 As shown, the method includes steps 110, 120, 130, and 140. The execution entity of the multi-agent simulation inference and game learning method provided by this invention can be a central server. Unless otherwise specified, the following embodiments will use a central server as an example for explanation.

[0033] Step 110: Receive local observation data from multiple distributed intelligent agents in the simulation environment, and construct global situation features based on the local observation data.

[0034] Specifically, the execution entity of the multi-agent simulation deduction and game learning method provided in this embodiment of the invention is a multi-agent simulation deduction and game learning device. This device can be implemented in software, such as a multi-agent simulation deduction and game learning program running on a computer; or it can be implemented in hardware, such as a computer or server that executes the multi-agent simulation deduction and game learning method.

[0035] In practical engineering applications, especially in mission scenarios with clear command hierarchies and complex adversarial relationships, traditional multi-agent reinforcement learning methods have revealed many limitations. Existing technologies suffer from the following three prominent problems: First, regarding the control structure, there is a lack of effective multi-level control mechanisms. Real-world collaborative tasks often require a clear command chain; for example, in military confrontations, multiple levels such as strategic command, tactical command, and unit execution are necessary. Traditional methods either distribute all decision-making power equally to each agent or over-centralize it to a single node, failing to achieve effective task decomposition and hierarchical decision-making. This lack of a robust control structure makes the system ill-suited for complex scenarios requiring multi-level, multi-granularity decision-making.

[0036] Secondly, regarding communication mechanisms, there is a lack of efficient global coordination methods. Existing methods typically employ fully connected or nearest-neighbor communication modes, resulting in a lack of organization and direction in information exchange between agents. This easily leads to communication storms, where communication overhead increases exponentially in high-agent-density scenarios, severely impacting system real-time performance. More importantly, the lack of a specially designed global communication channel makes it difficult to transmit high-level command intentions and for lower-level situational information to converge. Agents can only make decisions based on local information, failing to achieve globally optimal coordination.

[0037] Third, in terms of learning efficiency, there is a lack of hierarchical knowledge transfer paths. In traditional architectures, each agent needs to learn the complete task policy from scratch, failing to fully utilize the abstract knowledge learned by higher-level agents. This "each fighting their own battle" learning approach leads to low training efficiency, especially in scenarios with a large number of agents and complex tasks, resulting in slow convergence and difficulty in guaranteeing policy quality. Furthermore, due to the lack of hierarchical policy representation, the learned policies are often difficult to interpret, hindering their deployment and application in real-world systems.

[0038] Existing technologies have significant shortcomings in multi-level control structures, global communication mechanisms, and hierarchical learning, which restricts the practical application effectiveness of multi-agent systems in complex adversarial environments. Therefore, embodiments of this invention provide a multi-agent simulation inference and game theory learning method, applicable to application scenarios requiring multi-level decision-making and global coordination, such as intelligent game theory, drone swarms, and robot collaboration.

[0039] Figure 2 This is a schematic diagram of the overall architecture of the multi-agent simulation inference and game learning method provided by the present invention, as shown below. Figure 2 As shown, at the system architecture level, a hierarchical design is adopted to establish an operational mechanism for the collaborative work of the global simulation center and distributed intelligent agents. The global simulation center includes a game-solving engine, a dynamic task redistribution mechanism, and a situational simulation module. The game-solving engine is used to construct a multi-agent game model, modeling the task allocation problem as an incomplete information game; the dynamic task redistribution mechanism adjusts the task plan in real time according to changes in the battlefield situation; and the situational simulation module can predict the overall task progress and potential risks.

[0040] The global simulation center is responsible for constructing battlefield situation maps, planning tasks, and scheduling resources. It achieves multi-objective optimization decisions through an integrated game theory solver and uses mixed-integer programming to seek optimal solutions under complex constraints. Distributed agents autonomously execute tasks based on local observation information, employing reactive behavior planning to cope with dynamic environmental changes, while also being equipped with task monitoring and anomaly handling mechanisms to ensure execution efficiency. The system uses a simulation engine to simulate and evaluate different combat scenarios, providing a rich training environment for agent learning.

[0041] A simulation environment refers to a virtual adversarial space built based on a physics engine, terrain modeling, and environmental perturbation models, such as an intelligent game environment, a drone swarm adversarial environment, or a robot collaborative simulation environment. This simulation environment can simulate multi-level command and control relationships and complex adversarial situations in the real world.

[0042] Local observation data refers to the data set acquired by each distributed intelligent agent (such as a single UAV platform, a single combat unit, etc.) within its limited field of view using its own sensing payload. This local observation data typically includes, but is not limited to, the agent's own kinematic state (e.g., three-dimensional position coordinates, velocity, acceleration, heading angle, etc.), internal attribute state (e.g., remaining energy, payload state, etc.), and external environment detection information (e.g., relative positions of neighboring units captured within the local field of view, and information on detected enemy targets or obstacles, etc.).

[0043] When receiving the aforementioned data at the central server, to avoid "communication storms" in high-density multi-agent scenarios, the central server can establish a unified information exchange and communication hub to receive data uploaded from the underlying network. During the reception process, data compression and priority scheduling techniques can be employed. For example, higher transmission priority can be given to local observation data containing core enemy targets or sudden threats, thereby ensuring the real-time nature and reliability of critical situational information.

[0044] Because the observation perspective of a single distributed agent is limited and may have blind spots, the central server needs to clean, align, and fuse massive amounts of local observation data. The steps for constructing global situational awareness features based on local observation data can include: aligning the local observation data from different distributed agents in terms of timestamp space and transforming the coordinate system, for example, uniformly transforming the relative local coordinates of each agent to the global absolute coordinate system; subsequently, using multi-source information fusion algorithms to eliminate redundant and overlapping observation data, filling in blind spots in individual agent perception, and thus generating a complete "battlefield situation map."

[0045] The final output global situation features can be represented as multi-dimensional feature vectors, rasterized map features, or node-edge features of graph neural networks, which comprehensively characterize the overall force deployment, resource distribution, and confrontation evolution trend in the current simulation environment.

[0046] Step 120: Input the global situation features into the game solver for multi-objective optimization and generate scheduling instructions; the game solver is used to model the task allocation problem of the distributed agent as an incomplete information game.

[0047] Specifically, in realistic or highly realistic simulated adversarial scenarios, such as drone swarm battles, although the global simulation center can aggregate data from all parties, it often cannot fully grasp the true intentions, hidden forces, and underlying resource status of the enemy or non-cooperative parties. Therefore, this embodiment of the invention uses incomplete information game theory to model this process.

[0048] After receiving global situational characteristics, the game solver abstracts the network to which the distributed agents reside and the adversary target as game participants. Due to the existence of "fog of war" or sensor detection blind spots, the game solver models the task allocation problem of the distributed agents as an incomplete information game model. In this model, "type" and "belief" mechanisms are introduced. Specifically, the system defines the enemy's intentions, resource reserves, or performance parameters that cannot be directly observed as unknown "types," and constructs a "belief distribution," or probability distribution, about the enemy type based on global situational characteristics, including the enemy's historical behavior trajectory and current formation. On this basis, the task allocation of the friendly distributed agents is no longer a static allocation based on a deterministic environment, but is transformed into a Bayesian Nash equilibrium solution process under the enemy type probability distribution, thus enabling the most robust macroscopic decision-making under conditions of missing information.

[0049] Based on the aforementioned incomplete information game model, the game solver needs to perform multi-objective optimization. In actual adversarial tasks, decision objectives are often multiple and mutually constraining, such as simultaneously maximizing the probability of task completion, minimizing the cost of one's own resources or casualties, and minimizing the task execution time.

[0050] In this embodiment of the invention, the game solver can further integrate a mixed-integer programming method. This is achieved by setting the aforementioned multiple objectives as the optimization objective function, and by setting the physical performance boundaries of the distributed agent, such as maximum flight speed, endurance, communication bandwidth limitations, and spatial obstacle avoidance requirements, as complex constraints. The game solver seeks the optimal solution by searching the solution space constituted by these multi-dimensional constraints.

[0051] After multi-objective optimization, the output of the game solver is encapsulated as scheduling instructions. It should be noted that in this embodiment of the invention, the scheduling instructions issued by the central server are macro-level guiding instructions, not low-level micro-control instructions. For example, these scheduling instructions may include strategic objective allocation for a specific agent subgroup, regional patrol route planning, role division (such as reconnaissance, attack, and cover), and priority sequences for coordinated attacks. These macro-level scheduling instructions convey the globally optimal strategic intent while leaving ample autonomous decision-making space for subsequent distributed agents to perform "reactive behavior planning" based on their local observations.

[0052] Step 130: Input the global situation features into the global value function network, output the long-term expected return value to characterize the collaborative interaction state among the multiple distributed agents, and generate the policy update gradient based on the long-term expected return value.

[0053] Specifically, this step is mainly applied to the "centralized evaluation" stage in a hierarchical learning system. Its core purpose is to assess the overall development trend of the current system from a global perspective and to indicate the precise direction for policy optimization for each distributed agent.

[0054] The Global Value Function Network (GFCN) employs a deep neural network architecture. To accurately capture the dynamic coupling relationships between multiple distributed agents in complex adversarial environments, a multi-head attention mechanism is integrated into the GFCN. After inputting global situational features, the multi-head attention mechanism dynamically focuses on key local adversarial regions or high-value targets by calculating attention weights between the feature representations of different agents, thereby extracting hidden features containing spatial topological relationships. Based on these hidden features, the GFCN outputs the long-term expected return value.

[0055] Long-term expected return (LTR) is a comprehensive quantitative indicator calculated by a global value function network based on the global situation characteristics at the current moment. It is used to prospectively evaluate the overall future performance of a system. LTR not only reflects the immediate reward obtained by the system at the current moment, but also prospectively describes the cumulative return that the system is expected to obtain over a complete future projection cycle under the current global situation. It is a comprehensive quantitative indicator of the quality of collaborative interaction among multiple distributed intelligent agents.

[0056] The policy update gradient refers to the mathematical vector signal calculated by a centralized global evaluation network (such as a global value function network) during the training of a reinforcement learning model. It is based on the error (such as temporal difference error) between the expected cumulative reward of the system as a whole and the actual environmental reward. It is used to guide each distributed agent to adjust the parameters of its local policy network.

[0057] In multi-agent cooperative tasks, because environmental rewards are typically global and sparse, the system faces a serious "credit allocation" problem, namely, it is difficult to determine which specific agent's actions account for an increase or decrease in global returns. In this embodiment of the invention, a policy update gradient is generated based on the long-term expected return value.

[0058] In this embodiment of the invention, value decomposition technology is used to backpropagate the global temporal difference error, accurately extract and calculate the error feedback belonging to the individual contribution of a specific agent, and generate a policy update gradient. By distributing this gradient to the distributed agents, the system can force or guide the underlying agents to iteratively update the policy parameters according to the globally optimal cooperative direction, thereby effectively avoiding local optimum traps and promoting the rapid convergence of the overall cooperative strategy.

[0059] Step 140: Distribute the policy update gradient and the scheduling instruction to the corresponding distributed agents through a multi-level communication channel.

[0060] Specifically, the central server, acting as the unified information exchange hub of the entire multi-agent system, is responsible for establishing and maintaining multi-level communication channels connecting various distributed agents. In complex adversarial scenarios, such as military simulations and drone swarm collaboration, where there is a high density of agents, traditional fully connected or nearest-neighbor communication modes are prone to triggering "communication storms," ​​leading to an exponential increase in communication overhead. To address this issue, the central server in this embodiment of the invention employs a directional, hierarchical communication mechanism, delivering high-dimensional information through multi-level communication channels.

[0061] In this embodiment of the invention, firstly, the central server preprocesses the data to be sent. The central server uses a preset data compression algorithm to compress the policy update gradient and scheduling instructions to reduce channel bandwidth occupancy. Secondly, the central server uses priority scheduling technology for queue management. Based on the urgency of the instructions or the criticality of the task to which they belong, dynamic priorities are assigned to different scheduling instructions and policy update gradients. For example, scheduling instructions involving global survival, emergency evasion, or core adversarial targets are given the highest priority to ensure real-time transmission and priority delivery of critical information. Finally, the central server completes protocol conversion and sends out the data through a cross-level communication interface. The central server distributes the compressed and sorted data, along a multi-level communication channel, such as the command chain of "global simulation center - regional controller - unit intelligent agent," to the corresponding distributed intelligent agents level by level.

[0062] The scheduling instructions carry macro-level task planning and resource allocation constraints, used to convey higher-level command intentions downwards, ensuring that the local behavior of the distributed agent does not deviate from the optimal solution trajectory of the global multi-objective optimization. The policy update gradient carries the global abstract knowledge and individual contribution feedback obtained by the central server based on the global value function network evaluation. By establishing this top-down gradient propagation path, this embodiment of the invention can achieve cross-level knowledge transfer. The distributed agent does not need to learn through trial and error from scratch, but directly uses this policy update gradient to share parameters and learn lessons in the local policy network.

[0063] The multi-agent simulation inference and game learning method provided in this invention breaks through the "information island" dilemma caused by the limited local vision of a single agent by receiving local observation data from multiple distributed agents in a simulation environment and constructing global situation features. It effectively aggregates fragmented information at the bottom level, providing a macroscopic and comprehensive global perception foundation for the system. By inputting the global situation features into a game solver for multi-objective optimization, scheduling instructions are generated, and the task allocation problem is modeled as an incomplete information game, which highly matches the information asymmetry and uncertainty in real complex adversarial environments. Under complex constraints, scientific task decomposition and optimal resource scheduling are achieved, effectively avoiding decision conflicts and task overlaps among distributed individuals. By inputting the global situation features into a global value function network, the long-term expected payoff value representing the collaborative interaction state is output, and a strategy update is generated based on this value. The gradient mechanism constructs a hierarchical reinforcement learning mechanism of "global evaluation-local update" to solve the typical "credit allocation" problem in multi-agent reinforcement learning from a macro-level collaborative perspective. This enables each agent to optimize its policy in the direction of maximizing global benefits, thereby significantly accelerating the convergence speed of model training. By distributing policy update gradients and scheduling instructions to the corresponding distributed agents through multi-level communication channels, a hierarchical and guided information transmission channel is established. This ensures that macro-level command intentions and cross-level learned knowledge can be accurately and efficiently delivered to the bottom-level nodes, effectively alleviating the communication storm problem caused by massive node interactions and improving the overall system collaboration and communication efficiency. By establishing a hierarchical control structure and an efficient global communication mechanism, the mechanism solves the problems of decision conflict, low communication efficiency, slow learning convergence, and insufficient collaborative effectiveness in traditional multi-agent systems in complex tasks.

[0064] In some embodiments, the step of inputting the global situation features into the game solver for multi-objective optimization and generating scheduling instructions includes: Obtain the interaction history data of the adversarial target in the simulation environment; The interaction history data and the local observation data are input into a pre-trained adversary behavior modeling network, which outputs a long-term strategy intent prediction result for the adversary target; the long-term strategy intent prediction result is updated based on new observation evidence. The global situation characteristics and the long-term strategy intention prediction results are input into the game solver for multi-objective optimization to generate the scheduling instructions.

[0065] Specifically, new observational evidence is incremental information extracted from local observations that can correct the perception of adversarial targets, such as location and dynamic changes.

[0066] In embodiments of the present invention, the long-term strategy intent prediction result refers to the inference data inferred by the central server based on a pre-trained adversary behavior modeling network, which integrates the historical interaction sequence of the adversary target with the current local observation data, and uses feature extraction and intent recognition algorithms to characterize the macro-tactical objectives, action evolution trends, or collaborative planning patterns of the adversary target over a longer future observation period.

[0067] In complex multi-agent adversarial game environments such as drone swarm air combat and robot combat competitions, the behavior of the adversary, i.e., the target, is often not random, but follows a certain tactical logic. The central server extracts the adversary's interaction history data over a period of time. This data may include, but is not limited to, the adversary's spatial trajectory coordinates, velocity change curves, weapon formation evolution, radar frequency band switching records, etc.

[0068] Figure 3 This is a flowchart of the agent behavior prediction module of the multi-agent simulation inference and game learning method provided by the present invention, as shown in the figure. Figure 3 As shown, in this embodiment of the invention, an adversary behavior modeling network based on deep learning is introduced. The agent behavior inference includes the adversary behavior modeling network, a multi-timescale prediction model, an intent recognition algorithm, and a belief update mechanism. The adversary behavior modeling network is built based on deep learning technology; the multi-timescale prediction model is used to estimate future strategies; the intent recognition algorithm infers the agent's goal by observing historical behavior; and the belief update mechanism dynamically corrects the inference results based on new observational evidence.

[0069] The macroscopic interaction history data is fused with the local observation data reported in real time by the friendly agent and input into the adversary behavior modeling network. The adversary behavior modeling network extracts temporal and spatial interaction features and outputs long-term strategy intent prediction results for the adversary target. For example, it predicts whether the enemy group is attempting to "flank and encircle", "concentrate to break through", or "lure the enemy deep".

[0070] In particular, this inference process is not static, but incorporates a belief update mechanism. As the simulation progresses, the central server continuously acquires new local observation data, i.e., new observational evidence. The adversary behavior modeling network will, based on Bayesian inference or dynamic weight adjustment mechanisms, correct and update the previous long-term strategy intent prediction results in real time, thereby ensuring that the intent prediction can adapt to the rapidly changing battlefield environment and avoid misjudgments due to information lag.

[0071] After obtaining precise opponent intentions, the central server uses them as prior knowledge within the game environment, inputting them along with the current global situation characteristics of its own side into the game solver. Traditional game solving often faces significant uncertainty, but introducing long-term strategy intention prediction results greatly reduces the complexity of the search space in incomplete information games.

[0072] When performing multi-objective optimization, game solvers can transform passive responses into proactive suppression by simultaneously considering multiple conflicting objectives such as "maximizing one's own survival rate," "minimizing task completion time," and "maximizing enemy resource consumption." Through mathematical methods such as mixed-integer programming, game solvers can calculate Pareto optimal solutions under various physical and communication constraints, thereby generating scheduling instructions for distributed agents, such as tactical positioning, firepower allocation, and cover retreat instructions.

[0073] The multi-agent simulation inference and game learning method provided in this invention obtains the interaction history data of the adversary target, infers the long-term strategic intent prediction result of the adversary target, and generates scheduling instructions based on the result. This breaks the limitations of traditional reactive scheduling, enabling the central server to perform forward-looking resource scheduling and task planning based on future predictions, significantly improving the survivability and game winning rate of the multi-agent cluster in a highly dynamic adversarial environment.

[0074] In some embodiments, generating the policy update gradient based on the long-term expected return value includes: Based on the long-term expected return value, value decomposition technique is used to analyze the individual contribution. Based on the individual contribution, the policy update gradient is generated for different distributed agents.

[0075] Specifically, in complex distributed simulation environments, when multiple distributed agents jointly execute a collaborative task, the environment typically only provides a single global reward signal, i.e., the global long-term expected return value. Directly using this global reward signal to update all agents simultaneously can lead to a "free-rider" phenomenon, where poorly performing agents benefit from the performance of well-performing agents, or vice versa, severely slowing down the system's learning and convergence speed.

[0076] To overcome the aforementioned shortcomings, this invention introduces a value decomposition technique. The global value function network at the central server evaluates the global long-term expected return value, which characterizes the overall situation of the system. Through a preset decomposition algorithm mechanism, this macroscopic return is reverse-analyzed into the individual contributions of each distributed agent.

[0077] In practical engineering implementations, value decomposition techniques can employ, but are not limited to, linear decomposition mechanisms or nonlinear hybrid network mechanisms. For example, using a monotonic value decomposition network ensures that the global long-term expected return and the return of each individual agent satisfy a monotonically increasing mathematical constraint, thus keeping the direction of maximizing global return strictly consistent with the direction of maximizing individual return. Consequently, the central server can accurately extract the specific weight or responsibility of each distributed agent for global success or failure in the current joint interaction action.

[0078] Furthermore, after obtaining the precise individual contribution of each agent, the system, based on the policy gradient theorem and combined with the action probability distribution of each distributed agent, independently calculates and generates differentiated policy update gradients for each agent.

[0079] The multi-agent simulation and game learning method provided in this invention generates policy update gradients for different distributed agents by combining long-term expected returns with value decomposition technology. This breaks the ambiguity caused by traditional global unified rewards, enabling each distributed agent to obtain feedback signals that match its actual performance. The targeted policy update gradients allow excellent tactical actions to be quickly reinforced and incorrect tactical actions to be quickly corrected, greatly improving the joint training efficiency of large-scale clusters in complex game adversarial environments.

[0080] Figure 4 This is the second flowchart illustrating the multi-agent simulation inference and game learning method provided by this invention, as shown below. Figure 4 As shown, the method includes steps 410, 420, 430, 440, and 450. The executing entity of the multi-agent simulation inference and game learning method provided by this invention can be a distributed agent. Unless otherwise specified, subsequent embodiments will use a distributed agent as an example for illustration.

[0081] Step 410: Obtain local observation data and upload the local observation data to the central server.

[0082] Specifically, local observation data refers to the data set acquired by each distributed intelligent agent (such as a single UAV platform, a single combat unit, etc.) within its limited field of view using its own sensing payload. This local observation data typically includes, but is not limited to, the agent's own kinematic state (e.g., three-dimensional position coordinates, velocity, acceleration, heading angle, etc.), internal attribute state (e.g., remaining energy, payload state, etc.), and external environment detection information (e.g., the relative positions of neighboring units captured within the local field of view, and information on detected enemy targets or obstacles, etc.).

[0083] In complex multi-agent simulations such as drone swarm warfare and robot collaboration, due to the highly dynamic environment and the physical limitations of the sensor line of sight of individual agents, each distributed agent is typically engaged in an incompletely observable Markov decision process. Therefore, distributed agents need to utilize their onboard virtual sensors or situational observation interfaces called in the simulation engine to acquire local observation data within their field of view.

[0084] To break down the "information silos" between agents and enable the central server to coordinate the overall situation, distributed agents must report their observed local information upwards. However, in adversarial scenarios with high agent density, if all agents upload data frequently and in full, it can easily trigger a communication storm, leading to a surge in system latency or even communication link paralysis.

[0085] To address this technical challenge, distributed agents preferably employ data compression and priority scheduling techniques when performing upload operations. For example, the distributed agent first extracts features and performs dimensionality reduction compression on the original local observation data, filtering out redundant background information and extracting only key situational changes, such as newly emerging high-value enemy targets or sudden serious self-failures. Subsequently, the system assigns different communication priorities based on the urgency and tactical value of the information. For instance, alarm data indicating enemy fire control lock is given the highest priority, while routine patrol position heartbeat packets are given a lower priority and transmitted to the central server through cross-level communication interfaces.

[0086] Step 420: Receive the scheduling instructions and policy update gradients issued by the central server.

[0087] Specifically, the policy update gradient refers to the mathematical vector signal calculated by the centralized global evaluation network (such as the global value function network) during the training of the reinforcement learning model, based on the error (such as the temporal difference error) between the expected cumulative reward of the system as a whole and the actual environmental reward. It is used to guide each distributed agent to adjust the parameters of its local policy network.

[0088] Scheduling instructions are generated by the central server based on global situational characteristics and through multi-objective optimization simulations using a game theorist solver. In complex adversarial or simulation scenarios (such as UAV swarm operations and intelligent robot collaborative operations), distributed agents, limited by sensor detection range, often only acquire local observation information, making them prone to "short-sighted decision-making." Scheduling instructions issued by the central server typically include macro-level task allocation plans, route planning constraints, or resource access permissions. By receiving these instructions, distributed agents can obtain high-dimensional command intentions that transcend their local perspectives, ensuring that their local actions always align with the globally optimal tactical direction, effectively avoiding task overlap or resource conflicts among multiple agents.

[0089] In traditional reinforcement learning architectures, agents typically need to collect environmental feedback and calculate gradients independently. This not only consumes enormous edge computing power but also easily leads to excessively large gradient estimation variance and difficulty in learning convergence due to the limitations of local observations. However, in the hierarchical learning system of this invention, the distributed agent only acts as the gradient receiver and the local network parameter update execution end.

[0090] Figure 5 This is a hierarchical learning architecture diagram of the multi-agent simulation inference and game learning method provided by this invention, as shown in the figure. Figure 5 As shown, in this embodiment of the invention, a hierarchical learning system combining global inference and local decision-making is constructed. The hierarchical learning system includes a collaborative optimization algorithm and a multi-agent policy gradient algorithm. The collaborative optimization algorithm employs a centralized critic and a distributed executor architecture. The multi-agent policy gradient algorithm solves the credit allocation problem. The hierarchical learning system includes a global value function network and a local policy network.

[0091] The global value function network is responsible for evaluating the overall situation and development trend of the system. It uses a centralized state value function to characterize the long-term expected return, captures the interaction effects between agents through a multi-head attention mechanism, and analyzes individual contributions using value decomposition techniques. The local policy network builds a personalized decision-making model for each agent, generates action policies based on local observation information, optimizes decision performance using a policy gradient method, and improves training efficiency through parameter sharing and course learning mechanisms. The system achieves the organic unity of global objectives and local policies through a hierarchical learning mechanism, establishes gradient propagation paths to promote cross-level knowledge transfer, and uses multi-timescale update strategies to balance learning speed and stability. It should be noted that this system also integrates a multi-agent collaborative optimization algorithm, solves the credit allocation problem through a centralized evaluation and distributed execution architecture, and enhances the system's adaptability and robustness by combining opponent modeling and meta-learning techniques, ultimately forming a complete multi-agent simulation inference and game learning solution.

[0092] Furthermore, in the specific communication implementation process, the receiving step can also rely on data compression and priority scheduling techniques to ensure the real-time transmission of critical information. For example, when the communication channel in the simulation environment is limited or congested, the underlying communication protocol can set scheduling instructions (affecting immediate survival and task execution) as high-priority data streams for low-latency reception, while setting policy update gradients (affecting the long-term evolution of the model) as secondary-priority data streams for batch or delayed reception, thereby ensuring the robustness of the system in extreme adversarial environments.

[0093] Step 430: Update the network parameters of the pre-deployed local policy network of the distributed agent based on the policy update gradient.

[0094] Specifically, the local policy network pre-deployed locally by the distributed intelligent agent is usually constructed using a deep neural network architecture, such as a multilayer perceptron (MLP), a convolutional neural network (CNN), or a recurrent neural network (RNN). Its core function is to establish a direct mapping from local observation data to the probability distribution of specific actions or deterministic actions.

[0095] When the distributed agent receives the policy update gradient tailored for it by the central server through the communication channel, the agent's local computing unit will call the built-in optimizer, and apply the policy update gradient to each hidden layer node of the local policy network in combination with the preset learning rate or the dynamically decaying learning rate, and perform backpropagation and iterative update of its network parameters such as weights and biases.

[0096] Furthermore, in application scenarios with large-scale multi-agent clusters, parameter sharing mechanisms can be adopted among the local policy networks of multiple distributed agents to improve training efficiency. Although the basic network structure and some common parameters are shared, the policy update gradient received by each agent is calculated and distributed separately by the central server based on its "individual contribution." This gradient update method based on individual differences can effectively break the system symmetry, prompting each distributed agent to evolve personalized action policies under a unified goal orientation.

[0097] Step 440: Input the local observation data and the scheduling instructions into the local policy network, and output personalized action policies.

[0098] Specifically, scheduling instructions typically represent macro-level task planning or role allocation at the global level, while local observation data, acquired by the distributed agent's own sensors or radar, represents the real-time micro-situation around the agent, such as the location of obstacles within line of sight, sudden local threats, or the relative distance to nearby friendly forces.

[0099] In the actual model inference process, the local policy network deployed locally by the distributed agent receives the two heterogeneous inputs mentioned above. The system first performs feature encoding and alignment on these two types of data. For example, it reduces the dimensionality of high-dimensional scheduling instructions to conditional feature vectors and concatenates them with the state feature vectors of local observation data or uses an attention mechanism for weighted fusion. Subsequently, it performs nonlinear mapping through a multi-layer neural network of the local policy network, ultimately outputting a personalized action policy for the current dynamic environment. The personalized action policy can be a specific continuous control quantity, such as heading angle, acceleration, or throttle valve opening, or it can be an action probability distribution in a discrete action space.

[0100] Step 450: Execute the personalized action strategy in the simulation environment.

[0101] Specifically, personalized action strategies are usually expressed as a probability distribution in the action space (for discrete action spaces) or a specific operation parameter control vector (for continuous action spaces).

[0102] First, a mapping and parsing process is performed from policy to control commands. The control module of the distributed agent samples or parses the personalized action policies output by the local policy network to generate specific control commands that can be recognized by the underlying actuators. For example, in a drone swarm adversarial simulation scenario, these control commands may include, but are not limited to, target heading angle, desired flight speed, acceleration, radar activation commands, or evasion action trigger commands.

[0103] Secondly, there is physical feedback interaction based on the simulation engine. Control commands are input into the dynamics or kinematics engine within the simulation environment, driving the digital model of the distributed intelligent agent to undergo pose transitions or state changes within the current time step. The simulation environment calculates the execution result of the control command in real time based on physical laws such as gravity, collision detection, aerodynamics, and the interaction results with other intelligent agents or environmental obstacles.

[0104] Finally, environmental feedback is obtained to drive state transitions. After the control command is executed, the distributed agent obtains new local observation data from the simulation environment, namely the observation state at the next time step and the immediate reward obtained for performing the action. This new local observation data will be used for the next round of policy reasoning or uploaded to the central server, thus forming a continuous iterative closed loop of "observation-decision-execution-feedback".

[0105] The multi-agent simulation and game-theoretic learning method provided in this invention acquires local observation data and uploads it to a central server, providing an accurate data foundation for the central server to aggregate underlying information, construct global situational features, and perform macroscopic multi-objective optimization, thus breaking down information silos between distributed agents. By receiving scheduling instructions and policy update gradients, and updating the network parameters of the pre-deployed local policy networks of the distributed agents based on the policy update gradients, a top-down cross-level knowledge transfer and gradient propagation path is constructed. This enables the distributed agents to avoid blindly exploring from scratch and fully utilize a global perspective. The individual contribution values ​​are analyzed to guide the training of local models, which significantly improves the learning efficiency and convergence speed of local network models. By inputting local observation data and scheduling instructions issued by the superior into the local policy network, personalized action policies are output and executed in the simulation environment. This achieves an organic combination of macro-level global command intentions and micro-level dynamic environment perception, enabling distributed agents to strictly follow global task allocation and scheduling plans while making responsive and flexible adjustments based on the current local situation. This effectively avoids decision conflicts and task overlaps in complex adversarial environments and significantly improves the collaborative decision-making ability and environmental adaptability of multi-agent clusters.

[0106] In some embodiments, the method further includes: When a communication failure occurs between the distributed intelligent agent and the central server, a direct collaborative link is established with other distributed intelligent agents within a preset proximity range through a local communication protocol. The local observation data is exchanged with other distributed agents within a preset proximity range based on the direct collaborative link.

[0107] Specifically, in complex adversarial environments, such as those with electromagnetic interference, terrain obstruction, or the destruction of a central node, multi-agent clusters are prone to overall paralysis due to "single point of failure." Under normal operational conditions, distributed agents heavily rely on the global scheduling of a central server. However, in real battlefields or complex natural environments, long-distance centralized communication links are extremely vulnerable. To address this issue, this invention provides a smooth degradation communication and coordination strategy for distributed agents.

[0108] Figure 6 This is a schematic diagram of the communication mechanism of the multi-agent simulation inference and game learning method provided by the present invention, as shown below. Figure 6As shown in this embodiment of the invention, in terms of communication and coordination mechanisms, the system adopts a communication architecture that combines centralized deduction with distributed execution. The central deduction node establishes a unified information exchange hub, realizing global situational awareness distribution and command transmission through multi-level communication channels, and employing data compression and priority scheduling technologies to ensure real-time transmission of critical information. Distributed agents maintain relatively independent communication capabilities during execution, achieving direct coordination between neighboring units through local communication protocols, and establishing communication quality monitoring and fault tolerance mechanisms to improve system robustness.

[0109] First, the distributed agent will monitor the quality of its communication link with the central server in real time. Communication anomalies include, but are not limited to, consecutive heartbeat packet loss exceeding a preset threshold, communication latency exceeding a tolerance limit, or a received signal-to-noise ratio below a preset security standard. When any of these anomalies is detected, the distributed agent will automatically trigger a communication degradation contingency plan.

[0110] Once the contingency plan is triggered, the distributed agent will activate its onboard local communication protocols, such as Ad-Hoc Network (Ad-Hoc), Ultra-Wideband (UWB), Wi-Fi Direct, or tactical data links. It will actively broadcast handshake signals, scan for and discover other friendly distributed agents within a preset proximity range, such as the physical distance calculated based on a signal attenuation model or the radio frequency coverage radius. Through mutual authentication, these "disconnected" agents will quickly establish decentralized direct collaborative links.

[0111] Subsequently, based on the established direct collaborative links, adjacent distributed agents will frequently exchange local observation data. Each agent will share environmental features captured by its own sensors, the location of enemy targets, and its own current state (such as remaining battery power and ammunition) with its neighboring nodes. Through this low-level data communication, local clusters that are out of contact can use information fusion technology to piece together a local situational map.

[0112] The multi-agent simulation inference and game learning method provided in this invention, by adopting a communication architecture that combines centralized inference with distributed execution, endows the cluster with strong fault tolerance and autonomy. Even in extreme disadvantages such as the destruction of the central server or the complete suppression of global communication, the surviving distributed agents will not fight alone or wait in place, but will be able to quickly "band together" and maintain minimum formation coordination and autonomous risk avoidance based on exchanged local observation data and their own local policy networks, thereby maximizing the continuity of tasks and the survival rate of the cluster.

[0113] In some embodiments, inputting the local observation data and the scheduling instruction into the local policy network and outputting a personalized action policy includes: Acquire historical interaction data of adversarial targets in a simulated environment; The interaction history data and the local observation data are input into a pre-trained adversary prediction model, which outputs the short-term action prediction results of the adversary target; the short-term action prediction results are updated based on new observation evidence. The short-term action prediction results, the local observation data, and the scheduling instructions are input into the local policy network to output a personalized action policy.

[0114] Specifically, traditional distributed intelligent agents often rely solely on their current local observations to perform reactive planning when executing instructions from superiors. However, in game-theoretic situations involving incomplete information, such as close-range drone combat or robot combat, the opponent's strategy is dynamically changing. If an agent reacts only based on the opponent's current posture or position, it often misses opportunities or falls into traps due to decision-making delays.

[0115] To overcome this deficiency, this invention innovatively introduces a short-timescale adversary inference mechanism based on deep learning at the distributed agent level. It integrates an agent behavior inference module, constructs an adversary behavior model based on deep learning technology, focuses on the dynamic characteristics of key adversaries through an attention mechanism, analyzes agent behavior patterns using multi-timescale prediction and intent recognition algorithms, and continuously optimizes inference accuracy through a belief update mechanism, thereby achieving accurate prediction of the behavioral intentions of other agents.

[0116] First, the distributed intelligent agent uses its own sensors or local communication links to collect and acquire historical interaction data of specific adversarial targets in the simulation environment, such as the opponent's movement trajectory, speed changes, and frequency of fire strikes over a period of time.

[0117] Subsequently, the distributed agent combines the aforementioned historical interaction data with current local observation data and inputs them into an adversary prediction model pre-deployed locally within the agent. This model can be a short-term branch of a multi-timescale prediction model built on recurrent neural networks or time-series attention mechanisms. By extracting features from historical behavioral patterns, the model can quickly output short-term action predictions of the adversary, such as predicting the adversary's next movement direction or attack intent.

[0118] In particular, this embodiment of the invention also introduces a belief update mechanism. As the simulation progresses, the agent continuously acquires new local observation data, i.e., new observation evidence. The opponent prediction model uses Bayesian inference or implicit state update algorithms to dynamically correct the short-term action prediction results in real time, thereby avoiding prediction bias caused by the lack of early information and ensuring the accuracy of the prediction.

[0119] Finally, the agent concatenates or fuses the short-term action prediction results, local observation data, and scheduling instructions issued by the central server, and inputs them into its own local policy network to output the final personalized action policy.

[0120] The multi-agent simulation and game learning method provided in this invention upgrades the agent's micro-decision from "passive response" to "active response" by introducing short-term opponent action prediction, which greatly improves the survival rate and task completion rate in high-speed confrontation environment. By integrating macro-level scheduling instructions with micro-level dynamic prediction, it ensures that the agent's actions do not deviate from the overall strategy of the central server, and also gives the agent tactical autonomy to flexibly deal with changes in the enemy situation.

[0121] Figure 7 This is a schematic diagram of the system deployment of the multi-agent simulation inference and game learning method provided by this invention in a typical application scenario, such as... Figure 7 As shown, in actual deployment, the system's hierarchical structure is first determined based on task requirements. Taking a drone swarm mission as an example, a global simulation center is established to be responsible for overall task planning, multiple regional controllers are responsible for coordinating different functional unit groups, and multiple unit controllers directly control individual drone platforms. Then, communication configuration is performed. The configuration center's simulation nodes adopt reliable communication protocols to ensure the reliability of command transmission; the distributed intelligent agent communication network adopts efficient communication protocols to improve data transmission efficiency; and cross-level communication interfaces are established to achieve protocol conversion and data format unification. Afterward, the training process begins. System training is divided into several stages: unit-level pre-training, where each unit controller learns basic skills in a simple environment; regional-level coordination training, where regional controllers learn task decomposition and unit coordination; and global-level joint training, where the global simulation center learns overall task planning and resource allocation.

[0122] Through the aforementioned multi-stage hierarchical training and rigorous communication configuration, the control nodes and distributed agents at all levels within the system have acquired mature collaborative evolution and environmental adaptation capabilities, enabling them to effectively transfer learned experience to real adversarial environments. In intelligent game scenarios, the method provided in this invention demonstrates significant advantages. Through a multi-level control architecture, intelligent communication mechanisms, and an advanced behavior inference system, it achieves efficient collaborative decision-making in complex adversarial environments, outperforming traditional architectures in terms of task completion rate, resource utilization, and decision-making timeliness.

[0123] The apparatus provided in the embodiments of the present invention will be described below. The apparatus described below can be referred to in correspondence with the method described above.

[0124] Figure 8 This is one of the structural schematic diagrams of the multi-agent simulation inference and game learning device provided by the present invention, such as... Figure 8 As shown, the device is applied to the central server and includes a sensing module 810, a scheduling module 820, a strategy module 830, and a distribution module 840 connected in sequence.

[0125] The perception module 810 is used to receive local observation data from multiple distributed intelligent agents in a simulation environment, and to construct global situation features based on the local observation data. The scheduling module 820 is used to input the global situation features into the game solver for multi-objective optimization and generate scheduling instructions; the game solver is used to model the task allocation problem of the distributed agent as an incomplete information game. The strategy module 830 is used to input the global situation features into the global value function network, output a long-term expected reward value that characterizes the cooperative interaction state among the multiple distributed agents, and generate a strategy update gradient based on the long-term expected reward value. The distribution module 840 is used to distribute the policy update gradient and the scheduling instruction to the corresponding distributed agents through a multi-level communication channel.

[0126] The multi-agent simulation inference and game learning device provided in this invention receives local observation data from multiple distributed agents in a simulation environment and constructs global situation features. This breaks the "information island" dilemma caused by the limited local vision of a single agent, effectively aggregating fragmented information at the bottom level and providing a macroscopic and comprehensive global perception foundation for the system. By inputting the global situation features into a game solver for multi-objective optimization, scheduling instructions are generated, and the task allocation problem is modeled as an incomplete information game, which highly matches the information asymmetry and uncertainty in real complex adversarial environments. Under complex constraints, scientific task decomposition and optimal resource scheduling are achieved, effectively avoiding decision conflicts and task overlaps among distributed individuals. By inputting the global situation features into a global value function network, the device outputs a long-term expected return value representing the collaborative interaction state, and generates policy updates based on this value. The gradient mechanism constructs a hierarchical reinforcement learning mechanism of "global evaluation-local update" to solve the typical "credit allocation" problem in multi-agent reinforcement learning from a macro-level collaborative perspective. This enables each agent to optimize its policy in the direction of maximizing global benefits, thereby significantly accelerating the convergence speed of model training. By distributing policy update gradients and scheduling instructions to the corresponding distributed agents through multi-level communication channels, a hierarchical and guided information transmission channel is established. This ensures that macro-level command intentions and cross-level learned knowledge can be accurately and efficiently delivered to the bottom-level nodes, effectively alleviating the communication storm problem caused by massive node interactions and improving the overall system collaboration and communication efficiency. By establishing a hierarchical control structure and an efficient global communication mechanism, the mechanism solves the problems of decision conflict, low communication efficiency, slow learning convergence, and insufficient collaborative effectiveness in traditional multi-agent systems in complex tasks.

[0127] Figure 9 This is the second structural schematic diagram of the multi-agent simulation inference and game learning device provided by the present invention, as shown below. Figure 9 As shown, the device is applied to a distributed intelligent agent and includes an observation module 910, a receiving module 920, an update module 930, an action generation module 940, and an execution module 950 connected in sequence.

[0128] Observation module 910 is used to acquire local observation data and upload the local observation data to the central server. The receiving module 920 is used to receive the scheduling instructions and policy update gradients issued by the central server; Update module 930 is used to update the network parameters of the pre-deployed local policy network of the distributed agent based on the policy update gradient; Action generation module 940 is used to input the local observation data and the scheduling instructions into the local policy network and output personalized action policies; The execution module 950 is used to execute the personalized action strategy in a simulation environment.

[0129] The multi-agent simulation and game-playing learning device provided in this invention provides an accurate data foundation for the central server to aggregate underlying information, construct global situational features, and perform macroscopic multi-objective optimization by acquiring local observation data and uploading it to the central server, thus breaking down information silos between distributed agents. By receiving scheduling instructions and policy update gradients, and updating the network parameters of the pre-deployed local policy networks of the distributed agents based on these gradients, a top-down, cross-level knowledge transfer and gradient propagation path is constructed. This allows the distributed agents to avoid blindly exploring from scratch and fully utilize a global perspective. The individual contribution values ​​are analyzed to guide the training of local models, which significantly improves the learning efficiency and convergence speed of local network models. By inputting local observation data and scheduling instructions issued by the superior into the local policy network, personalized action policies are output and executed in the simulation environment. This achieves an organic combination of macro-level global command intentions and micro-level dynamic environment perception, enabling distributed agents to strictly follow global task allocation and scheduling plans while making responsive and flexible adjustments based on the current local situation. This effectively avoids decision conflicts and task overlaps in complex adversarial environments and significantly improves the collaborative decision-making ability and environmental adaptability of multi-agent clusters.

[0130] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communications bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communications bus 1040. The processor 1010 can call logical commands in the memory 1030 to execute the methods described in the above embodiments, for example: The system receives local observation data from multiple distributed agents in a simulation environment and constructs global situation features based on the local observation data. The global situation features are then input into a game solver for multi-objective optimization to generate scheduling instructions. The game solver models the task allocation problem of the distributed agents as an incomplete information game. The global situation features are input into a global value function network, which outputs a long-term expected payoff value characterizing the collaborative interaction state among the multiple distributed agents. Based on the long-term expected payoff value, a policy update gradient is generated. The policy update gradient and the scheduling instructions are distributed to the corresponding distributed agents through a multi-level communication channel.

[0131] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] The processor in the electronic device provided in this embodiment of the invention can call logical instructions in the memory to implement the above method. Its specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, which will not be repeated here.

[0133] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0134] The specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, so it will not be repeated here.

[0135] This invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0136] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent simulation inference and game theory learning method, applied to a central server, characterized in that, include: Receive local observation data from multiple distributed intelligent agents in a simulation environment, and construct global situation features based on the local observation data; The global situation features are input into the game solver for multi-objective optimization to generate scheduling instructions; the game solver is used to model the task allocation problem of the distributed agent as an incomplete information game. The global situation features are input into a global value function network, which outputs a long-term expected return value that characterizes the collaborative interaction state among the multiple distributed agents. Based on the long-term expected return value, a policy update gradient is generated. The policy update gradient and the scheduling instructions are distributed to the corresponding distributed agents through multi-level communication channels.

2. The multi-agent simulation inference and game learning method according to claim 1, characterized in that, The step of inputting the global situation features into the game solver for multi-objective optimization and generating scheduling instructions includes: Obtain the interaction history data of the adversarial target in the simulation environment; The interaction history data and the local observation data are input into a pre-trained adversary behavior modeling network, which outputs a long-term strategy intent prediction result for the adversary target; the long-term strategy intent prediction result is updated based on new observation evidence. The global situation characteristics and the long-term strategy intention prediction results are input into the game solver for multi-objective optimization to generate the scheduling instructions.

3. The multi-agent simulation inference and game learning method according to claim 1, characterized in that, The step of generating the policy update gradient based on the long-term expected return value includes: Based on the long-term expected return value, value decomposition technique is used to analyze the individual contribution. Based on the individual contribution, the policy update gradient is generated for different distributed agents.

4. A multi-agent simulation inference and game theory learning method, applied to distributed agents, characterized in that, include: Acquire local observation data and upload the local observation data to the central server; Receive scheduling instructions and policy update gradients issued by the central server; Based on the policy update gradient, update the network parameters of the pre-deployed local policy network of the distributed agent; The local observation data and the scheduling instructions are input into the local policy network to output personalized action policies. The personalized action strategy is executed in a simulation environment.

5. The multi-agent simulation inference and game learning method according to claim 4, characterized in that, The method further includes: When a communication failure occurs between the distributed intelligent agent and the central server, a direct collaborative link is established with other distributed intelligent agents within a preset proximity range through a local communication protocol. The local observation data is exchanged with other distributed agents within a preset proximity range based on the direct collaborative link.

6. The multi-agent simulation inference and game learning method according to claim 4, characterized in that, The step of inputting the local observation data and the scheduling instructions into the local policy network and outputting personalized action policies includes: Acquire historical interaction data of adversarial targets in a simulated environment; The interaction history data and the local observation data are input into a pre-trained adversary prediction model, which outputs the short-term action prediction results of the adversary target; the short-term action prediction results are updated based on new observation evidence. The short-term action prediction results, the local observation data, and the scheduling instructions are input into the local policy network to output a personalized action policy.

7. A multi-agent simulation inference and game learning device, applied to a central server, characterized in that, include: The perception module is used to receive local observation data from multiple distributed agents in a simulation environment and construct global situation features based on the local observation data. The scheduling module is used to input the global situation features into the game solver for multi-objective optimization and generate scheduling instructions; the game solver is used to model the task allocation problem of the distributed agent as an incomplete information game. The strategy module is used to input the global situation features into the global value function network, output the long-term expected reward value to characterize the collaborative interaction state among the multiple distributed agents, and generate a strategy update gradient based on the long-term expected reward value. The distribution module is used to distribute the policy update gradient and the scheduling instruction to the corresponding distributed agents through a multi-level communication channel.

8. A multi-agent simulation inference and game-playing learning device, applied to distributed intelligent agents, characterized in that, include: The observation module is used to acquire local observation data and upload the local observation data to the central server. The receiving module is used to receive scheduling instructions and policy update gradients issued by the central server; The update module is used to update the network parameters of the pre-deployed local policy network of the distributed agent based on the policy update gradient. The action generation module is used to input the local observation data and the scheduling instructions into the local policy network and output personalized action policies. The execution module is used to execute the personalized action strategy in the simulation environment.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the multi-agent simulation deduction and game learning method as described in any one of claims 1 to 3, or implements the multi-agent simulation deduction and game learning method as described in any one of claims 4 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-agent simulation deduction and game learning method as described in any one of claims 1 to 3, or implements the multi-agent simulation deduction and game learning method as described in any one of claims 4 to 6.

Images