A cross-domain cooperative simulation deduction method and system for complex adversarial systems

Abstract

The application discloses a cross-domain cooperative simulation deduction method and system for a complex confrontation system, enemy and ally combat units in a confrontation scene are modeled as multiple intelligent agents, and a unified cross-domain interaction framework is constructed, so that the intelligent agents in multiple combat domains are integrated into the same simulation environment for operation; cooperative deduction of combat unit mobility, sensing, interaction and result statistics is realized through a step-by-step simulation mode; a closed-loop interaction interface between the simulation system and a reinforcement learning algorithm is further established, a state space, an action space and a reward function are defined for the combat units, decision control based on a learning strategy is realized; and the strategy obtained through training is introduced into the simulation system for deduction, and the confrontation process under different strategy combinations is evaluated, so that the cross-domain cooperative simulation deduction under the complex confrontation scene is realized.

Description

Technical Field

[0001] This invention relates to the field of modeling and simulation, specifically to a cross-domain collaborative simulation and deduction method and system for complex adversarial systems. Background Technology

[0002] With the development of complex system modeling and computer simulation technologies, simulation exercises have become an important technical means for analyzing and evaluating complex adversarial scenarios, verifying solutions, and supporting decision-making. By modeling multiple types of combat units, environmental elements, and their interrelationships in a virtual environment, different strategy combinations and operational situations can be repeatedly simulated without relying on actual equipment conditions, thereby reducing testing costs and improving decision-making efficiency.

[0003] Existing simulation and deduction systems mostly employ rule-based or script-driven modeling methods, pre-setting the behavioral logic of combat units. While this approach can complete basic simulation tasks in small-scale scenarios with relatively fixed interaction relationships, it still has significant shortcomings when facing complex scenarios involving high dynamics, strong adversarial forces, and multi-domain collaboration. On the one hand, the behavioral decisions of combat units typically rely on manual experience rules, lacking the ability to autonomously learn and evolve based on changes in environmental states, making it difficult to realistically reflect the emergent characteristics of behavioral strategies during complex confrontations. On the other hand, the interaction relationships between different combat domains are often simplified to static or weakly coupled models, lacking a unified cross-domain interaction framework, making it difficult to characterize the co-evolutionary process of information flow, decision flow, and effects among multi-domain elements.

[0004] Furthermore, with the development of intelligent decision-making algorithms such as reinforcement learning, some studies have attempted to introduce learning algorithms into the simulation process to enhance the strategy generation capabilities of combat units. However, existing solutions often loosely couple reinforcement learning algorithms with simulation systems, lacking efficient and standardized interaction interfaces between the simulation platform and the learning algorithms. This results in delays or inconsistencies in state acquisition, action execution, and reward feedback, making it difficult to support stable training and real-time simulations in large-scale, multi-agent scenarios. Simultaneously, most systems lack the ability to control simulation parameters, agent behavior, and training strategies in real time, making it difficult to flexibly adjust algorithms or parameter configurations according to changes in the situation during simulations. This limits the applicability and engineering practical value of the simulation results.

[0005] In view of this, the present invention proposes a cross-domain collaborative simulation and deduction method and system for complex adversarial systems. Summary of the Invention

[0006] The purpose of this invention is to provide a cross-domain collaborative simulation and deduction method and system for complex adversarial systems, aiming to solve the problems of insufficient cross-domain collaborative modeling capability of multiple combat units, lack of autonomous evolution of decision-making rules of intelligent agents, and low coupling efficiency of simulation system and reinforcement learning algorithm, which makes it difficult to support efficient deduction of complex adversarial scenarios in existing simulation and deduction technologies.

[0007] In a first aspect, the present invention provides a cross-domain collaborative simulation and deduction method for complex adversarial systems, comprising the following steps:

[0008] The adversarial combat units in the scenario are modeled as a set of multiple agents. The set of multiple agents includes at least a main environment agent for maintaining the global state of the simulation and providing a cross-domain message interaction mechanism, and multiple combat unit agents located in the main environment agent. Each combat unit agent performs state synchronization and event interaction between different combat domains through the cross-domain message interaction mechanism.

[0009] A unified cross-domain collaborative simulation and deduction environment is constructed in the main environment intelligent agent. The multi-agent set is scheduled in a discrete step-by-step operation mode. In each simulation step, the combat unit generation, state update, perception processing, action execution and result statistics are executed in a fixed order.

[0010] At least one combat unit agent is selected as the decision-making subject. In the cross-domain collaborative simulation environment, a reinforcement learning interaction model that is strictly synchronized with the discrete simulation step is constructed for the decision-making subject. The reinforcement learning interaction model obtains state information from the simulation environment, outputs control actions, and receives environmental feedback through a predefined interface, thereby forming a closed-loop interaction between simulation and learning.

[0011] Based on the closed-loop interaction, the decision-making agent is trained to obtain a strategy model, and the strategy model is imported into the cross-domain collaborative simulation environment to drive the decision-making agent to perform simulation without changing the multi-agent modeling structure.

[0012] As a preferred technical solution of the first aspect of the present invention, the main environment agent is used to maintain simulation map information, simulation step size parameters and event scheduling queue, and distributes synchronization signals corresponding to each simulation step to each combat unit agent through the cross-domain message interaction mechanism, so as to constrain each combat unit agent to complete state update and action execution within the same simulation step.

[0013] As a preferred technical solution of the first aspect of the present invention, the combat unit intelligent agent is equipped with a spatial perception model. The spatial perception model is used to obtain the existence status and relative positional relationship of other combat unit intelligent agents within a preset spatial range, and to use the relative positional relationship as one of the state inputs of the reinforcement learning interaction model.

[0014] As a preferred technical solution of the first aspect of the present invention, the multi-agent set includes an enemy combat unit agent, which enters the simulation environment according to a preset generation rule and moves towards the target area along a preset path; when the enemy combat unit agent completes the preset path without being eliminated, the main environment agent records the corresponding defense failure state.

[0015] As a preferred technical solution of the first aspect of the present invention, the multi-agent set includes a weapon and ammunition agent triggered by a combat unit agent. The weapon and ammunition agent is used to perform attack behavior on the enemy combat unit agent within the simulation step, and update the state parameters of the attacked enemy combat unit agent after the attack hits.

[0016] As a preferred technical solution of the first aspect of the present invention, in each simulation step, the generation of combat units, state updates, perception processing, action execution, and result statistics are completed sequentially in a fixed order determined during the initialization phase of the simulation system.

[0017] As a preferred technical solution of the first aspect of the present invention, the reinforcement learning interaction model interacts with the simulation environment through a state acquisition interface, an action execution interface and a reward feedback interface, and each interface is called once in each simulation step to constrain the decision-making subject to perform at most one control action in each simulation step.

[0018] As a preferred technical solution of the first aspect of the present invention, the strategy training step S104 further includes: adjusting the generation frequency of the enemy combat unit agent according to the current training round number in multiple training rounds to form a training process from low complexity to high complexity; when obtaining the state information of the current simulation step, combining the state information corresponding to the current simulation step and the previous predetermined number of consecutive simulation steps into a historical state array, which is used as the input of the reinforcement learning interaction model.

[0019] As a preferred technical solution of the first aspect of the present invention, the strategy training step S104 further includes: in the training start stage, randomizing the simulation environment parameters to introduce environmental differences; and in the state acquisition process, processing the loss of some state information with a preset probability to simulate the situation of incomplete information acquisition.

[0020] In a second aspect, the present invention provides a cross-domain collaborative simulation and deduction system for complex adversarial systems, used to perform the first aspect, including:

[0021] The agent layer is used to perform multi-agent modeling of enemy and friendly combat units in complex adversarial scenarios. The agent layer includes multiple combat unit agents. Each combat unit agent uses a state control structure to describe its behavior state and state transition relationship, which is used to generate the corresponding combat unit's behavior requests and state change information.

[0022] The simulation layer is used to build a unified cross-domain collaborative simulation and inference environment. The simulation layer is configured to uniformly schedule the multiple combat unit agents according to discrete simulation steps. In each simulation step, it receives behavior requests from the agent layer, updates the state and spatial distribution of the combat units, and generates corresponding simulation state information.

[0023] The algorithm layer is used to construct a reinforcement learning decision model. The algorithm layer is configured to receive the simulation state information output by the simulation layer, perform policy training or inference calculation based on the simulation state information, and output decision control information to the simulation layer to drive the behavior of combat units.

[0024] The simulation layer is configured to export simulation state information to the algorithm layer in each simulation step and receive decision control information returned by the algorithm layer for application, so as to form a simulation inference closed loop that combines multi-agent modeling, cross-domain collaborative simulation and learning decision-making.

[0025] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0026] This invention models enemy and friendly combat units in complex adversarial scenarios as a collection of multiple intelligent agents and constructs a cross-domain message interaction mechanism in the main environment intelligent agent, enabling combat units in different combat domains to synchronize states and interact with events in a unified simulation environment, thereby avoiding the problem of collaborative failure caused by information fragmentation in multi-domain simulation.

[0027] A discrete step-by-step operation mode is adopted to uniformly schedule multiple agents and strictly synchronize the simulation steps with the reinforcement learning interaction model, so that the state evolution and decision update of the combat unit are consistent, reducing the timing deviation between the simulation system and the learning model. By directly importing the trained policy model into the existing simulation environment for inference without reconstructing the multi-agent modeling structure, a stable closed loop is formed between policy training and simulation execution, thereby improving the repeatability and controllability of simulation inference in complex adversarial scenarios. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0029] Figure 1 This is a flowchart of the cross-domain collaborative simulation and deduction method described in this invention;

[0030] Figure 2 This is a schematic diagram of the cross-domain collaborative simulation and deduction system architecture described in this invention.

[0031] Figure 3 This is a comparison chart of training effects under different reinforcement learning schemes in the cross-domain collaborative simulation and inference system described in this invention.

[0032] Figure 4 This is a comparison chart of task performance under different reinforcement learning schemes in the cross-domain collaborative simulation and inference system described in this invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings.

[0035] Throughout the accompanying drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. The described embodiments are only a part of the embodiments of this application, not all of them. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0036] Example 1

[0037] like Figure 1 This embodiment provides a cross-domain collaborative simulation and deduction method for complex adversarial systems, applicable to adversarial scenarios encompassing multiple operational domains such as airspace and seaspace. This method achieves collaborative deduction and strategy evaluation in complex adversarial scenarios through multi-agent cross-domain modeling, the construction of a unified simulation and deduction environment, and the introduction of a reinforcement learning decision-making mechanism.

[0038] The adversarial combat units in the scenario are modeled as multiple agents, and a unified cross-domain interaction framework is introduced. The multiple agents include at least the host environment agent, the friendly airspace combat unit agent, the enemy airspace combat unit agent, the enemy naval combat unit agent, and the weapon and ammunition agent.

[0039] Among them, the main environment intelligent agent is used to carry the unified simulation environment, maintain the GIS map, simulation step size, event scheduling queue and global state information, and provide cross-domain interaction channels for each sub-intelligent agent through the message mechanism.

[0040] Our airspace combat unit intelligent agents include at least internal states such as position, speed, heading, fuel value, payload, and combat status, and are set up with a fan-shaped perception range for real-time perception of enemy airspace and sea intelligent agents within the range; their fuel consumption rate is correlated with flight speed to reflect the impact of maneuvering behavior on resource consumption.

[0041] Enemy airspace combat unit agents and enemy seaspace combat unit agents are generated in the simulation environment according to preset generation rules and move towards the protected area; when the enemy combat unit completes the predetermined movement path without being destroyed, a defense failure event is triggered.

[0042] The weapon and ammunition agent is generated by our airspace combat unit agent and is used to attack enemy combat units. After a hit, the target's health status is updated according to preset rules.

[0043] All of the aforementioned intelligent agents publish their status and subscribe to events through the messaging mechanism provided by the host environment intelligent agent, thereby forming a unified interaction framework across airspace and seaspace.

[0044] The multiple agents are integrated into the same simulation environment to construct a cross-domain collaborative simulation and inference system. The simulation adopts a discrete step-by-step operation mode, with each simulation step corresponding to a fixed time length, which is used to uniformly schedule the state updates and action execution of each agent.

[0045] In each simulation step, the processes of generating enemy combat units, updating agent maneuvers, refreshing perception information, executing actions, determining attack hits, and updating statistical information are completed sequentially.

[0046] Simultaneously, a simulation interface interaction framework is constructed for real-time control of the simulation process. This framework supports adjustments to map scale parameters, combat unit generation rate, friendly combat unit maneuver parameters, and strategy selection. It also displays real-time simulation data such as the number of defense failures, cumulative damage, and remaining resources, thereby enabling dynamic adjustment of simulation parameters and visualized monitoring of the simulation process.

[0047] Using our airspace combat unit intelligent agent as the decision-making subject of reinforcement learning, a Markov decision process is established for it in the simulation system.

[0048] The state space of the Markov decision process includes at least the threat level information of enemy combat units, the number of enemy units within the protected area, the position and maneuver status of friendly combat units, the remaining resource information, and the position information of key targets. When the number of enemy combat units is insufficient to fill the state dimension, preset values ​​are used to fill it to ensure the consistency of the state space dimension.

[0049] The action space includes at least control actions such as attack, supply, speed adjustment and direction adjustment, and each simulation step can execute at most one action.

[0050] The reward function is calculated based on the attack hit effect, the completion status of the defense task, the resource consumption status, and the violation behavior, and is used to guide the learning strategy to achieve a balance between the defense success rate and the resource utilization efficiency.

[0051] In the simulation system, the intelligent agents of our airspace combat units are provided with interfaces for observation acquisition, action execution, and reward calculation, and a unified calling interface is provided in the main system entry point. This enables the reinforcement learning algorithm to complete state acquisition, action selection, environmental feedback, and state transition in each simulation step, thereby forming a closed-loop interaction between the simulation and the learning algorithm.

[0052] Based on the closed-loop interaction mechanism, reinforcement learning training is performed on our airspace combat unit agents, and a corresponding strategy model is generated after the training is completed.

[0053] The trained strategy model is imported into the cross-domain collaborative simulation system, and the strategy model is invoked through timed triggering to make decision-making inferences on the current simulation state, driving the combat unit to execute corresponding actions, thereby realizing simulation and inference based on the learned strategy.

[0054] To improve the stability and generalization ability of the simulation results, an incremental training mechanism, a historical state joint input mechanism, and an environmental disturbance mechanism were introduced during the training and simulation process. Different strategy combinations were compared and simulated under the same simulation conditions. The simulation effect was evaluated based on indicators such as round rewards, number of defense failures, and resource consumption.

[0055] Example 2

[0056] Based on Example 1, such as Figure 2 As shown, this embodiment uses multiple agents as the modeling carrier for combat units, providing a cross-domain collaborative simulation and deduction method for complex adversarial systems. A unified interaction framework enables cross-domain collaborative simulation, and a closed-loop deduction link is formed with reinforcement learning decision training; the method includes the following steps:

[0057] Step 1: Model the enemy and friendly combat units in the adversarial scenario as multiple intelligent agents and establish a unified framework for their cross-domain interaction.

[0058] Specifically, the steps to model enemy and friendly combat units in adversarial scenarios as multi-agent systems and establish a unified framework for their cross-domain interaction are as follows:

[0059] Step 101: Establish an "aircraft carrier" intelligent agent and restrict its movement trajectory through a state diagram: move between three nodes in the GIS map at a speed of 72 km / h, and at the same time generate our airspace intelligent agent and provide it with 36 fuel points per minute and 12 minutes / ammunition resupply.

[0060] Step 102: Establish an "Aircraft" agent in our airspace with a speed range of 500km / h~900km / h, an initial speed of 600km / h, a fuel value of 360, and a payload of 6. Establish predefined rules through state graphs and build a reinforcement learning interaction interface based on state graph transitions.

[0061] Step 103: Create a "VisualAgent" agent, which is generated around the "Aircraft" agent in real time at 30° intervals based on the "Aircraft" agent's position, to assist its movement.

[0062] Step 104: Establish the enemy airspace "Enemy" agent, with the default setting as... Each unit spawns at a speed of 400 km / h from a distant node, with a health value of 2. Its target is a random coordinate within our defense area; upon reaching it, it returns. A successful return is considered a failure of our defense. When its health value falls below half, its speed decreases to 360 km / h.

[0063] Step 105: Establish the enemy sea area "Enemiesboat" intelligent agent, defaulting to Each unit spawns at a speed of 108 km / h from a distant node, with a health value of 2. Its target is two nodes within our defense area; upon reaching them, it will return. A successful return is considered a failure of our defense. When its health value drops below half, its speed decreases to 90 km / h.

[0064] Step 106: Establish the "Missile" air-to-air missile agent, generated in our airspace, with a movement speed of 3600 km / h, targeting enemy airspace agents. Upon impact, the target's health value is reduced, with the specific reduction depending on the distance between the missile and the target at launch, as shown in Table 1.

[0065] Step 107: Establish the "Bomb" agent for anti-ship missiles. The agent is generated in our airspace and moves at a speed of 3600 km / h. The target is the enemy's maritime agent. When it hits, the target's health value is reduced. The specific reduction depends on the distance between the missile and the target at the time of launch, as shown in Table 1.

[0066] In step 102, when establishing predefined rules for our airspace agents, our agents are divided into a defense state and a resupply state. In the defense state, two internal transitions are established to launch missiles at two different types of enemy agents; in the resupply state, the agents return to the "aircraft carrier" agent for fuel and ammunition resupply.

[0067] In step 102, the fuel consumption rate is: / minute, of which This is the fuel consumption factor, which defaults to 1. For the speed of the intelligent agent, The initial velocity of the agent.

[0068] In step 102, when modeling our airspace agent, a 200km radius, 120° sector-shaped perception range is established for the agent, which can perceive enemy agents within the range in real time.

[0069] Step 2: Integrate the intelligent agents mentioned in Step 1 into the same environment to establish a cross-domain collaborative simulation and deduction system, and design a simulation interface interaction framework to realize functions such as dynamically adjustable simulation parameters and real-time collaborative interaction.

[0070] Specifically, a cross-domain collaborative simulation system is established, employing a step-by-step computation method, with each step lasting one minute. The simulation interface is designed using a slider, radio buttons, and checkboxes to link with GIS map parameters, agent parameters, and other parameters. The system features real-time map scale modification, dynamic adjustment of friendly agent parameters and enemy agent generation speed via sliders, and real-time viewing of combat data.

[0071] Step 3: Build a rapid interaction framework between the simulation system and reinforcement learning. For combat units in the scenario, define their Markov decision process and design key components such as state space, action space, and reward function.

[0072] Specifically, the steps for building a rapid interaction framework between the simulation system and reinforcement learning in step three are as follows:

[0073] Step 301: Define the Markov decision process: state space, action space, and reward function.

[0074] Step 302: For our intelligent agent, define relevant functions such as "getObservation", "action", and "calReward", and define relevant algorithm interfaces in "Main".

[0075] Step 303: Export the simulation system as a standalone Java program, write code based on the rl4j reinforcement learning library, and train it using the Double DQN algorithm; import the trained policy back into the simulation system and trigger decisions by establishing timed events.

[0076] Furthermore, step 301 defines a Markov decision process: state space, action space, and reward function. Specifically:

[0077] State space:

[0078] ;

[0079] ;

[0080] in: The danger coefficient of enemy maritime intelligent agents (based on the inverse square distance ratio); The danger coefficient of the enemy airspace agent (based on the inverse square distance ratio); To protect the number of intelligent agents in the marine area; To protect the number of airspace agents within the region; This represents the current ammunition load. To determine whether the state is in a defensive posture; and The latitude and longitude of our airspace intelligent agent; , , , , and These are the latitude and longitude coordinates of the three enemy airspace agents listed first in the index; , , , , and These are the latitude and longitude coordinates of the first three enemy sea area agents in the index; (if the number of agents is insufficient, use 0 instead of the above latitude and longitude coordinates). For the speed of our intelligent agent; This represents the fuel value of our intelligent agent; and This refers to the latitude and longitude of our aircraft carrier's intelligent agent.

[0081] Action space: ;

[0082] in: No action; and This is an attack action; and This is a resupply action; and For speed control actions; This is for directional control actions. At most one action can be executed per step.

[0083] Reward function: ;

[0084] in: Indicates a hit reward. Indicates task punishment. Indicates danger and punishment. It indicates control and punishment.

[0085] Specifically, the hit reward is as follows: when a missile hits an enemy agent, a reward of 100 times the damage value is obtained. The damage value calculation rules are shown in Table 1.

[0086] Table 1: Damage Value Calculation Rules

[0087]

[0088] In addition, when the target is within a 200km, 120° fan-shaped area of ​​our agent during an attack, we receive a reward of 100, which encourages our agent to track enemy agents.

[0089] When an enemy airspace agent or seaspace agent is completely defeated, you will receive a reward of 80 and 50 respectively (enemy airspace agents are faster, so the reward will be compensated accordingly); if the number of that agent type is greater than that of another type when defeated, you will receive an additional reward of 40, which encourages your agents to attack the enemy agent type that has a larger number of enemies.

[0090] The penalty for failing to destroy the enemy AI is -200 to -50. The punishment, among which The cumulative number of agents of this type that have not been destroyed in the current training round.

[0091] The danger penalty is: when an enemy agent is present within the defense zone, [the player receives] [a certain amount] per minute. The penalty is calculated separately for each type of enemy agent.

[0092] The control penalties are as follows: When a friendly agent exceeds the defense range, it incurs a penalty of -50 per minute; if a friendly agent performs a defensive action while in a resupply state, the action will fail, resulting in a penalty of -10; if a friendly agent remains in a defensive state even when no enemy agent is present within the defense area, it will incur a penalty of -0.001 per minute. The punishment.

[0093] Furthermore, the specific steps for defining the relevant algorithm interface in "Main" in step 302 are as follows: The Markov decision process described above is defined within the "Aircraft" agent using "getObservation," "action," and "calReward." In the "Main" interface, "getObservation," "action," "calReward," and "rewardReset" are defined to interact with the functions within the "Aircraft" agent. In the training code, these functions are called using the root. method for interaction.

[0094] Furthermore, the specific steps in step 303 to export the simulation system as an independent Java program are as follows: export the experiment of type "customExperiment" and clear all default code.

[0095] Furthermore, in step 303, the specific steps for writing code based on the rl4j reinforcement learning library and training using the Double DQN algorithm are as follows: import rl4j and related Anylogic dependency libraries, create the Training class, and implement the Markov Decision Process interface by creating the MDP environment, configuring the Double DQN algorithm and neural network hyperparameters, starting training, and saving the strategy and training logs. This establishes a framework connecting the Anylogic simulation environment and rl4j. Some algorithm hyperparameters and neural network hyperparameters are shown in Table 2.

[0096] Table 2 shows some algorithm hyperparameters and neural network hyperparameters:

[0097]

[0098] Furthermore, in step 303, the trained policy is imported back into the simulation system. The specific steps for establishing timed event-triggered decision-making are as follows: import the rl4j learning library and the trained policy into the simulation system, establish a timed event, and when triggered, give the optimal action for the current state based on the imported policy and the current state of the simulation environment.

[0099] Step 4: Improvements were made to the original reinforcement learning algorithm by designing modules such as course learning and training using historical states.

[0100] Specifically, the steps in step four that improve upon the original reinforcement learning algorithm are as follows:

[0101] Step 401, Add Course Learning: During training, adopt a training framework that progresses from easy to difficult, with the enemy agent generation frequency increasing from...

[0102]

[0103] The decision was made to gradually increase the number of training sessions, reaching a maximum of 300 rounds. This indicates the current training round number.

[0104] Step 402: Training is performed using historical states. During training, the state space of the three most recent time steps is integrated into 72 dimensions. An additional historical state array is maintained in AnyLogic, and the maintenance process is shown in Table 3.

[0105] Table 3 is the maintenance status table:

[0106]

[0107] Step 403: Add environmental randomization: At the start of each training round, dynamically change the weather conditions to simulate real-world weather changes and improve policy generalization performance. At the start of each training round, 10% of the weather conditions are severe, and 90% are normal. Under severe weather conditions, the energy consumption of all friendly agents increases by 1.05 times, the initial velocity of enemy agents in the sea area decreases from 30 m / s to 25 m / s, and missile accuracy decreases by 10%.

[0108] Step 404: Add a state loss mechanism: During training, there is a 5% probability of losing information in the state each time a state is obtained, and the lost information is replaced with 0.

[0109] Step 405: Add a reconnaissance probability mechanism: When an enemy agent enters the protected area, there is an 80% probability that it will be detected every 6 minutes. Only the detected agents can be used as attack targets.

[0110] Example 2

[0111] Building upon Example 1, a rapid interactive framework for the simulation system and reinforcement learning is constructed. For combat units in the scenario, a Markov decision process is defined, and key components such as the state space, action space, and reward function are designed. Step four includes two modules: course learning and training using historical states. This example follows the above steps for training, using the parameters shown in Tables 1 and 2. Training images are shown below. Figure 3 As shown. Figure 3In the graph, ① represents the training result of the original Double DQN; ② represents the training result of Double DQN + the course learning module; ③ represents the training result of Double DQN + the module using historical states; and ④ represents the training result of Double DQN + the course learning module + the module using historical states. All results were trained three times under different seeds. The shaded area in the graph indicates that the reward for the current round has a 95% confidence level within the shaded area. From the graph, all methods converge after approximately 600 rounds, and the Double DQN + course learning module + the module using historical states ultimately achieves the highest convergent reward.

[0112] Next, the task performance of this embodiment will be evaluated. The task performance comparison diagram of each combined strategy in this cross-domain collaborative simulation and inference system is shown in the figure below. Figure 4 As shown.

[0113] As seen in the images, adding course learning alone results in a slight decrease in damage due to environmental differences; the Double DQN+ course learning module plus the module for training using historical states achieves the highest total damage per round and the fewest number of agents failing to defend, demonstrating the best performance.

[0114] Finally, Table 4 shows the average rewards of the four strategy combinations proposed in this example under 10 different seed simulation environments.

[0115] Table 4 shows the average reward under 10 different seed simulation environments:

[0116]

[0117] In summary, under the same verification environment, the Double DQN+course learning module+training module using historical states yields the highest average reward.

[0118] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A cross-domain collaborative simulation and deduction method for complex adversarial systems, characterized in that, Includes the following steps: The adversarial combat units in the scenario are modeled as a set of multiple agents. The set of multiple agents includes at least a main environment agent for maintaining the global state of the simulation and providing a cross-domain message interaction mechanism, and multiple combat unit agents located in the main environment agent. Each combat unit agent performs state synchronization and event interaction between different combat domains through the cross-domain message interaction mechanism. A unified cross-domain collaborative simulation and deduction environment is constructed in the main environment intelligent agent. The multi-agent set is scheduled in a discrete step-by-step operation mode. In each simulation step, the combat unit generation, state update, perception processing, action execution and result statistics are executed in a fixed order. At least one combat unit agent is selected as the decision-making subject. In the cross-domain collaborative simulation environment, a reinforcement learning interaction model that is strictly synchronized with the discrete simulation step is constructed for the decision-making subject. The reinforcement learning interaction model obtains state information from the simulation environment, outputs control actions, and receives environmental feedback through a predefined interface, thereby forming a closed-loop interaction between simulation and learning. Based on the closed-loop interaction, the decision-making agent is trained to obtain a strategy model, and the strategy model is imported into the cross-domain collaborative simulation environment to drive the decision-making agent to perform simulation without changing the multi-agent modeling structure.

2. The cross-domain collaborative simulation and deduction method for complex adversarial systems according to claim 1, characterized in that, The main environment agent is used to maintain simulation map information, simulation step size parameters, and event scheduling queues. It also distributes synchronization signals corresponding to each simulation step to each combat unit agent through the cross-domain message interaction mechanism, so as to constrain each combat unit agent to complete state updates and action execution within the same simulation step.

3. The cross-domain collaborative simulation and deduction method for complex adversarial systems according to claim 1, characterized in that, The combat unit agent is equipped with a spatial perception model, which is used to obtain the existence status and relative positional relationship of other combat unit agents within a preset spatial range, and to use the relative positional relationship as one of the state inputs of the reinforcement learning interaction model.

4. The cross-domain collaborative simulation and deduction method for complex adversarial systems according to claim 1, characterized in that, The multi-agent set includes enemy combat unit agents. The enemy combat unit agents enter the simulation environment according to preset generation rules and move towards the target area along a preset path. When the enemy combat unit agents complete the preset path without being eliminated, the main environment agent records the corresponding defense failure state.

5. The cross-domain collaborative simulation and deduction method for complex adversarial systems according to claim 1, characterized in that, The multi-agent set includes weapon and ammunition agents triggered by combat unit agents. These weapon and ammunition agents are used to perform attack actions against enemy combat unit agents within a simulation step and update the state parameters of the attacked enemy combat unit agents after a successful attack.

6. The cross-domain collaborative simulation and deduction method for complex adversarial systems according to claim 1, characterized in that, In each simulation step, the generation of combat units, status updates, perception processing, action execution, and result statistics are completed sequentially in a fixed order determined during the initialization phase of the simulation system.

7. The cross-domain collaborative simulation and deduction method for complex adversarial systems according to claim 1, characterized in that, The reinforcement learning interaction model interacts with the simulation environment through a state acquisition interface, an action execution interface, and a reward feedback interface. Each of these interfaces is called once in each simulation step to constrain the decision-making entity to perform at most one control action in each simulation step.

8. The cross-domain collaborative simulation and deduction method for complex adversarial systems according to claim 1, characterized in that, The strategy training step S104 further includes: adjusting the generation frequency of the enemy combat unit agent according to the current training round number in multiple training rounds to form a training process from low complexity to high complexity; when obtaining the state information of the current simulation step, combining the state information corresponding to the current simulation step and the previous predetermined number of consecutive simulation steps into a historical state array, which is used as the input of the reinforcement learning interaction model.

9. The cross-domain collaborative simulation and deduction method for complex adversarial systems according to claim 1, characterized in that, The strategy training step S104 further includes: in the initial training phase, randomizing the simulation environment parameters to introduce environmental differences; and in the state acquisition process, processing the loss of some state information with a preset probability to simulate the situation of incomplete information acquisition.

10. A cross-domain collaborative simulation and deduction system for complex adversarial systems, used to execute the cross-domain collaborative simulation and deduction method for complex adversarial systems as described in any one of claims 1-9, characterized in that, include: The agent layer is used to perform multi-agent modeling of enemy and friendly combat units in complex adversarial scenarios. The agent layer includes multiple combat unit agents. Each combat unit agent uses a state control structure to describe its behavior state and state transition relationship, which is used to generate the corresponding combat unit's behavior requests and state change information. The simulation layer is used to build a unified cross-domain collaborative simulation and inference environment. The simulation layer is configured to uniformly schedule the multiple combat unit agents according to discrete simulation steps. In each simulation step, it receives behavior requests from the agent layer, updates the state and spatial distribution of the combat units, and generates corresponding simulation state information. The algorithm layer is used to construct a reinforcement learning decision model. The algorithm layer is configured to receive the simulation state information output by the simulation layer, perform policy training or inference calculation based on the simulation state information, and output decision control information to the simulation layer to drive the behavior of combat units. The simulation layer is configured to export simulation state information to the algorithm layer in each simulation step and receive decision control information returned by the algorithm layer for application, so as to form a simulation inference closed loop that combines multi-agent modeling, cross-domain collaborative simulation and learning decision-making.

Images