DPPO deep reinforcement learning agent deduction method

Abstract

The invention discloses a DPPO deep reinforcement learning agent deduction method comprising the following steps: 1) carrying out deduction scene initialization, uniformly loading a game deduction engine, and deploying a DPPO in a task planning scene; 2) a DPPO side outputs a random strategy and supports a discrete / continuous mixed action space, an immediate reward and a situation at the next moment are returned by a deduction engine, and an independent experience playback mechanism is adopted; the DPPO has the advantages of macroscopic task planning and'first learning and second competition '; according to the method, real-time and autonomous learning type game deduction in a sea-air action situation scene is realized, and the method can be directly used for action scheme evaluation of an offshore platform aircraft.

Description

Technical Field

[0001] This invention relates to the field of intelligent agents, specifically a DPPO deep reinforcement learning intelligent agent inference method. Background Technology

[0002] With the deep integration of artificial intelligence, modern game theory is shifting towards a paradigm of "agent-based decision-making": the electromagnetic spectrum is being densely suppressed and cleverly jammed, and factors such as weather amplify environmental uncertainty. Traditional command models relying on manual map interpretation, voice coordination, and hierarchical reporting are no longer sufficient to generate situational awareness and guidance commands within seconds or even milliseconds. Therefore, command systems must introduce self-learning and self-evolving agents across the entire "perception-judgment-decision-execution" chain. Utilizing distributed reinforcement learning, these agents can search for optimal strategies in real-time within a massive state-action space, providing low-latency, high-confidence, and continuously optimized support for maritime platform aircraft, thereby gaining a crucial preemptive time advantage in future high-intensity confrontations.

[0003] Currently, mainstream deep reinforcement learning methods can be summarized into two main categories: one is the traditional rule-based decision-making framework represented by Behavior Tree (BT), which relies on manually preset node logic and state transition rules to achieve condition-action mapping. It performs stably in scenarios with clear rules and fixed scenarios, but lacks online learning capabilities; the other is the Deep Reinforcement Learning (DRL) framework, which autonomously learns trajectory scheduling and cooperative scheduling strategies in a continuous action space through a "trial and error-reward" mechanism, and has the characteristics of adaptability and fast convergence.

[0004] Existing DRL schemes still have shortcomings: 1) The training framework adopts a single-agent perspective, and the platform units lack collaborative channels; 2) The reward function relies on sparse win / loss signals, and high-value intermediate actions do not receive timely feedback, making it difficult for the policy network to learn fine-grained actions; 3) Discrete instructions and continuous parameters constitute a mixed action space, resulting in large policy gradient variance and a significant increase in convergence time; 4) The network hyperparameters are sensitive to the initial unit deployment and situational weather, and the win rate drops significantly when switching to different scenarios, leading to performance degradation. Therefore, researching an agent algorithm with fast convergence, strong generalization, and collaborative decision-making capabilities has direct practical significance for improving the technical support efficiency of maritime platform aircraft. Summary of the Invention

[0005] The purpose of this invention is to provide a DPPO deep reinforcement learning agent inference method, comprising the following steps:

[0006] Step 1) Based on the situational parameter set of the scenario template, load the wargaming engine uniformly and deploy the DPPO agent in the task planning scenario;

[0007] The DPPO agent employs a distributed training architecture, including multiple Actor nodes and Learner nodes; multiple Actor nodes collect data in parallel, and Learner nodes aggregate gradients and update the global policy model.

[0008] Step 2) Utilize the DPPO agent to output actions and write them into the experience cache;

[0009] Step 3) Use the inference engine to return the immediate reward and the next moment's situation of the DPPO agent, and complete the reward and situation feedback;

[0010] Step 4) Employ an independent experience replay mechanism to calculate the GAE advantage estimate for the DPPO cache, perform gradient ascent according to the PPO-Clip objective until the preset convergence judgment condition is met, and output the planning strategy for the agent's aerial action mission to be deployed.

[0011] Furthermore, in step 1), the steps of uniformly loading the wargaming engine include: uniformly loading it into the wargaming engine through the Environment::reset() interface and initializing the simulation environment.

[0012] Furthermore, the simulation environment is a Markov decision process;

[0013] Furthermore, the situational parameter set includes action space modeling, functional regions, state characteristics, and action space parameters;

[0014] Among them, the situational awareness focus area in the functional area is used to dynamically characterize the real-time spatial distribution of all detected, identified and confirmed aerial targets in flight within the current effective coverage area of ​​the friendly sensor network.

[0015] Status characteristics include unit type, carrying device type, and detection type;

[0016] Action space parameters include task type, target location, task unit, task time, and execution details; the task unit is the intelligent agent that performs the task; execution details include task parameter configuration and electromagnetic spectrum management strategy.

[0017] Furthermore, in step 2), the DPPO agent employs GAE-dominance estimation to calculate the dominance function. and the advantage function Substitute the values ​​into the objective function to complete the policy update;

[0018] Furthermore, in the distributed DPPO architecture, after each Actor node of the DPPO agent completes trajectory sampling and gradient calculation locally, the Learner node aggregates all gradients and performs a global parameter update. Subsequently, the updated policy network parameters are broadcast to each Actor to complete a round of collaborative training.

[0019] Furthermore, the intelligent agent's output of the DPPO intelligent agent's aerial action mission planning strategy includes mission type, mission parameters, mission area, mission triggering conditions, and dynamic adjustment mechanism.

[0020] The mission types include patrol missions and strike missions;

[0021] The task parameters include the number of agents executing the task, the task execution interval, and the duration.

[0022] The dynamic adjustment mechanism is used to adjust task parameters and task area.

[0023] Furthermore, the calculation of the DPPO objective function in step 3) takes into account distributed data collection and parameter synchronization;

[0024] Each worker generates data independently, and the advantage function is calculated uniformly under the old strategy parameters to ensure consistent estimation.

[0025] The central node, Learner, aggregates the gradients of all Workers and calculates the global update;

[0026] The distributed data in DPPO has undergone normalization processing;

[0027] Furthermore, DPPO adopts a three-layer structure that combines a centrally coordinating Learner with distributed computing Actors. The top layer is designed with Learner nodes as the control center, the middle layer is designed with a shared gradient region as a gradient cache pool, and the bottom layer is designed with N Actors running in parallel, each containing a local PPO model and an environment interaction module.

[0028] Furthermore, Learner extrapolates environmental interactions and accumulates experience, providing Actors with a basis for decision-making, and during testing, weighing the trade-off between trying new actions and performing known high-reward actions in unknown environments;

[0029] The Learner training process is as follows: retrieve batch data from the queue, calculate the objective function and gradient and update the parameters, and periodically save the model and performance metrics;

[0030] The Actor workflow is as follows: retrieve the latest model from the Learner via _update_model(), perform multi-step interaction using _nstep_rollout(), calculate the GAE advantage estimate, and push data to the Learner via ZMQ.

[0031] The technical effects of this invention are undeniable, and its beneficial effects are as follows:

[0032] Efficient agent decision-making and fast convergence: Agents operate independently, each completing a single-game simulation under a unified simulation rhythm. Distributed proximal policy optimization achieves second-level task instruction output in a continuous-discrete hybrid action space; the average decision latency is reduced to less than 100 ms, and the total number of actions monotonically decreases with the number of training rounds. The results show that the DPPO win rate is significantly higher than the comparative algorithms, validating the model's fast convergence speed and low steady-state loss.

[0033] Strong generalization and scenario transfer capabilities: Cross-validation was conducted on three types of agents using four different initial scenarios. Each agent ran independently for 100 rounds under the same conditions, and the average win rate and swap ratio were statistically analyzed. The results show that the DPPO agent can still stably output cooperative strike routes after any scenario switch, and the win rate decreases the least, verifying the model's good generalization and transfer capabilities.

[0034] Online deployment and real-time decision support: The converged DPPO model is saved as a .ckpt file and inferred offline in the same simulation environment. The time taken for a single simulation is approximately 1 times that of the original real-time simulation. This can provide decision reference for subsequent access to the command system, improving the surprise of mission execution and the probability of survival. Since DPPO directly outputs trajectory commands in the continuous action space, no additional planning is required during inference. This can provide real-time reference for the terminal maneuver of the aircraft, potentially improving the surprise of the attack. Attached Figure Description

[0035] Figure 1 This is a behavior tree structure diagram; Figure 2 This is a framework diagram of the DPPO model; Figure 3 A diagram of the Markov decision process for the Actor-Critic reinforcement learning algorithm; Figure 4 Here is a diagram of the DDPG structure; Figure 5 This is a schematic diagram of the fitting of an Actor network multilayer perceptron (MLP). Figure 6 Develop architectures for intelligent agents; Figure 7 This refers to the deployment and iteration methods of intelligent agents; Figure 8 PyTorch training framework diagram; Figure 9 Diagram of the TensorFlow training framework; Figure 10 The reward value for the training process of a reinforcement learning algorithm with 5000 training iterations; Figure 11 To classify the different regions as zone1, zone2, zone3 and zone4 in sequence, and to define the regional division situation map of the situation awareness focus area; Figure 12-14 This refers to the distance between Category A and Category B vessels on both sides (Red and Blue), the distance between Category A vessels on both sides, and the distance between adjacent vessels. Figure 15-16 The comparison of win rates of behavior tree agents and the comparison of various indicators of the red team in behavior tree are presented. Figure 17-18This section showcases a comparison of DPPO agent win rates and a comparison of various capabilities of the DPPO Red Team. Figures 19-20 This section showcases a comparison of the win rates of DDPG agents and a comparison of various capabilities of the DDPG red team. Figure 21-22 It shows a comparison of average win rates and a comparison of the capabilities of the three types of agents. Detailed Implementation

[0036] The present invention will be further described below with reference to embodiments, but it should not be construed that the scope of the present invention is limited to the following embodiments. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the above-described technical concept of the present invention should be included within the scope of protection of the present invention.

[0037] Example 1:

[0038] See Figures 1 to 22 A DPPO deep reinforcement learning agent inference method includes the following steps:

[0039] Step 1) Based on the situational parameter set of the scenario template, load the wargaming engine uniformly and deploy the DPPO agent in the task planning scenario;

[0040] The DPPO agent employs a distributed training architecture, including multiple Actor nodes and Learner nodes; multiple Actor nodes collect data in parallel, and Learner nodes aggregate gradients and update the global policy model.

[0041] Step 2) Utilize the DPPO agent to output actions and write them into the experience cache;

[0042] Step 3) Use the inference engine to return the immediate reward and the next moment's situation of the DPPO agent, and complete the reward and situation feedback;

[0043] Step 4) Employ an independent experience replay mechanism to calculate the GAE advantage estimate for the DPPO cache, perform gradient ascent according to the PPO-Clip objective until the preset convergence judgment condition is met, and output the planning strategy for the agent's aerial action mission to be deployed.

[0044] Example 2:

[0045] A DPPO deep reinforcement learning agent inference method, with the same technical content as in Embodiment 1, further, in step 1), the step of uniformly loading the wargaming engine includes: uniformly loading to the wargaming engine through the Environment::reset() interface and initializing the simulation environment.

[0046] Example 3:

[0047] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of Embodiments 1-2, further wherein the simulation environment is a Markov decision process, defined as follows:

[0048] (1)

[0049] In the formula, Represent the state space and action space respectively; Indicates the state transition probability; Represents the reward function; Indicates the discount factor; This represents the Markov decision process.

[0050] Example 4:

[0051] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-3, further comprising a situation parameter set including action space modeling, functional regions, state features and action space parameters;

[0052] Among them, the situational awareness focus area in the functional area is used to dynamically characterize the real-time spatial distribution of all detected, identified and confirmed aerial targets in flight within the current effective coverage area of ​​the friendly sensor network.

[0053] The situational awareness focus area is shown below:

[0054] 'zone_1': ['AI-AO-1', 'rp2', 'rp3', 'rp4'];

[0055] 'zone_2': ['rp2', 'AI-AO-2', 'rp5', 'rp3'];

[0056] 'zone_3': ['rp3', 'rp5', 'AI-AO-3', 'rp6'];

[0057] 'zone_4': ['rp4', 'rp3', 'rp6', 'AI-AO-4'];

[0058] In the formula, AI-AO-1, ​​AI-AO-2, AI-AO-3, and AI-AO-4 represent the core target points, respectively; rp2, rp3, rp4, rp5, and rp6 represent reference points; and zone_1, zone_2, zone_3, and zone_4 represent the situational awareness focus areas.

[0059] Status characteristics include unit type, carrying device type, and detection type;

[0060] Action space parameters include task type, target location, task unit, task time, and execution details; the task unit is the intelligent agent that performs the task; execution details include task parameter configuration and electromagnetic spectrum management strategy.

[0061] Example 5:

[0062] A method for inferring a DPPO deep reinforcement learning agent, with the same technical content as any one of embodiments 1-4, further wherein the DPPO agent in step 2) employs GAE-dominance estimation to calculate the dominance function. and the advantage function Substitute the values ​​into the objective function to complete the policy update;

[0063] The advantage function is shown below:

[0064] (2)

[0065] In the formula, , The time-series difference error at different times; For GAE attenuation parameters; Discount factor; For instant rewards; , Here, T is the state value function; T is the total trajectory length; t is the current time step.

[0066] The objective function is as follows:

[0067] (3)

[0068] In the formula, The objective function is... For expectations; , New and old strategies; For pruning hyperparameters; This is the clipping function.

[0069] Example 6:

[0070] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-5, further wherein, under the distributed DPPO architecture, after each Actor node of the DPPO agent completes trajectory sampling and gradient calculation locally, the Learner node aggregates all gradients and performs a global parameter update, and then broadcasts the updated policy network parameters to each Actor to complete a round of collaborative training.

[0071] Example 7:

[0072] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-6, further wherein the agent's aerial action mission planning strategy output by the DPPO agent includes mission type, mission parameters, mission area, mission triggering conditions, and dynamic adjustment mechanism.

[0073] The mission types include patrol missions and strike missions;

[0074] The task parameters include the number of agents executing the task, the task execution interval, and the duration.

[0075] The dynamic adjustment mechanism is used to adjust task parameters and task area.

[0076] Example 8:

[0077] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-7, further wherein the calculation of the DPPO objective function in step 3) considers distributed data collection and parameter synchronization;

[0078] Each worker generates data independently, and the advantage function is calculated uniformly under the old policy parameters to ensure consistent estimation.

[0079] (4)

[0080] In the formula, , For global policy parameters, old policy;

[0081] The central node, Learner, aggregates the gradients of all Workers and calculates the global update, i.e.:

[0082] (5)

[0083] In the formula, For global gradient; This represents the number of Worker nodes; Let be the local gradient of the i-th Worker;

[0084] The distributed data in DPPO has undergone normalization processing, namely:

[0085] (6)

[0086] In the formula, , The dominance functions before and after normalization; , The mean and standard deviation of the dominance function are given.

[0087] Example 9:

[0088] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-8. Further, DPPO adopts a three-layer structure that combines a central coordinating Learner with distributed computing Actors. The top layer is designed with Learner nodes as the control center, the middle layer is designed with a shared gradient region as a gradient cache pool, and the bottom layer is designed with N Actors running in parallel, each containing a local PPO model and an environment interaction module.

[0089] The objective function of DPPO is shown below:

[0090] (7)

[0091] In the formula, The objective function of DPPO is...

[0092] Example 10:

[0093] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-9, further wherein the learner infers the interaction and experience accumulation of the environment to provide the actor with a decision-making basis, and during testing, weighs the attempt to try new actions versus the execution of known high-reward actions in an unknown environment;

[0094] The Learner training process is as follows: retrieve batch data from the queue, calculate the objective function and gradient and update the parameters, and periodically save the model and performance metrics;

[0095] The parameters are updated as follows:

[0096] (8)

[0097] In the formula, , To update parameters before and after; The gradient of the combined loss function; The learning rate;

[0098] The Actor workflow is as follows: retrieve the latest model from the Learner via _update_model(), perform multi-step interaction using _nstep_rollout(), calculate the GAE advantage estimate, and push data to the Learner via ZMQ.

[0099] The Actor's GAE advantage is estimated as follows:

[0100] (9)

[0101] In the formula, This refers to timing difference error; For GAE attenuation parameters; This is the discount factor.

[0102] Example 11:

[0103] See Figures 1-19 A DPPO deep reinforcement learning agent inference method includes the following steps:

[0104] 1) Initialize the simulation scenario, load the wargame engine, units and situation parameters in a unified manner, and then deploy the three types of intelligent agents: behavior tree, DPPO and DDPG in sequence;

[0105] 2) During the sampling phase, the three types of agents output actions independently within the same simulation cycle, and continuous / discrete instructions are written into their respective experience buffers;

[0106] 3) The simulation engine returns the immediate reward and the situation at the next moment, completing the reward and situation feedback;

[0107] 4) An independent experience replay mechanism is adopted to calculate the GAE advantage estimate for the DPPO cache, and to obtain the gradient signal by the difference between the state-action value output by the Critic network and the target value for the DDPG cache, and update their respective networks respectively.

[0108] 5) During the policy network update phase, the behavior tree remains unchanged and is not updated with gradients. DPPO performs gradient ascent according to the PPO-Clip objective; DDPG updates the Critic first, and then updates the Actor based on the Q value output by the Critic.

[0109] 6) Use an average win rate that is consistently ≥90% or reaches a preset number of training rounds as the convergence criterion, then freeze the model parameters and export a deployable version;

[0110] 7) Conduct comparative simulations and performance evaluations. Under the same scenario, each agent independently engages in combat and statistically analyzes indicators such as win rate and exchange ratio to verify that DPPO is significantly superior to the other two types of agents.

[0111] Example 12:

[0112] A DPPO deep reinforcement learning agent inference method, with the same technical content as in Examples 1-11, further, in step 1), by configuring core parameters such as unit structure, carrying device loading scheme and scoring weight, a standardized situational environment is established for agent training and evaluation.

[0113] A task planning agent is an autonomous entity that integrates environmental perception, situational understanding, decision-making and planning, and action execution, possessing the intelligent behavioral capability to achieve multi-objective optimization in uncertain environments. The basic template for a task planning agent includes five elements: task, state / action space analysis, interface, learning strategy, and model program.

[0114] In the mission planning scenario, the DPPO agent is deployed; in the route planning scenario, the DDPG agent is deployed; and the behavior tree agent is used for comparison. The three types of agents run independently under the same simulation cycle.

[0115] Example 13:

[0116] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-12, further, in step 2), the sampling stage is used to represent the entire cycle of collecting training data, and a simulation beat update is completed through the environment advancement interface. The three types of agents run independently and output actions within a unified simulation beat, and the experience cache is stored separately.

[0117] Example 14:

[0118] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-13, further wherein after the decision system executes an action, it obtains feedback signals such as rewards, penalties and new observations from the environment, and the agent makes corresponding actions based on these feedbacks and the current strategy;

[0119] The PPO algorithm uses a Long Short-Term Memory (LSTM) network as the core architecture of its policy network. The LSTM outputs a hidden state at each time step, and finally generates an action through FC3. The model interacts with the environment based on the action to obtain a reward and a new state (obs_next), forming an experience tuple for training.

[0120] In the Actor-Critic algorithm, the Actor is responsible for interacting with the environment, generating a probability distribution of actions based on the state of the environment, then selecting an action and acquiring interaction data (state, action, reward, next state). Essentially, it is a policy network that can learn a policy to map states to actions.

[0121] The DDPG model employs an MLP (Multilayer Perceptron) / LSTM-fitted Actor policy network that maps states to actions using a policy function μ(s) to maximize long-term expected rewards. A deep feedforward neural network-fitted Critic value network evaluates actions using a value function Q(s,a) and updates parameters with temporal difference error. The target network stabilizes the main network training process, and experience replay is stored in a buffer and randomly sampled to reduce correlation. The core design of the loss function is to minimize the mean squared error between the Critic's Q-value prediction and the target value, and to maximize the Critic's output Q-value according to the Actor's policy gradient. Ornstein-Uhlenbeck noise enhancement is added to the Actor's output actions to explore ways to avoid premature policy convergence.

[0122] Example 15:

[0123] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-14, further, in step 4), the experience replay cache is maintained independently by each agent and is not shared across algorithms, and samples are taken from their respective caches during training;

[0124] The DPPO agent uses GAE-dominance estimation, according to

[0125] (twenty two)

[0126] Calculate the advantage function and substitute it into...

[0127] (twenty three)

[0128] Complete the strategy update;

[0129] The DDPG algorithm employs an experience replay mechanism, storing transition samples generated by the agent's interaction with the environment in a replay pool. When updating network parameters, it randomly samples historical experience data in a fixed batch size (mini-batch). The final output action of the Actor network is...

[0130] (twenty four)

[0131] The parameter updates of the Actor network follow the policy gradient principle, aiming to maximize the expected reward. The network generates action instructions by fitting a deterministic policy function based on the next environmental state. The update process of the Actor network is as follows:

[0132] (25)

[0133] The update process of the Actor and Critic target networks is as follows:

[0134] (26)

[0135] Example 16:

[0136] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-15, further wherein, in step 5), the behavior tree retains the rules unchanged;

[0137] The policy network of the PPO algorithm uses a Long Short-Term Memory (LSTM) network as its core architecture. The key innovation is the introduction of memory units (Cell State) and a gating system. The state observations of the environment (obs (obs current state, action executed by action, obs_next next state, reward, reward obtained, and done flag indicating whether the process has ended)) are processed by FC1 and FC2 (two fully connected layers) to extract features, and then input into the LSTM for processing step by step.

[0138] PPO-Truncation limits the magnitude of policy updates through clipping, with the objective function being...

[0139] (27)

[0140] The clipping operation limits the range of strategy ratios; .

[0141] The specific limitations of the clip operation are as follows:

[0142] (28)

[0143] PPO-Clip controls the magnitude of policy updates through clipping;

[0144] The Actor is responsible for selecting actions based on the current policy and optimizing this policy through learning. By interacting with the environment and receiving rewards, the Actor adjusts its parameters θ to improve the policy, with the aim of maximizing long-term rewards.

[0145] (29)

[0146] Actor networks use deep neural networks (MLP or LSTM) to fit a set of states. The agent learns a control policy to apply actions to the environment and maximize its gains. Smooth loss avoids gradient explosion, mitigates gradient vanishing, and is robust to outliers. Whether the loss function is averaged or summed depends on the learning rate; the result is the same, so the program uses the Adam optimizer. The value network adjusts parameters to reduce prediction error, while the policy network adjusts parameters to increase the value function.

[0147] Stochastic Gradient Descent Optimizer (SGD) Learning rate

[0148] (30)

[0149] The Critic, as an auxiliary model, provides information to help Actors understand which actions are advantageous, thereby guiding Actors to update their strategies.

[0150] (31)

[0151] The Critic network's loss function is constructed by minimizing the difference between its predicted value and its target estimate. This loss function guides the Critic network's parameter updates and indirectly optimizes the Actor network's parameters through its output value assessment signal, ultimately maximizing the cumulative total return. Its loss function is:

[0152] (32)

[0153] Where N is the minimum batch size; Let Q be an estimate of Q, where Q is the Critic network;

[0154] (33)

[0155] in, The target network for Critic; The target network for the Actor.

[0156] Example 17:

[0157] A DPPO deep reinforcement learning agent inference method, with technical content identical to any one of embodiments 1-16, constructs a loss function by minimizing the difference between the predicted value function of the Critic network and its target estimate. This loss function guides the parameter updates of the Critic network and indirectly optimizes the parameters of the Actor network through its output value evaluation signal, ultimately maximizing the cumulative total reward. Its loss function is as follows:

[0158] (34)

[0159] Where N is the minimum batch size; Let Q be an estimate of Q, where Q is the Critic network;

[0160] (35)

[0161] in, The target network for Critic; The target network for the Actor;

[0162] Example 18:

[0163] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-17, further includes agent model iteration, client deployment of single / multiple agents, and starting the control inference server by encapsulating inference system object classes / methods, Lua language behavior control, and gRPC communication control API, selecting scenarios, and acquiring inference information in real time. It adopts three deployment methods: one-to-one, one-to-many, and many-to-one / many, and conducts task planning agent development and training to realize different adversarial modes such as non-intelligent agent, human agent, agent agent+, and agent human, as well as human-in-the-loop agent upgrade and iteration mode.

[0164] Example 19:

[0165] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-18. Furthermore, a comparative win rate analysis shows that behavior trees are suitable for fixed scenarios with complete prior knowledge, but their ability to adapt to random situations is insufficient; deep reinforcement learning algorithms (DPPO / DDPG) are more advantageous in uncertain environments—DPPO has outstanding generalization ability, while DDPG has leading real-time performance but large fluctuations.

[0166] Behavior tree agents perform stably in scenarios with well-defined rules, but their dynamic adaptability is weak. DPPO agents achieve efficient learning and robust decision-making in high-dimensional state space tasks through distributed policy optimization. DDPG agents still lag behind DPPO agents in continuous control tasks.

[0167] Example 20:

[0168] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-19, further wherein Python is used to construct the training process and TensorFlow is used as the backend framework for policy network and gradient update.

[0169] TensorFlow uses a data flow graph as its core, and its distributed training process includes:

[0170] 1) Tensor Calculation

[0171] Efficient forward and backward propagation is achieved through computational graphs.

[0172] (36)

[0173] 2) Automatic Differentiation

[0174] Automatic gradient calculation using the chain rule

[0175] (37)

[0176] 3) Optimizer

[0177] (38)

[0178] (39)

[0179] (40)

[0180] (41)

[0181] 4) Distributed aggregation

[0182] Gradient averaging or asynchronous updates can be used to achieve parallel acceleration, and the aggregated global gradient is:

[0183] (42)

[0184] 5) Loss Function

[0185] Parameters such as Mean Squared Error (MSE) and Cross-Entropy Loss (for multi-class classification) directly correspond to high-level APIs.

[0186] MSE

[0187] (43)

[0188] Cross-entropy loss

[0189] (44)

[0190] The core values ​​of the PyTorch framework include: flexibility and ease of use, distributed scalability, and a rich ecosystem.

[0191] Example 21:

[0192] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-20, further comprising a situation parameter set, the specific content of which mainly covers the following types of data:

[0193] 1) Operational space modeling and regional division

[0194] The operational space includes all combat units of the Red Force and all combat units of the Blue Force that are detected and identified in real time by the Red Force's sensor system;

[0195] The area is divided into multiple functional zones. Simultaneously, a Situational Awareness Focus Zone (SAW Zone) is defined. This zone dynamically represents the real-time spatial distribution of all detected, identified, and confirmed airborne targets (aircraft) of the Blue Team within the current effective coverage area of ​​the friendly sensor network.

[0196] 'zone_1': ['AI-AO-1', 'rp2', 'rp3', 'rp4'],

[0197] 'zone_2': ['rp2', 'AI-AO-2', 'rp5', 'rp3'],

[0198] 'zone_3': ['rp3', 'rp5', 'AI-AO-3', 'rp6'],

[0199] 'zone_4': ['rp4', 'rp3', 'rp6', 'AI-AO-4']}

[0200] 2) State Feature Construction

[0201] It employs one-hot encoding and normalization processing, and mainly includes three types: unit type, weapon type, and detection type;

[0202] 3) Motion space parameters

[0203] Mission types include, for example, surface target missions and patrol missions;

[0204] The target location refers to the coordinates / azimuth of the patrolled airspace / sea area or the target surface target;

[0205] A mission unit is a combat unit that performs a mission (such as an air combat aircraft or an anti-surface target aircraft).

[0206] Task time refers to the specific point in time when the sub-action begins execution;

[0207] Execution details include mission parameter configuration, tactical doctrine application, and electromagnetic spectrum management strategies.

[0208] Example 22:

[0209] A DPPO deep reinforcement learning agent inference method, with the same technical content as any one of embodiments 1-21, further wherein the calculation of the DPPO objective function considers distributed data collection and parameter synchronization;

[0210] Each worker generates data independently, and the advantage function needs to be calculated uniformly under the old policy parameters to ensure consistent estimation.

[0211] (4)

[0212] The central node (Learner) aggregates the gradients of all Workers and calculates the global update. The objective function needs to be synchronized among all Workers to prevent trust region failure caused by policy differences.

[0213] (5)

[0214] Distributed data may have different advantage values ​​for different workers, so normalization (such as subtracting the mean or dividing by the standard deviation) is performed to stabilize training.

[0215] Example 23:

[0216] A DPPO deep reinforcement learning agent inference method is provided, with technical content identical to any one of embodiments 1-22. Furthermore, both the DPPO agent and the DDPG agent use the same set of situational parameters as input, and share a unified state space definition and feature construction method in the inference environment.

[0217] The outputs of the DPPO agent and the DDPG agent are the mission planning scheme and the route planning scheme, respectively. The mission planning scheme mainly includes the following key elements:

[0218] 1) Task type definition

[0219] Anti-surface target missions are missions to strike surface targets (Category B ships) of the Blue Force; patrol missions are divided into two types: defensive patrols and offensive patrols; air combat missions are combat missions specifically targeting aerial targets.

[0220] 2) Task parameter configuration

[0221] Defensive patrols will be carried out by selecting 4 out of 8 air combat aircraft; offensive patrols will be carried out by selecting 8 out of 12 air combat aircraft; anti-surface target patrols will be carried out by selecting units from 4 anti-surface target-air combat aircraft.

[0222] The task start interval is uniformly configured to 2 minutes, and the task duration is set according to the desired requirements;

[0223] 3) Regional Deployment Strategy

[0224] Defensive patrol zones use four standard zones: zone_1, zone_2, zone_3, and zone_4; offensive patrol zones use the saw_zone (situational awareness focus area); the vertices of patrol zones are defined using fixed coordinate points.

[0225] 4) Tactical doctrine settings: Configure them specifically for different mission types;

[0226] 5) Task execution logic

[0227] The mechanism is triggered when a blue team surface target is detected and enters its effective strike range, the red team has a usable anti-surface target platform, or the rules of engagement (ROE) are met.

[0228] 6) Dynamic adjustment mechanism

[0229] The conditions for updating a mission are the integrity of existing patrol missions, adjustment of patrol routes and troop deployments, and replenishment of damaged / consumed combat units.

[0230] Example 24:

[0231] A DPPO deep reinforcement learning agent inference method, in which an agent controls the red team to perform inference;

[0232] The unit sequences of both sides are shown in Table 1:

[0233] Table 1. Two-way unit sequence

[0234]

[0235] The weapons carried by both sides are shown in Table 2:

[0236] Table 2 Comparison of weapons between the two sides

[0237]

[0238] The Red Team aims to cripple the Blue Team's Category A ships and aircraft take-off and landing capabilities, eliminate the Blue Team's Category B ships, weaken its anti-aircraft target network, control key waterways, and cut off the Blue Team's supply lines.

[0239] The Blue Team's objectives are to destroy the Red Team's Category A ships, seize electromagnetic dominance, protect the Red Team's Category B ships' cooperative combat capability nodes, establish a safe zone, and ensure the arrival of subsequent teams.

[0240] The task has a time limit of 2 hours.

[0241] The scoring criteria are shown in Table 3. Destroying or eliminating the opponent's target earns positive points, while the red team's target being destroyed or eliminated earns negative points.

[0242] Table 3 Scoring Criteria for Both Sides

[0243]

[0244] When the simulation ends, if Red's score > Blue's score, Red wins; if Red's score < Blue's score, Blue wins; if Red's score equals Blue's score, it's a draw. During the simulation, Blue's actions are based on the following four different scenarios:

[0245] Scenario 1: The Blue Team's task is a randomized scheme.

[0246] 2) Scenario 2: The Blue team's task is A0.

[0247] At the initial moment, an eight-aircraft formation (1, 2, 3, 4, 5, 6, 7, 8) carrying anti-surface target guided aircraft and anti-air target guided aircraft took off from the Class A ship and carried out an attack mission against the Red Force ship along route A.

[0248] 3) Scenario 3: Blue Team's Mission Plan B0

[0249] At the initial moment, a four-aircraft formation (1, 2, 3, 4, 5, 6, 7, 8) carrying anti-surface target guided aircraft and anti-air target guided aircraft took off from the Class A ship and carried out an attack mission against the Red Force ship along route B.

[0250] Scenario 4: Blue Team's Mission Plan A0B0

[0251] Maritime Strike A0, Route A: At the initial moment, a four-aircraft formation (1, 2, 3, 4) carrying anti-surface target guided aircraft and anti-air target guided aircraft takes off from a Class A ship and carries out an attack mission against the Red Force ship along Route A.

[0252] Maritime Strike B0, Route B: At the initial moment, a four-aircraft formation (5, 6, 7, 8) carrying anti-surface target guided aircraft and anti-air target guided aircraft takes off from a Class A ship and carries out an attack mission against the Red Force ship along Route B.

[0253] The relevant coordinates in the above task planning are as follows: RP5 (26°29´19´´N, 156°03´19´´E), RP6 (26°30´54´´N, 156°39´03´´E), RP7 (25°39´41´´N, 156°39´04´´E), RP8 (25°39´59´´N, 156°02´21´´E), RP129 (24°49´53´´N, 156°03´49´´E), RP130 (24°49´37´´N, 156°37´48´´).

[0254] Example 25:

[0255] A DPPO deep reinforcement learning agent inference method, behavior tree agent inference simulation; it is worth noting that Examples 25-27 are all simulation inference processes, and the contents involved are all computer simulation results.

[0256] 1) Imagine a

[0257] When the Blue team faces a random mission, the win rate, derived through behavior tree analysis, is 80%. All simulations were performed 100 times, and 10 of these simulations were randomly selected for comparative analysis, as shown in Table 4. The table clearly shows that the Red team's average aircraft exchange ratio is 1:1.11, but its ability to engage Blue team surface targets and its own anti-air targets is relatively high.

[0258] Table 4. Results of Behavior Tree Scenario 1 Inference

[0259]

[0260] 2) Imagine two

[0261] When the blue team is A0, the win rate is 90% based on behavior tree analysis. All analyses were performed 100 times, and 10 of these were randomly selected for comparative analysis, as shown in Table 5. The table clearly shows that the red team's average aircraft exchange ratio is 1:1.21, but its ability to strike blue team surface targets and its own anti-air capabilities is relatively high.

[0262] Table 5. Results of Behavior Tree Scenario 2 Derivation

[0263]

[0264] 3) Think three

[0265] When the blue team is B0, the win rate obtained through behavior tree analysis is 85%. All simulations were performed 100 times. Ten of these simulations were randomly selected for comparative analysis, as shown in Table 6. The table clearly shows that the red team's average aircraft exchange ratio is 1:1.23, but its ability to strike blue team surface targets and its own anti-air targets is relatively high.

[0266] Table 6. Results of the three-stage deduction of the behavior tree scenario

[0267]

[0268] 4) Scenario 4

[0269] When the blue team is A0B0, the win rate is 90% based on behavior tree analysis. All simulations were performed 100 times. Table 7 shows 10 randomly selected simulations for comparative analysis. The table clearly shows that the red team's average aircraft exchange ratio is 1:1.23, but its ability to engage blue team surface targets and its own anti-air targets is relatively high.

[0270] Table 7. Results of the fourth scenario deduction of the behavior tree

[0271]

[0272] like Figure 12-13 As shown, by comparing the outcomes of the four scenarios, when the Blue Team's mission changes, the Red Team, through prior learning and training, is already able to respond well to the Blue Team's mission plans. Furthermore, the Red Team has the highest win rate when the Blue Team's mission plan is A0. This indicates that when the Blue Team launches a surprise attack along flight path A, the Red Team has ample time to react and deploy. However, in terms of air combat capabilities, except for the random mission, the air combat capabilities of the other three combat scenarios are basically equal. This may be due to the agent's still insufficient ability to handle random missions.

[0273] Example 26:

[0274] A DPPO deep reinforcement learning agent inference method, DPPO agent inference simulation;

[0275] 1) Imagine a

[0276] When the Blue team is assigned a random mission, the DPPO simulation yielded a 90% win rate. All simulations were conducted 100 times, and 10 of these simulations were randomly selected for comparative analysis, as shown in Table 8. The table clearly shows that the Red team's average aircraft exchange ratio is 1:1.45, and they possess high capabilities in attacking Blue team surface targets and their own anti-air targets.

[0277] Table 8. DPPO Scenario 1: Calculation Results

[0278]

[0279] 2) Scenario 2: When the Blue side is A0, the final win rate obtained through DPPO simulation is 95%. All simulations were conducted 100 times. Ten of these simulations were randomly selected for comparative analysis, as shown in Table 9. The table clearly shows that the Red side's average aircraft exchange ratio is 1:1.56, but its ability to strike Blue side surface targets and its own anti-air targets is relatively high.

[0280] Table 9. DPPO Scenario 2 Derivation Results

[0281]

[0282] 3) Scenario 3: When the Blue side is B0, the final win rate obtained through DPPO simulation is 93%. All simulations were conducted 100 times. The following 10 simulations were randomly selected for comparative analysis, as shown in Table 10. It is not difficult to see from the table that the Red side's average aircraft exchange ratio is 1:1.34, but its ability to strike Blue side surface targets and its own anti-air targets is relatively high.

[0283] Table 10 DPPO Scenario 3 Calculation Results

[0284]

[0285] 4) Scenario 4: When the Blue side is A0B0, the final win rate obtained through DPPO simulation is 95%. 100 simulations were conducted, and 10 of them were randomly selected for comparative analysis, as shown in Table 11. The table clearly shows that the Red side's average aircraft exchange ratio is 1:1.15, but its ability to strike Blue side surface targets and its own anti-air targets is relatively high.

[0286] Table 11 DPPO Scenario 4 Calculation Results

[0287]

[0288] like Figure 14-15 As shown, by comparing the four scenarios, it can be seen that the DPPO's simulation win rate is above 90%, and the red team's win rate is the highest when the blue team's mission plan is A0. This also shows that when the blue team launches a surprise attack along route A, the red team has ample time to react and deploy. However, in terms of aircraft air combat capabilities, the aircraft controlled by DPPO, after learning and training, have significantly improved their ability to deal with the blue team's random mission aircraft, indicating that DPPO's control and learning capabilities are stronger than those of the behavior tree.

[0289] Example 27:

[0290] A DPPO deep reinforcement learning agent inference method, DDPG agent inference simulation;

[0291] 1) Imagine a

[0292] When the Blue team faces a random mission, the win rate obtained through DDPG simulations is 87%. All simulations were conducted 100 times, and 10 of them were randomly selected for comparative analysis, as shown in Table 12. The table clearly shows that the Red team's average aircraft exchange ratio is 1:1.19, but its ability to engage Blue team surface targets and its own anti-air targets is relatively high.

[0293] Table 12 DDPG Scenario 1 Derivation Results

[0294]

[0295] 2) Imagine two

[0296] When the Blue side is A0, the win rate obtained through DDPG simulation is 95%. All simulations were conducted 100 times. Ten of these simulations were randomly selected for comparative analysis, as shown in Table 13. The table clearly shows that the Red side's average aircraft exchange ratio is 1:1.10, but its ability to engage Blue side surface targets and its own anti-air targets is relatively high.

[0297] Table 13 DDPG Scenario 2 Derivation Results

[0298]

[0299] 3) Scenario 3: When the Blue side is B0, the win rate obtained through DDPG simulation is 90%. All simulations were conducted 100 times. Ten of these simulations were randomly selected for comparative analysis, as shown in Table 14. The table clearly shows that the Red side's average aircraft exchange ratio is 1:1.10, but its ability to engage Blue side surface targets and its own anti-air targets is relatively high.

[0300] Table 14 DDPG Scenario 3 Derivation Results

[0301]

[0302] 4) Scenario 4: When the Blue team is A0B0, the win rate obtained through DDPG simulation is 91%. All simulations were conducted 100 times. Ten of these simulations were randomly selected for comparative analysis, as shown in Table 15. The table clearly shows that the Red team's average aircraft exchange ratio is 1:1.23, but its ability to engage Blue team surface targets and its own anti-air targets is relatively high.

[0303] Table 15 DDPG Scenario 4 Derivation Results

[0304]

[0305] like Figure 16-17 As shown, comparing the four scenarios, the inference win rate is above 85%, which is worse than behavior tree overall, but not as good as DPPO. In terms of air combat capability, its air combat capability is similar to behavior tree, but inferior to DPPO. The training stability of DDPG agent is not as good as DPPO, and it does not have the function of outward exploration.

[0306] Example 28:

[0307] A deep reinforcement learning agent inference method, DPPO, was proposed. Comparative analysis showed that DPPO agents achieved an average win rate 7% higher than behavior tree agents and 2.5% higher than DDPG agents. The aircraft controlled by DPPO agents exhibited the best mission execution performance, while those controlled by DDPG agents performed the worst. This is because DPPO agents have better training stability and their output is a stochastic policy, inherently exploratory. DDPG training, on the other hand, is relatively unstable, and its output consists of deterministic actions, with exploration entirely dependent on noise added to the Actor output. There was no significant difference in attack and defense capabilities among the three agents. Furthermore, the distance at which the aircraft controlled by the DDPG agent attacks surface targets was varied; comparisons revealed that 80,000.0m was the optimal distance.

Claims

1. A method for deriving a DPPO deep reinforcement learning agent, characterized in that, Includes the following steps: Step 1) Based on the situational parameter set of the scenario template, load the wargaming engine uniformly and deploy the DPPO agent in the task planning scenario; The DPPO agent employs a distributed training architecture, including multiple Actor nodes and Learner nodes; multiple Actor nodes collect data in parallel, and Learner nodes aggregate gradients and update the global policy model. Step 2) Utilize the DPPO agent to output actions and write them into the experience cache; Step 3) Use the inference engine to return the immediate reward and the next moment's situation of the DPPO agent, and complete the reward and situation feedback; Step 4) Using an independent experience replay mechanism, calculate the GAE advantage estimate for the DPPO cache, perform gradient ascent according to the PPO-Clip objective until the preset convergence judgment condition is reached, and output the planning strategy for the agent's aerial action mission to be deployed. 2.The DPPO deep reinforcement learning agent deduction method according to claim 1, characterized in that: In step 1), the steps of uniformly loading the wargaming engine include: uniformly loading it into the wargaming engine through the Environment::reset() interface and initializing the simulation environment. 3.The DPPO deep reinforcement learning agent deduction method of claim 2, wherein: The simulation environment is a Markov decision process, defined as follows: ；（1） In the formula, Represent the state space and action space respectively; Indicates the state transition probability; Represents the reward function; Indicates the discount factor; This represents the Markov decision-making process.

4. The DPPO deep reinforcement learning agent inference method according to claim 2, characterized in that: The situational parameter set includes action space modeling, functional regions, state characteristics, and action space parameters; Among them, the situational awareness focus area in the functional area is used to dynamically characterize the real-time spatial distribution of all detected, identified and confirmed aerial targets in flight within the current effective coverage area of ​​the friendly sensor network. The situational awareness focus area is shown below: 'zone_1': ['AI-AO-1', 'rp2', 'rp3', 'rp4']; 'zone_2': ['rp2', 'AI-AO-2', 'rp5', 'rp3']; 'zone_3': ['rp3', 'rp5', 'AI-AO-3', 'rp6']; 'zone_4': ['rp4', 'rp3', 'rp6', 'AI-AO-4']; In the formula, AI-AO-1, ​​AI-AO-2, AI-AO-3, and AI-AO-4 represent the core target points, respectively; rp2, rp3, rp4, rp5, and rp6 represent reference points; and zone_1, zone_2, zone_3, and zone_4 represent the situational awareness focus areas. Status characteristics include unit type, carrying device type, and detection type; Action space parameters include task type, target location, task unit, task time, and execution details; the task unit is the intelligent agent that performs the task; execution details include task parameter configuration and electromagnetic spectrum management strategy.

5. The DPPO deep reinforcement learning agent deduction method according to claim 1, characterized in that: The DPPO agent in step 2) adopts GAE-advantage estimation to calculate an advantage function and substitutes the advantage function into an objective function to complete policy updating; The advantage function is shown below: ；（2） wherein , is the timing difference error at different times; is the GAE decay parameter; is the discount factor; is the immediate reward; , is the state value function; T is the total length of the trajectory; t is the current time step; The objective function is as follows: ；（3） wherein is the objective function; is the expectation; , is the new, old policy; is the clipping hyperparameter; is the clipping function.

6. The DPPO deep reinforcement learning agent deduction method according to claim 1, characterized in that: In the distributed DPPO architecture, after each Actor node of the DPPO agent completes trajectory sampling and gradient calculation locally, the Learner node aggregates all gradients and performs a global parameter update. Then, the updated policy network parameters are broadcast to each Actor to complete a round of collaborative training.

7. The DPPO deep reinforcement learning agent deduction method according to claim 1, characterized in that: The DPPO agent outputs an agent-based aerial operations mission planning strategy that includes mission type, mission parameters, mission area, mission triggering conditions, and dynamic adjustment mechanism. The mission types include patrol missions and strike missions; The task parameters include the number of agents executing the task, the task execution interval, and the duration. The dynamic adjustment mechanism is used to adjust task parameters and task area. 8.The DPPO deep reinforcement learning agent deduction method of claim 1, wherein: The calculation of the DPPO objective function described in step 3) takes into account distributed data collection and parameter synchronization; Each worker generates data independently, and the advantage function is calculated uniformly under the old policy parameters to ensure consistent estimation. ；（4） In the formula, , is a global policy parameter, an old policy; The central node, Learner, aggregates the gradients of all Workers and calculates the global update, i.e.: ； （5） In the formula, For global gradient; This represents the number of Worker nodes. Let be the local gradient of the i-th Worker; The distributed data in DPPO has undergone normalization processing, namely: ； （6） wherein , are the normalized and non-normalized merit functions, respectively; , are the merit function mean and standard deviation, respectively. 9.The DPPO deep reinforcement learning agent inference method of claim 1, wherein: DPPO adopts a three-layer structure that combines a centrally coordinating Learner with distributed computing Actors. The top layer is designed with Learner nodes as the control center, the middle layer is designed with a shared gradient region as a gradient cache pool, and the bottom layer is designed with N Actors running in parallel, each containing a local PPO model and an environment interaction module. The objective function of DPPO is shown below: ；（7） In the formula, is the objective function of DPPO.

10. The DPPO deep reinforcement learning agent deduction method of claim 1, wherein: Learners infer environmental interactions and accumulate experience, providing Actors with a basis for decision-making. During testing, they weigh trying new actions against performing known high-reward actions in unknown environments. The Learner training process is as follows: retrieve batch data from the queue, calculate the objective function and gradient and update the parameters, and periodically save the model and performance metrics; The parameters are updated as follows: ；（8） wherein , are the pre-update and post-update parameters; is the gradient of the combined loss function; is the learning rate; The Actor workflow is as follows: retrieve the latest model from the Learner via _update_model(), perform multi-step interaction using _nstep_rollout(), calculate the GAE advantage estimate, and push data to the Learner via ZMQ. The Actor's GAE advantage is estimated as follows: ；（9） wherein is the timing difference error; is the GAE decay parameter; is the discount factor.

Images