Three-dimensional cooperative maneuver decision-making method based on adaptive entropy regularization multi-agent proximal policy optimization algorithm

Abstract

This invention provides a three-dimensional cooperative maneuver decision-making method based on an adaptive entropy regularization multi-agent proximal policy optimization algorithm, belonging to the field of cooperative maneuver decision-making for unmanned aerial vehicles (UAVs). First, a 6v6 cooperative adversarial Dec-POMDP model is constructed using a simulation engine. Next, normalized observation vectors containing individual, formation, and opponent situations are extracted as input. Under the CTDE architecture, shared MLP features are used to extract the backbone representing 72-dimensional global situational features. Simultaneously, an adaptive entropy regularization mechanism based on Lagrange duality is introduced, and the exploration and utilization during the training process are balanced by dynamically adjusting the temperature coefficient. Finally, a composite reward mechanism including mission final value rewards and reward shaping based on potential energy functions is implemented to achieve efficient confidence allocation and cooperative tactical emergence in a 6v6 scale. This invention significantly enhances the cooperative combat effectiveness of formations in highly dynamic and complex environments.

Description

Technical Field

[0001] This invention relates to the fields of air combat decision optimization and artificial intelligence technology, and in particular to a three-dimensional cooperative maneuver decision-making method based on an adaptive entropy regularized multi-agent near-end strategy optimization algorithm. Background Technology

[0002] Modern air combat situations are constantly changing. Beyond-visual-range (BVR) combat scenarios exhibit significant characteristics such as high dynamism, strong coupling, and partial observability. Traditional decision-making methods based on classic optimization techniques such as rules or particle swarm optimization generally face technical bottlenecks such as low computational efficiency, slow convergence speed, and easy entrapment in local optima when dealing with high-dimensional, multi-constraint, and multi-target environments.

[0003] While Multi-Agent Reinforcement Learning (MARL) offers a new direction for multi-aircraft cooperative decision-making, standard algorithms such as Proximity Policy Optimization (MAPPO) still face significant challenges in sparse-reward air combat environments. The core issue lies in the fact that the lack of high-quality feedback in the early stages of training can easily lead to an imbalance between exploration and exploitation, causing agents to prematurely converge to mediocre strategies such as avoidance and hovering or merely pursuing survival. Furthermore, existing simulation environments often overemphasize low-level control surfaces or lack standardized interfaces, resulting in a severe disconnect between physical dynamics constraints and high-level tactical logic, making it difficult to directly support the generation and evolution of high-value cooperative tactics. Therefore, for beyond-visual-range (BVR) combat environments, developing an improved cooperative maneuver decision-making algorithm that balances search depth and training stability while deeply adapting to flight physics constraints has become a key requirement for enhancing the effectiveness of unmanned swarm operations. Summary of the Invention

[0004] To address the problems of low agent exploration efficiency and premature policy convergence caused by extremely sparse reward signals and complex state spaces in existing cooperative air combat, this invention proposes a three-dimensional cooperative maneuver decision-making method based on the Adaptive Entropy Multi-Agent Proximal Policy Optimization (AE-MAPPO) algorithm, which significantly enhances the cooperative combat effectiveness of formations in highly dynamic and complex environments.

[0005] This invention provides a three-dimensional cooperative maneuver decision-making method based on the AE-MAPPO algorithm, comprising the following steps:

[0006] Step S1: Construct a cooperative adversarial distributed partially observable Markov decision process (Dec-POMDP) ​​model: Define the 6v6 cooperative adversarial task as an octet consisting of a set of agents, a global state space, a joint action space, a state transition function, a set of reward functions, an observation space, observation functions, and a discount factor. The nonlinear motion state transition is driven by the B-ACE dynamics simulation engine of flight physics and three-dimensional air combat.

[0007] Step S2: Extract and normalize multi-source heterogeneous situational features: Construct local observation vectors in real time that include the aircraft's own flight status, friendly aircraft cooperation features, and adversary threat situational features. The physical parameters are mapped to using a linear normalization method. interval;

[0008] Step S3: Design an action mapping mechanism with physical constraints adaptive: the agent outputs a four-dimensional continuous vector. Through highly adaptive overload mapping and high threshold triggering decision-making, the transformation from tactical intent to motion state is realized;

[0009] Step S4: Construct the AE-MAPPO optimization framework based on maximum entropy reinforcement learning: Under the centralized training and decentralized execution CTDE architecture, the policy entropy regularization term is introduced into the trust domain constraint objective function of PPO to reconstruct the composite loss function of the Actor network.

[0010] Step S5: Perform adaptive adjustment of entropy coefficients based on Lagrange duality: Construct a dual loss function and dynamically update the temperature coefficient using a negative feedback closed loop. ;

[0011] Step S6: Implement a composite reward-driven approach to generate guided cooperative tactics: combine sparse rewards guided by the mission's final value with dense rewards guided by the potential function to smooth the gradient propagation path and guide the formation's emergent cooperative behavior.

[0012] Furthermore, the nonlinear motion state transition equation in step S1 is:

[0013]

[0014] in, Indicates the current time The physical state vector of an aircraft includes its three-dimensional position, attitude angles, and velocity. Indicates the current time The action command vector output by the agent As a high-impact factor, It is a non-linear mapping function.

[0015] Further, step S2 specifically includes:

[0016] Step S21, construct individual observation vectors Real-time acquisition of the agent's own three-dimensional position coordinates Heading angle Flight speed and information on the preset number of remaining missiles ;

[0017] Step S22, construct formation cooperative features : Integrate the three-dimensional position, velocity, and remaining munitions of all friendly aircraft in the formation except itself, and extract the status of missiles en route, including remaining flight time and target lock status. ;

[0018] Step S23: Construct the opponent's situation analysis vector Extracting relative distances for priority tracking targets Height difference , entrance angle Target azimuth Maximum missile launch range and the no-escape zone ;

[0019] Step S24, perform data normalization processing: transform individual observation vectors Formation coordination characteristics and opponent situation analysis vector By splicing and fusing the data, a complete 38-dimensional local observation vector for the agent at the current moment is constructed, and physical feature data for each dimension of this local observation vector is analyzed. Using the formula Map them uniformly to the interval [-1.0, 1.0], where, These are the normalized eigenvalues. and These are the minimum and maximum values ​​of the characteristic quantity in the real physical environment, respectively.

[0020] Furthermore, the motion mapping and physical constraints in step S3 are specifically as follows:

[0021] Step S31, Implement height adjustment constraints: Based on a preset reference height layer, adjust the height control component in the motion vector. Mapping height adjustment amount And the final flight altitude will be forcibly constrained within the preset tactical envelope;

[0022] Step S32, establish a linear available overload constraint model: the available overload function is expressed as:

[0023]

[0024] in, For the aircraft's current real-time flight altitude The maximum available overload that can be achieved under these conditions. The preset standard operational altitude, For maximum available overload, This is a high correction factor;

[0025] Step S33, execute overload command adaptive mapping: adjust the maneuver intensity component in the action vector. Dynamic mapping to the target overload is represented as:

[0026]

[0027] in, This represents the target maneuver overload command obtained from the final solution. The basic gravity overload of the aircraft in level flight;

[0028] Step S34, set a high threshold trigger mechanism for weapon firing: when a weapon firing decision command is issued... When the threshold value is greater than or equal to a preset threshold, the high threshold condition is triggered, and the aircraft performs a weapon firing action towards the locked target.

[0029] Furthermore, step S4 specifically includes:

[0030] Step S41: Construct an Actor-Critic neural network architecture based on parameter sharing: Employ a parameter-sharing network model based on a centralized training and decentralized execution CTDE architecture, enabling all agents' Actor networks to share the underlying multilayer perceptron (MLP) feature extraction backbone with the centralized Critic network. Global situation feature vectors are determined based on the core feature dimensions of the agent to improve training efficiency;

[0031] Step S42, Construct a decentralized Actor policy distribution: Employ a Gaussian policy network. The system represents four-dimensional continuous control commands for heading, altitude, overload, and launch. Indicated by network parameters Parameterized first Actor policy distribution of each agent Indicates the first An intelligent agent in The action command vector output at each moment, Indicates the first An intelligent agent in The local observation vector input at time step; Represents the multivariate Gaussian distribution function. Let be the mean vector of this Gaussian distribution. Let V be the variance of the Gaussian distribution. It is the identity matrix;

[0032] in, The differential entropy of the 3D action space is calculated using the following analytical expression:

[0033]

[0034] in, Indicates the first Each agent has the current policy parameters The differential entropy of the action, The dimension representing the action space. An index variable representing the dimension of the action space. Indicates that, given a local observation vector Under the condition that, the Gaussian policy distribution output by the Actor network is the first... The standard deviation of the action components;

[0035] Step S43, construct the Actor composite loss function under the trust region constraint: Composite Loss Function Represented as:

[0036]

[0037] in, This is the estimated global dominance function generated by the centralized Critic network. The preset cutoff coefficient, This represents a truncation function used to truncate variables. The value is forcibly constrained in Within the interval, The weighting coefficients representing the policy entropy regularization term, i.e., the temperature coefficients, This represents the probability ratio between the old and new strategies.

[0038] Furthermore, the parameter-sharing Actor-Critic neural network architecture in step S41 includes:

[0039] Multi-source situational input layer: Receives local observation vectors from 6 friendly agents, and simultaneously inputs a 72-dimensional global situational feature vector into the centralized evaluation network during the training phase. The 72-dimensional global situational feature vector is composed of six-dimensional core physical attributes of 12 agents from both the red and blue sides. The six-dimensional core physical attributes include three-dimensional position coordinates X, Y, Z, flight speed, heading angle, and pitch angle.

[0040] Shared feature extraction backbone The backbone network consists of a multilayer perceptron (MLP) network and layer standardization modules.

[0041] Decentralized execution head and centralized evaluation network: Two processing branches are derived from the shared feature extraction backbone. The first branch contains 6 independent Actor policy networks, which output their respective maneuver decision parameters based on the shared features.

[0042] The second branch is the soft value function network. In other words, a centralized evaluation network is used to assess the expected gains of the current global situation and output the global state value. Then calculate the global advantage function. Global advantage function Represented as:

[0043]

[0044] in, For timing difference error, As a discount factor, To estimate the smoothing coefficient of GAE for generalized advantage, As the reward for the current step, This is the index of the time step for future prediction in the GAE algorithm.

[0045] Further, step S5 specifically includes:

[0046] Step S51, construct the optimization objective based on Lagrange duality: construct the objective with respect to the temperature coefficient using the Lagrange duality method. Dual loss function :

[0047]

[0048] in, Indicates the current time The probability distribution of action policies in the Actor network. This represents the action instruction vector that the agent samples from the policy distribution. The input represents the global situational characteristics at the current moment. Indicates the current policy distribution The expected value is obtained by sampling the motion trajectory. The preset target entropy value;

[0049] Step S52, execute temperature coefficient Gradient update: through the gradient of the dual loss function Online parameter updates are represented as follows:

[0050]

[0051] in, The real-time differential entropy of the policy network at the current moment;

[0052] Step S53, construct a negative feedback adaptive closed loop for tactical evolution: utilize the updated temperature coefficient Dynamically adjust the weight of the entropy regularization term in the Actor network loss function when the real-time entropy value is lower than the target entropy. At that time, temperature coefficient Automatically increases; when the real-time entropy value is higher than the target entropy. At that time, temperature coefficient Automatically decreases.

[0053] Further, step S6 specifically includes:

[0054] Step S61: Set the task's final value-oriented sparse reward. : The sparse reward Defined as a reward for completing a task Enemy damage reward and the punishment for personal injury The linear summation combination formed, i.e. ;

[0055] Step S62, implement PBRS based on potential energy function: introduce differential auxiliary reward that satisfies the invariance of optimal policy. ,in, This represents the global state at the current moment. This represents the global state at the next moment after the state transition. Discount factor; and The system is in state respectively and The heuristic potential function value obtained from the calculation is used to apply the differential auxiliary reward through a preset constant reward weighting coefficient. This is added to the underlying sparse reward to generate the total reward used for network training. ;

[0056] Step 63, perform collaborative reliability allocation based on a centralized evaluation network: the reliability allocation logic uses a global advantage function. The sign of the positive or negative sign and the magnitude of the value are used to achieve this: when When, it indicates that the intelligent agent is The coordinated actions executed at any given time contribute significantly more to the overall formation gain than the baseline expectation. This allows for the positive reinforcement of the generation probability of this high-value collaborative maneuver behavior through the Actor policy gradient, thereby achieving implicit confidence allocation in a 6v6 scale.

[0057] Furthermore, the heuristic potential function Represented as:

[0058]

[0059] in, and These are the preset weighting constants for the potential energy term. For the mission's propulsion potential term, For boundary penalty constraints, The target azimuth angle.

[0060] Furthermore, the task propulsion potential energy term The calculation method is the reciprocal of the normalized relative distance between the agent's current 3D position and the target's spatial center point, with boundary penalty constraints. This is implemented based on logical condition judgment. When a radar unlock, blind firing, or missile miss-target invalid maneuver event is detected, this item outputs a preset fixed penalty constant; otherwise, it is 0.

[0061] The beneficial technical effects of this invention are as follows: By deeply integrating trust domain constraints and adaptive entropy regularization mechanisms, it systematically solves the problems of "cold start" and "strategy homogenization" caused by the extremely sparse reward signals in beyond-visual-range cooperative air combat decision-making. While ensuring monotonically converging and stable training, it significantly enhances the agent's exploration depth in unknown high-value tactical areas. It utilizes shared MLP features to extract backbone representations of 72-dimensional global situational features, thereby improving the feature extraction efficiency of large-scale agent training. It employs Lagrange dual logic to achieve a dynamic balance of exploration intensity, enabling the agent to effectively escape the trap of merely pursuing local optima for survival avoidance. It also guides the formation to spontaneously emerge with complex cooperative tactics such as decoy cover, angle management, and avoiding overlapping fire, greatly improving... This invention improves the decision-making efficiency and strike accuracy of the OODA loop. By dynamically adjusting the temperature coefficient to balance exploration and utilization during training, it solves the problem of premature convergence in sparse reward environments. It implements a composite reward mechanism that includes mission final value rewards and reward shaping based on potential energy functions, achieving efficient confidence allocation and collaborative tactical emergence in 6v6 scenarios. Furthermore, by combining an altitude-adaptive flight physics constraint model and parameter sharing architecture, this invention not only ensures the reliability of decision-making schemes in variable altitude physics environments but also demonstrates excellent strategic resilience and cross-situation generalization capabilities. It can effectively cope with defensive counterattack missions under asymmetric disadvantage openings, achieving Pareto improvement in combat effectiveness and significantly enhancing the collaborative combat effectiveness of formations in highly dynamic and complex environments. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart illustrating a three-dimensional cooperative maneuver decision-making method based on the adaptive entropy regularized multi-agent proximal policy optimization algorithm (AE-MAPPO) provided in an embodiment of the present invention.

[0064] Figure 2 This invention provides a parameter-sharing network topology and a 72-dimensional global situation feature representation structure diagram under a centralized training and decentralized execution (CTDE) architecture.

[0065] Figure 3 This is a multi-index comprehensive comparison radar chart of the combat effectiveness of 6v6 beyond-visual-range cooperative maneuver warfare provided by an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] like Figure 1 As shown, a three-dimensional cooperative maneuver decision-making method based on an adaptive entropy regularized multi-agent proximal policy optimization algorithm includes the following steps:

[0068] Step 1, construct a cooperative adversarial distributed partially observable Markov decision process (Dec-POMDP) ​​model:

[0069] Step 11, establish the mathematical framework for multi-agent game theory: define the 6v6 cooperative adversarial task as an octet consisting of a set of agents, a global state space, a joint action space, a state transition function, a set of reward functions, an observation space, an observation function, and a discount factor. This framework defines the rules for agent interaction, state transition, and observation and perception in highly dynamic air combat scenarios.

[0070] Step 12, encapsulate and drive the dynamic model: Based on the Godot game engine and its provided CharacterBody3D class, encapsulate the three-dimensional position, attitude angle, flight speed, acceleration, and other core physical properties of the aircraft, and establish a nonlinear motion state transition equation. This nonlinear motion state transition equation serves as the state transition function in Step 11. The underlying physical driving logic is specifically represented as follows:

[0071]

[0072] in, Indicates the current time The agent's physical state vector includes its three-dimensional position, attitude angle, and velocity; Indicates the current time The motor control commands (i.e., action commands) output by the intelligent agent; The altitude influence factor is used to correct for the effect of air density at different altitudes on lift and drag coefficients. In this embodiment, the altitude influence factor... Based on the International Standard Atmosphere (ISA) model, this reflects the nonlinear decay characteristics of atmospheric density with altitude, and its specific expression is as follows:

[0073]

[0074] in, This represents the real-time flight altitude of the aircraft. For height Atmospheric density at that location The standard atmospheric density at sea level (approximately) ), This is the atmospheric elevation constant within the troposphere.

[0075] The nonlinear mapping function The specific expression is based on time step Discretized kinematic update equations:

[0076]

[0077]

[0078] in, and They are respectively The velocity and position components in the middle. The thrust is driven by the action command. Air resistance is affected by highly influential factors. For the mass of the aircraft, Using gravitational acceleration as the equation, a smooth transfer of the agent's discrete decision-making actions to a continuous three-dimensional physical space state is achieved.

[0079] Step 13, integrate standardized environment interaction interface: adapt the simulation environment to the Gymnasium PettingZoo standard protocol through a customized wrapper, use the Godot-RL plugin to realize cross-platform interaction between Python scripts and simulation engine, set the mapping relationship between decision cycle and simulation physical step size, and ensure that the agent acquires battlefield situation and outputs decision instructions at a preset frequency (preferably 1Hz).

[0080] Step 2, extract and normalize the multi-source heterogeneous situational features:

[0081] Step 21, Construct individual observation vectors Real-time acquisition of its own three-dimensional position coordinates Heading angle Flight speed and information on the preset number of remaining missiles ;

[0082] Step 22, Construct formation cooperative features : Integrate the three-dimensional position, velocity, and remaining munitions of all friendly aircraft in the formation except itself, and extract the status of missiles en route, including remaining flight time and target lock status. ;

[0083] Step 23, Construct the opponent's situation analysis vector Extracting relative distances for priority tracking targets Height difference Aspect Angle ), target azimuth (Antenna Train Angle) Maximum missile launch range and the No Escape Zone );

[0084] Step 24, Perform data normalization: This involves processing the individual observation vectors extracted in steps 21 to 23 above. Formation coordination characteristics and opponent situation analysis vector By splicing and fusing the data, a complete 38-dimensional local observation vector for the agent at the current moment is constructed; for each dimension of the local observation vector, physical feature data is then analyzed. Using the formula Map them uniformly to the interval [-1.0, 1.0], where, These are the normalized eigenvalues. and These are the minimum and maximum values ​​of the feature quantity in the real physical environment (for example, for relative distance features, the minimum and maximum values ​​are 0 to 50 nautical miles; for angle features such as the angle of entry, the minimum and maximum values ​​are -180° to 180°).

[0085] The local observation vectors after the above normalization process will be directly used as input data for the Actor policy network (multi-source situation input layer) corresponding to each agent in the subsequent step 4.

[0086] Step 3: Design an action mapping mechanism with physical constraints adaptive: This mechanism is used to receive the four-dimensional continuous action vectors output by the Actor neural network in subsequent steps. ,in, For heading control components, To control the components at a high degree, For the motor strength component, We assign weapon launch decision commands, decode these abstract numerical commands, and map them into actual flight control variables constrained by physical limits, thereby driving the simulation environment in step 1. The specific mapping logic is as follows:

[0087] Step 31, Implement height adjustment constraints: Based on a preset reference height layer, adjust the height control component in the motion vector. Mapping height adjustment amount The final flight altitude will be forcibly constrained within a preset tactical envelope, which is 1,000 to 50,000 feet.

[0088] Step 32, establish a linear available overload constraint model: the available overload function follows linear decay logic, at 25,000 feet ( (This is) the standard operational altitude; set the maximum available overload at this altitude. 9g:

[0089]

[0090] in, The preset standard combat altitude, Indicates the current real-time flight altitude of the aircraft. The maximum achievable overload (i.e., the pneumatic physical limit) is determined by a height correction factor. Real-time calculation of different flight altitudes The physical limits below ensure that maneuver decisions conform to the aerodynamic and physical characteristics of the thin atmosphere at high altitudes.

[0091] Step 33, execute overload command adaptive mapping: adjust the maneuver intensity component in the action vector. (Its numerical range is limited by the activation function of the network output layer.) The interval is dynamically mapped to the target overload, and the calculation formula is as follows:

[0092]

[0093] in, This represents the final calculated target maneuver overload command (i.e., the physical overload control quantity sent to the underlying layer). This represents the basic gravity overload of an aircraft in level flight. This formula enables adaptive adjustment of maneuver intensity at different altitude levels.

[0094] Step 34, Set the weapon launch high threshold triggering mechanism: The weapon launch decision command This is the fourth component of the four-dimensional continuous action vector output by the Actor neural network in subsequent steps (its value is also mapped to by the activation function). (interval); if and only if When this high threshold condition is triggered, the aircraft executes a weapon firing maneuver towards the locked target. This high threshold mechanism effectively filters out exploration noise in the early stages of reinforcement learning training.

[0095] Step 4: Construct the AE-MAPPO optimization framework based on maximum entropy reinforcement learning:

[0096] Step 41, construct the Actor-Critic neural network architecture based on parameter sharing:

[0097] like Figure 2 As shown, this embodiment constructs a parameter-sharing network model based on a centralized training-decentralized execution (CTDE) architecture. This network model is the underlying Actor-Critic neural network architecture that implements the aforementioned Multi-Agent Proximal Policy Optimization (MAPPO) algorithm. The network model specifically includes the following hierarchical structure:

[0098] Multi-source situational awareness input layer: Receives local observation vectors from each of the 6 friendly aircraft, which are the 38-dimensional vectors constructed and normalized in step 2; Simultaneously, during the training phase, a 72-dimensional global situational awareness feature vector is input to the centralized evaluation network. The 72-dimensional feature vector is composed of 6-dimensional core physical attributes of 12 agents from both the red and blue sides. The 6-dimensional core physical attributes specifically include: three-dimensional position coordinates (X, Y, Z), flight speed, heading angle, and pitch angle.

[0099] Shared feature extraction backbone The backbone network consists of a multilayer perceptron (MLP) network and a layer normalization module. Its function is to perform feature dimensionality reduction and high-level semantic extraction on the data received from the multi-source situational input layer. The Actor network of all agents and the centralized Critic network (i.e., the centralized evaluation network) jointly reuse the parameters of this underlying network. It extracts general battlefield semantic representations when processing different local observation vectors in the input, significantly improving feature extraction efficiency and sample utilization in 6v6 scale.

[0100] Decentralized execution head and centralized evaluation network: Two processing branches are derived from the shared feature extraction backbone:

[0101] The first branch contains 6 independent Actor policy networks (which serve as parameter generators for the Actor policy distribution in step 42), each outputting its own maneuver decision parameters based on shared features;

[0102] The second branch is the soft value function network. That is, a centralized evaluation network (composed of parameters) The multilayer perceptron implementation is used to evaluate the expected gains of the current global situation and output the global state value. Then calculate the global advantage function. Global advantage function The expression is calculated based on the generalized advantage estimation (GAE):

[0103]

[0104] in, For timing difference error, As a discount factor, This is the GAE smoothing coefficient. As the reward for the current step, This is the index of the time step for future prediction in the GAE algorithm.

[0105] Figure 2 The feedback path shown is specifically described as: the global dominance function estimate calculated by the centralized evaluation network. As an error feedback signal, it is transmitted to each decentralized Actor policy network. The policy gradient ascent method guides the weight update of the Actor network and the shared backbone network, thereby increasing the probability of maneuver actions that can generate higher global advantage values.

[0106] Step 42, Construct a decentralized Actor policy distribution: Employ a Gaussian policy network.

[0107]

[0108] in, Indicated by network parameters Parameterized first Actor policy distribution for each agent; Indicates the first An intelligent agent in The action command vector output at each moment (corresponding to the four-dimensional continuous action vector in step 3). Indicates the first An intelligent agent in The local observation vector input at time step; Represents the multivariate Gaussian distribution function; The mean vector of this Gaussian distribution is directly output by the Actor network based on the input situation, representing the deterministic action instruction of the current optimal decision; The variance of this Gaussian distribution represents the exploration noise (action randomness) of the policy during training. It is an identity matrix. The calculation method is as follows: the local observation vector of the agent... Input its corresponding Actor Gaussian policy network, the network outputs the mean With variance During the training phase, by applying a Gaussian distribution Sampling During the testing and evaluation phase, the average value is taken directly. As a deterministic action.

[0109] This formula represents the four-dimensional continuous control commands for heading, altitude, overload, and launch; among which... The differential entropy of the dimensional action space is expressed analytically as:

[0110]

[0111] Calculations are performed to measure the exploration potential of the strategy, whereby... Indicates the first Each agent has the current policy parameters The differential entropy of the action; Represents the dimension of the action space; The index variable representing the action space dimension (with values...) to ); Indicates that, given a local observation vector Under the condition that, the Gaussian policy distribution output by the Actor network is the first... Standard deviation of the action components.

[0112] Step 43, Construct the Actor composite loss function under the trust domain constraint: Introduce a soft policy iteration mechanism into the policy network update objective, and truncate the trust domain objective function of near-end policy optimization (PPO). With policy entropy regularization Perform weighted fusion to construct a composite loss function :

[0113]

[0114] in, This represents the time steps sampled in the experience playback buffer. Calculate the mathematical expectation of the state-action trajectory. This represents a truncation function used to truncate variables. The value is forcibly constrained in Within the range; The weighting coefficient of the policy entropy regularization term is the temperature coefficient (this parameter will be adaptively and dynamically adjusted in step 5 to balance exploration and exploitation). The probability ratio between the old and new strategies is calculated using the following formula: ,in, This represents the old Actor policy network before the current parameter update. This is an estimate of the global dominance function generated by the centralized Critic network; The preset cutoff coefficient is used to determine the cutoff value. Limit the range of updates between old and new strategies to ensure training monotonicity and strategy stability in a highly dynamic and non-stationary adversarial environment.

[0115] Step 5: Perform adaptive adjustment of the entropy coefficient based on Lagrange duality:

[0116] Step 51: Construct an optimization objective based on Lagrange duality: Transform the control of exploration intensity into a constrained optimization problem, requiring the policy to maximize external rewards while maintaining its average entropy at or above a preset target entropy value. Constructing information about the temperature coefficient using the Lagrange duality method Dual loss function :

[0117]

[0118] in, Indicates the current time The probability distribution of action policies in the Actor network; This represents the action instruction vector that the agent samples and outputs from the policy distribution. The motion control variable used to drive the physics engine in the nonlinear motion state transition equation of step 12 above is the same physical quantity, realizing the direct mapping from algorithm decision to physical control; The input represents the global situational characteristics at the current moment; Indicates the current policy distribution The expected value is obtained by sampling the motion trajectory.

[0119] Target Entropy Preferred based on the dimensions of the action space Set as .

[0120] Step 52, Execute temperature coefficient Gradient update: via the gradient of the dual loss function:

[0121]

[0122] right Perform online parameter updates; among them, This represents the real-time differential entropy of the policy network at the current moment.

[0123] Step 53, construct a negative feedback adaptive closed loop for tactical evolution: utilize the updated temperature coefficient Dynamically adjust the weights of the entropy regularization term in the Actor network loss function to form the following closed-loop logic:

[0124] Forced exploration mechanism: When the real-time entropy value is lower than the target entropy At that time, temperature coefficient Automatically increasing the randomness of actions by forcing the algorithm to explore a wider range of maneuvering space, thereby breaking through the local optimum trap.

[0125] Precise convergence mechanism: When the real-time entropy value is higher than the target entropy, the temperature coefficient... Automatic reduction causes the strategy distribution to collapse towards deterministic, high-return actions, ensuring tactical stability and ultimate win rate in 6v6 squad battles.

[0126] Step 6, Implement a composite reward-driven approach to generate guided collaborative tactics:

[0127] Step 61, Set up a task end-value oriented sparse reward : The sparse reward Defined as a reward for completing a task Enemy damage reward and the punishment for personal injury The linear summation combination formed (i.e. This sparse reward itself serves as a global benefit anchor for measuring the overall combat effectiveness of the formation, providing underlying constraints for the evolution of formation strategies.

[0128] Step 62, Implement Potential-Based Reward Shaping (PBRS): Introduce differential auxiliary rewards that satisfy the invariance of the optimal policy. ,in, This represents the global state at the current moment. This represents the global state at the next moment after the state transition. Discount factor; and The system is in state respectively and The heuristic potential function value obtained from the calculation is used to apply the differential auxiliary reward through a preset constant reward weighting coefficient. This is added to the underlying sparse reward to generate the total reward used for network training. This allows for the construction of a gradient guiding field in the state space, smoothing the gradient propagation path.

[0129] Step 63, Construct the heuristic potential function Its specific calculation expression is defined as follows: ,in, and These are the preset weighting constants for the potential energy term. For the mission's propulsion potential term, Boundary penalty constraints; mission propulsion potential energy term The calculation method is the reciprocal of the normalized relative distance between the agent's current 3D position and the target's spatial center point; the cosine value of the target azimuth angle (Antenna Train Angle) through which the continuously tracked excitation term passes is also used. Calculations are performed to provide positive incentives for tracking situations at small angles; boundary penalty constraints are also implemented. This is implemented based on logical condition judgment. When invalid maneuvering events such as radar unlocking, blind firing, and missile missing the target are detected, this item outputs a preset fixed penalty constant; otherwise, it is 0.

[0130] Step 64: Perform collaborative confidence allocation based on a centralized value network: the centralized Critic network is the centralized evaluation network constructed in step 41 above. A parameter-sharing architecture is used to enable the updating of Actor gradients to help the centralized Critic network accurately fit the soft value function containing the entropy term. The specific fitting method is to use the time-series difference algorithm to minimize the prediction value and the augmented target return (that is, to incorporate the product of the temperature coefficient and the policy entropy in step 43). The mean squared error between the cumulative returns (after which the confidence level is determined). Based on this, the confidence allocation logic is specifically implemented through a global advantage function. The sign of the positive or negative sign and the magnitude of the value are used to achieve this: when When, it indicates that a specific intelligent agent is The coordinated actions executed at any given time contribute significantly more to the overall formation gain than the baseline expectation. The algorithm then reinforces the generation probability of this high-value collaborative maneuver behavior through the Actor policy gradient, thereby achieving implicit confidence allocation in a 6v6 scale.

[0131] To verify the effectiveness of the AE-MAPPO algorithm described in this invention in three-dimensional cooperative maneuver decision-making, this embodiment conducted a 6v6 adversarial experiment in the B-ACE simulation environment, and selected IPPO, MAPPO, High-MAPPO and MASAC algorithms as benchmark comparison algorithms.

[0132] Figure 3 This paper presents a performance comparison of the AE-MAPPO algorithm of this invention with various benchmark algorithms across five dimensions: Win Rate, Annihilation, Normalized Kills, Survival Rate, and Normalized Reward. Figure 3 As shown, the AE-MAPPO algorithm provided by this invention (red shaded area in the figure) significantly outperforms other algorithms in overall performance, specifically in the following ways:

[0133] Complete win rate and kill efficiency: Thanks to the adaptive entropy adjustment mechanism described in step 5, the algorithm maintains a high degree of exploration in the early stage of training, and locks in high-yield tactics through precise convergence in the later stage, so that the complete win rate index is close to 1.0, far exceeding the conventional MAPPO algorithm.

[0134] Improved Survival Rate: With the potential energy function reward shaping designed in steps 62 and 63, the agent exhibits more reasonable defensive and evasive cooperative behaviors, resulting in a significant improvement in survival rate compared to the baseline algorithm.

[0135] Maximizing global benefits: By allocating credibility through the centralized value network in step 64, the problem of "credit allocation" in multi-machine collaboration is effectively solved, so that the normalized reward of the formation reaches the highest level.

[0136] In summary, the method described in this invention has higher decision-making efficiency and tactical emergence capability when dealing with highly dynamic, large-scale collaborative air combat decision-making tasks.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional cooperative maneuver decision-making method based on an adaptive entropy regularized multi-agent proximal policy optimization algorithm, characterized in that, The method includes: Step S1: Construct a cooperative adversarial distributed partially observable Markov decision process (Dec-POMDP) ​​model: Define the 6v6 cooperative adversarial task as an octet consisting of a set of agents, a global state space, a joint action space, a state transition function, a set of reward functions, an observation space, observation functions, and a discount factor. The nonlinear motion state transition is driven by the B-ACE dynamics simulation engine of flight physics and three-dimensional air combat. Step S2: Extract and normalize multi-source heterogeneous situational features: Construct local observation vectors in real time that include the aircraft's own flight status, friendly aircraft cooperation features, and adversary threat situational features. The physical parameters are mapped to using a linear normalization method. interval; Step S3: Design an action mapping mechanism with physical constraints adaptive: the agent outputs a four-dimensional continuous vector. Through highly adaptive overload mapping and high threshold triggering decision-making, the transformation from tactical intent to motion state is realized; Step S4: Construct the AE-MAPPO optimization framework based on maximum entropy reinforcement learning: Under the centralized training and decentralized execution CTDE architecture, the policy entropy regularization term is introduced into the trust domain constraint objective function of PPO to reconstruct the composite loss function of the Actor network. Step S5: Perform adaptive adjustment of entropy coefficients based on Lagrange duality: Construct a dual loss function and dynamically update the temperature coefficient using a negative feedback closed loop. ; Step S6: Implement a composite reward-driven approach to generate guided cooperative tactics: combine sparse rewards guided by the mission's final value with dense rewards guided by the potential function to smooth the gradient propagation path and guide the formation's emergent cooperative behavior.

2. The method according to claim 1, characterized in that, The nonlinear motion state transition equation in step S1 is: in, Indicates the current time The physical state vector of an aircraft includes its three-dimensional position, attitude angles, and velocity. Indicates the current time The action command vector output by the agent Indicates a high-impact factor. This represents a nonlinear mapping function.

3. The method according to claim 1, characterized in that, Step S2 specifically includes: Step S21, construct individual observation vectors Real-time acquisition of the agent's own three-dimensional position coordinates Heading angle Flight speed and information on the preset number of remaining missiles ; Step S22, construct formation cooperative features : Integrate the three-dimensional position, velocity, and remaining munitions of all friendly aircraft in the formation except itself, and extract the status of missiles en route, including remaining flight time and target lock status. ; Step S23: Construct the opponent's situation analysis vector Extracting relative distances for priority tracking targets Height difference , entrance angle Target azimuth Maximum missile launch range and the no-escape zone ; Step S24, perform data normalization processing: transform individual observation vectors Formation coordination characteristics and opponent situation analysis vector By splicing and fusing the data, a complete 38-dimensional local observation vector for the agent at the current moment is constructed, and physical feature data for each dimension of this local observation vector is analyzed. Using the formula Map them uniformly to the interval [-1.0, 1.0], where, These are the normalized eigenvalues. and These are the minimum and maximum values ​​of the characteristic quantity in the real physical environment, respectively.

4. The method according to claim 1, characterized in that, The motion mapping and physical constraints in step S3 are specifically as follows: Step S31, Implement height adjustment constraints: Based on a preset reference height layer, adjust the height control component in the motion vector. Mapping height adjustment amount And the final flight altitude will be forcibly constrained within the preset tactical envelope; Step S32, establish a linear available overload constraint model: the available overload function is expressed as: in, For the aircraft's current real-time flight altitude The maximum available overload that can be achieved under these conditions. The preset standard combat altitude, For maximum available overload, This is a high correction factor; Step S33, execute overload command adaptive mapping: adjust the maneuver intensity component in the action vector. Dynamic mapping to the target overload is represented as: in, This represents the target maneuver overload command obtained from the final solution. The basic gravity overload of the aircraft in level flight; Step S34, set a high threshold trigger mechanism for weapon firing: when a weapon firing decision command is issued... When the threshold value is greater than or equal to a preset threshold, the high threshold condition is triggered, and the aircraft performs a weapon firing action towards the locked target.

5. The method according to claim 1, characterized in that, Step S4 specifically includes: Step S41: Construct an Actor-Critic neural network architecture based on parameter sharing: Employ a parameter-sharing network model based on a centralized training and decentralized execution CTDE architecture, enabling all agents' Actor networks to share the underlying multilayer perceptron (MLP) feature extraction backbone with the centralized Critic network. Global situation feature vectors are determined based on the core feature dimensions of the agent to improve training efficiency; Step S42, Construct a decentralized Actor policy distribution: using a Gaussian policy network. The system represents four-dimensional continuous control commands for heading, altitude, overload, and launch. Indicated by network parameters Parameterized first Actor policy distribution of each agent Indicates the first An intelligent agent in The action command vector output at each moment, Indicates the first An intelligent agent in The local observation vector input at time step; Represents the multivariate Gaussian distribution function. Let be the mean vector of this Gaussian distribution. Let V be the variance of the Gaussian distribution. It is the identity matrix; in, The differential entropy of the 3D action space is calculated using the following analytical expression: in, Indicates the first Each agent has the current policy parameters The differential entropy of the action, The dimension representing the action space. Index variable representing the dimension of the action space. Indicates that, given a local observation vector Under the condition that, the Gaussian policy distribution output by the Actor network is the first... The standard deviation of the action components; Step S43, construct the Actor composite loss function under the trust region constraint: Composite Loss Function Represented as: in, The global dominance function is estimated by the centralized Critic network. The preset cutoff coefficient, This represents a truncation function used to truncate variables. The value is forcibly constrained in Within the interval, The weighting coefficients representing the policy entropy regularization term, i.e., the temperature coefficients, This represents the probability ratio between the old and new strategies.

6. The method according to claim 5, characterized in that, The parameter-sharing Actor-Critic neural network architecture in step S41 includes: Multi-source situational input layer: Receives local observation vectors from 6 friendly agents, and simultaneously inputs a 72-dimensional global situational feature vector into the centralized evaluation network during the training phase. The 72-dimensional global situational feature vector is composed of six-dimensional core physical attributes of 12 agents from both the red and blue sides. The six-dimensional core physical attributes include three-dimensional position coordinates X, Y, Z, flight speed, heading angle, and pitch angle. Shared feature extraction backbone The backbone network consists of a multilayer perceptron (MLP) network and layer standardization modules. Decentralized execution head and centralized evaluation network: Two processing branches are derived from the shared feature extraction backbone. The first branch contains 6 independent Actor policy networks, which output their respective maneuver decision parameters based on the shared features. The second branch is the soft value function network. In other words, a centralized evaluation network is used to assess the expected gains of the current global situation and output the global state value. Then calculate the global advantage function. Global advantage function Represented as: in, For timing difference error, As a discount factor, To estimate the smoothing coefficient of GAE for generalized advantage, As the reward for the current step, This is the index of the time step for future prediction in the GAE algorithm.

7. The method according to claim 6, characterized in that, Step S5 specifically includes: Step S51, construct the optimization objective based on Lagrange duality: construct the objective with respect to the temperature coefficient using the Lagrange duality method. Dual loss function : in, Indicates the current time The probability distribution of action policies in the Actor network. This represents the action instruction vector that the agent samples from the policy distribution. The input represents the global situational characteristics at the current moment. Indicates the current policy distribution The expected value is obtained by sampling the motion trajectory. The preset target entropy value; Step S52, execute temperature coefficient Gradient update: through the gradient of the dual loss function Online parameter updates are represented as follows: in, The real-time differential entropy of the policy network at the current moment; Step S53, construct a negative feedback adaptive closed loop for tactical evolution: utilize the updated temperature coefficient Dynamically adjust the weight of the entropy regularization term in the Actor network loss function when the real-time entropy value is lower than the target entropy. At that time, temperature coefficient Automatically increases; when the real-time entropy value is higher than the target entropy. At that time, temperature coefficient Automatically decreases.

8. The method according to claim 1, characterized in that, Step S6 specifically includes: Step S61: Set the task's final value-oriented sparse reward. : The sparse reward Defined as a reward for completing a task Enemy damage reward and the punishment for personal injury The linear summation combination formed, i.e. ; Step S62, implement PBRS based on potential energy function: introduce differential auxiliary reward that satisfies the invariance of optimal policy. ,in, This represents the global state at the current moment. This represents the global state at the next moment after the state transition. Discount factor; and The system is in state respectively and The heuristic potential function value obtained from the calculation is used to apply the differential auxiliary reward through a preset constant reward weighting coefficient. This is added to the underlying sparse reward to generate the total reward used for network training. ; Step 63, perform collaborative reliability allocation based on a centralized evaluation network: the reliability allocation logic uses a global advantage function. The sign of the positive or negative sign and the magnitude of the value are used to achieve this: when When, it indicates that the intelligent agent is The coordinated actions executed at any given time contribute significantly more to the overall formation gain than the baseline expectation. This allows for the positive reinforcement of the generation probability of this high-value collaborative maneuver behavior through the Actor policy gradient, thereby achieving implicit confidence allocation in a 6v6 scale.

9. The method according to claim 8, characterized in that, The heuristic potential function Represented as: in, and These are the preset weighting constants for the potential energy term. For the mission's propulsion potential term, For boundary penalty constraints, The target azimuth angle.

10. The method according to claim 9, characterized in that, The mission propulsion potential energy term The calculation method is the reciprocal of the normalized relative distance between the agent's current 3D position and the target's spatial center point, with boundary penalty constraints. This is implemented based on logical condition judgment. When a radar unlock, blind firing, or missile miss-target invalid maneuver event is detected, this item outputs a preset fixed penalty constant; otherwise, it is 0.

Images