A hierarchical autonomous behavior decision-making modeling method for air force based on PPO and HIRO algorithms

Abstract

This invention discloses a hierarchical autonomous behavior decision-making modeling method for air force based on PPO and HIRO algorithms, belonging to the field of computer-generated force confrontation decision-making technology. It proposes a joint operational behavior decision-making model based on reinforcement learning algorithms, supporting exploration and optimization in operational scenarios without prior experience. A hierarchical joint operational behavior decision-making model is proposed, avoiding problems such as network training non-convergence and low training efficiency caused by high-dimensional decision spaces in complex operational scenarios, and iterative optimization is performed based on reinforcement learning algorithms. A training process for the overall hierarchical behavior decision-making model based on a pre-trained model is proposed, which can avoid the policy non-stationarity problem caused by simultaneous training of behavior decision-making models at different decision levels, helping to improve agent training efficiency and making the policy learning process more stable. The technical solution of this invention can effectively complete hierarchical behavior decision-making modeling and efficiently generate a joint operational hierarchical behavior decision-making model through pre-training and overall training using reinforcement learning algorithms.

Description

Technical Field

[0001] This invention relates to the field of computer-generated force confrontation decision-making technology, and more specifically to a hierarchical autonomous behavior decision-making modeling method for air force based on PPO and HIRO algorithms. Background Technology

[0002] Combat simulation and evaluation based on adversarial simulation systems are important decision-making tools in the battlefield environment and also serve as virtual battlefields for studying adversarial tactics. Since the Iraq War, joint operations have become the main mode of combat, making the battlefield environment more complex and exponentially increasing battlefield situational information. This has placed higher demands on response speed, and traditional combat models relying on commanders are no longer adequate for the needs of current operations. There is an urgent need for intelligent decision-making technologies to improve the effectiveness and flexibility of combat plans and enhance the adaptability and decision-making capabilities of decision-making units in adversarial environments. This poses new requirements for adversarial combat simulation.

[0003] In military simulation systems, Computer Generated Forces (CGFs) are virtual combat forces created by computers and capable of autonomously controlling or guiding all or part of their actions and behaviors. They are a crucial means and effective approach to supporting adversarial simulations. Introducing CGFs into combat simulations helps expand the simulation scale and enables flexible control and prediction. The core task of building CGFs is to model the behavior of combat entities in the battlefield environment. The goal is to provide a behavioral model that allows CGFs to output actions or behaviors based on the observed situation, essentially modeling human behavior. Command and Control (C2) behavioral decision-making modeling of computer-generated forces is a core technology in adversarial simulations. Intelligent force behavioral decision-making modeling is a research result that transfers intelligent game theory to the combat domain and is a key focus of adversarial simulation research under joint operations systems. Using neural networks as the main technical means, intelligent force behavioral decision-making modeling can make rapid and autonomous decisions in complex adversarial simulation environments, and is key to solving the intelligentization of combat plan generation, mission planning, and on-the-spot decision-making in the context of joint operations. Therefore, in the context of information-based warfare, there is an urgent need for high-quality and intelligent force behavior decision-making modeling methods in force confrontation simulations to support the increasingly complex joint force confrontation simulation requirements.

[0004] Traditional force behavior decision-making modeling often relies on expert experience, lacking flexibility and exploratory capabilities. Reinforcement learning, a method for solving sequential decision-making problems, has spawned several branches since its inception, including deep reinforcement learning, multi-agent reinforcement learning, and hierarchical reinforcement learning. Reinforcement learning can continuously interact with the environment through an "interaction-trial-and-error" mechanism, thereby learning effective decision-making schemes. This end-to-end learning approach provides a new perspective for force behavior decision-making modeling, and the development of military intelligent game platforms makes this approach possible: researchers can construct intelligent behavior decision-making models based on reinforcement learning, interact with combat simulation platforms by receiving situational awareness and outputting actions, thereby optimizing the decision-making model and evaluating its capabilities. Research on intelligent adversarial force behavior decision-making modeling based on these game platforms can provide effective technical assistance for generating efficient combat decision-making schemes and improving joint operational command levels, possessing significant theoretical and applied value.

[0005] This invention primarily considers using reinforcement learning methods to model the hierarchical behavioral decision-making model of our formation in the context of air combat. Currently, the main types of algorithms used for joint operational behavioral decision-making modeling with reinforcement learning include: 1) Single-agent reinforcement learning algorithms, which use a single agent to implement all behavioral decisions in a joint operational scenario, such as DQN (Deep Q-network) and PPO; this approach can naturally represent the collaborative cooperation between multiple agents, but as the number of agents increases, the joint action space grows exponentially, making it impractical. 2) Multi-agent reinforcement learning algorithms, which often use the Centralized Training with Decentralized Execution (CTDE) paradigm for decision-making, such as QMIX and VDN; this can solve the dimensionality explosion caused by the increase in the number of agents, but it cannot model the hierarchical behavioral decisions in actual joint operations, nor can it solve the problem of the increased decision space in hierarchical decision-making. 3) Hierarchical reinforcement learning algorithms, such as FuN and HIRO; these algorithms can naturally express the hierarchical behavioral decision-making characteristics during joint operations, while also decomposing the decision space, and have the potential to effectively solve the problem of exponential growth of the action space.

[0006] Furthermore, training instability and difficulty in convergence are currently key factors restricting further performance improvements of hierarchical reinforcement learning algorithms. This is mainly because the alternating iteration of behavioral decision-making models at each layer exacerbates the non-stationarity of reinforcement learning training. Simultaneously, the decision objectives of each layer are complex, making direct optimization and iteration of the entire model inefficient. Therefore, optimizing the training process is necessary to reduce training non-stationarity and improve model optimization training stability and convergence speed. Methods for optimizing the hierarchical reinforcement learning training process include offline sample correction, hierarchical iteration, and pre-training. Therefore, joint operations autonomous decision-making behavioral decision-making modeling technology based on hierarchical reinforcement learning algorithms and optimized training processes has significant theoretical and military application value for solving game simulation and adversarial problems in complex environments and forming an effective joint operations behavioral decision-making modeling method. Summary of the Invention

[0007] For complex joint air force operations, this invention employs a hierarchical training approach. This type of algorithm includes option-based algorithms (such as HOC and HIPPO) and objective-based algorithms (such as FuN and HIRO). Objective-based algorithms are more suitable for scenarios where each decision-making layer is relatively independent, offering better reproducibility and flexibility. HIRO is an objective-based algorithm that effectively solves hierarchical behavioral decision-making problems. Given that the decision-making entities for each behavioral decision-making layer are different in this invention, the HIRO algorithm is used to model the joint operational behavior decision-making of the formation. To address the difficulty in convergence during training of hierarchical reinforcement learning models, this invention adopts a pre-training-overall training process. The PPO reinforcement learning algorithm is used to pre-train each behavioral decision-making layer, and then the pre-trained model is used to perform overall training of the joint operational behavior decision-making model, improving the training stability and convergence speed of HRL.

[0008] This invention establishes a hierarchical autonomous behavior decision-making model for joint air force operations based on the HIRO and PPO algorithms. In typical joint operational scenarios, the overall combat mission can often be broken down into several specific, well-defined, and interrelated sub-tasks. Directly modeling the overall behavior decision-making would lead to an explosion in the dimensionality of the state and action spaces. Therefore, to introduce a hierarchical reinforcement learning algorithm, the joint air force combat mission is first hierarchically divided, such as... Figure 1 As shown.

[0009] Different decision-making levels need to execute different operational tasks: the overall command level schedules tasks across multiple regions, including selecting operational areas and regional operational formations; the regional operational level allocates tasks to weapon platforms within the region, including weapon platform target allocation and interception zone selection; and the weapon platform level assigns tasks that can be performed by individual soldiers, including interception operations, target assault, and patrol route planning. After the upper-level decision-making level makes a decision, the lower-level decision-making levels need to execute the decisions of the upper-level decision-making level within a certain time step, and their temporal relationship is as follows: Figure 2 As shown, after constructing the hierarchical behavioral decision-making structure for joint air force operations, the corresponding state space, action space, and reward function are constructed. A joint operational force behavior decision-making model based on hierarchical reinforcement learning is then built, providing methodological support for joint operational behavior decision-making.

[0010] In constructing a hierarchical behavior decision-making model based on the HIRO algorithm, this invention addresses the issues of poor stability and difficulty in convergence inherent in hierarchical reinforcement learning algorithms by employing a pre-training + overall training process for optimization. When using different behavior decision-making layers to make behavioral decisions, both the individual's own task objectives and the decision instructions from the upper-level decision-making layers need to be considered simultaneously. This multi-objective training mode suffers from poor convergence. In this invention, the PPO algorithm is first used for pre-training with the individual's own task objectives as the decision-making targets. After obtaining pre-trained behavior decision-making models at each layer capable of completing the corresponding tasks, the models from each decision layer are combined and then trained using the HIRO algorithm to achieve overall training, ultimately realizing the complete modeling of an intelligent behavior decision-making model for joint air force operations. The overall process of hierarchical behavior decision modeling is as follows: Figure 3 As shown, the training process of the hierarchical behavioral decision-making model is as follows: Figure 4 As shown.

[0011] This invention establishes a hierarchical autonomous behavior decision-making model for joint air force operations based on the HIRO and PPO algorithms, comprising the following steps:

[0012] S1: The joint operations behavior decision-making model is hierarchically divided from top to bottom as follows: Overall Command - Regional Operations - Weapon Platforms. The command layer is the high-level decision-making layer, responsible for formulating the overall operational plan and allocating deployment locations for regional operations; the formation layer is the mid-level decision-making layer, responsible for target allocation to weapon platforms; and the weapon platform layer is the low-level decision-making layer, specifically determining the operational and mission execution methods of each weapon platform. This ultimately forms a hierarchical autonomous behavior decision-making framework based on a hierarchical reinforcement learning algorithm.

[0013] S2: Based on the characteristics of the overall combat mission, feature vectors are extracted to construct the input state space of each layer of reinforcement learning algorithms, so as to effectively represent the combat situation of concern at each layer of formation cooperative combat.

[0014] S3: Based on the operational characteristics and tasks of different decision-making levels in the overall operational mission, design different behavioral decision-making spaces for each decision-making level in the coordinated operations of the formation.

[0015] S4: Extract feature vectors and design reward functions for reinforcement learning algorithms for combat objectives at different decision levels.

[0016] S5: Based on the PPO algorithm, the reinforcement learning state space, action space and reward function design mentioned above are integrated to pre-train each decision layer.

[0017] S6: Based on the HIRO algorithm and the behavioral decision-making models of each decision layer obtained from the pre-training in step 5, the reinforcement learning state space, action space and reward function design designed above are integrated, and the reward function of each decision layer is modified based on the overall reward function of joint operations, forming a complete operation process for iterative optimization training of the joint operations behavioral decision-making model.

[0018] Furthermore, the specific process of step S1 is as follows:

[0019] S1-1: Addressing the multi-layered nature of coordinated operations, joint operational decision-making is hierarchically divided into three levels: overall command, regional operations, and weapon platforms. This results in a model as follows: Figure 1 The hierarchical behavioral decision-making framework shown.

[0020] S1-2: The overall command employs reinforcement learning algorithms to make decisions regarding area selection during regional operations. When a regional combat formation, including but not limited to early warning aircraft, fighter jets, and bombers, receives operational instructions from the Red Force command center, the formation will proceed to the corresponding area to conduct patrol missions and prepare for subsequent combat missions.

[0021] S1-3: The regional combat layer employs a behavioral model that integrates reinforcement learning algorithms and traditional rule sets for decision-making, including task allocation and target selection. Taking fighter jet formations as an example, a rule set is used for target allocation decisions, while a reinforcement learning algorithm is used to select the interception zone, guiding the fighter jets to the designated area to perform interception missions. The target allocation rule set is shown in Table 1.

[0022] Table 1. Set of Target Allocation Rules for Fighter Aircraft Formations

[0023]

[0024] S1-4: The weapon platform layer employs a behavioral model that integrates reinforcement learning algorithms and traditional rule sets to make decisions regarding the combat behavior of combat aircraft. Taking fighter jets as an example, rule sets are used to select combat behaviors, as shown in Table 2. When a fighter jet performs an interception mission, a reinforcement learning algorithm is used to plan its flight path.

[0025] Table 2. Fighter Aircraft Combat Rules Set

[0026] 1 Insufficient fuel Return 2 Sufficient fuel and regional operational layer allocation of interception targets Execute interception task 3 Sufficient fuel and no interception targets assigned to the area combat layer Patrol the designated area

[0027] Furthermore, the specific process of step S2 is as follows:

[0028] S2-1: In joint operations, the battlefield scope, tactical targets, situational elements, content, and information precision that are of concern to personnel at different levels vary. Therefore, the operational situation that different command levels need to acquire differs. The overall command level acquires overall battlefield situational information, which may include command and control area map G. c Air attack range diagram G aa Air detection range map G ad Maritime attack range map G sa Map G of the sea exploration range sd These situational diagrams, together, constitute the state input space for the overall command-level decision-making model, for example:

[0029] S t = <G c G aa G ad G sa G sd >

[0030] S2-2: The situational awareness acquired at the regional operational layer is regional situational information. Taking a regional collaborative combat system that includes, but is not limited to, fighter formations, bomber formations, and early warning aircraft as an example, the three-dimensional coordinates (X, Y, Z) of all friendly aircraft within the region can be selected. r ,Y r Z r ), Troop Type LX r Heading H r Speed ​​V r Ammunition capacity D_num r The three-dimensional coordinates of the enemy's incoming target (X) b ,Y b Z b ), Troop Type LX b Heading H b Speed ​​V b These, together, constitute the state input space for the regional operational layer decision-making model, for example:

[0031] S m = <X r ,Y r Z r LX r H r ,D_num r Vr ,I r ,X b ,Y b Z b LX b H b V b >

[0032] S2-3: The situational awareness acquired at the weapon platform level is the local situational awareness when performing a single-aircraft mission. Taking a fighter jet performing an interception mission as an example, the force type LX can be selected. r Current fuel quantity F r Current remaining ammunition quantity D_num r The relative position of the intercepted target (X) rb ,Y rb Z rb ), Troop Type LX b Speed ​​V b Angle of entry (AOT), line-of-sight (ATA), etc., together constitute the state input space for the weapon platform-level decision-making model, for example:

[0033] S l = <LX r ,F r ,D_num r ,X rb ,Y rb Z rb LX b V b AOT, ATA>

[0034] Furthermore, the specific process of step S3 is as follows:

[0035] S3-1: Taking area patrol operations as an example, design the behavioral decision space of the overall command layer. Divide the battlefield into n1 different patrol areas. Assuming there are k area combat formations, the overall command layer decision model based on reinforcement learning algorithm outputs the k area patrol positions pos. t ={p i} i＝1,2…k ,in Each regional combat formation will proceed to its designated area and carry out patrol missions according to predetermined positions.

[0036] S3-2: Taking the interception zone planning task as an example, design the behavioral decision space of the regional combat layer. The combat area of ​​each regional formation is divided into n² different interception zones. For each incoming enemy target, the regional combat layer decision model based on reinforcement learning algorithm outputs the corresponding interception zone. The weapon platform will proceed to the designated interception zone to carry out the interception mission.

[0037] S3-3: Taking interception operations as an example, this invention designs the behavioral decision space for the weapon platform layer. For fighter jets, this invention constructs seven basic maneuvers as the behavioral decision space, including constant speed forward flight, deceleration forward flight, acceleration forward flight, left turn, right turn, climb, and dive.

[0038] Furthermore, the specific process of step S4 is as follows:

[0039] S4-1: The selection of reward function elements is crucial. For different combat missions, the reward function feature vector driving model learning should consist of different elements. Taking patrol area selection as an example, this invention selects the following basic features: detection coverage C1, enemy unit coverage C2, enemy threat area coverage C3, patrol area overlap rate R, and interception response time T. Based on the above basic features, the reward function feature vector f of the overall command layer is designed. t It can be represented as:

[0040] f t =<C1,C2,C3,R,T>

[0041] S4-2: Taking interception zone planning as an example, this invention designs the reward function for the regional combat layer. Considering the need to maximize the advantage over the enemy at the start of the interception operation during the decision-making process, the following basic features are selected in this invention: the interception target's line-of-sight angle (ATA), the interception target's entry angle (AOT), the relative distance (D), and the radial relative velocity (V). r Based on the above fundamental characteristics, the reward function feature vector f of the regional combat layer is designed. m It can be represented as:

[0042] f m = <ATA,AOT,D,V r >

[0043] S4-3: Taking interception operations as an example, the reward function of the weapon platform layer is designed. The following basic features are selected in this invention: remaining payload B. r Number of hits T1, number of hits T2. Based on the above basic characteristics, design the reward function feature vector f of the weapon platform layer. l It can be represented as:

[0044] f l = r ,T1,T2>

[0045] S4-4: Design the overall reward function for joint operations. This invention selects the following fundamental features: interceptor missile consumption M, AWACS aircraft damage rate D. w Bomber damage rate D b Fighter jet damage rate D a ​The interception rate P against enemy fighter aircraft. Based on the above fundamental characteristics, the feature vector f of the overall return function for joint operations is designed. c It can be represented as:

[0046] f c = <M,D w D b D a ,P>

[0047] S4-5: Set the weights of each element in the feature vector of the reward function for each layer to form the reward function. Taking the weapon platform layer as an example, its reward function r l It can be represented as:

[0048]

[0049] In the formula, This represents the weight vector of the weapon platform layer reward function, where the sum of its elements is 1.

[0050] Furthermore, the specific process of step S5 is as follows:

[0051] S5-1: Taking a fighter jet performing interception operations as an example, design the pre-training process of the behavior decision model at each layer. First, construct a sub-environment for pre-training the model, namely a one-to-one interception operation training environment.

[0052] S5-2: Initialize the battlefield situation and PPO neural network parameters, and initialize various hyperparameters such as learning rate lr, batch size, reward discount factor γ, truncation hyperparameter ∈, and number of training times per sample K.

[0053] S5-3: Based on the content in S2-3, extract the current state space information s of the PPO network from the battlefield situation. t ,in

[0054] s t = <LX r ,F r ,D_num r ,X rb ,Y rb Z rb LX b V b AOT, ATA>

[0055] S5-4: Obtain the current situation information s t Inputting the decision network of the PPO network yields the probability p of performing different actions in the current situation. t Randomly based on probability p t The action of the fighter jet was obtained by sampling. t Simultaneously, the situation information st The input state-value network yields the state value v. t .

[0056] S5-5: The fighter jet completes the maneuver. t Afterwards, a new battlefield situation was obtained. t and the round end marker d t And calculate the reward r obtained after the action in the previous frame was executed according to the reward function. t , will {s t ,a t ,logp t ,v t ,r t ,d t It is stored as a sample in the sample pool.

[0057] S5-6: If the number of samples in the sample pool is less than batch_size, return to S5-3; otherwise, execute S5-7.

[0058] S5-7: Extract all samples from the experience pool for training. First, calculate the sample advantage function:

[0059]

[0060] Where T represents the step number of the last sample in the training samples.

[0061] S5-8: Set state s t Input the target decision network and the target state value network, and output the target action probability p′. t and target state value v′ t And calculate the ratio ρ between the old and new strategies. t Compared to cutting off the old and new strategies

[0062]

[0063]

[0064] S5-9: Calculate the loss function L a (θ) is used for backpropagation and updating the target decision network and L. c (θ) Parameters of the target state value network:

[0065]

[0066]

[0067] S5-10: Repeat S5-8 to S5-9 K times, and update the policy network and state value network with the parameters of the target policy network and the target state value network.

[0068] S5-11: Determine if the air combat has ended. If so, execute S5-12; otherwise, return to S5-3.

[0069] S5-12: Determine if the conditions for stopping training are met. If yes, terminate the current process; otherwise, return to S5-3.

[0070] S5-13: After the algorithm training is completed, the pre-training of the behavioral decision-making model for fighter jet interception operations is finished. The resulting model can make autonomous decisions on interception operations. The same steps can be used to complete the pre-training of behavioral decision-making models at the overall command level and the regional operations level.

[0071] Furthermore, the specific process of step S6 is as follows:

[0072] S6-1: Initialize the complete battlefield situation, and initialize the HIRO neural network parameters with the pre-trained PPO neural network parameters, so that each decision layer has the initial ability to make behavioral decisions to complete its own combat mission.

[0073] S6-2: Optimize the training of the overall command level behavioral decision-making model. When training the overall command level, fix the neural network parameters of the area operations layer and the weapon platform layer, and use the joint operations overall reward function feature vector f from S4-4. c The reward function of the overall command level is modified to obtain a new reward function r′. t It can be represented as:

[0074]

[0075] Where, α t It is the global reward weight modified from the overall command level reward function. This is the global reward function weight vector, where the sum of its elements is 1. The original reward function r... t Replace with the new reward function r′ t Then, repeat S-3 to S-7N. t Next, N tests were conducted on the overall command level's behavioral decision-making model. t The training is optimized in iterations, where N t It sets the number of optimization iterations for the overall command level behavior decision-making model in a single overall iteration.

[0076] S6-3: Optimize the training model for the regional operational layer's behavioral decision-making. Fix the neural network parameters for the overall command layer and weapon platform layer, and refine the regional operational layer's behavioral decision-making model according to S6-2. m The training is optimized in iterations, where N m It sets the number of optimization iterations for the overall command level behavior decision-making model in a single overall iteration.

[0077] S6-4: Optimize the training weapon platform-level behavioral decision-making model. Fix the neural network parameters for the overall command level and the area operations level, and refine the weapon platform-level behavioral decision-making model according to S6-2. l The training is optimized in iterations, where N l It sets the number of optimization iterations for the overall command level behavior decision-making model in a single overall iteration.

[0078] S6-5: Repeat S6-2 to S6-4 until the update magnitude of the reward function of each layer is less than the given threshold, then end the policy optimization and output the optimized HIRO network parameters θ. * As parameters for the optimal hierarchical autonomous behavior decision-making model.

[0079] The beneficial effects of this invention are as follows:

[0080] (1) Compared with traditional force behavior decision modeling methods, this invention introduces reinforcement learning algorithms into joint operations behavior decision modeling, which can realize the self-evolution of command strategies and support exploration and strategy optimization in new combat scenarios without prior experience.

[0081] (2) Compared with using single agent reinforcement learning algorithm to model force behavior decision, this invention hierarchically divides the joint operation behavior decision model and decouples the state space and action space of each behavior decision layer, avoiding problems such as non-convergence of network training and low training efficiency caused by high-dimensional decision space in complex combat scenarios.

[0082] (3) Compared with training the hierarchical behavior decision model as a whole, this invention pre-trains each behavior decision layer and trains the joint operation behavior decision model as a whole based on the pre-trained model. This can avoid the policy non-stationarity problem caused by training behavior decision models of different decision layers at the same time, which helps to improve the training efficiency of the agent and makes the policy learning process more stable. Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Referring to the accompanying drawings will provide a clearer understanding of the features and advantages of the present invention. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. Wherein:

[0084] Figure 1 It is a hierarchical classification of joint air force operations.

[0085] Figure 2 It is a hierarchical behavioral decision-making model with inference time sequence relationships.

[0086] Figure 3 It is the overall process of hierarchical behavioral decision modeling.

[0087] Figure 4 It is the training process of a hierarchical behavioral decision-making model. Detailed Implementation

[0088] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. In the following description, in conjunction with... Figure 1 The overall framework diagram illustrates many specific details to provide a full understanding of the invention. However, the invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0089] Specifically, a hierarchical autonomous behavior decision-making model for joint air force operations based on HIRO and PPO algorithms includes the following steps:

[0090] S1: The joint operations behavior decision-making model is hierarchically divided from top to bottom as follows: Overall Command - Regional Operations - Weapon Platforms. The command layer is the high-level decision-making layer, responsible for formulating the overall operational plan and allocating deployment locations for regional operations; the formation layer is the mid-level decision-making layer, responsible for target allocation to weapon platforms; and the weapon platform layer is the low-level decision-making layer, specifically determining the operational and mission execution methods of each weapon platform. This ultimately forms a hierarchical autonomous behavior decision-making framework based on a hierarchical reinforcement learning algorithm.

[0091] S2: Based on the characteristics of the overall combat mission, feature vectors are extracted to construct the input state space of each layer of reinforcement learning algorithms, so as to effectively represent the combat situation of concern at each layer of formation cooperative combat.

[0092] S3: Based on the operational characteristics and tasks of different decision-making levels in the overall operational mission, design different behavioral decision-making spaces for each decision-making level in the coordinated operations of the formation.

[0093] S4: Extract feature vectors and design reward functions for reinforcement learning algorithms for combat objectives at different decision levels.

[0094] S5: Based on the PPO algorithm, the reinforcement learning state space, action space and reward function design mentioned above are integrated to pre-train each decision layer.

[0095] S6: Based on the HIRO algorithm and the behavioral decision-making models of each decision layer obtained from the pre-training in step 5, the reinforcement learning state space, action space and reward function design designed above are integrated, and the reward function of each decision layer is modified based on the overall reward function of joint operations, forming a complete operation process for iterative optimization training of the joint operations behavioral decision-making model.

[0096] Furthermore, the specific process of step S1 is as follows:

[0097] S1-1: Addressing the multi-layered nature of coordinated operations, joint operational decision-making is hierarchically divided into three levels: overall command, regional operations, and weapon platforms. This results in a model as follows: Figure 1 The hierarchical behavioral decision-making framework shown.

[0098] S1-2: The overall command employs reinforcement learning algorithms to make decisions regarding area selection during regional operations. When a regional combat formation, including but not limited to early warning aircraft, fighter jets, and bombers, receives operational instructions from the Red Force command center, the formation will proceed to the corresponding area to conduct patrol missions and prepare for subsequent combat missions.

[0099] S1-3: The regional combat layer employs a behavioral model that integrates reinforcement learning algorithms and traditional rule sets for decision-making, including task allocation and target selection. Taking fighter jet formations as an example, a rule set is used for target allocation decisions, while a reinforcement learning algorithm is used to select the interception zone, guiding the fighter jets to the designated area to perform interception missions. The target allocation rule set is shown in Table 1.

[0100] Table 3. Set of Target Allocation Rules for Fighter Aircraft Formations

[0101]

[0102]

[0103] S1-4: The weapon platform layer employs a behavioral model that integrates reinforcement learning algorithms and traditional rule sets to make decisions regarding the combat behavior of combat aircraft. Taking fighter jets as an example, rule sets are used to select combat behaviors, as shown in Table 2. When a fighter jet performs an interception mission, a reinforcement learning algorithm is used to plan its flight path.

[0104] Table 4. Fighter Aircraft Combat Rules Set

[0105] 1 Insufficient fuel Return 2 Sufficient fuel and regional operational layer allocation of interception targets Execute interception task 3 Sufficient fuel and no interception targets assigned to the area combat layer Patrol the designated area

[0106] Furthermore, the specific process of step S2 is as follows:

[0107] S2-1: In joint operations, the battlefield scope, tactical targets, situational elements, content, and information precision that are of concern to personnel at different levels vary. Therefore, the operational situation that different command levels need to acquire differs. The overall command level acquires overall battlefield situational information, which may include command and control area map G. c Air attack range diagram G aa Air detection range map G ad Maritime attack range map G saMap G of the sea exploration range sd These situational diagrams, together, constitute the state input space for the overall command-level decision-making model, for example:

[0108] S t = <G c G aa G ad G sa G sd >

[0109] S2-2: The situational awareness acquired at the regional operational layer is regional situational information. Taking a regional collaborative combat system that includes, but is not limited to, fighter formations, bomber formations, and early warning aircraft as an example, the three-dimensional coordinates (X, Y, Z) of all friendly aircraft within the region can be selected. r ,Y r Z r ), Troop Type LX r Heading H r Speed ​​V r Ammunition capacity D_num r The three-dimensional coordinates of the enemy's incoming target (X) b ,Y b Z b ), Troop Type LX b Heading H b Speed ​​V b These, together, constitute the state input space for the regional operational layer decision-making model, for example:

[0110] S m = <X r ,Y r Z r LX r H r ,D_num r V r ,I r ,X b ,Y b Z b LX b H b V b >

[0111] S2-3: The situational awareness acquired at the weapon platform level is the local situational awareness when performing a single-aircraft mission. Taking a fighter jet performing an interception mission as an example, the force type LX can be selected. r Current fuel quantity F r Current remaining ammunition quantity D_num r The relative position of the intercepted target (X) rb ,Y rb Z rb ), Troop Type LXb Speed ​​V b Angle of entry (AOT), line-of-sight (ATA), etc., together constitute the state input space for the weapon platform-level decision-making model, for example:

[0112] S l = <LX r ,F r ,D_num r ,X rb ,Y rb Z rb LX b V b AOT, ATA>

[0113] Furthermore, the specific process of step S3 is as follows:

[0114] S3-1: Taking area patrol operations as an example, design the behavioral decision space of the overall command layer. Divide the battlefield into n1 different patrol areas. Assuming there are k area combat formations, the overall command layer decision model based on reinforcement learning algorithm outputs the k area patrol positions pos. t ={p i} i＝1,2…k ,in Each regional combat formation will proceed to its designated area and carry out patrol missions according to predetermined positions.

[0115] S3-2: Taking the interception zone planning task as an example, design the behavioral decision space of the regional combat layer. The combat area of ​​each regional formation is divided into n² different interception zones. For each incoming enemy target, the regional combat layer decision model based on reinforcement learning algorithm outputs the corresponding interception zone. The weapon platform will proceed to the designated interception zone to carry out the interception mission.

[0116] S3-3: Taking interception operations as an example, this invention designs the behavioral decision space for the weapon platform layer. For fighter jets, this invention constructs seven basic maneuvers as the behavioral decision space, including constant speed forward flight, deceleration forward flight, acceleration forward flight, left turn, right turn, climb, and dive.

[0117] Furthermore, the specific process of step S4 is as follows:

[0118] S4-1: The selection of reward function elements is crucial. For different combat missions, the reward function feature vector driving model learning should consist of different elements. Taking patrol area selection as an example, this invention selects the following basic features: detection coverage C1, enemy unit coverage C2, enemy threat area coverage C3, patrol area overlap rate R, and interception response time T. Based on the above basic features, the reward function feature vector f of the overall command layer is designed. t It can be represented as:

[0119] f t =<C1,C2,C3,R,T>

[0120] S4-2: Taking interception zone planning as an example, this invention designs the reward function for the regional combat layer. Considering the need to maximize the advantage over the enemy at the start of the interception operation during the decision-making process, the following basic features are selected in this invention: the interception target's line-of-sight angle (ATA), the interception target's entry angle (AOT), the relative distance (D), and the radial relative velocity (V). r Based on the above fundamental characteristics, the reward function feature vector f of the regional combat layer is designed. m It can be represented as:

[0121] f m = <ATA,AOT,D,V r >

[0122] S4-3: Taking interception operations as an example, the reward function of the weapon platform layer is designed. The following basic features are selected in this invention: remaining payload B. r Number of hits T1, number of hits T2. Based on the above basic characteristics, design the reward function feature vector f of the weapon platform layer. l It can be represented as:

[0123] f l = r ,T1,T2>

[0124] S4-4: Design the overall reward function for joint operations. This invention selects the following fundamental features: interceptor missile consumption M, AWACS aircraft damage rate D. w Bomber damage rate D b Fighter jet damage rate D a The interception rate P against enemy fighter aircraft. Based on the above fundamental characteristics, the feature vector f of the overall return function for joint operations is designed. c It can be represented as:

[0125] f c = <M,D w D b D a ,P>

[0126] ​S4-5: Set the weights of each element in the feature vector of the reward function for each layer to form the reward function. Taking the weapon platform layer as an example, its reward function r l It can be represented as:

[0127]

[0128] In the formula, This represents the weight vector of the weapon platform layer reward function, where the sum of its elements is 1.

[0129] Furthermore, the specific process of step S5 is as follows:

[0130] S5-1: Taking a fighter jet performing interception operations as an example, design the pre-training process of the behavior decision model at each layer. First, construct a sub-environment for pre-training the model, namely a one-to-one interception operation training environment.

[0131] S5-2: Initialize the battlefield situation and PPO neural network parameters, and initialize various hyperparameters such as learning rate lr, batch size, reward discount factor γ, truncation hyperparameter ∈, and number of training times per sample K.

[0132] S5-3: Based on the content in S2-3, extract the current state space information s of the PPO network from the battlefield situation. t ,in

[0133] s t = <LX r ,F r ,D_num r ,X rb ,Y rb Z rb LX b V b AOT, ATA>

[0134] S5-4: Obtain the current situation information s t Inputting the decision network of the PPO network yields the probability p of performing different actions in the current situation. t Randomly based on probability p t The action of the fighter jet was obtained by sampling. t Simultaneously, the situation information s t The input state-value network yields the state value v. t .

[0135] S5-5: The fighter jet completes the maneuver. t Afterwards, a new battlefield situation was obtained. t and the round end marker d t And calculate the reward r obtained after the action in the previous frame was executed according to the reward function.t , will {s t ,a t ,logp t ,v t ,r t ,d t It is stored as a sample in the sample pool.

[0136] S5-6: If the number of samples in the sample pool is less than batch_size, return to S5-3; otherwise, execute S5-7.

[0137] S5-7: Extract all samples from the experience pool for training. First, calculate the sample advantage function:

[0138]

[0139] Where T represents the step number of the last sample in the training samples.

[0140] S5-8: Set state s t Input the target decision network and the target state value network, and output the target action probability p′. t and target state value v′ t And calculate the ratio ρ between the old and new strategies. t Compared to cutting off the old and new strategies

[0141]

[0142]

[0143] S5-9: Calculate the loss function L a (θ) is used for backpropagation and updating the target decision network and L. c (θ) Parameters of the target state value network:

[0144]

[0145]

[0146] S5-10: Repeat S5-8 to S5-9 K times, and update the policy network and state value network with the parameters of the target policy network and the target state value network.

[0147] S5-11: Determine if the air combat has ended. If so, execute S5-12; otherwise, return to S5-3.

[0148] S5-12: Determine if the conditions for stopping training are met. If yes, terminate the current process; otherwise, return to S5-3.

[0149] S5-13: After the algorithm training is completed, the pre-training of the behavioral decision-making model for fighter jet interception operations is finished. The resulting model can make autonomous decisions on interception operations. The same steps can be used to complete the pre-training of behavioral decision-making models at the overall command level and the regional operations level.

[0150] Furthermore, the specific process of step S6 is as follows:

[0151] S6-1: Initialize the complete battlefield situation, and initialize the HIRO neural network parameters with the pre-trained PPO neural network parameters, so that each decision layer has the initial ability to make behavioral decisions to complete its own combat mission.

[0152] S6-2: Optimize the training of the overall command level behavioral decision-making model. When training the overall command level, fix the neural network parameters of the area operations layer and the weapon platform layer, and use the joint operations overall reward function feature vector f from S4-4. c The reward function of the overall command level is modified to obtain a new reward function r′. t It can be represented as:

[0153]

[0154] Where, α t It is the global reward weight modified from the overall command level reward function. This is the global reward function weight vector, where the sum of its elements is 1. The original reward function r... t Replace with the new reward function r′ t Then, repeat S-3 to S-7N. t Next, N tests were conducted on the overall command level's behavioral decision-making model. t The training is optimized in iterations, where N t It sets the number of optimization iterations for the overall command level behavior decision-making model in a single overall iteration.

[0155] S6-3: Optimize the training model for the regional operational layer's behavioral decision-making. Fix the neural network parameters for the overall command layer and weapon platform layer, and refine the regional operational layer's behavioral decision-making model according to S6-2. m The training is optimized in iterations, where N m It sets the number of optimization iterations for the overall command level behavior decision-making model in a single overall iteration.

[0156] S6-4: Optimize the training weapon platform-level behavioral decision-making model. Fix the neural network parameters for the overall command level and the area operations level, and refine the weapon platform-level behavioral decision-making model according to S6-2. l The training is optimized in iterations, where N l It sets the number of optimization iterations for the overall command level behavior decision-making model in a single overall iteration.

[0157] S6-5: Repeat S6-2 to S6-4 until the update magnitude of the reward function of each layer is less than the given threshold, then end the policy optimization and output the optimized HIRO network parameters θ. * As parameters for the optimal hierarchical autonomous behavior decision-making model.

[0158] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hierarchical autonomous behavior decision-making modeling method for air force based on PPO and HIRO algorithms, characterized in that: The method specifically includes the following steps: S1: The joint operations behavior decision-making model is divided into layers from top to bottom: overall command - regional operations - weapon platforms. The command layer is the high-level decision-making layer, responsible for formulating the overall operational plan and allocating the deployment positions for regional operations; the formation layer is the mid-level decision-making layer, which allocates targets to weapon platforms; the weapon platform layer is the bottom-level decision-making layer, which specifically determines the operation and mission execution methods of each weapon platform, ultimately forming a hierarchical autonomous behavior decision-making framework based on hierarchical reinforcement learning algorithms. S2: Based on the characteristics of the overall combat mission, feature vectors are extracted to construct the input state space of each layer of reinforcement learning algorithms, so as to effectively represent the combat situation of concern at each layer of formation cooperative combat; S3: Based on the operational characteristics and tasks of different decision-making levels in the overall operational mission, design different behavioral decision-making spaces for each decision-making level in the coordinated operations of the formation; S4: Extract feature vectors and design reward functions for reinforcement learning algorithms for combat objectives at different decision levels. S5: Based on the PPO algorithm, the reinforcement learning state space, action space and reward function design mentioned above are integrated to pre-train each decision layer; S6: Based on the HIRO algorithm and the behavior decision models of each decision layer obtained by pre-training in step 5, the reinforcement learning state space, action space and reward function design designed above are integrated, and the reward function of each decision layer is modified based on the overall reward function of joint operations to form a complete operation process for iterative optimization training of the joint operations behavior decision model. The specific process of S5 is as follows: S5-1: Taking a fighter jet performing interception operations as an example, design the pre-training process of the behavior decision model at each layer. First, construct a sub-environment for pre-training the model, namely a one-to-one interception operation training environment. S5-2: Initialize the battlefield situation and PPO neural network parameters, and initialize various hyperparameters, including the learning rate. Single training sample size Return Discount Factor truncation of hyperparameters Number of training sessions per sample ; S5-3: Based on the content in S2-3, extract the current state space information of the PPO network from the battlefield situation. , in, As for troop type, This is the current oil level. This represents the current remaining ammunition. To intercept the relative position of the target, As for troop type, For speed, For the entry angle, The viewing angle; S5-4: Obtain the current situation information By inputting the decision network of the PPO network, we obtain the probabilities of performing different actions in the current situation. Randomly based on probability The action of the fighter jet was obtained by sampling. Simultaneously, situation information will be provided. Input state value network to obtain state value ; S5-5: The fighter jet completes the maneuver. Afterwards, a new battlefield situation was obtained. and the end-of-round marker And calculate the reward obtained after the action in the previous frame was executed based on the reward function. ,Will It is stored as a sample in the sample pool; S5-6: If the number of samples in the sample pool is less than Return to S5-3; otherwise, execute S5-7. S5-7: Extract all samples from the experience pool for training. First, calculate the sample advantage function: in, This indicates the step number of the last sample in the training dataset; S5-8: Change the status Input the target decision network and the target state value network, and output the target action probability. and target state value And calculate the ratio of the old strategy to the new strategy. Compared to cutting off the old and new strategies : S5-9: Calculate the loss function Used for backpropagation and updating the target decision network and Parameters of the target state value network: S5-10: Repeat Then from S5-8 to S5-9, the policy network and state value network are updated with the parameters of the target policy network and the target state value network; S5-11: Determine if the air combat has ended. If so, execute S5-12; otherwise, return to S5-3. S5-12: Determine whether the conditions for stopping training are met. If yes, terminate the current process; otherwise, return to S5-3. S5-13: After the algorithm training is completed, the pre-training of the behavioral decision-making model for interception operations of fighter jets is completed. The obtained model can make autonomous decisions on interception operations. The same steps can be used to complete the pre-training of the behavioral decision-making model at the overall command level and the regional combat level. The specific process of S6 is as follows: S6-1: Initialize the complete battlefield situation, initialize the HIRO neural network parameters with the pre-trained PPO neural network parameters, so that each decision layer has the initial ability to make behavioral decisions to complete its own combat mission. S6-2: Optimize the training of the overall command level behavior decision-making model. When training the overall command level, fix the neural network parameters of the area operations layer and the weapon platform layer, and use the joint operations overall reward function feature vector from S4-4. The reward function of the overall command level is modified to obtain a new reward function. It can be represented as: in, It is the global reward weight modified from the overall command level reward function. This is the global reward function weight vector, and the sum of its elements is... , The feature vector of the overall reward function for joint operations; the original reward function Replace with a new reward function Then, repeat S-3 to S-7. Next, the behavioral decision-making model of the overall command level was conducted. The training is optimized in the next iteration, where It sets the number of optimization iterations for the overall command-level behavior decision-making model in a single overall iteration; S6-3: Optimize the training area operational layer behavioral decision-making model, fix the neural network parameters of the overall command layer and weapon platform layer, and refer to S6-2 to refine the area operational layer behavioral decision-making model. The training is optimized in the next iteration, where It sets the number of optimization iterations for the overall command-level behavior decision-making model in a single overall iteration; S6-4: Optimize the training weapon platform-level behavior decision-making model, fix the neural network parameters of the overall command level and the area combat level, and refer to S6-2 to refine the weapon platform-level behavior decision-making model. The training is optimized in the next iteration, where It sets the number of optimization iterations for the overall command-level behavior decision-making model in a single overall iteration; S6-5: Repeat S6-2 to S6-4 until the update magnitude of the reward function of each layer is less than the given threshold, then end the policy optimization and output the optimized HIRO network parameters. As parameters for the optimal hierarchical autonomous behavior decision-making model.

2. The method according to claim 1, characterized in that, The specific process of S1 is as follows: S1-1: In view of the multi-layered nature of formation-coordinated operations, the joint operations behavior decision-making is divided into layers, and the behavior decision-making model is divided into three layers: overall command, regional operations and weapon platform, forming a hierarchical behavior decision-making framework as shown in Figure 1. S1-2: The overall command uses reinforcement learning algorithms to make decisions on the selection of areas during regional operations. When a regional combat formation, including but not limited to early warning aircraft, fighter jets, and bombers, receives combat instructions from the Red Force command center, the formation will go to the corresponding area to carry out patrol missions and prepare for subsequent combat missions. S1-3: The regional operations layer uses a behavioral model that integrates reinforcement learning algorithms and traditional rule sets for decision-making, including at least task allocation and target selection; Taking fighter jet formation as an example, a set of rules is used to make target allocation decisions, and a reinforcement learning algorithm is used to select the interception zone and guide the fighter jets to the designated area to perform the interception mission. The target allocation rule set is shown in Table 1. Table 1. Set of Target Allocation Rules for Fighter Aircraft Formations S1-4: The weapon platform layer adopts a behavior model that integrates reinforcement learning algorithms and traditional rule sets to make decisions on the combat behavior of combat aircraft. Taking the fighter jet as an example, the rule set is used to select combat behavior, as shown in Table 2. When the fighter jet performs an interception mission, the reinforcement learning algorithm is used to plan the route for the fighter jet. Table 2. Fighter Aircraft Combat Rules Set 。 3. The method according to claim 1, characterized in that, The specific process of S2 is as follows: S2-1: In joint operations, the battlefield scope, tactical targets, situational elements and content, and information accuracy that combat personnel at different levels are concerned with vary. Therefore, the operational situation that different command levels need to obtain is different. The overall command level obtains information on the overall battlefield situation, and can select multiple situation maps, which must include at least: a command and control area map. Air attack range diagram Air detection range map Maritime attack range map Map of the sea exploration range Multiple situational diagrams together constitute the state input space for the overall command level decision-making model, which is represented as follows: S2-2: The situational awareness acquired at the regional operational layer is regional situational information. Taking a regional collaborative combat system that includes, but is not limited to, fighter formations, bomber formations, and early warning aircraft as an example, multiple regional situational information can be selected. These multiple regional situational information must include at least the three-dimensional coordinates of all friendly aircraft within the region. Types of troops ,course ,speed Ammunition load 3D coordinates of the enemy's incoming target Types of troops ,course ,speed Multiple regional situational information collectively constitute the state input space for the regional operational layer decision-making model, which is represented as: S2-3: The situational awareness acquired at the weapon platform level is a local situational awareness during a single-aircraft mission. Taking a fighter jet performing an interception mission as an example, the selected local situational awareness data should include at least the following information: force type. Current fuel level Current remaining ammunition The relative position of the intercepted target Types of troops ,speed , entrance angle ( ), line of sight ( Multiple local situations together constitute the state input space for the weapon platform-level decision-making model, which is represented as follows: 。 4. The method according to claim 1, characterized in that, The specific process of S3 is as follows: S3-1: Taking regional patrol operations as an example, design the behavioral decision-making space of the overall command level, dividing the battlefield into... There are several different patrol areas, assuming there are... For each regional operational formation, the output of the overall command decision-making model based on reinforcement learning algorithms is... Patrol locations in each area ,in Each regional combat formation will proceed to its designated area and carry out patrol missions according to its predetermined positions; S3-2: Taking the interception zone planning mission as an example, design the behavioral decision space of the regional combat layer, dividing the combat area of ​​each regional formation into... For each incoming enemy target, a regional combat layer decision model based on reinforcement learning algorithms outputs a corresponding interception zone, representing a different interception zone. The weapon platform will proceed to the designated interception zone to carry out the interception mission; S3-3: Taking interception operations as an example, the behavior decision space of the weapon platform layer is designed. For fighter jets, seven basic maneuvers are constructed as the behavior decision space, including uniform forward flight, deceleration forward flight, acceleration forward flight, left turn, right turn, climb and dive.

5. The method according to claim 1, characterized in that, The specific process of S4 is as follows: S4-1: The selection of reward function elements is crucial. For different combat missions, the feature vector of the reward function driving model learning should consist of different elements. For patrol area selection, the following basic features were chosen: detection coverage. Enemy unit coverage Coverage of enemy-threatened areas Patrol area overlap rate Interception response time Based on the above fundamental characteristics, the reward function feature vector of the overall command layer is designed. It can be represented as: S4-2: Taking interception zone planning as an example, design the reward function for the regional combat layer; considering the need to maximize the advantage over the enemy at the start of the interception operation during the decision-making process, the following basic features are selected: the line-of-sight angle of the intercepted target. Intercepting the target's entry angle Relative distance Radial relative velocity Based on the above fundamental characteristics, the reward function feature vector for the regional combat layer is designed. It can be represented as: S4-3: Taking interception operations as an example, the reward function of the weapon platform layer was designed, selecting the following basic features: remaining payload. Number of hits Number of times hit Based on the above fundamental characteristics, the reward function feature vector of the weapon platform layer is designed. It can be represented as: S4-4: Design the overall reward function for joint operations, selecting the following basic characteristics: interceptor missile consumption. Early warning aircraft damage rate Bomber damage rate Fighter jet damage rate Interception rate against enemy fighter aircraft Based on the above fundamental characteristics, design the feature vector of the overall return function for joint operations. It can be represented as: S4-5: Set the weights of each element in the feature vector of the reward function for each layer to form the reward function; taking the weapon platform layer as an example, its reward function... It can be represented as: In the formula, This represents the weight vector of the weapon platform layer reward function, where the sum of its elements is... .

Images