Air combat deep reinforcement learning hierarchical algorithm based on time scale separation theory

Abstract

The invention discloses an air combat deep reinforcement learning hierarchical algorithm based on a time scale separation theory, which combines long-period decision and short-period control, and realizes unmanned aerial vehicle autonomous air combat maneuver decision optimization through a two-stage training strategy. Firstly, the state quantity of the unmanned aerial vehicle is separated into a long-period state quantity and a short-period state quantity according to the time scale, the long-period state quantity comprises the position and the relative speed, and the short-period state quantity comprises the roll angle speed, the pitch angle speed, the yaw angle speed and the like. The long-period decision is optimized through a PPO algorithm, and expected values of short-period control, such as an expected roll angle, a pitch angle and a speed, are output; and the short-period control realizes accurate adjustment of the attitude of the unmanned aerial vehicle through a PID controller. The algorithm adopts two-stage training, in the first stage, convergence is accelerated through a single-step reward strategy, and in the second stage, combat performance is optimized through a final reward. The maneuverability and autonomous decision-making ability of the unmanned aerial vehicle in a complex air combat environment are improved, and the method has the characteristics of quick response and efficient learning and is widely applied to autonomous air combat tasks of the unmanned aerial vehicle.

Description

Technical Field

[0001] The patent of this invention relates to the field of autonomous control and decision-making technology in the field of artificial intelligence and aerospace, and specifically, to a hierarchical algorithm for deep reinforcement learning in air combat based on time scale separation theory. Background Art

[0002] In modern air combat scenarios, the development of automated drone technology has brought about tremendous changes in military operations. Drones have become an important role in air combat due to their high speed, flexibility and versatility. Especially in complex air combat environments, drones need to have autonomous maneuvering decision-making capabilities to cope with dynamically changing tactical environments, enemy threats and multi-dimensional maneuvering needs. This places higher demands on the intelligent decision-making algorithms of drones. Existing drone control technologies mostly rely on traditional flight controllers, which mainly achieve flight operations through pre-set paths and fixed maneuvers. However, in real-time adversarial air combat scenarios, this preset strategy is often difficult to cope with complex and changeable air combat situations, so it is particularly important to study more flexible and intelligent decision-making systems.

[0003] With the rapid development of deep reinforcement learning (DRL) technology, it has demonstrated excellent autonomous learning and decision-making capabilities in fields such as games, robot control, and autonomous driving. Deep reinforcement learning can continuously optimize strategies through interaction with the environment, enabling the agent to achieve optimal performance in a given task. In the field of drone air combat, deep reinforcement learning also shows great application potential. Through reinforcement learning algorithms, drones can autonomously learn complex air combat maneuvering strategies from historical confrontation data, thereby improving their flexibility and response capabilities when facing enemy aircraft.

[0004] However, traditional deep reinforcement learning algorithms also face significant challenges in drone air combat applications. First, the six-degree-of-freedom model of drones has a complex multi-dimensional state space and a continuous action space, which leads to slow convergence of reinforcement learning algorithms in such high-dimensional environments and makes it difficult to quickly learn effective strategies. Second, air combat scenarios are highly dynamic, with extremely fast state changes, and traditional reinforcement learning algorithms have difficulty responding to these rapid changes in a timely manner. Third, existing reinforcement learning methods often rely on empirically designed reward functions, which limits the generalization ability of the algorithm in air combat environments and makes it difficult to adapt to different air combat mission scenarios. In addition, during air combat, drones also need to deal with different time scale issues: for example, the change cycle of variables such as attitude angular velocity is very short, while the change cycle of states such as position and speed is longer. How to reasonably distinguish these two types of state quantities in the algorithm to achieve more efficient learning and control is another key issue.

[0005] In order to meet the above challenges, a hierarchical algorithm for deep reinforcement learning in air combat based on the theory of time scale separation came into being. The algorithm divides the state quantity of the UAV into two parts, long-period and short-period, and effectively reduces the complexity of the state space by utilizing the difference in time scales. Specifically, the short-period state quantity includes attitude parameters such as roll angular velocity, pitch angular velocity, and yaw angular velocity, while the long-period state quantity includes position, relative speed, etc. Through this division, the algorithm can perform hierarchical processing on state quantities of different time scales, further improving the decision-making efficiency and response speed of the UAV in air combat.

[0006] In long-term decision-making, a deep reinforcement learning strategy based on the proximal policy optimization (PPO) algorithm is used. This strategy optimizes the trajectory planning of the drone by continuously interacting with the air combat environment, enabling it to make strategic decisions on a longer time scale. In short-term control, a proportional-integral-differential (PID) controller is used, combined with the expected value output of the PPO algorithm, to adjust the attitude and speed of the drone in real time to achieve refined maneuver control. Through the combination of long and short cycles, the algorithm can not only effectively respond to rapidly changing tactical scenarios, but also perform strategic planning on a longer time scale.

[0007] In addition, the algorithm also introduces a two-stage training strategy. In the first stage, a dense single-step reward function is designed to speed up the initial convergence speed and help the intelligent agent quickly learn basic maneuvers. In the second stage, the single-step reward is removed, only the final sparse reward is retained, and the frequency of long-term decisions is increased, thereby optimizing the drone's air combat strategy and improving the final combat performance. This two-stage training method overcomes the problem of traditional reinforcement learning's over-reliance on single-step reward functions, and effectively improves the generalization ability of drones in different air combat scenarios.

[0008] Compared with the existing reinforcement learning methods, the hierarchical algorithm based on the time scale separation theory effectively improves the autonomous decision-making ability of drones in air combat by reasonably dividing the state quantity, introducing the combination of PPO and PID controller and two-stage training strategy. Experiments show that the drone agent using this algorithm can achieve faster strategy learning speed in complex air combat scenarios, and its combat performance against different types of enemy aircraft is significantly better than traditional reinforcement learning methods. This innovation provides a new solution for autonomous maneuvering decision-making of drones in the field of air combat, which has broad application prospects.

[0009] In summary, the deep reinforcement learning hierarchical algorithm based on the time scale separation theory not only effectively improves the maneuverability and intelligence of UAVs in autonomous air combat by hierarchically processing long and short cycle state quantities, introducing advanced strategy optimization and control algorithms, and two-stage training strategies, but also provides more efficient decision-making support for UAVs in complex and changeable combat environments. This lays the foundation for the future autonomous air combat decision-making system of UAVs and shows broad development prospects. Summary of the invention

[0010] The purpose of the present invention is to solve the problems raised in the background technology, and further proposes an air combat deep reinforcement learning hierarchical algorithm based on time scale separation theory.

[0011] The present invention is implemented in this way: a hierarchical algorithm for deep reinforcement learning of air combat based on time scale separation theory is characterized in that: the algorithm combines long-term decision-making with short-term control, and realizes the optimization of autonomous air combat maneuvering decision of UAV through a two-stage training strategy, which specifically includes the following steps:

[0012] Based on the principle of time scale separation, the state quantity of the drone is divided into long-period state quantity and short-period state quantity. The long-period state quantity includes position (x, y, z), relative speed (v x , v y , v z ), the short-period state quantities include roll angular velocity (p), pitch angular velocity (q), yaw angular velocity (r), pitch angle (θ), roll angle (φ) and velocity (V) to achieve dimensionality reduction of air combat state space;

[0013] Long-term decisions are optimized through the PPO (Proximal Policy Optimization) algorithm. The long-term strategy outputs the expected values ​​required for short-term decisions, including the expected roll angle, pitch angle, sideslip angle and speed.

[0014] The attitude and speed of the drone are adjusted through the PID controller in a short period to achieve precise control of the expected values;

[0015] Two-stage training: The first stage adopts a single-step reward strategy, and the second stage removes the single-step reward, retaining only the final sparse reward and increasing the decision frequency of the long-term strategy.

[0016] As a preferred technical solution of the present invention: the short-cycle control uses a PID controller to perform posture and speed control, and the output control variable of the PID controller is:

[0017]

[0018] Among them, K p , K i , Kd Represent the gain parameters of proportional, integral and differential controllers respectively, proportional control part: K p e(t), where K p is the proportional gain parameter, which indicates the influence of the error signal on the control output; e(t) is the error between the target value and the actual output;

[0019] Integral control part: Where K i is the integral gain parameter, which indicates the influence of the error integral on the control output;

[0020] Differential control part: Where K d is the differential gain parameter, which indicates the influence of the error change rate on the control output;

[0021] As a preferred technical solution of the present invention: Roll angle controller: used to control the rotation of the drone around the longitudinal axis. When the drone turns on the horizontal plane, it is necessary to adjust its roll angle φ to achieve the purpose of turning. Therefore, the roll angle controller needs to have the ability of fast response and minimum overshoot. The controller receives the desired roll angle φ c As input, and generate the corresponding δ a The command makes the drone gradually reach the desired roll angle. The inner loop of the roll angle controller uses the roll angle velocity p, and the outer loop uses the roll angle φ feedback. The whole control process is as follows: Figure 3 (a), its mathematical expression is as follows:

[0022] δ a =-[k φ (φ c -φ)]-k p p

[0023] Pitch angle controller: When controlling the longitudinal flight of the drone, that is, changing its altitude or speed, the pitch angle of the drone needs to be adjusted through the pitch angle control system. The controller receives the desired pitch angle θ c As input, and generate the corresponding δ e The command makes the drone gradually reach the desired pitch angle. The inner loop of the pitch angle controller uses the pitch angle velocity q, and the outer loop uses the pitch angle θ feedback. The whole control process is as follows: Figure 3 (b).

[0024] δ e =-[k θ (θ c -θ)]-k q q

[0025] Side slip angle controller: Since the UAV will generate a sideslip angle β during the turning process, it is necessary to eliminate the sideslip angle by manipulating the rudder to maintain the stability of the UAV. The controller receives the expected sideslip angle β c As input, the corresponding δ is output r The inner loop of the sideslip angle controller uses the yaw angle velocity r, and the outer loop uses the sideslip angle feedback. Considering the problem of steady-state error in the sideslip angle control, the proportional integral control method is adopted. The whole control process is as follows Figure 3 (c), whose mathematical expression is as follows:

[0026]

[0027] Throttle speed controller: adjust the throttle lever delta τ The power output of the drone can be changed to adjust the speed of the drone. By working with the pitch angle control system, the speed of the drone can be controlled. The throttle speed controller receives the expected value V of the speed. c As input, and output the corresponding δ τ Instructions are given to adjust the speed of the drone to achieve the desired goal. The feedback of the control system is speed V. Proportional integral control is used to solve the steady-state error problem. The entire control process is as follows: Figure 3 (d), which is mathematically expressed as follows:

[0028]

[0029] As the preferred technical solution of the present invention: Long-term decision-making adopts a deep reinforcement learning strategy based on the PPO algorithm, whose goal is to maximize the long-term cumulative reward To optimize the strategy, where:

[0030]

[0031] γ is the discount factor, r k is the reward obtained at time k, and the objective function of the PPO algorithm is:

[0032]

[0033] in, is the advantage function estimate, ∈ is the clipping coefficient, which is used to limit the amplitude of the policy update.

[0034] As a preferred technical solution of the present invention: the long-period state quantity is transformed into a relative state through the following coordinate system transformation, and the transformation matrix M is defined as:

[0035]

[0036] in, Used to convert state quantities to a new coordinate system.

[0037] As a preferred technical solution of the present invention: the algorithm is based on a time scale separation architecture of long cycle and short cycle, the frequency of long cycle decision is 0.5Hz, the frequency of short cycle control is 10Hz, the short cycle control receives the expected value of the long cycle strategy output, and realizes real-time posture adjustment through PID control.

[0038] As a preferred technical solution of the present invention: the reward function includes the following parts: Relative position reward: reward our drone for being behind the enemy and facing the enemy, the formula is:

[0039] Reward position =k1cos(θ1)+k2cos(θ2)

[0040] Among them, θ1 is the angle between our speed and the enemy's relative position, and θ2 is the angle between the enemy's speed and our relative position;

[0041] Distance bonus: Encourages the drone to get close to the enemy target. The formula is:

[0042]

[0043] Where r is the distance between the two drones.

[0044] As a preferred technical solution of the present invention: the strike reward is designed to give a reward when our drone enters the enemy's attack range and attacks. The specific calculation is:

[0045]

[0046] Among them, r is the distance between drones, ω is the attack angle, and ω0 is the threshold of the attack angle.

[0047] As a preferred technical solution of the present invention: a two-stage strategy training is adopted during the training process, wherein:

[0048] Phase 1: The drone’s air combat behavior is guided by designing a single-step reward function. In this phase, the PPO network’s update frequency is 0.5 Hz, and the dense reward function is used to accelerate convergence. Specific rewards include relative position rewards, distance rewards, and strike rewards.

[0049] Phase 2: Remove single-step rewards, retain only final rewards, and increase the decision frequency of the PPO network to 2Hz to improve the decision-making efficiency and combat capability of the agent.

[0050] As a preferred technical solution of the present invention: the final reward function is used to evaluate the status at the end of the air combat mission, including loss of altitude control, exceeding the distance limit, timeout, victory or failure. The specific formula is:

[0051]

[0052] Among them: r1 represents the penalty given when the drone altitude is not within the range of [1000m, 10000m];

[0053] r2 indicates that when the distance of the drone exceeds the set maximum distance max the punishment given when

[0054] r3 indicates that when the task time exceeds the maximum allowed time end Rewards given when

[0055] r4 represents the reward given when the drone wins an air battle;

[0056] r5 represents the penalty given when the drone fails in an air combat.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] Improve decision-making efficiency and accuracy: This invention separates the long-term and short-term state quantities of the UAV by time scale separation, uses the PPO algorithm for long-term trajectory planning, and uses the PID controller for short-term attitude control, effectively reducing the complexity of the state space and improving the efficiency of strategy learning and control accuracy. The layered processing method ensures the real-time response capability of the UAV in air combat, especially in the face of complex dynamic environments, so that it can make fast and accurate decisions.

[0059] Enhanced strategy generalization ability: Through a two-stage training strategy, the agent accelerates initial convergence through dense single-step rewards in the first stage, and further optimizes the strategy through sparse final rewards in the second stage, avoiding the limitations of single-step rewards on strategy learning. This design improves the generalization ability of the drone strategy, enabling it to show stronger adaptability and response capabilities in different air combat tasks and scenarios, significantly enhancing its combat effectiveness.

[0060] Air combat maneuver optimization and control accuracy improvement: The reward function design of the present invention reasonably guides the UAV to obtain a favorable position in the air combat, approach the enemy aircraft and carry out effective strikes. By combining relative position rewards, distance rewards and strike rewards, it ensures that the UAV always has a tactical advantage during the air combat. At the same time, the final reward further optimizes the maneuver control accuracy and combat performance by comprehensively evaluating the UAV's mission completion, thereby enhancing the UAV's comprehensive combat capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1: Schematic diagram of the time scale separation of state quantities, showing the division of short-period and long-period state quantities.

[0062] Figure 2 : A time-division air combat decision framework showing the interface relationship between long-cycle and short-cycle control strategies.

[0063] Figure 3 : Detailed design of roll angle, pitch angle, sideslip angle and throttle controller.

[0064] Figure 4 : A detailed diagram of the final reward calculation process.

[0065] Figure 5 : Convergence process and performance comparison curve of the two-stage training strategy. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0067] The implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0068] The same or similar numbers in the drawings of this embodiment correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the drawings, it is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limitations on this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0069] Example 1: System modeling and state separation of UAV air combat missions

[0070] This embodiment describes the system modeling of a UAV in an air combat mission and processes state quantities based on time scale separation.

[0071] 1.1 UAV six-degree-of-freedom modeling

[0072] In air combat missions, the six-degree-of-freedom model is used to describe the motion state of the drone. The motion of the drone can be divided into two parts: translation and rotation. The specific state quantities include: position (x, y, z); velocity (V x , V y , V z); roll angle φ; pitch angle θ; yaw angle ψ; corresponding angular velocity (ω φ ,ω θ ,ω ψ ).

[0073] These state quantities are collected in real time through sensors and serve as input for subsequent control systems.

[0074] 1.2 Time scale separation of state quantities

[0075] The state quantities of drones are divided into short-term and long-term state quantities according to their rate of change. Short-term state quantities include attitude angles (roll angle, pitch angle, yaw angle) and their angular velocities (p, q, r). Long-term state quantities include position (x, y, z), velocity (V x , V y , V z ) and relative speed. By separating the time scales, the dimension of the state space is reduced and the computational complexity is reduced.

[0076] Figure 1 The division of short-period and long-period state quantities and their corresponding control strategies are demonstrated.

[0077] Example 2: Design and implementation of short-cycle controller

[0078] This embodiment describes in detail the application of the PID controller in short-cycle motion control.

[0079] 2.1 Roll Angle Controller Design

[0080] In short-cycle motion, the roll angle controller is used to control the UAV to rotate around the longitudinal axis, thereby achieving steering in air combat. The input of the controller is the desired roll angle φ c , the output is the adjustment amount δ of the control surface a The governing equation is as follows:

[0081] δ a =-k φ (φ c -φ)-k p p

[0082] Where p is the current roll angular velocity, k φ and k p is the control gain parameter.

[0083] 2.2 Pitch Angle Controller

[0084] The pitch angle controller is used to adjust the pitch attitude of the drone to control its climb or dive. The controller controls the pitch angle θ and pitch angle velocity q through feedback, and the formula is:

[0085] δ e =-kθ (θ c -θ)-k q q

[0086] Through this controller, the drone can accurately adjust the flight altitude and speed.

[0087] 2.3 Sideslip Angle Controller

[0088] The side slip angle controller is used to eliminate the side slip angle β during the turning process to ensure the stability of the drone. The controller is adjusted through the rudder, and its mathematical model is:

[0089] δ r =k β (β c -β)+k r r

[0090] Where r is the yaw rate. To solve the problem of steady-state error, the sideslip angle control adopts the proportional integral control method. This design ensures that the UAV remains stable during high-speed maneuvers.

[0091] 2.4 Throttle speed controller

[0092] The throttle control system is used to adjust the speed of the drone, and the feedback variable is the speed V. The controller formula is as follows:

[0093]

[0094] This controller, combined with the pitch angle controller, ensures that the drone maintains the appropriate speed during different maneuvering tasks. Figure 3 The detailed structure of the roll angle, pitch angle, sideslip angle and throttle control system is shown.

[0095] Example 3: PPO algorithm for long-term trajectory planning

[0096] This embodiment introduces the long-term decision-making part, mainly using the Proximal Policy Optimization (PPO) algorithm for trajectory optimization.

[0097] 3.1 Working Principle of PPO Algorithm

[0098] The PPO algorithm is an advanced deep reinforcement learning strategy optimization algorithm suitable for solving long-term decision-making problems in drone air combat. The goal of PPO is to maximize the cumulative discounted reward R t :

[0099]

[0100] Among them, γ is the discount factor, r tis the instant reward for each step. In order to prevent the strategy update from being too large, PPO introduces a clipping mechanism, and its objective function is:

[0101]

[0102] 3.2 Interface for long and short cycle control

[0103] The long-term decision part outputs the desired roll angle φ through the PPO strategy. c , pitch angle θ c and speed V c , these expected values ​​serve as inputs for short-cycle control. The long-cycle strategy is updated at a low frequency (0.5Hz), while the short-cycle control is updated at a high frequency (10Hz). This separation ensures computational efficiency and control accuracy. Figure 2 The interface and time-division framework for long-cycle and short-cycle control are demonstrated.

[0104] Example 4: Reward Function Design

[0105] This example demonstrates the design of the reward function, which directly affects the strategy learning effect of the drone. The reward function includes relative position reward, distance reward, strike reward and final reward.

[0106] 4.1 Relative Position Reward

[0107] When our drone is behind the enemy drone and facing the enemy drone, a reward is given. The calculation formula for relative position reward is:

[0108] r position =k1·cos(θ1)+k2·cos(θ2)

[0109] Among them, θ1 is the angle between our speed and the enemy's relative position, and θ2 is the angle between the enemy's speed and our relative position.

[0110] 4.2 Distance Reward

[0111] Encourage our drones to approach enemy drones. The distance reward is calculated as follows:

[0112]

[0113] Where r is the distance between the two drones.

[0114] 4.3 Strike Rewards

[0115] When our drone enters the attack range of the enemy aircraft and attacks, a reward is given. The formula for the strike reward is:

[0116]

[0117] Among them, r is the distance, ω is the attack angle, and ω0 is the attack angle threshold.

[0118] 4.4 End-game rewards

[0119] The final reward function is used to evaluate the status at the end of the air combat mission, including loss of altitude control, exceeding the distance limit, timeout, victory or failure. The specific formula is:

[0120]

[0121] Among them: r1 represents the penalty given when the drone altitude is not within the range of [1000m, 10000m];

[0122] r2 indicates that when the distance of the drone exceeds the set maximum distance max the punishment given when

[0123] r3 indicates that when the task time exceeds the maximum allowed time end Rewards given when

[0124] r4 represents the reward given when the drone wins an air battle;

[0125] r5 represents the penalty given when the drone fails in an air combat.

[0126] Figure 4 The final reward calculation process is shown.

[0127] Example 5: Two-stage training strategy

[0128] In order to improve the convergence speed of drone strategy learning, this embodiment designs a two-stage training strategy.

[0129] 5.1 Phase 1: Single-step reward training

[0130] In the first stage, a dense single-step reward function is used to guide the drone to learn basic maneuvers. At this time, the update frequency of the PPO algorithm is 0.5Hz, in order to give the model enough time to explore.

[0131] 5.2 Phase 2: Final Reward Training

[0132] In the second stage, the single-step reward is removed, and only the final reward is retained. The decision frequency of the PPO network is increased to 2Hz, allowing the drone to quickly respond to complex maneuvering scenarios in air combat. This stage focuses on the final performance and guides the drone to learn complex strategies through the design of the reward function.

[0133] Figure 5 The convergence curve and performance comparison of the two-stage training strategy are shown.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A hierarchical algorithm for deep reinforcement learning in air combat based on time scale separation theory, characterized by: The algorithm combines long-term decision-making with short-term control and optimizes the autonomous air combat maneuvering decision of the UAV through a two-stage training strategy, which specifically includes the following steps: Based on the principle of time scale separation, the state quantity of the drone is divided into long-period state quantity and short-period state quantity. The long-period state quantity includes position (x, y, z), relative speed (v x ,v y ,v z ), the short-period state quantities include roll angular velocity (p), pitch angular velocity (q), yaw angular velocity (r), pitch angle (θ), roll angle (φ) and velocity (V) to achieve dimensionality reduction of air combat state space; Long-term decisions are optimized through the PPO algorithm, and the long-term strategy outputs the expected values ​​required for short-term decisions, including the expected roll angle, pitch angle, sideslip angle and speed; The attitude and speed of the drone are adjusted through the PID controller in a short period to achieve precise control of the expected values; Two-stage training: The first stage adopts a single-step reward strategy, and the second stage removes the single-step reward, retaining only the final sparse reward and increasing the decision frequency of the long-term strategy.

2. The hierarchical method for deep reinforcement learning in air combat based on time scale separation theory according to claim 1 is characterized in that: The short-cycle control uses a PID controller for attitude and speed control, and the output control variable of the PID controller is: Among them, K p ,K i ,K d Represent the gain parameters of the proportional, integral and differential controllers respectively, the proportional control part: K p e(t), where K p is the proportional gain parameter, which indicates the influence of the error signal on the control output; e(t) is the error between the target value and the actual output; Integral control part: Where K i is the integral gain parameter, which indicates the influence of the error integral on the control output; Differential control part: Where K d It is the differential gain parameter, which indicates the influence of the error change rate on the control output.

3. The hierarchical method for deep reinforcement learning in air combat based on time scale separation theory according to claim 1 is characterized in that: Roll angle controller: used to control the rotation of the drone around the longitudinal axis. When the drone turns on the horizontal plane, it needs to adjust its roll angle φ to achieve the purpose of turning. Therefore, the roll angle controller needs to have the ability of fast response and minimum overshoot. The controller receives the desired roll angle φ c As input, and generate the corresponding δ a The command makes the drone gradually reach the desired roll angle. The inner loop of the roll angle controller uses the roll angle velocity p, and the outer loop uses the roll angle φ feedback. The mathematical expression is as follows: d a =-[k φ (f c -φ)]-k p p Pitch angle controller: When controlling the longitudinal flight of the drone, that is, changing its altitude or speed, the pitch angle of the drone needs to be adjusted through the pitch angle control system. The controller receives the desired pitch angle θ c As input, and generate the corresponding δ e The command makes the drone gradually reach the desired pitch angle. The inner loop of the pitch angle controller uses the pitch angle velocity q, and the outer loop uses the pitch angle θ feedback. d e =-[k θ (i c -θ)]-k q q Side slip angle controller: Since the UAV will generate a sideslip angle β during the turning process, it is necessary to eliminate the sideslip angle by manipulating the rudder to maintain the stability of the UAV. The controller receives the expected sideslip angle β c As input, the corresponding δ is output r Instruction, the inner loop of the sideslip angle controller uses the yaw angle velocity r, and the outer loop uses the sideslip angle feedback. Considering the problem of steady-state error in the sideslip angle control, the proportional integral control method is adopted, and its mathematical expression is as follows: Throttle speed controller: adjust the throttle lever delta τ The power output of the drone can be changed to adjust the speed of the drone. By working with the pitch angle control system, the speed of the drone can be controlled. The throttle speed controller receives the expected value V of the speed. c As input, and output the corresponding δ τ Instructions are used to adjust the speed of the drone to achieve the desired goal. The feedback of the control system is speed V. Proportional integral control is used to solve the steady-state error problem. Its mathematical expression is as follows: 。 4. The air combat deep reinforcement learning hierarchical algorithm based on time scale separation theory according to claim 1 is characterized in that: Long-term decision-making uses a deep reinforcement learning strategy based on the PPO algorithm, whose goal is to maximize the long-term cumulative reward To optimize the strategy, where: γ is the discount factor, r k is the reward obtained at time k, and the objective function of the PPO algorithm is: in, is the advantage function estimate, and e is the clipping coefficient, which is used to limit the amplitude of the policy update.

5. The air combat deep reinforcement learning hierarchical algorithm based on time scale separation theory according to claim 1 is characterized in that: The long-period state quantity is transformed into relative state through the following coordinate system transformation, and the transformation matrix M is defined as: in, Used to convert state quantities to a new coordinate system.

6. The air combat deep reinforcement learning hierarchical algorithm based on time scale separation theory according to claim 1 is characterized in that: The algorithm is based on a long-cycle and short-cycle time scale separation architecture. The frequency of long-cycle decision-making is 0.5Hz, and the frequency of short-cycle control is 10Hz. The short-cycle control receives the expected value of the long-cycle strategy output and realizes real-time posture adjustment through PID control.

7. The air combat deep reinforcement learning hierarchical algorithm based on time scale separation theory according to claim 1 is characterized in that: The reward function consists of the following parts: Relative position reward: rewards our drone for being behind the enemy and facing the enemy, the formula is: Reward position =k1cos(θ1)+k2cos(θ2) Among them, θ1 is the angle between our speed and the enemy's relative position, and θ2 is the angle between the enemy's speed and our relative position; Distance bonus: Encourages the drone to get closer to the enemy target. The formula is: Where r is the distance between the two drones.

8. The air combat deep reinforcement learning hierarchical algorithm based on time scale separation theory according to claim 7 is characterized in that: The strike reward is designed to give rewards when our drone enters the enemy's attack range and attacks. The specific calculation is: Among them, r is the distance between drones, ω is the attack angle, and ω0 is the threshold of the attack angle.

9. The air combat deep reinforcement learning hierarchical algorithm based on time scale separation theory according to claim 1 is characterized in that: The training process adopts a two-stage strategy training, in which: Phase 1: The drone’s air combat behavior is guided by designing a single-step reward function. In this phase, the PPO network’s update frequency is 0.5 Hz, and the dense reward function is used to accelerate convergence. Specific rewards include relative position rewards, distance rewards, and strike rewards. Phase 2: Remove single-step rewards, retain only final rewards, and increase the decision frequency of the PPO network to 2Hz to improve the decision-making efficiency and combat capability of the agent.

10. The air combat deep reinforcement learning hierarchical algorithm based on time scale separation theory according to claim 9 is characterized in that: The final reward function is used to evaluate the status at the end of the air combat mission, including loss of altitude control, exceeding the distance limit, timeout, victory or failure. The specific formula is: Among them: r1 represents the penalty given when the drone altitude is not within the range of [1000m, 10000m]; r2 indicates that when the distance of the drone exceeds the set maximum distance max the punishment given when r3 indicates that when the task time exceeds the maximum allowed time end Rewards given when r4 represents the reward given when the drone wins an air battle; r5 represents the penalty given when the drone fails in an air combat.

Images