The invention discloses an
air combat deep
reinforcement learning hierarchical algorithm based on a time
scale separation theory, which combines long-period decision and short-period control, and realizes unmanned aerial vehicle autonomous
air combat maneuver decision optimization through a two-stage training strategy. Firstly, the state quantity of the unmanned aerial vehicle is separated into a long-period state quantity and a short-period state quantity according to the time scale, the long-period state quantity comprises the position and the relative speed, and the short-period state quantity comprises the roll angle speed, the
pitch angle speed, the
yaw angle speed and the like. The long-period decision is optimized through a PPO
algorithm, and expected values of short-period control, such as an expected roll angle, a
pitch angle and a speed, are output; and the short-period control realizes accurate adjustment of the attitude of the unmanned aerial vehicle through a
PID controller. The
algorithm adopts two-stage training, in the first stage, convergence is accelerated through a single-step reward strategy, and in the second stage, combat performance is optimized through a final reward. The maneuverability and autonomous decision-making ability of the unmanned aerial vehicle in a complex
air combat environment are improved, and the method has the characteristics of quick response and efficient learning and is widely applied to autonomous air combat tasks of the unmanned aerial vehicle.