A missile guidance method based on hierarchical reinforcement learning

Abstract

This invention provides a missile guidance method based on hierarchical reinforcement learning, belonging to the field of control strategies. The steps are as follows: First, a missile-target kinematic model and a missile velocity decay model are introduced; second, based on the missile-target kinematic model, process rewards and outcome rewards are designed to obtain a total reward function, guiding the missile to pursue the target; third, after completing the reward function design, the Flexible Action-Evaluation Algorithm (SAC) is used to initially train the missile under the condition that the target is moving in a straight line, obtaining an initial guidance strategy; finally, based on the initial guidance strategy, the enemy aircraft adopts an evasion strategy, and the SAC is used to continue training the missile, ultimately obtaining a high-quality guidance strategy. This invention enables missiles to autonomously adjust their flight paths in complex combat situations, overcoming interference and challenges in complex environments, thereby significantly improving guidance accuracy and attack success rate, and enhancing missile strike effectiveness and survivability.

Description

Technical Field

[0001] This invention belongs to the field of sensor management and optimization technology, and relates to a missile guidance strategy, particularly a missile guidance method based on hierarchical reinforcement learning. This method enables the missile agent to autonomously adjust its flight path in complex combat situations, overcome interference and challenges in complex environments, thereby significantly improving guidance accuracy and attack success rate, and enhancing missile strike effectiveness and survivability. Background Technology

[0002] In modern air combat, maneuvering strategies are not the only factor determining victory. Missiles, as long-range strike weapons, play a decisive role. How to achieve precise guidance in a highly dynamic, nonlinear, and uncertain battlefield environment has become crucial for further improving air combat efficiency and win rates. Existing technologies [Chinese Invention Patent CN116227343A: Design Method for Terminal Angle Attack Guidance Law Satisfying Field-of-View Constraints for Intercepting Maneuvering Targets] and [Chinese Invention Patent CN110008502A: Integrated Three-Dimensional Guidance and Control Design Method Considering Field-of-View Constraints of All-Slip Linked Seekers] face the following difficulties when facing guided attacks:

[0003] (1) High dynamic environment. The maneuverability of modern fighter jets, drones or ballistic missiles is constantly breaking through, and guidance algorithms need to complete state estimation, trajectory prediction and control command generation in milliseconds, which puts extremely high demands on computing hardware and algorithm efficiency.

[0004] (2) The combat environment is complex. Deceptive (such as false radar signals) or suppressive jamming (such as noise jamming) released by the enemy can disrupt the target tracking capability of the seeker (radar / infrared).

[0005] (3) Optimization under strong constraints. Balancing maneuverability and range under limited fuel conditions presents a difficult optimization problem.

[0006] Unlike close-range UAV maneuvering strategies, missile guidance strategies not only require intelligent agents with strong environmental awareness capabilities but also demand efficient decision-making in target tracking, interference avoidance, and evasion of enemy countermeasures. In missile guidance, the uncertainty and randomness of target maneuverability and detection delay lead to a significant decline in pursuit performance. Compared to traditional radar and laser guidance methods, developing intelligent rapid guidance methods based on neural networks can maximize guidance efficiency while reducing decision-making time, thereby enhancing missile strike capabilities and is of great significance. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides a missile guidance method based on hierarchical reinforcement learning. It establishes a one-to-one missile-target three-dimensional pursuit and escape model. Based on the hierarchical learning concept, it performs temporal fusion of dense and sparse reward functions to promote the joint guidance of human experience and strategy exploration on the learning process, thereby achieving the goal of high missile hit rate.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A missile guidance method based on hierarchical reinforcement learning, the missile guidance method comprising the following steps:

[0010] Step 1: Introduce the missile-target kinematic model and the missile velocity decay model to provide a high-fidelity simulation environment for the pursuit-escape game;

[0011] Step 1.1: Construct a missile-target kinematic model;

[0012] A six-degree-of-freedom system is used to describe the missile-target kinematics model, treating the missile as a particle and ignoring roll angle variations to simplify the model. The missile and target share a unified three-dimensional inertial coordinate system. Defining the missile as M and the target as TG, the kinematic model of the missile and target is as follows:

[0013] (1)

[0014] Among them, superscript This indicates that the variable corresponds to a missile or target; If it is the coordinates of the missile or target, then... It is the projection component of the flight velocity on the three coordinate axes in a three-dimensional inertial coordinate system; Indicates velocity, its direction is determined by the pitch angle. and yaw angle definition; The acceleration commands in the overload, yaw, and pitch directions are indicated by... Control; g represents the gravitational constant; This indicates the change in yaw angle; This indicates the change in pitch angle.

[0015] The three-dimensional coordinates between the missile and the target are represented as follows:

[0016] (2)

[0017] in, and A vector representing the difference in three-dimensional coordinates between the missile and the target in a three-dimensional inertial coordinate system; and A vector representing the velocity difference between the missile and the target in a three-dimensional inertial coordinate system; This represents the target's three-dimensional coordinates in a three-dimensional inertial coordinate system. This represents the missile's three-dimensional coordinates in a three-dimensional inertial coordinate system. This represents the projection components of the target velocity onto the three coordinate axes in a three-dimensional inertial coordinate system. This represents the projected components of the missile's velocity onto the three coordinate axes in a three-dimensional inertial coordinate system.

[0018] Step 1.2, introduce the missile velocity decay model;

[0019] During flight, a missile's speed gradually decreases due to air resistance, gravity, and other external factors. The speed decay rate is an important parameter describing the change in missile speed over time, and it is usually closely related to the missile's flight altitude, speed, aerodynamic characteristics, and environmental conditions (such as atmospheric density). The speed decay rate not only affects the missile's effective strike range but also directly relates to whether the missile can complete its interception mission on time. A missile speed decay model, as shown in formula (3), is introduced to calculate the missile's speed decay.

[0020] (3)

[0021] in, It is the missile's altitude in a three-dimensional inertial coordinate system; It is the missile in a three-dimensional inertial coordinate system coordinate; It is the missile in a three-dimensional inertial coordinate system coordinate; It is the missile in a three-dimensional inertial coordinate system coordinate; It is dynamic pressure; It's the missile's speed; It is the status update time; It is the missile's yaw angle; This is the effective area, set to 0.1; coefficient. Set to - ,coefficient Set to - ; This is the aerodynamic coefficient, which varies with velocity;

[0022] Step 2: Based on the missile-target kinematics model, design process rewards and outcome rewards to guide the missile to pursue the target;

[0023] Step 2.1: During missile guidance, the position information of both sides is transformed into situational information for situation assessment. The flight direction vectors of the missile and the enemy aircraft are defined as follows: and The position vectors of both sides are and The expression is:

[0024] (4)

[0025] in, Represents the missile's flight vector; Represents the target's flight vector; This represents the coordinate difference vector between the target and the missile. This represents the coordinate difference vector between the missile and the target. Indicates the missile's yaw angle; Indicates the target yaw angle; Indicates the missile's elevation angle; Indicates the target's pitch angle; Represents the missile's three-dimensional coordinates; Represents the three-dimensional coordinates of the target.

[0026] Based on the coordinate relationship shown in formula (4), the missile's flight vector is used. Flight vector of the target Obtain the angle between the two. The calculation formula is:

[0027] (5)

[0028] Step 2.2, the process reward has two components: distance reward and angle reward.

[0029] The distance reward Depend on The calculation is described as follows:

[0030] (6)

[0031] in, Indicates the distance between the missile and the target at the current moment; express The distance between the two at any given time is obtained through the action given by the reinforcement learning algorithm, the missile kinematics model, and the numerical solution of the Euler differential equation. Represents the missile's flight vector; This represents the target's flight vector. The distance between them... A positive reward is given when the amount decreases, and a negative reward is given when the amount increases.

[0032] The angle reward The description is as follows:

[0033] (7)

[0034] in, Indicates the angle between the missile and the target's flight vector;

[0035] The angle bonus is determined based on the calculated distance bonus. Since the range of the angle between the missile and the target is... Therefore, the value of the angle reward is determined by the sign of the distance reward. Specifically, this means that when the distance between the two objects is decreasing, i.e. When the interception angle The greater the distance, the higher the probability that the missile will intercept the target. In any case, as the distance between the two increases, Then the intercept angle is required. As small as possible, the missile reorients itself to intercept the target.

[0036] Step 2.3, Result Reward The description is as follows:

[0037] (8)

[0038] If the missile hits the enemy in this round, you win and receive a positive reward. If the target is not hit by the end of the round, a negative reward is given. .

[0039] Step 2.4, the overall reward function is:

[0040] (9)

[0041] in, As the weight of distance reward, As the weight of the angle reward, Weighting of the reward for the outcome. This represents the total reward earned in this round.

[0042] Step 3: After completing the reward function design, the missile is initially trained using the SoftActor-Critic (SAC) algorithm under the condition that the target is moving in a straight line, so as to obtain the initial guidance strategy;

[0043] The enemy aircraft did not adopt evasive tactics, i.e., pitch angle With yaw angle Under the condition that everything remains unchanged, the missile undergoes initial strategy training to obtain an initial guidance strategy. In this case, the missile will gradually learn and update the strategy from its initial, somewhat convoluted guidance trajectory, continuously optimizing the guidance trajectory. Subsequently, the initial guidance strategy is extracted, and higher-quality guidance strategy training is conducted based on this.

[0044] Step 4: Based on the initial guidance strategy, the enemy aircraft adopts an evasion strategy. The Flexible Action-Evaluation Algorithm (SAC) is used to continue training the missile, and finally a high-quality guidance strategy is obtained.

[0045] After completing the initial training in Step 3, training is conducted under enemy aircraft evasion strategies, including pull-up evasion and turn evasion. The initial guidance strategy developed in Step 3 is used as a foundation to continue learning guidance strategies. As training progresses, the missile's guidance strategy will be gradually optimized, enabling the missile to intercept targets more accurately and efficiently, avoiding problems such as large-angle turns and uneven trajectories.

[0046] The beneficial effects of this invention are as follows:

[0047] (1) The present invention describes the trajectory, acceleration and attitude changes of the missile more accurately through the missile-target kinematic model and velocity decay model in step 1.

[0048] (2) The present invention decomposes the complex guidance task into multiple levels of sub-tasks through hierarchical reinforcement learning in steps 3 and 4, so that the missile can focus on learning the simpler and local tasks, avoiding the complexity of learning the entire task directly from scratch, and improving the training speed.

[0049] (3) The reinforcement learning adopted in this invention can significantly improve decision-making efficiency, and the high-precision guidance technology can significantly enhance the battlefield adaptability and operational flexibility of the missile system, effectively improving the comprehensive combat effectiveness of the weapon system. Attached Figure Description

[0050] Figure 1 This is a flowchart of the present invention.

[0051] Figure 2 Initial training diagram for target-free evasion strategy guidance.

[0052] Figure 3 Mid-term training diagram for target-free evasion strategy guidance.

[0053] Figure 4 The training image shows the later stages of a target-oriented, non-evasive strategy.

[0054] Figure 5 The later stage of the training diagram for target-free evasion strategy guidance.

[0055] Figure 6 The result diagram of the target-oriented evasion strategy.

[0056] Figure 7 The guidance result diagram of the turning avoidance strategy for the target. Detailed Implementation

[0057] The present invention will be further described below with reference to specific embodiments.

[0058] A missile guidance strategy based on hierarchical reinforcement learning includes the following steps:

[0059] Step 1: Introduce the missile-target kinematic model and the missile velocity decay model to provide a high-fidelity simulation environment for the pursuit-escape game;

[0060] Step 1.1: Construct a missile-target kinematic model;

[0061] A six-degree-of-freedom system is used to describe the missile-target kinematics model, treating the missile as a particle and ignoring roll angle variations to simplify the model. The missile and target share a unified three-dimensional inertial coordinate system. In addition to the three-dimensional position coordinates of both parties, pitch and yaw angles are used to define their attitude information, which is then analyzed through overload. control.

[0062] Step 1.2, introduce the missile velocity decay model;

[0063] Considering the gradual decrease in speed of a missile during flight due to air resistance, gravity, and other external factors, a velocity decay rate is introduced. The velocity decay rate is an important parameter describing the change of missile velocity over time, and it is usually closely related to the missile's flight altitude, speed, aerodynamic characteristics, and environmental conditions (such as atmospheric density).

[0064] Step 2: Based on the missile-target kinematics model, design process rewards and outcome rewards to guide the missile to pursue the target;

[0065] Step 2.1: During missile guidance, the position information of both sides is transformed into situational information for situation assessment. The flight direction vectors of the missile and the enemy aircraft are defined as follows: and The position vectors of both sides are and Based on the coordinate relationship, the missile's velocity vector is used. velocity vector of the target The angle between the two can be obtained. In order to design the reward function.

[0066] Step 2.2, the process reward has two components: distance reward and angle reward.

[0067] The reward function mainly consists of two parts: process reward and outcome reward. The process reward has two components: distance reward and angle reward. The distance reward is determined by... Calculate, where, This indicates the distance between the missile and the target at the current moment. express The distance between the two targets at any given time. A positive reward is given when the distance decreases, and a negative reward is given when it increases. The angular reward is determined based on the calculated distance reward. Since the angle between the missile and the target ranges from... Therefore, the value of the angle reward is determined by the sign of the distance reward. Specifically, this means that when the distance between the two objects is decreasing, i.e. When the interception angle The greater the distance, the higher the probability that the missile will intercept the target. In any case, as the distance between the two increases, Then the intercept angle is required. As small as possible, the missile reorients itself to intercept the target.

[0068] Step 2.3, Result Reward :

[0069] The result reward indicates the outcome of this round. When the maximum number of moves is reached, a positive reward is given if the missile hits the target, and a negative reward is given if it misses.

[0070] Step 2.4, Total Reward Function:

[0071] The overall reward function is a weighted sum of process rewards and outcome rewards.

[0072] Step 3: After completing the design of the reward function, the SAC algorithm is used to perform preliminary training on the missile under the condition that the target takes a straight line motion to obtain the initial guidance strategy. The parameter settings are shown in Table 1.

[0073] Table 1. Experimental parameter settings for the initial guidance strategy

[0074]

[0075] The enemy aircraft did not adopt evasive tactics, i.e., pitch angle With yaw angle Under the condition that everything remains unchanged, the missile is trained with preliminary strategies to obtain the initial guidance strategy. Table 2 shows the initial coordinates (initial position), speed, pitch angle, and yaw angle of the enemy aircraft and the missile in this case.

[0076] Table 2, Information on the initial guidance strategy

[0077]

[0078] Figure 2 The training trajectory in the early stages of the exercise was shown, with the missile beginning to intercept enemy aircraft. However, due to a lack of experience, the missile trajectory exhibited many twists and turns, indicating that the missile's movements were constantly changing, including sudden changes in angle, which is clearly unrealistic. Figure 3The training trajectory during the middle of the training process is shown. At this point, the reinforcement learning algorithm has solved the problem of large pitch angle direction changes in the previous guidance process through continuous learning. However, for targets moving in uniform linear motion, there is a clear problem of altitude reduction, which indicates that the guidance strategy is of low quality at this time. Figure 4 The training trajectory was shown in the later stages of the training, and the results showed that the missile trajectory had become relatively smooth, and it had learned to intercept enemy aircraft more quickly. However, there were still shortcomings: significant jitter in the middle of the flight trajectory, rapid changes in the missile yaw angle, and a lack of sensitivity in maneuver selection. Figure 5 The training trajectory in the later stages of training was shown. Through continuous learning, the missile guidance trajectory became smooth, and the interception of the target was completed in the shortest time, thus obtaining the initial guidance strategy.

[0079] Step 4: Based on the initial guidance strategy, the enemy aircraft adopts an evasion strategy. The missile is then trained using SAC (Self-Controlled Ambient Aircraft) to ultimately obtain a high-quality guidance strategy. Specifically:

[0080] After completing initial guidance strategy training, training will proceed under enemy aircraft evasion strategies. The initial guidance strategy will be used as a foundation for the missile to continue learning guidance strategies. This includes training on enemy aircraft employing rapid climb and sharp turn evasion strategies. Initial information for both sides is shown in Table 3.

[0081] Table 3. Information on high-quality guidance strategies

[0082]

[0083] Figure 6 The diagram shows the guidance results when the target adopts an evasive maneuver strategy. This indicates that as training progresses, the missile's guidance strategy is gradually optimized, enabling the missile to complete the interception mission more accurately and efficiently, avoiding problems such as large-angle turns and uneven trajectories. Figure 7 The diagram shows the guidance results when the target employs a turning evasion strategy. The missile has mastered a variety of guidance strategies and can quickly change its flight trajectory even if it misses an interception opportunity, creating a second strike opportunity.

[0084] The above-described embodiments are merely illustrative of the implementation methods of the present invention, but should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A missile guidance method based on hierarchical reinforcement learning, characterized in that, The missile guidance method includes the following steps: Step 1: Introduce the missile-target kinematic model and the missile velocity decay model to provide a high-fidelity simulation environment for the pursuit-escape game; specifically: Step 1.1, construct the missile-target kinematic model as follows: （1）； Among them, superscript This indicates that the variable corresponds to a missile or target; If it is the coordinates of the missile or target, then... It is the projection component of the flight velocity on the three coordinate axes in a three-dimensional inertial coordinate system; Indicates velocity, its direction is determined by the pitch angle. and yaw angle definition; The acceleration commands in the overload, yaw, and pitch directions are indicated by... Control; g represents the gravitational constant; This indicates the change in yaw angle; This indicates the change in pitch angle; The three-dimensional coordinates between the missile and the target are represented as follows: （2）； in, and A vector representing the difference in three-dimensional coordinates between the missile and the target in a three-dimensional inertial coordinate system; and A vector representing the velocity difference between the missile and the target in a three-dimensional inertial coordinate system; This represents the target's three-dimensional coordinates in a three-dimensional inertial coordinate system. This represents the missile's three-dimensional coordinates in a three-dimensional inertial coordinate system. This represents the projection components of the target velocity onto the three coordinate axes in a three-dimensional inertial coordinate system. This represents the projected components of the missile's velocity onto the three coordinate axes in a three-dimensional inertial coordinate system. Step 1.2, introduce the missile velocity decay model; A missile velocity decay model as shown in formula (3) is introduced to calculate the missile velocity decay. （3）； in, It is the missile's altitude in a three-dimensional inertial coordinate system; It is the missile in a three-dimensional inertial coordinate system coordinate; It is the missile in a three-dimensional inertial coordinate system coordinate; It is the missile in a three-dimensional inertial coordinate system coordinate; It is dynamic pressure; It's the missile's speed; It is the status update time; It is the missile's yaw angle; This is the effective area, set to 0.1; , Indicates coefficient; The aerodynamic coefficient; Step 2: Based on the missile-target kinematics model, design process rewards and outcome rewards to obtain the overall reward function, guiding the missile to pursue the target; specifically: Step 2.1: During the missile guidance process, the position information of both sides is converted into situational information for situational assessment. Step 2.2, the process reward has two components: distance reward and angle reward; Step 2.3, Result Reward The description is as follows: （8）； If the missile hits the enemy in this round, you win and receive a positive reward. If the target is not hit by the end of the round, a negative reward is given. ; Step 2.4, the overall reward function is: （9）； in, As the weight of distance reward, As the weight of the angle reward, Weighting of the reward for the outcome; This is the total reward earned in this round; As a distance reward; As an angle reward; Step 3: After completing the reward function design, the missile is initially trained using the Flexible Action-Evaluation Algorithm (SAC) under the condition that the target is moving in a straight line, so as to obtain the initial guidance strategy. Step 4: Based on the initial guidance strategy, the enemy aircraft adopts an evasion strategy. The Flexible Action-Evaluation Algorithm (SAC) is used to continue training the missile, and finally a high-quality guidance strategy is obtained.

2. The missile guidance method based on hierarchical reinforcement learning according to claim 1, characterized in that, In step 1.2, Set to 0.1; coefficient Set to -1.15×10 −4 ,coefficient Set to -1.62×10 −4 .

3. The missile guidance method based on hierarchical reinforcement learning according to claim 1, characterized in that, In step 2: Step 2.1 specifically involves defining the flight direction vectors of the missile and the enemy aircraft as follows: and The position vectors of both sides are and The expression is: （4）； in, Represents the missile's flight vector; Represents the target's flight vector; This represents the coordinate difference vector between the target and the missile. This represents the coordinate difference vector between the missile and the target. Indicates the missile's yaw angle; Indicates the target yaw angle; Indicates the missile's elevation angle; Indicates the target's pitch angle; Represents the missile's three-dimensional coordinates; Represents the target's three-dimensional coordinates; Based on the coordinate relationship shown in formula (4), the missile's flight vector is used. Flight vector of the target Obtain the angle between the two. ; Step 2.2 specifically involves: The distance reward Depend on The calculation is described as follows: （6）； in, Indicates the distance between the missile and the target at the current moment; express The distance between the two at any given time is obtained through the action given by the reinforcement learning algorithm, the missile kinematics model, and the numerical solution of the Euler differential equation; Represents the missile's flight vector; Represents the target's flight vector; the distance between them. When the amount decreases, a positive reward is obtained; conversely, a negative reward is obtained. The angle reward The description is as follows: （7）； in, Indicates the angle between the missile and the target's flight vector; in, As the weight of distance reward, As the weight of the angle reward, Weighting of the reward for the outcome; This represents the total reward earned in this round.

4. The missile guidance method based on hierarchical reinforcement learning according to claim 3, characterized in that, In step 2.1, the included angle The calculation formula is: （5）。 5. A missile guidance method based on hierarchical reinforcement learning according to claim 3, characterized in that, In step 2.2, the angle reward is determined based on the obtained distance reward; the range of the angle between the missile and the target is... The value of the angle reward is determined by the sign of the distance reward.

6. The missile guidance method based on hierarchical reinforcement learning according to claim 3, characterized in that, Step 4 specifically involves: After completing the initial training in step 3, training is conducted under enemy aircraft evasion strategies, including climb evasion strategies and turn evasion strategies; the initial guidance strategy completed in step 3 is extracted as the basis for further learning of guidance strategies. As training progresses, the missile's guidance strategy will be gradually optimized, enabling the missile to accurately and efficiently complete the interception mission.

Images