A drone avoidance method based on deep reinforcement learning

Abstract

This invention provides a drone evasion method based on deep reinforcement learning, belonging to the field of control strategies. First, a three-dimensional kinematic model of the drone is established as a training and learning environment for the evasion strategy. Second, a set of drone actions is set to ensure the completeness of the maneuver strategy. Third, based on the drone-missile pursuit model and the action set, process rewards and outcome rewards are designed to obtain an evasion strategy reward function, which guides the drone to evade missile pursuit. Finally, after constructing the training and learning environment and the evasion strategy reward function, training is performed under the condition that the missile adopts an intelligent guidance strategy until the drone successfully evades the target and obtains a high-quality evasion strategy. This invention enables drones to master a composite evasion strategy that combines energy management and spatial maneuvering, effectively countering the interception threat of high-speed missiles, enabling rapid and accurate responses under extreme conditions, and significantly improving their survivability.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous control strategies for drones, and relates to a drone avoidance strategy, and in particular to a drone avoidance method based on deep reinforcement learning. For threat scenarios involving high-speed pursuit and strong maneuverability of missiles, the drone generates optimal avoidance maneuver instructions and autonomously adjusts its flight path to avoid attacks, particularly improving the drone's survivability in complex electromagnetic environments. Background Art

[0002] With the widespread application of modern intelligent technology in the military, the battlefields faced by drones are becoming increasingly challenging. Beyond improving maneuvering decision-making capabilities and attack success rates, drones' evasive maneuvering strategies for missile avoidance are crucial for increasing air superiority. Existing technologies [Chinese invention patent CN116893690A: A reinforcement learning-based method for calculating input data for drone evasion attacks] and [Chinese invention patent CN112051863A: A method for autonomous drone counter-reconnaissance and enemy attack avoidance] face the following difficulties in missile evasion:

[0003] (1) Limited real-time performance. Most evasion strategies rely on pre-programmed maneuvers or passive interference such as releasing heat flares.

[0004] (2) Lack of adaptability. Rule-based control relies on accurate missile motion modeling and is prone to failure in complex electromagnetic interference environments, resulting in escape failure.

[0005] (3) It is difficult to obtain high-quality avoidance strategies. Traditional avoidance strategies have difficulty balancing short-term avoidance with long-term goals and are prone to falling into local optimality.

[0006] Compared to traditional rule-based and electronic countermeasures approaches, research on intelligent drone avoidance strategies based on neural networks, adapted to modern battlefield environments, can significantly reduce the probability of being shot down, ensure mission completion rates, and adapt to changes in missile speed and maneuverability, as well as complex battlefield environments. Furthermore, avoidance strategies trained using deep reinforcement learning require only forward computation (millisecond-level response) during the inference phase, meeting the timeliness requirements for drones to avoid supersonic missiles, which is of great significance. Summary of the Invention

[0007] In response to the problems existing in the existing technology, the present invention provides a drone avoidance method based on deep reinforcement learning. Based on a one-to-one drone-missile kinematic model, a drone-missile pursuit model is established as an avoidance strategy learning environment. A reward function consisting of the relative posture and flight status of the two is designed to achieve rapid avoidance of the drone after being locked and attacked by a missile.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A drone avoidance method based on deep reinforcement learning, the drone avoidance method comprising the following steps:

[0010] Step 1: Establish a 3D kinematic model of the UAV as a learning environment for avoidance strategy training;

[0011] The three-dimensional UAV model is described using a six-degree-of-freedom system. The three-dimensional kinematic model of the UAV is expressed as follows:

[0012] (1)

[0013] in, Indicates flight speed The projected components on the coordinate axes in the three-dimensional inertial coordinate system; is the gravitational acceleration constant; Characterizes tangential overload (defined as the ratio of thrust to weight), which serves as a control input to adjust flight rate; represents the normal overload (the ratio of lift to weight), which is used as a control input to achieve pitch attitude control; Represents the roll angle (positive for right), which is used as a control input to adjust the rotation of the aircraft around the velocity vector; Indicates the pitch angle, reflecting the angle between the flight trajectory and the horizontal plane, and determines the climb / dive state of the flight trajectory; Indicates the yaw angle, which reflects the lateral tilt of the drone. Indicates the change in yaw angle; Indicates the change in the speed of the drone; Indicates the speed of the drone; Indicates the pitch angle change.

[0014] Step 2: Set the drone action set to ensure the completeness of the maneuver strategy;

[0015] To ensure practicality and avoid the control difficulty caused by overly complex tactical actions, seven basic maneuvers in the basic air combat maneuver system are selected to form the UAV's action space. The seven basic maneuvers include constant speed, acceleration, deceleration, left turn, right turn, pull-up, and dive. The three-dimensional control instructions of tangential overload, normal overload and roll angle are used. The UAV's maneuvering actions are parameterized and encoded, and the physical constraints are directly reflected in the control instructions through normalization processing, which not only ensures the feasibility of maneuvering but also realizes the discrete representation of the action space.

[0016] Based on the established UAV motion space coding system, combined with the three-dimensional kinematic model shown in formula (1) in step 1, the dynamic update calculation of the UAV state can be realized. By inputting the discretized maneuver instructions into the three-dimensional kinematic model, the numerical integration method is used to solve formula (1), thereby solving the position and attitude state parameters of the UAV in real time, which can be expressed as:

[0017] (2)

[0018] Where, represents the maneuver selected by our UAV during the decision cycle; represents the state transition function; Indicates the system status at the current moment; Indicates the system status at the next moment.

[0019] Using the Euler numerical integration method, the aircraft speed, three-dimensional coordinates, pitch angle and yaw angle of formula (1) can be used to calculate the system state at the current moment. Next, iteratively calculate the system state at the next moment .

[0020] Step 3: Based on the drone-missile pursuit model and action set, design process rewards and result rewards to obtain the avoidance strategy reward function to guide the drone to avoid missile pursuit;

[0021] Step 3.1: During the training of the avoidance strategy, the position information of both parties is converted into situation information for situation assessment. The missile-UAV situation information is as follows:

[0022] (3)

[0023] in, Represents the displacement of the missile relative to the drone, also known as the line of sight; , and Represents the position of the missile in three-dimensional space; , and Represents the position of the drone in three-dimensional space.

[0024] The distance between the missile and the drone is:

[0025] (4)

[0026] definition , are the velocity vectors of the UAV and missile respectively, then the missile speed relative to the UAV for:

[0027] (5)

[0028] Step 3.2: Based on the situation information established in step 3.1, construct the corresponding real-time information of the sight line and angle, specifically:

[0029] Line of sight change rate (positive if moving away) for:

[0030] (6)

[0031] Line of sight angular velocity for:

[0032] (7)

[0033] Line of sight angular velocity size:

[0034] (8)

[0035] definition and are the projections of the velocity vectors of the UAV and missile on the horizontal plane XOY respectively. Expressed as:

[0036] (9)

[0037] in, Indicates missile coordinate; Indicates missile coordinate; Indicates drone coordinate; Indicates drone coordinate;

[0038] Then the aircraft lead angle in the horizontal plane (the angle between the velocity direction and the sight line) for:

[0039] (10)

[0040] in, Indicates the yaw angle of the drone;

[0041] The missile entry angle in the horizontal plane is for:

[0042] (11)

[0043] in, Indicates the missile's yaw angle;

[0044] Step 3.3: The avoidance strategy reward function consists of two parts: the process reward function and the result reward function. These two parts guide the drone to avoid missiles and accelerate the convergence of the algorithm. Specifically:

[0045] The rewards for the process are as follows:

[0046] (12)

[0047] in, is the initial distance between the missile and the UAV; is the distance between the missile and the UAV at the current moment; is the rate of change of the line of sight; is the maximum absolute value of the rate of change of the line of sight; is the weight coefficient of each reward, i is 1, 2, 3, 4, 5; A bonus for the distance between missiles and aircraft; A reward for the missile and aircraft distance change rate; Bonus for the drone's front angle; is the absolute value of the aircraft lead angle; reward for missile entry corner; is the absolute value of the missile entry angle; The purpose of the flight altitude reward is to prevent the drone from exceeding the altitude limit; The flight altitude of the drone; Give a reward value for each step of the process.

[0048] Result reward function as follows:

[0049] (13)

[0050] in, Indicates the reward when the drone successfully avoids the attack. Rewards for being hit by missiles.

[0051] Total reward function for:

[0052] (14)

[0053] Step 4: After constructing the training learning environment and the avoidance strategy reward function, training is performed under the condition that the missile adopts the intelligent guidance strategy until the UAV successfully avoids the target and obtains a high-quality avoidance strategy. Specifically:

[0054] The drone's avoidance strategy is primarily trained using the Proximal Policy Optimization (PPO) algorithm. The missile employs an intelligent guidance strategy, and the drone uses the algorithm for avoidance training. The first 1,000 rounds of training are set as the exploration phase, during which network parameters are not updated. This allows for the accumulation of diverse state-action experience samples. Network parameters are then updated normally. Ultimately, a composite avoidance strategy combining energy management and spatial maneuverability is achieved, enhancing the drone's survivability in high-risk environments. This continues until a high-quality avoidance strategy is achieved.

[0055] The beneficial effects of the present invention are:

[0056] (1) Through the UAV kinematic model in step 1 and the UAV action set in step 2, the trajectory, acceleration and attitude changes of the UAV are described more accurately.

[0057] (2) Through the detailed reward function setting in step 3, in addition to the win and loss rewards of the training round, dense process rewards are provided to accelerate the reinforcement learning process and implement the avoidance strategy more quickly.

[0058] (3) The adoption of intelligent guidance strategies in step 4 can effectively improve the survivability of drones in high-risk situations and demonstrate excellent adaptability in complex battlefields. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Flowchart of the present invention.

[0060] Figure 2 It is a three-dimensional UAV kinematic model.

[0061] Figure 3 A collection of drone actions.

[0062] Figure 4 It is the missile-UAV kinematic model.

[0063] Figure 5 Training early stage situations for avoidance strategies.

[0064] Figure 6 Training for avoidance strategies in late situations. DETAILED DESCRIPTION

[0065] The present invention will be further described below with reference to specific embodiments.

[0066] A deep reinforcement learning-based avoidance strategy includes the following steps:

[0067] Step 1: Establish a 3D kinematic model of the UAV as a learning environment for avoidance strategy training;

[0068] The three-dimensional UAV model is described by a six-degree-of-freedom system, such as Figure 2 As shown, a three-dimensional kinematic model of the UAV's six-degree-of-freedom kinematics is established. Indicates flight speed The projected components on the coordinate axes in the three-dimensional inertial coordinate system; is the gravitational acceleration constant; Characterizes tangential overload (defined as the ratio of thrust to weight), which serves as a control input to adjust flight rate; represents the normal overload (the ratio of lift to weight), which is used as a control input to achieve pitch attitude control; Represents the roll angle (positive for right), which is used as a control input to adjust the rotation of the aircraft around the velocity vector; Indicates the pitch angle, reflecting the angle between the flight trajectory and the horizontal plane, and determines the climb / dive state of the flight trajectory; Indicates the yaw angle, reflecting the lateral tilt of the drone.

[0069] Step 2: Set the drone action set to ensure the completeness of the maneuver strategy;

[0070] In order to ensure practicality and avoid the control difficulty caused by overly complex tactical actions, seven basic maneuvers (constant speed, acceleration, deceleration, left turn, right turn, pull up, and dive) are selected to form the action space of the drone, such as Figure 3 As shown. Using three-dimensional control instructions of tangential overload, normal overload and roll angle The parametric encoding of UAV maneuvers not only ensures the feasibility of maneuvers, but also realizes the discrete representation of the action space.

[0071] Based on the established UAV motion space coding system and the obtained three-dimensional kinematic model shown in formula (1), the dynamic update calculation of the UAV state is realized. By inputting the discretized maneuver instructions into formula (1), the numerical integration method is used to solve the UAV position and attitude state parameters in real time.

[0072] Step 3: Design process rewards and outcome rewards based on the drone-missile pursuit model and action set to guide the drone to avoid missile pursuit;

[0073] Step 3.1: During the training of the avoidance strategy, the position information of both parties is converted into situation information for situation assessment.

[0074] The situation information of both sides is as follows Figure 4 As shown. Among them, Represents the displacement of the missile relative to the drone, also known as the line of sight; , and Represents the position of the missile in three-dimensional space; , and Represents the position of the drone in three-dimensional space. Definition , are the velocity vectors of the UAV and missile respectively.

[0075] Step 3.2: Based on the situation information established in step 3.1, construct the corresponding real-time information of the sight line and angle, specifically:

[0076] definition and The projections of the velocity vectors of the UAV and missile on the horizontal plane XOY are respectively. , line of sight change rate , angular velocity of the line of sight , aircraft lead angle in the horizontal plane , missile entry angle in the horizontal plane The aiming line and angle information describe the current UAV and missile information.

[0077] Step 3.3: The avoidance strategy reward function consists of two parts: the process reward function and the result reward function. These two parts guide the drone to avoid missiles and accelerate the convergence of the algorithm. Specifically:

[0078] Construction Process Rewards Reward with results The total reward function is the sum of the two reward functions. The process reward is determined by factors such as the initial distance between the missile and the drone, the current distance between the missile and the drone, the rate of change of the line of sight, the maximum absolute value of the rate of change of the line of sight, the absolute value of the aircraft's lead angle, the absolute value of the missile's approach angle, and the drone's flight altitude. The outcome reward is determined by the evasion result of the current round. The drone receives a positive reward if it successfully evades the attack, and a negative reward if it fails.

[0079] Step 4: After constructing the training learning environment and the avoidance strategy reward function, training is performed under the condition that the missile adopts an intelligent guidance strategy until the UAV successfully avoids the target and obtains a high-quality avoidance strategy. The parameter settings are shown in Table 1.

[0080] Table 1. Parameter settings for the avoidance strategy experiment

[0081]

[0082] Figure 5The flight trajectory of the UAV in the early stage of evasion training is shown. In the early stage of training, the UAV only mastered the basic advantage strategy, which is to obtain higher rewards by flying at higher altitudes. It has not yet developed a more complex and effective maneuvering evasion strategy system. This limitation makes it difficult for the UAV to successfully evade high-speed pursuit. Through continuous accumulation of experience, Figure 6 In the final stages of training, the drone performed a wide-angle loop, rapidly adjusting its flight angle and speed, entering the missile's blind spot and successfully avoiding a strike. This demonstrates that a fully trained drone has mastered a complex evasion strategy combining energy management and spatial maneuvering, effectively countering the interception threat of high-speed missiles and responding quickly and accurately under extreme conditions, significantly enhancing its survivability.

[0083] The above-described embodiments merely express the implementation methods of the present invention, but should not be understood as limiting the scope of the patent of the present invention. It should be pointed out that for those skilled in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A drone avoidance method based on deep reinforcement learning, characterized in that: The drone avoidance method comprises the following steps: Step 1: Build a 3D kinematic model of the drone as a learning environment for avoidance strategy training. Step 2: Set the drone action set to ensure the completeness of the maneuver strategy; specifically: The drone action set includes seven basic maneuvers that constitute the drone's action space; the drone's maneuvers are encoded and normalized; the discretized maneuver instructions are input into a three-dimensional kinematic model to obtain the drone's position and attitude state parameters in real time; Step 3: Based on the drone-missile pursuit model and action set, design process rewards and result rewards to obtain the avoidance strategy reward function to guide the drone to avoid missile pursuit; specifically: Step 3.1: During the training of the avoidance strategy, the position information of both parties is converted into situation information for situation assessment to obtain the distance between the missile and the drone. ; and define , are the velocity vectors of the UAV and missile respectively, and the relative velocity of the missile to the UAV is obtained ; Step 3.2: Based on the situation information established in step 3.1, construct the corresponding real-time information of the sight line and angle; obtain the sight line change rate , angular velocity of the line of sight , angular velocity of the line of sight ;definition and They are respectively the projections of the velocity vectors of the UAV and the missile on the horizontal plane XOY; through the azimuth of the line of sight Get the aircraft lead angle in the horizontal plane , missile entry angle in the horizontal plane ; Step 3.3, the avoidance strategy reward function consists of two parts: the process reward function and the result reward function; Step 4: After constructing the training learning environment and the avoidance strategy reward function, training is performed under the condition that the missile adopts the intelligent guidance strategy until the UAV successfully avoids the target and obtains a high-quality avoidance strategy; In step 3.1: The missile-UAV situation information is as follows: （3）； in, Represents the displacement of the missile relative to the drone, also known as the line of sight; , and Represents the position of the missile in three-dimensional space; , and Represents the position of the drone in three-dimensional space; The distance between the missile and the drone is: （4）； The missile's relative speed to the drone for: （5）； In step 3.2: The line of sight change rate for: （6）； The line of sight angular velocity for: （7）； The line of sight angular velocity size: （8）； The line of sight azimuth Expressed as: （9）； in, Indicates missile coordinate; Indicates missile coordinate; Indicates drone coordinate; Indicates drone coordinate; The aircraft lead angle in the horizontal plane for: （10）； in, Indicates the yaw angle of the drone; The missile entry angle in the horizontal plane for: （11）； in, Indicates the missile's yaw angle; In step 3.3: The rewards for the process are as follows: （12）； in, is the initial distance between the missile and the UAV; is the distance between the missile and the UAV at the current moment; is the rate of change of the line of sight; is the maximum absolute value of the rate of change of the line of sight; is the weight coefficient of each reward, i is 1, 2, 3, 4, 5; A bonus for the distance between missiles and aircraft; A reward for the missile and aircraft distance change rate; Bonus for the drone's front angle; is the absolute value of the aircraft lead angle; reward for missile entry corner; is the absolute value of the missile entry angle; The purpose of the flight altitude reward is to prevent the drone from exceeding the altitude limit; The flight altitude of the drone; Give reward values ​​for each step of the process; Result reward function as follows: （13）； in, Indicates the reward when the drone successfully avoids the attack; correspondingly, Indicates the reward obtained by being hit by a missile; Total reward function for: （14）。 2. The method for avoiding drones based on deep reinforcement learning according to claim 1, characterized in that: In step 1, a six-degree-of-freedom system is used to describe the three-dimensional UAV model. The three-dimensional kinematic model of the UAV is expressed as follows: （1）； in, Indicates flight speed The projected components on the coordinate axes in the three-dimensional inertial coordinate system; is the gravitational acceleration constant; Characterize tangential overload; Indicates normal overload; represents the roll angle, which is used as a control input to adjust the degree of rotation of the body around the velocity vector; Indicates the pitch angle; represents the yaw angle, then Indicates the change in yaw angle; Indicates the change in the speed of the drone; Indicates the speed of the drone; Indicates the pitch angle change.

3. The method for avoiding drones based on deep reinforcement learning according to claim 2, characterized in that: In step 2, the three-dimensional control instructions of tangential overload, normal overload and roll angle are used. Parameterize the UAV's maneuvers and directly reflect the physical constraints in the control instructions through normalization. By inputting the discretized maneuvering instructions into the three-dimensional kinematic model and solving formula (1), the position and attitude state parameters of the UAV can be calculated in real time, which can be expressed as: （2）； Where, represents the maneuver selected by our UAV during the decision cycle; represents the state transition function; Indicates the system status at the current moment; Indicates the system state at the next moment; Using the Euler numerical integration method, the aircraft speed, three-dimensional coordinates, pitch angle and yaw angle of formula (1) are used to calculate the system state at the current moment. The system state at the next moment is calculated iteratively by formula (2) .

4. The method for avoiding drones based on deep reinforcement learning according to claim 3, characterized in that: In step 2, the seven basic maneuvers include constant speed, acceleration, deceleration, left turn, right turn, pull-up, and dive.

5. The method for avoiding drones based on deep reinforcement learning according to claim 1, characterized in that: Specifically, step 4 includes the following steps: the drone's avoidance strategy is trained primarily using a proximal strategy optimization algorithm; the missile adopts an intelligent guidance strategy, and the drone performs avoidance training through the algorithm; the first 1,000 rounds of training are set as an exploration phase, without updating network parameters, to accumulate diverse state-action experience samples, and then the network parameters are updated normally; and ultimately, a composite avoidance strategy combining energy management and spatial maneuvering is implemented until a high-quality avoidance strategy is obtained.

Images