Cooperative trajectory planning method for decoy penetration aircraft based on deep reinforcement learning
By constructing an aircraft trajectory planning model based on deep reinforcement learning, the problem of trajectory planning in dynamic environments using traditional methods is solved. This enables rapid trajectory generation and collaborative planning of aircraft in dynamic threat areas, improving mission success rate and real-time responsiveness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SICHUAN UNIV
- Filing Date
- 2023-11-23
- Publication Date
- 2026-05-19
AI Technical Summary
Traditional aircraft trajectory planning methods are difficult to achieve effective trajectory planning in dynamic environments, especially in penetration missions with high real-time requirements. They lack exploration capabilities and cannot meet the needs of collaborative trajectory planning for multiple aircraft.
A deep reinforcement learning-based approach is used to construct an aircraft motion trajectory model and a Markov decision model. A reward function is designed, and a deep deterministic policy gradient algorithm based on an actor-commentator network framework is used to generate the aircraft's action policy. The heading angle changes of the penetration aircraft and the decoy aircraft are planned collaboratively, and their positions are updated to generate the trajectory.
It improves the trajectory generation speed and success rate of aircraft in dynamic threat areas, enhances the real-time responsiveness and adaptability of aircraft, improves the success rate of multi-aircraft collaborative missions, and is applicable to various aircraft types.
Smart Images
Figure CN117784807B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of aircraft path planning technology, and more specifically, to a cooperative trajectory planning method for decoy penetration aircraft based on deep reinforcement learning. Background Technology
[0002] Effective threat avoidance by aircraft is crucial for successful combat. As penetration missions become increasingly complex, the demands on aircraft control are also rising, leading to the development of multi-aircraft cooperative trajectory planning technology. Traditional trajectory planning methods, based on inference planning and biomimetic optimization, lack exploratory capabilities and struggle to solve trajectory planning problems in dynamic environments and those with high real-time requirements. Summary of the Invention
[0003] The present invention provides a cooperative trajectory planning method for decoy penetration aircraft based on deep reinforcement learning, which can solve the above-mentioned problems.
[0004] To solve the above problems, the technical solution adopted by the present invention is as follows:
[0005] A cooperative trajectory planning method for decoy penetration aircraft based on deep reinforcement learning includes:
[0006] Construct and simplify the flight trajectory models of penetration aircraft and decoy aircraft;
[0007] A Markov decision model for the aircraft penetration process is constructed, including a state-space model, an action-space model, and a reward function, wherein the reward function is designed based on the threat region.
[0008] A deep deterministic policy gradient algorithm model is constructed based on the actor-critic network penetration framework. Using the states of the penetrating aircraft and the decoy aircraft as input, the algorithm calculates the action policy. Based on the Markov decision model of the aircraft penetration process, the action policy is mapped to the change in heading angle of the penetrating aircraft and the decoy aircraft.
[0009] Change in heading angle By substituting a simplified aircraft motion trajectory model, the aircraft's motion is obtained, and then the aircraft's position is updated to obtain the motion trajectories of the penetration aircraft and the decoy aircraft.
[0010] In a preferred embodiment of the present invention, the flight trajectory models of the penetration aircraft and the decoy aircraft are the same, both being:
[0011]
[0012] Where x, y, and z represent the coordinate components in the coordinate system, and v represents the velocity vector of the aircraft.x v y v z These represent the velocity components of the aircraft along the three coordinate axes, where g represents gravitational acceleration and β is the trajectory inclination angle. φ is the heading angle, φ is the roll angle, and n x and n z These are the tangential and normal overloads of the aircraft, respectively.
[0013] Assuming the penetrating aircraft and decoy aircraft penetrate the defense in the oxy plane, with the aircraft trajectory inclination angle β = 0° and roll angle φ = 0°, the aircraft trajectory model can be simplified as follows:
[0014]
[0015] In a preferred embodiment of the present invention, the state-space model is defined as:
[0016]
[0017] Among them, o TF (t) represents the location information of the penetrating aircraft, o YE (t) represents the position information of the decoy aircraft, and l(t) represents the distance between the penetrating aircraft and the decoy aircraft and the target area. The heading angle of the aircraft itself; o TF (t)=[x T ( F ),ty T ( F [] represents the coordinate components of the penetrating aircraft at time t, o YE (t)=[x YE (t),y YE [l(t)] represents the coordinate components of the decoy aircraft at time t, l(t) = [l TF (t),l YE [(t)], and:
[0018]
[0019]
[0020] Where, x g and y g These are the x and y coordinates of the center point of the target area, respectively.
[0021] In a preferred embodiment of the present invention, the action space model is as follows:
[0022]
[0023] In the formula, The change in heading angle of the penetrating aircraft, This refers to the change in the heading angle of the decoy aircraft. The change in heading angle of the penetrating aircraft and the decoy aircraft within two adjacent simulation time steps is given. Substituting the change in heading angle into the simplified aircraft motion trajectory model, the positions of the penetrating aircraft and the decoy aircraft at the next moment can be updated.
[0024] In a preferred embodiment of the present invention, the reward function is:
[0025] reward(t) = r1 + r2 + r3 + r4 + r5;
[0026] The mission completion conditions for the penetration aircraft are set as follows:
[0027]
[0028] In the formula, t is a discrete value, 1s represents a simulation step size, maxT is the maximum time per round, and d(t) represents the distance from the penetrating aircraft to the center of the target area;
[0029] r1 is the reward function for the penetrating aircraft reaching the target area.
[0030]
[0031] Once the penetrating aircraft reaches the target area, set the end flag done=1 to end the training round;
[0032] r2 is the reward function used to guide the penetrating aircraft towards the target area and avoid the threat area.
[0033]
[0034] In the formula, This represents the angle between the center of mass of the penetrating aircraft at time t and the mission direction and true north. Indicates flight direction and mission direction The deviation, c1 is the coefficient to ensure the rationality of the reward function;
[0035] r3 is the reward function when both the penetrating aircraft and the decoy aircraft are in the threat zone.
[0036]
[0037] In the formula, R is the radius of the threat zone, and r YE (t) represents the distance between the decoy aircraft and the center of the threat zone, and λ is the weighting coefficient;
[0038] r4 is the collision reward between penetration aircraft and decoy aircraft.
[0039]
[0040] Among them, D min Let D be the minimum safe distance between the penetration aircraft and the decoy aircraft.
[0041]
[0042] r5 is the reward function when the penetration aircraft successfully completes its mission.
[0043] r5 = -c2d(t)
[0044] In the formula, d(t) is the distance from the penetrating aircraft to the target area, and c2 is a coefficient to ensure the rationality of the reward function.
[0045] In a preferred embodiment of the present invention, the deep deterministic policy gradient algorithm model includes an Actor network, a Critic network, an Actor target network, and a Critic target network, wherein the action policy a = μ(s|θ) is... μ This is mapped to the change in heading angle of the penetrating aircraft and the decoy aircraft. During the process, the Actor network produces the maximum Q(s,a|θ) through its output. q The Critic network updates the target Q(s,a|θ) with action a as the objective, in order to accurately evaluate the output Q(s,a|θ). q The target network is updated using a soft update method, where Q(s,a|θ) is the target network. q () indicates that in state s, after selecting action a, based on network parameters θ q The long-term value generated.
[0046] Compared with the prior art, the beneficial effects of the present invention are:
[0047] 1) The trajectory planning method based on deep reinforcement learning can generate trajectories for penetrating aircraft and decoy aircraft relatively quickly while dealing with dynamic threat areas.
[0048] 2) By constructing flight trajectory models for penetration aircraft and decoy aircraft, and a reward function designed based on threat areas, it is helpful to improve the identification and avoidance of threats by penetration aircraft, thereby increasing the success rate of penetration missions and reducing potential risks.
[0049] 3) By employing deep reinforcement learning with an actor-critic framework, aircraft can collaboratively plan trajectories to better achieve penetration targets. This helps penetration aircraft work better with decoy aircraft, increasing the chances of mission success.
[0050] 4) Deep reinforcement learning methods allow systems to learn and adjust strategies in real-time environments to cope with constantly changing threats and environmental conditions, which improves the real-time responsiveness of the system.
[0051] 5) The actor-critic framework method and the use of deep neural networks enable the system to adapt to different situations, which increases the system's ability to cope with complex environments.
[0052] 6) This method can be applied to various types of aircraft, not just specific models or brands, making it more widely applicable.
[0053] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, embodiments of the present invention are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0054] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0055] Figure 1 A diagram illustrating the relationships within a breakthrough scenario;
[0056] Figure 2 The flowchart of the decision-making process algorithm for penetrating decoy aircraft. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.
[0058] This invention proposes a cooperative trajectory planning method for decoy penetration aircraft based on deep reinforcement learning, as detailed below:
[0059] 1. Constructing a model of the aircraft's motion trajectory
[0060] The trajectory models for penetration aircraft and decoy aircraft are the same, as follows:
[0061]
[0062] Where x, y, and z represent the coordinate components in the coordinate system, and v represents the velocity vector of the aircraft. x v y v zThese represent the velocity components of the aircraft along the three coordinate axes, where g represents gravitational acceleration and β is the trajectory inclination angle. φ is the heading angle, φ is the roll angle, and n x and n z These are the tangential overload and normal overload of the aircraft, respectively.
[0063] Assuming that the penetrating aircraft and the decoy aircraft are penetrating in the oxy plane, with the aircraft trajectory inclination angle β = 0° and the roll angle φ = 0°, the aircraft motion trajectory model can be simplified to the following form:
[0064]
[0065] In addition, the flight path of an aircraft is limited by the minimum track length and the maximum turning radius.
[0066] 2. Construct a Markov decision model for the aircraft penetration process, including a state-space model, an action-space model, and a reward function.
[0067] 1) Constructing a state-space model
[0068] For the scenario of decoy aircraft penetrating defenses, the state-space model is defined as:
[0069]
[0070] This includes the location information of penetrating aircraft. TF (t), decoy aircraft position information o YE (t), the distance l(t) between the penetrating aircraft and the decoy aircraft and the target area, and the self-flight heading angle. o TF (t)=[x TF (t),y TF [(t)] represents the coordinate components of the penetrating aircraft at time t, o YE (t)=[x YE (t),y YE [l(t)] represents the coordinate components of the decoy aircraft at time t. l(t) = [l TF (t),l YE [t], calculate as follows:
[0071]
[0072]
[0073] In the formula, x g and y g These are the x and y coordinates of the center point of the target area, respectively.
[0074] 2) Constructing the action space model
[0075] Based on the definition of control variables in flight motion models, the action space model is designed to include the change in heading angle of the penetrating aircraft. Change in heading angle of the decoy aircraft The specific form is as follows:
[0076]
[0077] In the formula, The change in heading angle of the penetrating aircraft and the decoy aircraft within two adjacent simulation time steps is represented by the change in heading angle. Substituting the change in heading angle into the simplified aircraft motion trajectory model, the positions of the penetrating aircraft and the decoy aircraft at the next moment can be updated.
[0078] 3) Reward function design
[0079] The objective of decoy-based aircraft penetration is to allow the penetrating aircraft to reach a designated target area, with target area being point A. T The conditions for a penetration aircraft mission to complete are designed as follows: (The original text appears to be incomplete and contains several errors. A more accurate translation would require the full context.)
[0080]
[0081] In the formula, t is a discrete value, 1s represents a simulation step size, maxT is the maximum time per round, and d(t) represents the distance from the penetrating aircraft to the center of the target area.
[0082] The reward for a penetration aircraft reaching the target area is set as follows:
[0083]
[0084] Once the aircraft reaches the target area, the end flag "done=1" is set, and the training round ends.
[0085] Based on the mission completion conditions of the penetration aircraft, the following reward function is designed to guide the penetration aircraft to the target area and avoid threat areas:
[0086]
[0087] In the formula, This represents the angle between the center of mass of the penetrating aircraft at time t and the mission direction and true north. Indicates flight direction and mission direction The closer the aircraft gets to the target area, the higher the reward. Entering the threat zone incurs continuous penalties. c1 is a coefficient to ensure the rationality of the reward function.
[0088] Figure 1 For the breakthrough scenario relationship diagram, based on Figure 1 It can be seen that when Angle range in The aircraft moved toward the target area, and The closer d(t) is to 0 rad, the smaller d(t) is, the easier it is for the aircraft to reach the target area.
[0089] The greater the attraction of the decoy aircraft to the target, the higher the probability of the penetration aircraft successfully penetrating the defenses. To ensure that the penetration aircraft reaches the designated target area with the shortest possible detour, based on the decoy aircraft's attraction to the threat zone, the reward function is set as follows when both the penetration aircraft and the decoy aircraft are in the threat zone:
[0090]
[0091] In the formula, R is the radius of the threat zone, and r YE (t) represents the distance between the decoy aircraft and the center of the threat zone, and λ is the weighting coefficient. The closer the decoy aircraft is to the center of the threat, the greater its attraction to the target. In addition, the longer it stays in the threat zone, the greater the probability of being attacked. The reward decays over time, and when it exceeds a certain time step, it is considered to have been attacked.
[0092] Let D min The minimum safe distance between the penetration aircraft and the decoy aircraft is less than distance D. min A collision will be considered to have occurred. The collision reward between the penetrating aircraft and the decoy aircraft is defined as follows:
[0093]
[0094] D is the distance between the penetration aircraft and the decoy aircraft, as shown in the following formula:
[0095]
[0096] When a penetration aircraft successfully completes its mission, it receives a reward; the longer the flight distance, the smaller the reward. The reward function is defined as follows:
[0097] r5 = -c2d(t)
[0098] In the formula, d(t) is the distance from the penetrating aircraft to the target area, and c2 is a coefficient to ensure the rationality of the reward function.
[0099] The final single-step simulation reward is the sum of the above rewards:
[0100] reward(t) = r1 + r2 + r3 + r4 + r5
[0101] 3. Solving the aircraft penetration model based on deep reinforcement learning
[0102] Reinforcement learning seeks the optimal policy in a given Markov decision process. This primarily involves using the fitting capabilities of deep neural networks to model the transformation from state s to action value function q, given the state s. π (s,a) or state value function v π The mapping relationship between (s,a).
[0103] The Deep Deterministic Policy Gradient (DDPG) algorithm is based on the actor-critic breakthrough framework and is obtained by improving the deep Q network. The DDPG algorithm consists of four neural networks (Actor network, Critic network, Actor target network, and Critic target network).
[0104] The network parameters of the Actor network and the target Actor network are defined as θ. μ and After inputting the current state and the next state into the network, the outputs are represented as μ(s|θ). μ )and That is, the actions at the current moment and the next moment are denoted as a. t =μ(s|θ) μ )and Where μ(s|θ) μ ) indicates that in the current state s, according to the network policy θ μ The resulting action is a continuous value between [-1, 1].
[0105] The network parameters of the Critic network and the target Critic network are defined as θ. q and Network input s t a t and s t+1 a t+1 The resulting profit value is Q(s,a|θ) q )and Where Q(s,a|θ) q () indicates that in state s, after selecting action a, based on network parameters θ q The long-term value generated.
[0106] The Actor network produces the maximum Q(s,a|θ) at its output. q The action 'a' is used as the target for updating, and the Critic outputs Q(s,a|θ) accurately. q The function is used to update the target network, which is used to stabilize the training process and is updated using a soft update method. Based on the Markov decision model of the aircraft penetration process, μ(s|θ) is used to update the target network. μ This is mapped to the change in heading angle of the penetrating aircraft and the decoy aircraft.
[0107] The Deep Deterministic Policy Gradient (DDPG) algorithm utilizes an experience replay mechanism. An experience pool is used to store state information (S) of the penetrating and decoy aircraft acquired from the environment. t ,a t ,r t ,s t+1 ), where s represents t Current state information, a t Indicates the action at the current moment, r t Indicates an immediate reward, s t+1 Indicates that in state s t and action a t The state at the next moment is obtained through interaction with the environment. An experience replay mechanism is employed, randomly selecting a small batch of data from the experience pool to train the neural network each time. This breaks down the correlation between data points, increases the stability of the sample data, and avoids data being discarded after use, thus increasing the utilization rate of the sample data.
[0108] The loss function value of the DDPG algorithm is defined as follows:
[0109] Loss=(y t -Q(s t ,a t |θ q )) 2
[0110] In the formula, y t The cumulative return obtained over a long period is expressed as follows:
[0111]
[0112] The done flag is the end indicator; done = 1 when the state is terminated, and 0 when the state is otherwise terminated.
[0113] If the number of samples in the batch sampling in the experience pool is set to , then the loss function of the Critic network can be expressed as
[0114]
[0115] The Critic network updates by minimizing the loss value, and the gradient formula for the update is:
[0116]
[0117] Actor networks are used to find the maximum Q(s,a|θ). q Action a is updated using gradient ascent, and its update formula is:
[0118]
[0119] The DDPG algorithm uses both an Actor target network and a Critic target network to make the training process more stable. The target network is updated using a soft update method, which is as follows:
[0120]
[0121]
[0122] τ represents the trajectory strategy during the exploration process, and its value is usually τ∈(0,1).
[0123] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A cooperative trajectory planning method for decoy penetration aircraft based on deep reinforcement learning, characterized in that, include: Construct and simplify the flight trajectory models of penetration aircraft and decoy aircraft; A Markov decision model for the aircraft penetration process is constructed, including a state-space model, an action-space model, and a reward function. The reward function is designed based on the threat region. The reward function is as follows: ； The mission completion conditions for the penetration aircraft are set as follows: In the formula, These are discrete values, where 1 second represents a simulation step. The maximum time for each round, Indicates the distance from the penetrating aircraft to the center of the target area; This represents the distance vector between the penetrating aircraft and the decoy aircraft and the target area. The reward function for penetrating aircraft reaching the target area. After the penetrating aircraft reaches the target area, it sets an end marker. The training round is over; This is a reward function used to guide penetrating aircraft towards the target area and avoid threat zones. In the formula, express The angle between the center of gravity of the attack aircraft and the mission direction and true north should be constantly monitored. The heading angle of the aircraft itself. Indicates flight direction and mission direction deviation, The coefficients are used to ensure the rationality of the reward function; Let the reward function be defined when both the penetrating aircraft and the decoy aircraft are within the threat zone. In the formula, The radius of the threat zone. The distance between the decoy aircraft and the center of the threat zone. These are the weighting coefficients; The reward is for collisions between penetration aircraft and decoy aircraft. in, To determine the minimum safe distance between the penetration aircraft and the decoy aircraft, The distance between the penetration aircraft and the decoy aircraft. A reward function for a penetrating aircraft that successfully completes its mission. Indicates that the penetrating aircraft is The coordinate components at time, Indicates that the decoy aircraft is in The coordinate components at time; In the formula, The coefficients are used to ensure the rationality of the reward function; A deep deterministic policy gradient algorithm model is constructed based on the actor-critic network penetration framework. Using the states of the penetrating aircraft and the decoy aircraft as input, the algorithm calculates the action policy. Based on the Markov decision model of the aircraft penetration process, the action policy is mapped to the change in heading angle of the penetrating aircraft and the decoy aircraft. ; Change in heading angle By substituting a simplified aircraft motion trajectory model, the aircraft's motion is obtained, and then the aircraft's position is updated to obtain the motion trajectories of the penetration aircraft and the decoy aircraft.
2. The method for cooperative trajectory planning of decoy penetration aircraft based on deep reinforcement learning according to claim 1, characterized in that, The trajectory models of penetration aircraft and decoy aircraft are the same, both being: in, Represents the coordinate components in the coordinate system. Represents the velocity vector of the aircraft. These represent the velocity components of the aircraft on the three coordinate axes. Represents gravitational acceleration. For the inclination angle of the flight path, For heading angle, For roll angle, and These are the tangential and normal overloads of the aircraft, respectively. Assuming the penetration aircraft and the decoy aircraft are in To penetrate defenses in a plane, the aircraft's trajectory tilt angle at this time... Roll angle The aircraft trajectory model is then simplified to: 。 3. The method for cooperative trajectory planning of decoy penetration aircraft based on deep reinforcement learning according to claim 2, characterized in that, The state-space model is defined as: in, To obtain the location information of the penetrating aircraft, For the location information of the decoy aircraft, , , ,and: in, and These are the x and y coordinates of the center point of the target area, respectively.
4. The method for cooperative trajectory planning of decoy penetration aircraft based on deep reinforcement learning according to claim 3, characterized in that, The action space model is as follows: In the formula, The change in heading angle of the penetrating aircraft, This refers to the change in the heading angle of the decoy aircraft. The change in heading angle of the penetrating aircraft and the decoy aircraft within two adjacent simulation time steps is given. Substituting the change in heading angle into the simplified aircraft motion trajectory model, the positions of the penetrating aircraft and the decoy aircraft at the next moment can be updated.
5. The method for cooperative trajectory planning of decoy penetration aircraft based on deep reinforcement learning according to claim 4, characterized in that, The deep deterministic policy gradient algorithm model includes an Actor network, a Critic network, an Actor-target network, and a Critic-target network, which integrates action policies. Mapped to the change in heading angle of the penetrating aircraft and the decoy aircraft During the process, the Actor network produces the maximum output. action The Critic network is updated to accurately evaluate the output in order to achieve the target. To update the targets, both target networks are updated using a soft update method. Representing state Next, select an action. Then based on network parameters The long-term value generated; Indicates the current state According to network policy The resulting actions.