Autonomous Evasion Maneuver Decision-making Method for UAV Air Combat Based on Deep Reinforcement Learning

Abstract

The present invention discloses a method for autonomous evasive maneuvering decision-making in unmanned aerial vehicle (UAV) air combat based on deep reinforcement learning, comprising the following steps: determining multiple evasive maneuvering tactical requirements for UAVs; establishing a three-dimensional UAV-missile pursuit model; obtaining battlefield states at three consecutive moments t-1, t, and t+1; inputting multiple tactical requirements, preset weights corresponding to each tactical requirement, and the battlefield states at three consecutive moments t-1, t, and t+1 into a pre-trained LSTM-Dueling DDQN, determining the UAV's maneuvering control parameters at moment t+1 using a deep reinforcement learning algorithm; and inputting the maneuvering control parameters into the three-dimensional UAV-missile pursuit model to obtain the battlefield state at moment t+2. The evasive maneuvering strategy derived from the above-mentioned autonomous evasive maneuvering decision-making method for UAVs in air combat can reflect different tactical requirements for air combat while ensuring the safety of the UAV itself, thereby increasing the success probability of the attack mission.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a method for autonomous evasive maneuvering decision-making in air combat of unmanned aerial vehicles (UAVs) based on deep reinforcement learning. Background Art

[0002] With the development of drone technology, drones, with their unique performance advantages, are becoming a vital component of future air combat. The core principle of air combat is to defeat the enemy and preserve oneself. Ensuring drone safety is the foundation and prerequisite for achieving all air combat missions. In air combat environments, drones often face the threat of air-to-air missiles. The ability to reduce the lethality of incoming air-to-air missiles through evasive maneuvers and ultimately escape the threat is crucial to improving drone survivability.

[0003] In the relevant technology, methods for solving drone evasion strategies mainly include: numerical analysis, optimal control, model prediction, and differential game methods. With the development of artificial intelligence technology, intelligent algorithms are also beginning to be used to solve drone evasion maneuvers. Based on the above methods, it is possible to obtain optimal or suboptimal solutions for drones under certain conditions. However, the existing technology only considers the miss margin target, that is, only considers the safety of the drone, and ignores the impact of evasive maneuvers on the drone's overall air combat mission. Air combat confrontation is usually a continuous process involving multiple rounds of missile counterattacks. Therefore, the evasive maneuvers of drones should comprehensively consider the overall air combat effectiveness. That is, while maintaining a certain level of survivability, the tactical advantage should be maintained as much as possible to increase the success rate of the attack mission. Summary of the Invention

[0004] To address the aforementioned issues in the prior art, the present invention provides a method for autonomous evasive maneuvering decision-making in UAV air combat based on deep reinforcement learning. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0005] The present invention provides a method for autonomous evasive maneuvering decision-making in air combat of unmanned aerial vehicles based on deep reinforcement learning, comprising:

[0006] Determine the tactical requirements for various drone evasive maneuvers;

[0007] Establish a three-dimensional space pursuit model for drones and missiles;

[0008] Obtain the battlefield status at three consecutive moments t-1, t and t+1, including the relative distance r between the UAV and the missile, the flight altitude y of the UAV, and the distance between the UAV and the missile. u , horizontal distance r between the UAV and the missile xz , line of sight angle change rate and the rate of change of sight angle

[0009] The multiple tactical requirements, the preset weights corresponding to each tactical requirement, and the battlefield states at the three consecutive times t-1, t, and t+1 are input into a pre-trained LSTM-Dueling DDQN. The deep reinforcement learning algorithm is used to determine the maneuver control parameters of the UAV at time t+1.

[0010] The maneuver control parameters are input into the UAV-missile three-dimensional space pursuit model to obtain the battlefield state at time t+2.

[0011] In one embodiment of the present invention, the step of establishing a three-dimensional UAV-missile pursuit model includes:

[0012] According to the UAV motion model, UAV constraint model, missile motion model, missile guidance command response model and missile constraint model, the UAV-missile three-dimensional space pursuit model is determined.

[0013] In one embodiment of the present invention, the UAV motion model is:

[0014]

[0015] x u (t0) = x u0

[0016] Among them, the state vector x u =[x u ,y u ,z u ,v u ,θ u ,ψ u ] T , x u0 Represents the initial state vector of the drone, x u 、z u Indicates the horizontal coordinate of the drone in the geographic coordinate system, y u Indicates the flight altitude of the drone, v u represents the speed of the drone, θ u represents the track inclination of the UAV, ψ u Represents the track deviation angle of the UAV; the UAV maneuver control vector u=[n xu ,n yu ,n zu ] T , where n xu 、n yu 、n zu Respectively represent the control instructions of the UAV in terms of speed, pitch angle and yaw direction;

[0017] The UAV constraint model is:

[0018]

[0019] Among them, n xc 、n yc 、n zc Actual control overload for the UAV; n xmax 、n umax They represent the maximum available overload of the UAV in the tangential and normal directions, v umin Indicates the minimum speed limit of the drone, v umax Indicates the maximum speed limit of the drone, y umin Indicates the minimum height limit of the drone, y umax Indicates the maximum altitude limit of the drone.

[0020] In one embodiment of the present invention, the missile motion model is:

[0021]

[0022] x m (t0) = x m0

[0023] Among them, the state vector x m =[x m ,y m ,z m ,v m ,θ m ,ψ m ] T , x m0 is the initial state vector of the missile, x m 、y m 、z m represents the coordinates of the missile in the geographic coordinate system, v m represents the missile's velocity, θ m represents the missile's track inclination, ψ m Indicates the missile's track deviation angle;

[0024] The guidance command response model of the missile is:

[0025]

[0026]

[0027] Among them, n myc 、n mzc They represent the pitch and yaw command overloads generated by the proportional guidance law, τ m represents the time constant of the missile guidance control system;

[0028] The missile constraint model is:

[0029]

[0030] Among them, n mmax is the maximum normal overload of the missile.

[0031] In one embodiment of the present invention, the UAV-missile three-dimensional space pursuit model is:

[0032]

[0033]

[0034]

[0035]

[0036] Among them, x r =x u -x m ,y r =y u -y m , z r =z u -z m ,

[0037] In one embodiment of the present invention, the tactical requirements include miss distance, total energy consumption of the drone, and terminal avoidance advantage of the drone.

[0038] In one embodiment of the present invention, the reward function of the deep reinforcement learning algorithm includes dense rewards and sparse rewards;

[0039] During the training process of LSTM-Dueling DDQN, the proportion of dense rewards gradually decreases with the increase of iteration number, and the proportion of sparse rewards increases with the increase of iteration number.

[0040] In one embodiment of the present invention, the reward function of the deep reinforcement learning algorithm is R t =r t +e -kn s t ; Among them, e -kn represents the temporal degradation factor, k is a constant used for the decay rate of dense rewards, n represents the current number of iterations, r t represents the total sparse reward of each tactical requirement at time t, s t represents the dense reward at time t, R t represents the total reward at time t.

[0041] In one embodiment of the present invention, the avoidance cutoff condition of the drone includes an avoidance failure condition and an avoidance success condition.

[0042] In one embodiment of the present invention, the avoidance failure condition is: r(t)≤r d ,v m (t)≥v mmin and t≤t max , the successful avoidance condition is: t≥t max or v m (t) <v mmin ; Among them, r(t) represents the relative distance between the UAV and the missile at time t, v m (t) represents the velocity of the missile at time t, v mmin represents the minimum controlled speed of the missile, t max Indicates the energy working time on the missile, r d Indicates the missile damage radius.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention comprehensively considers multiple tactical requirements and establishes a tactical requirement-coupled UAV autonomous avoidance maneuver strategy generation method model. The LSTM-Dueling DDQN is used to solve the UAV autonomous avoidance maneuver strategy generation problem. Without relying on human expert experience, it can achieve autonomous mapping from state to avoidance strategy, has good convergence speed and learning effect, and the derived avoidance maneuver strategy can reflect different air combat tactical requirements while ensuring the safety of the UAV itself, thereby improving the success probability of the attack mission.

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of a method for autonomous evasive maneuvering decision-making in UAV air combat based on deep reinforcement learning provided by an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of a three-dimensional space pursuit model of a UAV and a missile provided by an embodiment of the present invention;

[0048] Figure 3 This is a structural diagram of the LSTM-Dueling DDQN provided by an embodiment of the present invention;

[0049] Figure 4 is a Q estimation value comparison curve provided by an embodiment of the present invention;

[0050] Figure 5is a comparison curve of cumulative reward values provided by an embodiment of the present invention;

[0051] Figure 6 This is a comparison diagram of air combat avoidance maneuver trajectories when the tactical requirement is a miss distance, provided by an embodiment of the present invention;

[0052] Figure 7 This is a comparison diagram of air combat avoidance maneuver trajectories when the tactical requirement is energy consumption, provided by an embodiment of the present invention;

[0053] Figure 8 This is a comparison diagram of air combat evasive maneuver trajectories when the tactical requirement is terminal advantage, provided by an embodiment of the present invention;

[0054] Figure 9 These are the terminal values of the tactical requirement functions of the three tactical requirements provided by the embodiment of the present invention under different weights. DETAILED DESCRIPTION

[0055] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0056] Figure 1 This is a flow chart of a method for autonomous evasive maneuvering decision-making in UAV air combat based on deep reinforcement learning provided by an embodiment of the present invention. Figure 1 The embodiment of the present invention provides a method for autonomous evasive maneuver decision-making in air combat of a UAV based on deep reinforcement learning, comprising:

[0057] S1. Determine the tactical requirements for various drone evasive maneuvers;

[0058] S2. Establish a three-dimensional space pursuit model for drones and missiles;

[0059] S3. Obtain the battlefield status at three consecutive moments t-1, t, and t+1. The battlefield status includes the relative distance r between the UAV and the missile, the flight altitude y of the UAV, and the distance between the UAV and the missile. u , horizontal distance r between the UAV and the missile xz , line of sight angle change rate and the rate of change of sight angle

[0060] S4. Input multiple tactical requirements, the preset weights corresponding to each tactical requirement, and the battlefield status at three consecutive times t-1, t, and t+1 into the pre-trained LSTM-Dueling DDQN, and use the deep reinforcement learning algorithm to determine the maneuver control parameters of the UAV at time t+1;

[0061] S5. Input the maneuver control parameters into the UAV-missile three-dimensional space pursuit model to obtain the battlefield status at time t+2.

[0062] In this embodiment, the tactical requirements include miss distance, total UAV energy consumption, and UAV terminal avoidance advantage. Specifically, MD (Miss Distance) refers to the minimum distance between the UAV and the missile. Therefore, maximizing MD means maximizing the UAV's survival probability. Minimizing EC (Energy Consumption) can provide energy advantage for subsequent air combat confrontation processes, and maximizing TS (Terminal Superiority) can provide an initial tactical advantage for the next round of air combat confrontation.

[0063] In addition, the battlefield status in step S5 includes the relative distance between the UAV and the missile at time t+2, the flight altitude of the UAV, the horizontal distance between the UAV and the missile, the rate of change of the line of sight inclination angle, and the rate of change of the line of sight deflection angle.

[0064] Figure 2 This is a schematic diagram of a drone-missile three-dimensional space pursuit model provided by an embodiment of the present invention. Optionally, in step S2 above, the step of establishing a drone-missile three-dimensional space pursuit model includes:

[0065] According to the UAV motion model, UAV constraint model, missile motion model, missile guidance command response model and missile constraint model, the UAV-missile three-dimensional space pursuit model is determined.

[0066] Optionally, the drone motion model is:

[0067]

[0068] x u (t0) = x u0

[0069] Among them, the state vector x u =[x u ,y u ,z u ,v u ,θ u ,ψ u ] T , x u0 Represents the initial state vector of the drone, x u 、z u Indicates the horizontal coordinate of the drone in the geographic coordinate system, y u Indicates the flight altitude of the drone, v u represents the speed of the drone, θ u represents the track inclination of the UAV, ψ u Represents the track deviation angle of the UAV; the UAV maneuver control vector u=[n xu ,n yu ,n zu ]T , where n xu 、n yu 、n zu Respectively represent the control instructions of the UAV in terms of speed, pitch angle and yaw direction;

[0070] The UAV constraint model is:

[0071]

[0072] Among them, n xc 、n yc 、n zc Actual control overload for the UAV; n xmax 、n umax They represent the maximum available overload of the UAV in the tangential and normal directions, v umin Indicates the minimum speed limit of the drone, v umax Indicates the maximum speed limit of the drone, y umin Indicates the minimum height limit of the drone, y umax Indicates the maximum altitude limit of the drone.

[0073] The missile motion model is:

[0074]

[0075] x m (t0) = x m0

[0076] Among them, the state vector x m =[x m ,y m ,z m ,v m ,θ m ,ψ m ] T , x m0 is the initial state vector of the missile, x m 、y m 、z m represents the coordinates of the missile in the geographic coordinate system, v m represents the missile's velocity, θ m represents the missile's track inclination, ψ m Indicates the missile's track deviation angle;

[0077] The missile's guidance command response model is:

[0078]

[0079]

[0080] Among them, n myc 、nmzc They represent the pitch and yaw command overloads generated by the proportional guidance law, τ m represents the time constant of the missile guidance control system;

[0081] The missile constraint model is:

[0082]

[0083] Among them, n mmax is the maximum normal overload of the missile.

[0084] Furthermore, in order to calculate the control instructions of the missile, it is necessary to establish a relative motion model between the UAV and the missile. The relative motion state of the missile and the UAV is generally taken as [r,ε m ,β m ], the UAV-missile three-dimensional space pursuit model is:

[0085]

[0086]

[0087]

[0088]

[0089] Among them, x r =x u -x m ,y r =y u -y m , z r =z u -z m ,

[0090] It should be understood that the state space in deep reinforcement learning can be represented by several relative motion features during the UAV avoidance process. Considering the requirements of the avoidance maneuver tactics and facilitating the description of the entire avoidance process, this embodiment selects the relative distance r between the UAV and the missile, the flight height y of the UAV, and the relative distance r between the UAV and the missile. u , horizontal distance r between the UAV and the missile xz , line of sight angle change rate and the rate of change of sight angle As the state space in deep reinforcement learning, In addition, in order to eliminate the adverse effects of differences in the value ranges of different variables on learning results, it is necessary to normalize each parameter by multiplying the corresponding constant coefficient before each parameter so that the value range of each parameter is roughly within the same range.

[0091] Furthermore, based on the above-mentioned drone-missile three-dimensional space pursuit model, a drone action space is established to implement a strategy for evading missile attacks through different maneuver types and maneuver amplitude combinations. According to the drone constraint model, this embodiment designs 6 basic action types, namely horizontal straight line, horizontal turn, plumb plane climb, plumb plane dive, turn climb and turn dive, and each action type contains multiple amplitude levels. To ensure that the action amplitudes of different maneuver types are within the same range, n is set. xi 、n yi 、n zi , where: n xi ∈[-sin(θ u ),0.5-sin(θ u )],i=1,2,3,n yi ∈[1-cos(θ u ),1.5-cos(θ u )],i=1,2,3,n yi ∈[-cos(θ u ),1-cos(θ u )],i=4,5,6,n zi ∈[-4,4],i=1,2,3,4,5,6.

[0092] In this embodiment, the reward function of the deep reinforcement learning algorithm includes dense rewards and sparse rewards;

[0093] During the training process of LSTM-Dueling DDQN, the proportion of dense rewards gradually decreases with the increase of iteration number, and the proportion of sparse rewards increases with the increase of iteration number.

[0094] In this embodiment, the miss distance MD tactical requirement function is f md (t) = r(t) / r0, where r0 represents the initial distance between the UAV and the missile, and r(t) represents the relative distance between the UAV and the missile at time t. The tactical demand function for minimizing energy consumption EC is: Among them, n xu (t), n yu (t), n zu (t) are the energy consumption of the UAV at time t and the control instructions of the UAV in the speed, pitch and yaw directions. The tactical demand function for maximizing the terminal advantage TS is f ts (t) = y u (t) / y u0 , where y u (t) represents the altitude of the UAV at time t, y u0 Indicates the initial altitude of the drone.

[0095] Since it is difficult for random strategies to obtain rewards by sparse rewards alone in the initial stage of training, it is necessary to guide the agent to learn through dense rewards. For example, s md (t) is the dense reward when the tactical requirement is MD, s ec (t) is the dense reward when the tactical requirement is EC, s ts (t) is the dense reward when the tactical requirement is TS, and is the elimination of the dense reward function s md (t), s ec (t) and s ts (t) The impact of the difference in value range on the reward function. In this embodiment, nine positive coefficients C1, C2, ..., C9 are set, as shown below:

[0096]

[0097] s ec (t) = C5 / f ec (t)-C6ε m (t)

[0098] s ts (t) = C7f ts (t)+C8ε m (t)+C9β m (t)

[0099] Total dense reward s(t) = α1s md (t)+α2s ec (t)+α3s ts (t).

[0100] In order to reflect the impact of different tactical requirements on the generation of autonomous avoidance maneuver strategies of the UAV, in this embodiment, α1, α2, and α3 are weight coefficients of the three tactical requirements, and α1+α2+α3=1. The size of the weight coefficient represents the importance of different tactical requirements.

[0101] Furthermore, the sparse reward design is as follows:

[0102]

[0103] Where r1, r2, and r3 represent the sparse rewards when the tactical requirements are MD, EC, and TS, respectively; r represents the total sparse reward; R d1 、R d2 、R e1 、R e2 、R y1 、R y2 are all constants, indicating the rewards obtained at the end of the round, D, E, Y are constants, r f 、e f 、yuf They represent the relative distance between the drone and the missile, the total energy consumption of the drone, and the flight altitude of the drone at the end of the round.

[0104] Figure 3 This is a structural diagram of the LSTM-Dueling DDQN provided by an embodiment of the present invention. The reward function of the reinforcement learning algorithm includes two parts: dense reward and sparse reward. This embodiment further incorporates the idea of exploration curriculum learning and introduces the time series degradation factor e -kn , that is, the total reward of the reinforcement learning algorithm is R t =r t +e -kn s t , where e -kn represents the temporal degradation factor, k is a constant used for the decay rate of dense rewards, n represents the current number of iterations, r t represents the total sparse reward of each tactical requirement at time t, s t represents the dense reward at time t, R t represents the total reward at time t.

[0105] In the early stages of LSTM-Dueling DDQN training, dense rewards account for a large proportion, and the agent's learning is mainly guided by humans. As the number of iterations n increases, the proportion of sparse rewards gradually increases, and the agent's learning gradually shifts from human guidance to an autonomous exploration strategy.

[0106] For the LSTM-Dueling DDQN algorithm, the long-term reward function is Where R represents the reward function, that is, the agent is in a given state s t , take action a t The resulting returns are discounted by a factor γ to make the long-term returns converge.

[0107] The solution method of the return estimation function is the same as that of the DDQN (Double DQN) algorithm, that is, y t =R+γQ(s t+1 ,argmaxQ(s t+1,a;ω);ω'), by using the estimation network to obtain the action that maximizes the estimated return value function, and then inputting the action into the target network to calculate the estimated return value function, this method comprehensively considers the parameter influence of the two networks, can eliminate the maximization deviation, and improve the stability of the algorithm. The solution method of the state behavior value function is the same as that of the Dueling DDQN algorithm, that is, Q(s,a;w)=V(s;w)+A(s,a;w), where A(s,a;w) is the advantage function; V(s;w) is the state value function. In order to solve the problem that there are infinite decompositions of the same Q(s,a;w), we can add a quantity derived from the advantage function, and add the average value of the advantage function after Q(s,a;w), so that the obtained advantage function retains fixed characteristics and the solution is unique, that is, in,

[0108] In this embodiment, in order to enhance the agent's ability to perceive state variables and make decision-making actions more consistent, LSTM (Long Short Term Memory) is introduced to extract temporal features.

[0109] Please continue to see Figure 3 , the input of the LSTM network is the state variables (s t-1 ,s t ,s t+1 ), the input of the LSTM unit is (s t ,h t-1 ,c t-1 ), the output is (h t ,c t ), where h t is the output of the hidden node, c t This is the LSTM unit state and persists throughout the entire LSTM chain. Connect the Dueling DDQN algorithm network after the LSTM network. After passing through the LSTM layer, select the output of the last time t+1 in the LSTM as the basic feature for calculating the Q value, and calculate the Q value using the above method.

[0110] During the training process of LSTM-Dueling DDQN, the current state and tactical requirements (MD, EC, TS) of three consecutive moments are input into LSTM-Dueling DDQN to obtain the current UAV maneuver control parameters. After executing the maneuver, the avoidance maneuver reward R is obtained. t And store this experience in the experience pool for updating network parameters. By continuously iterating the above process, the UAV avoidance maneuver strategy that meets the requirements can be output eventually.

[0111] In this embodiment, the avoidance cut-off conditions of the drone include an avoidance failure condition and an avoidance success condition.

[0112] For example, the avoidance failure condition is: r(t)≤r d ,v m (t)≥v mmin and t≤t max , r(t) represents the relative distance between the UAV and the missile at time t, v m (t) represents the velocity of the missile at time t, v mmin represents the minimum controlled speed of the missile, t max Indicates the energy working time on the missile, r d Indicates the missile damage radius; when the flight trajectory given by the evasion strategy causes the drone to exceed the given altitude limit y umin Also considered a failure of avoidance.

[0113] The successful avoidance condition is: t≥t max or v m (t) <v mmin , that is, the simulation time is greater than the energy working time on the missile or the terminal speed of the missile is lower than the minimum controlled speed, it is considered to be a successful avoidance; when r(t f )>r d If the terminal miss distance is greater than the missile damage radius, the avoidance is also considered successful. f The moment when the UAV and the target approach speed is 0, defined as:

[0114] The following simulation experiment further illustrates the above-mentioned UAV air combat autonomous avoidance maneuver decision-making method based on deep reinforcement learning.

[0115] Set the initial states of the drone and air-to-air missile. In this scenario, the drone and the missile are at the same altitude and 12 km apart, as shown in Table 1:

[0116] Table 1

[0117]

[0118] (1) Comparative analysis of the performance of deep reinforcement learning algorithms for autonomous evasive maneuvers

[0119] To further validate the effectiveness of the proposed algorithm, we conducted comparative tests using the existing DQN algorithm, the DQN algorithm with an exploration course, the DDQN algorithm, and the Dueling DDQN algorithm. DQN_traditional represents the existing DQN algorithm, while DQN represents the DQN algorithm with an exploration course. The exploration course was also used in all three algorithms: DDQN, Dueling DDQN, and LSTM-Dueling DDQN.

[0120] Figure 4 is a Q estimation value comparison curve provided by an embodiment of the present invention, Figure 5 is a comparison curve of the cumulative reward value provided by the embodiment of the present invention. Figure 4-5 As can be seen, the DQN_traditional algorithm suffers from severe overestimation and exhibits significant fluctuations during training. The DQN algorithm reduces this overestimation and fluctuations, but these issues still persist. The DDQN and Dueling DDQN algorithms largely address the overestimation issue, but exhibit significant fluctuations early in the training process. The LSTM-Dueling DDQN algorithm not only eliminates the overestimation issue but also exhibits minimal fluctuations throughout the training process, significantly outperforming the other algorithms. Compared to the DQN_traditional, DQN, DDQN, and Dueling DDQN algorithms, the LSTM-Dueling DDQN algorithm converges the fastest and achieves the highest reward. In summary, the LSTM-Dueling DDQN algorithm achieves superior results and a more stable training process compared to the DQN_traditional, DQN, DDQN, and Dueling DDQN algorithms.

[0121] (2) Verification of autonomous evasive maneuver strategy generation in air combat under various tactical requirements

[0122] Figure 6 This is a comparison diagram of air combat avoidance maneuver trajectories when the tactical requirement provided by the embodiment of the present invention is the miss distance. Figure 6 As shown in Figure 2, under this strategy, the UAV mainly performs a dive and right turn maneuver, using the higher resistance at low altitude to effectively reduce the speed of the incoming missile, and turns right in the process to further maximize MD. md (t f )=0.141,f ec (t f )=75.396,f ts (t f )=0.679.

[0123] Figure 7This is a comparison diagram of air combat avoidance maneuver trajectories when the tactical requirement provided by the embodiment of the present invention is energy consumption. Figure 7 As shown, the drone's maneuvers under this strategy are similar to those under the optimal MD, primarily performing dives. However, this strategy dives to a lower altitude and eliminates turning maneuvers, thereby reducing energy consumption. The figure shows the three tactical requirements under this strategy. Compared to the optimal MD strategy, this strategy relatively reduces MD and lowers its own safety, but it also reduces EC while successfully avoiding missiles, providing an energy advantage for subsequent air combat. At this time, f md (t f )=0.030,f ec (t f )=10.361,f ts (t f )=0.678.

[0124] Figure 8 This is a comparison diagram of air combat evasive maneuver trajectories when the tactical requirement provided by the embodiment of the present invention is terminal advantage. Figure 8 As shown in the figure, under this strategy, the UAV mainly performs a climb and right turn maneuver. Since the high-altitude drag is small, the missile's deceleration effect is not obvious, so the MD under this strategy is small. This strategy has a small MD and consumes more energy, but it provides the UAV with a larger TS, providing a higher initial tactical advantage for the next round of air combat. At this time, f md (t f )=0.087,f ec (t f )=93.095,f ts (t f )=1.262.

[0125] (3) Comparative analysis of three tactical requirements with different weights

[0126] Figure 9 It is the terminal value of each tactical requirement function under different weights of the three tactical requirements provided by the embodiment of the present invention. In order to verify the effectiveness and applicability of the algorithm model under different tactical requirements, simulations are performed when the weights of the three tactical requirements are different, where α1 is the MD tactical requirement weight, α2 is the EC tactical requirement weight, and α3 is the TS tactical requirement weight. Figure 9 As shown, when α1=0.6, α2=0.3, α3=0.1, f md (t f )=0.088,f ec (t f )=39.787,f ts (t f )=0.748, compared with α1=1, α2=0, α3=0, the energy consumption is reduced, the UAV’s dive and turning amplitude is relatively small, f md(t f ) value also decreases accordingly, but a relatively large MD is still obtained. When α1=0.1, α2=0.3, α3=0.6, f md (t f )=0.033,f ec (t f )=49.983,f ts (t f )=1.103, compared with α1=0, α2=0, α3=1, the energy consumption is reduced and the TS value is also reduced, but a relatively large TS value is still obtained.

[0127] In summary, a tactical requirement function value with a better tactical requirement will lead to relatively poor values of the other two objectives. These results further verify the effectiveness and applicability of the UAV evasive maneuver strategy generation algorithm proposed in this invention, and also reflect the contradiction and coupling of multiple tactical requirements in air combat evasive maneuvers.

[0128] It can be seen from the above embodiments that the beneficial effects of the present invention are:

[0129] The present invention comprehensively considers multiple tactical requirements and establishes a tactical requirement-coupled UAV autonomous avoidance maneuver strategy generation method model. The LSTM-Dueling DDQN is used to solve the UAV autonomous avoidance maneuver strategy generation problem. Without relying on human expert experience, it can achieve autonomous mapping from state to avoidance strategy, has good convergence speed and learning effect, and the derived avoidance maneuver strategy can reflect different air combat tactical requirements while ensuring the safety of the UAV itself, thereby improving the success probability of the attack mission.

[0130] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0131] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0132] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other changes to the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0133] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for autonomous evasive maneuvering decision-making in UAV air combat based on deep reinforcement learning, characterized in that: include: Determine the tactical requirements for various drone evasive maneuvers; Establish a three-dimensional space pursuit model for drones and missiles; Obtain the battlefield status at three consecutive moments t-1, t and t+1, including the relative distance r between the UAV and the missile, the flight altitude y of the UAV, and the distance between the UAV and the missile. u , horizontal distance r between the UAV and the missile xz , line of sight angle change rate and the rate of change of sight angle The multiple tactical requirements, the preset weights corresponding to each tactical requirement, and the battlefield states at the three consecutive times t-1, t, and t+1 are input into a pre-trained LSTM-Dueling DDQN. The deep reinforcement learning algorithm is used to determine the maneuver control parameters of the UAV at time t+1. Inputting the maneuver control parameters into the UAV-missile three-dimensional space pursuit model to obtain the battlefield state at time t+2; The steps of establishing the UAV-missile three-dimensional space pursuit model include: Determine the UAV-missile three-dimensional pursuit model based on the UAV motion model, UAV constraint model, missile motion model, missile guidance command response model, and missile constraint model; Said tactical requirements include miss distance, total UAV energy consumption and UAV avoidance terminal advantage; The avoidance cut-off conditions of the UAV include avoidance failure conditions and avoidance success conditions; The avoidance failure condition is: r(t)≤r d ,v m (t)≥v mmin and t≤t max , the successful avoidance condition is: t≥t max or v m (t) <v mmin ; Among them, r(t) represents the relative distance between the UAV and the missile at time t, v m (t) represents the velocity of the missile at time t, v mmin represents the minimum controlled speed of the missile, t max Indicates the energy working time on the missile, r d Indicates the missile damage radius.

2. The UAV air combat autonomous avoidance maneuver decision-making method based on deep reinforcement learning according to claim 1 is characterized in that: The UAV motion model is: x u (t0)=x u0 Among them, the state vector x u =[x u ,y u ,z u ,v u ,θ u ,ψ u ] T , x u0 Represents the initial state vector of the drone, x u 、z u Indicates the horizontal coordinate of the drone in the geographic coordinate system, y u Indicates the flight altitude of the drone, v u represents the speed of the drone, θ u represents the track inclination of the UAV, ψ u Represents the track deviation angle of the UAV; the UAV maneuver control vector u=[n xu ,n yu ,n zu ] T , where n xu 、n yu 、n zu Respectively represent the control instructions of the UAV in terms of speed, pitch angle and yaw direction; The UAV constraint model is: Among them, n xc 、n yc 、n zc Actual control overload for the drone; n xmax 、n umax They represent the maximum available overload of the UAV in the tangential and normal directions, v umin Indicates the minimum speed limit of the drone, v umax Indicates the maximum speed limit of the drone, y umin Indicates the minimum height limit of the drone, y umax Indicates the maximum altitude limit of the drone.

3. The method for autonomous evasive maneuvering decision-making in UAV air combat based on deep reinforcement learning according to claim 2 is characterized in that: The missile motion model is: x m (t0)=x m0 Among them, the state vector x m =[x m ,y m ,z m ,v m ,θ m ,ψ m ] T , x m0 is the initial state vector of the missile, x m 、y m 、z m represents the coordinates of the missile in the geographic coordinate system, v m represents the missile's velocity, θ m represents the missile's track inclination, ψ m Indicates the missile's track deviation angle; The guidance command response model of the missile is: Among them, n myc 、n mzc They represent the pitch and yaw command overloads generated by the proportional guidance law, τ m represents the time constant of the missile guidance control system; The missile constraint model is: Among them, n mmax is the maximum normal overload of the missile.

4. The method for autonomous evasive maneuvering decision-making in UAV air combat based on deep reinforcement learning according to claim 3 is characterized in that: The UAV-missile three-dimensional space pursuit model is: Among them, x r 、y r 、z r are all intermediate variables, x r =x u -x m ,y r =y u -y m , z r =z u -z m , 5. The method for autonomous evasive maneuvering decision-making in UAV air combat based on deep reinforcement learning according to claim 1 is characterized in that: The reward function of the deep reinforcement learning algorithm includes dense rewards and sparse rewards; During the training process of LSTM-Dueling DDQN, the proportion of dense rewards gradually decreases with the increase of iteration number, and the proportion of sparse rewards increases with the increase of iteration number.

6. The method for autonomous evasive maneuvering decision-making in UAV air combat based on deep reinforcement learning according to claim 5 is characterized in that: The reward function of the deep reinforcement learning algorithm is R t =r t +e -kn s t ; Among them, e -kn represents the temporal degradation factor, k is a constant used for the decay rate of dense rewards, n represents the current number of iterations, r t represents the total sparse reward of each tactical requirement at time t, s t represents the dense reward at time t, R t represents the total reward at time t.

Images