A UAV Maneuver Decision-Making Approach Combining Hesitant Fuzzy Model and Dynamic Deep Reinforcement Learning

Abstract

The present invention discloses a UAV maneuver decision method combining hesitant fuzzy and dynamic deep reinforcement learning, including first, establishing a UAV air combat motion model, establishing a decision model based on weighted optimization targets according to the attack parameters of the enemy and our side and the energy parameter difference based on the situation; secondly, according to the hesitant fuzzy theory, the maximum deviation method is used to determine the optimal weight of the decision model of the optimization target in real time; then, constructing the state space and action space of air combat maneuver decision reinforcement learning; then, merging the UAV states at multiple moments into a state set as a neural network input, constructing a dynamic deep Q network for UAV maneuver decision training; finally, obtaining the optimal maneuver decision by the trained dynamic deep Q network. This method mainly solves the UAV maneuver decision problem under the condition of incomplete environmental information, and considers the influence of the air combat process in the decision-making process, which is more in line with the needs of actual air combat.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle air combat decision-making, in particular to a unmanned aerial vehicle maneuver decision-making method combining hesitant fuzzy and dynamic deep reinforcement learning. Technical Background

[0002] Unmanned Combat Aerial Vehicle (UCAV) needs to make the best tactical plan or maneuver according to the complex battlefield situation information during air combat. The quality of the UAV decision-making mechanism is the key to whether the air combat mission can be successfully completed. As the air combat environment becomes more and more complex and unknown, improving the intelligence level of UAVs, enabling them to independently perceive the battlefield environment and automatically generate control commands to complete maneuvering choices in air combat is the main research direction of UAV air combat.

[0003] In recent years, with the rapid development of artificial intelligence technology, deep learning and machine learning have shown great potential in the field of UAV air combat decision-making. Reinforcement learning is an unsupervised machine learning method. The aircraft is rewarded by interacting with the environment, learns how to adapt to the environment according to the principle of obtaining the maximum reward, and updates the learned experience and stores it in the Q-value table. In real air combat, when there are too many states and the dimension is too high, the Q-value table is obviously not applicable. The deep reinforcement learning algorithm and the neural network fitting Q-value function can solve this problem. However, this method is mainly for air combat maneuvering decisions when the environmental parameters are known. However, in the actual air combat decision-making process, different air combat situations have different requirements for environmental parameters, and each optimization target parameter has a certain degree of fuzziness and imprecision. This type of method cannot meet such requirements.

[0004] Therefore, the present invention proposes a UAV maneuver decision-making method that combines hesitant fuzzy and dynamic deep reinforcement learning to address the above problems. The hesitant fuzzy maximum deviation method is used to determine the weight value of the optimization target at each moment, which solves the problem that the weights of multiple optimization targets in traditional reinforcement learning are fixed and unreasonable. The state set composed of multiple moment states is used as the input of the neural network, and the neural network is trained based on the principle of obtaining the maximum return. Since the use of the state set takes into account the impact of the air combat process on the result, rather than just considering the impact of the current moment on the result, it is more in line with the actual air combat. Summary of the invention

[0005] The purpose of the present invention is to provide a UAV maneuver decision method combining hesitant fuzzy and dynamic deep reinforcement learning, which can enable the UAV to update the objective function weight in real time using hesitant fuzzy theory in a fuzzy environment, and make decisions using a dynamic dual-depth Q network to obtain the optimal UAV air combat maneuver decision.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The UAV maneuver decision-making method combining hesitant fuzzy and dynamic deep reinforcement learning includes:

[0008] Step 1: Establish a UAV air combat motion model, and establish a decision model based on weighted optimization objectives according to the attack parameters of the enemy and our own side and the energy parameter difference based on the situation.

[0009] Step 2: Based on the hesitant fuzzy theory, the maximum deviation method is used to determine the optimal weight of the decision model of the optimization target in real time.

[0010] Step 3: Construct the state space and action space of air combat maneuver decision reinforcement learning.

[0011] Step 4: Combine the drone states at multiple moments into a state set as the neural network input to build a dynamic deep Q network.

[0012] Step 5: Let the immediate reward obtained by taking the action in the current state be the decision model based on the weighted optimization objective, and use the reward to train the dynamic deep Q network for UAV maneuver decision-making.

[0013] Step 6: Input the current state set of the UAV into the trained dynamic deep Q network to obtain the optimal maneuver decision.

[0014] The present invention has the following advantages:

[0015] 1. The present invention utilizes hesitant fuzzy theory and adopts the maximum deviation method to determine parameter weights in real time, and uses the weights to perform weighted summation on multiple objectives, thereby solving the problem that the weights of multiple optimization objectives in the optimization process of traditional reinforcement learning are fixed and unreasonable.

[0016] 2. The present invention introduces a dynamic deep Q network and uses a state set composed of multiple time states as the input of the neural network. It takes into account the impact of the air combat process on the decision-making, rather than just considering the impact of the current moment on the result, making the decision result more reasonable. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flow chart of the method of the present invention

[0018] Figure 2 Schematic diagram of the air combat situation between the enemy and us

[0019] Figure 3 Air combat trajectory diagram for enemy drones adopting the "S-shaped maneuver" strategy

[0020] Figure 4 The change curve of each optimized target value for the enemy UAV to adopt the "S-shaped maneuver" strategy

[0021] Figure 5Air combat trajectory diagram for enemy drones using the "pure tracking" strategy

[0022] Figure 6 The change curve of each optimized target value for the "pure tracking" strategy adopted for the enemy UAV DETAILED DESCRIPTION

[0023] The technical solution of the present invention is described in detail with reference to the accompanying drawings.

[0024] The unmanned aerial vehicle maneuvering decision-making method combining hesitant fuzzy and dynamic deep reinforcement learning of the present invention specifically includes the following steps:

[0025] Step 1: Establish the UAV air combat motion model, according to the missile attack parameters ξ of both sides i , T And the energy parameter difference ΔW based on the situation, a decision model based on weighted optimization objectives is established, specifically:

[0026] (1.1) The UCAV is regarded as a particle, without considering the specific rigid body motion and flight control algorithm, and a three-degree-of-freedom particle model is used to describe its motion state. Its motion model is:

[0027]

[0028] In the formula, x, y, and z represent the position of the aircraft in the inertial coordinate system, v is the aircraft's flight speed, θ is the track inclination, which represents the angle between the speed and the xOy plane, and ψ is the heading angle, which represents the projection of the speed on the xOy plane. The angle with the y-axis, where v′ represents the projection of the velocity on the xOy plane, g is the gravitational acceleration at the current position, [η x , η z , φ] is the control quantity of the UAV, where η x is the overload along the speed direction, representing the thrust of the aircraft, η z is the overload along the top direction, that is, the normal overload, and φ is the roll angle around the velocity vector.

[0029] (1.2) According to the above model, a decision model based on weighted optimization objectives is established, specifically:

[0030] (1.2.1) Attack parameter modeling based on weapon performance

[0031] The purpose of air combat is to shoot down the enemy and protect our side, and the purpose of making maneuvering decisions is to form conditions for our side to launch weapons while avoiding the enemy from forming conditions for launching weapons. Therefore, this patent combines angle, distance, weapon configuration and weapon range based on the attack performance of airborne weapons, and proposes a new attack parameter ζ as an optimization target.

[0032] Assume that both the enemy and our side carry air-to-air missiles, and their attack areas are as follows: Figure 2 As shown in the figure, Indicates the sight angle of our drone, is the target UAV sight angle, V U 、V T Denote the flight speed of our UAV and the target UAV respectively, and R is the distance between the enemy and our UAV. Define the attack parameter ξ of missile i i for:

[0033]

[0034] Where k is the attack parameter adjustment factor, here k=1, R g The missile is along the line of sight of our drone Maximum attack distance in direction.

[0035] If the number of missiles carried by our UCAV is n, and the specific missile numbers are U1, U2, ..., Un, the attack parameters of all our missiles are obtained by equation (2): U1 ,ξ U2 , ..., ξ Un}, take the maximum value as our current attack parameter ξ U , the specific formula is:

[0036] ξ U = arg max{ξ U1 ,ξ U2 , ..., ξ Un} (3)

[0037] If the number of missiles carried by the enemy UCAV is m, and the specific missile numbers are t1, t2, ..., Tm, the attack parameters of all enemy missiles {ξ T1 ,ξ T2 , ..., ξ Tm}, take the maximum value as the enemy's current attack parameter ξ T , the specific formula is:

[0038] ξ T = arg max{ξ T1 ,ξ T2 , ..., ξ Tm} (4)

[0039] (1.2.2) Energy parameter difference modeling based on UCAV situation

[0040] The completion of the actions of unmanned combat aircraft is based on the consumption of energy. The higher the energy, the more selective the actions are, which is more conducive to gaining an advantage in air combat. Assume that W Uis our energy parameter and W T is the enemy energy parameter, and the energy parameter difference based on the UCAV situation is defined as ΔW = W U -W T , where W U and W T Specifically:

[0041]

[0042] In the formula, ξ Up and W Uk are our gravitational potential energy and kinetic energy respectively, m U is the quality of our drone, E Tp and W Tk are the gravitational potential energy and kinetic energy of the target UAV, m T is the mass of the target UAV, k1 and k2 are the energy adjustment parameters of our UAV and the target UAV respectively, both of which are set to 1 here.

[0043] (1.2.3) Establish a decision model based on weighted optimization objectives

[0044] The missile attack parameters ξ of both sides calculated based on the current enemy and friendly states i , T And the energy parameter difference ΔW based on the situation, perform weighted summation to obtain the decision model of the optimization target, which is:

[0045] f(ξ U ,ξ T , ΔW; ω * )=ω1ξ U +ω2ξ T +ω3ΔW (6)

[0046] Where ω1, ω2 and ω3 are weights.

[0047] Step 2: According to the hesitant fuzzy theory, the maximum deviation method is used to determine the optimal weight ω of the decision model of the optimization target in real time. * , specifically:

[0048] (2.1) Based on the multi-time UAV situation, the hesitant fuzzy evaluation matrix of the optimized target weight is constructed. The specific method is as follows:

[0049] Let A = {A1, A2, ..., A n} is the decision set of the UAV at n consecutive moments, X = {ξ U ,ξ T , ΔW} is the optimization parameter set, then at the i-th moment, a hesitant fuzzy set A about X i for:

[0050]

[0051] In the formula, Indicates the optimization target x at the i-th moment j The set of possible memberships under .

[0052] When x j Take U hour, Indicates the optimization target ξ at the i-th moment U The possible membership set under Our current attack parameter ξ on the enemy Ui The larger the value is, the greater the air combat advantage is; otherwise, the smaller the air combat advantage is. The corresponding hesitant fuzzy set under this attribute is Specifically:

[0053]

[0054] Among them, ρ is the attack factor coefficient, and here ρ=0.2 is taken.

[0055] When x j Take T hour, Indicates the optimization target ξ at the i-th moment T The possible membership set under The enemy's current attack parameter ξ Ti The larger the value is, the smaller the air combat advantage is; otherwise, the greater the air combat advantage is. The corresponding hesitant fuzzy set under this attribute is Specifically:

[0056]

[0057] Among them, ρ is the attack factor coefficient, and here ρ=0.2 is taken.

[0058] When x j When ΔW is taken, It represents the possible membership set under the optimization target energy parameter difference ΔW at the i-th moment, which is abbreviated as Energy parameter difference ΔW at the current moment i The larger the value is, the greater the air combat advantage is; otherwise, the smaller the air combat advantage is. The corresponding hesitant fuzzy set under this attribute is Specifically:

[0059]

[0060] Among them, γ is the attack factor coefficient, and here γ=0.3.

[0061] Let H be the hesitant fuzzy decision matrix, which consists of n×3 hesitant elements, where n is the number of selected moments. The hesitant fuzzy decision matrix H is specifically:

[0062]

[0063] (2.2) The optimal weight of the objective function is determined based on the maximum deviation method, specifically:

[0064] For the optimization target x j ∈X, the hesitant fuzzy set A at the i-th moment i The deviation relative to all other moments under this optimization objective is expressed as:

[0065]

[0066] In the formula, h ij is the i-th row and j-th column element in the hesitant fuzzy decision matrix H, h kj is the kth row and jth column element in the hesitant fuzzy decision matrix H, m is the number of optimization targets, d(h ij ,h kj ) is the hesitant element h in the fuzzy decision matrix H ij and h kj The hesitant Euclidean distance is specifically defined as:

[0067]

[0068] Among them, l is the number of values ​​in the hesitant element.

[0069] For the optimization target x j ∈X, the deviation of all moments relative to other moments under the optimization objective is expressed as:

[0070]

[0071] Among them, ω j is the weight coefficient.

[0072] Construct a nonlinear model for determining the weight vector ω so that the deviation value of all optimization target parameters is maximized, specifically:

[0073]

[0074] The solution of the above model is transformed into a constrained optimization problem, and the Lagrangian function f(ω, λ) is constructed as shown in the following formula:

[0075]

[0076] Where λ is the Lagrange multiplier.

[0077] The partial differential of f(ω, λ) is calculated as follows:

[0078]

[0079] Solving the above equations gives the weight vector ω = (ω1, ω2, ..., ω j , ..., ω m ), here ω j for:

[0080]

[0081] Substituting formula (14) into formula (18), the above formula can be transformed into:

[0082]

[0083] Then ω={ω1,ω2,...,ω m}Normalize to get the standardized optimal weight vector Specifically:

[0084]

[0085] In the formula,

[0086] Step 3: Construct the state space S and action space A of air combat maneuver decision reinforcement learning.

[0087] (3.1) The state space S of the air combat maneuver decision reinforcement learning includes all the situation factors of both sides that affect the calculation of the air combat advantage function, specifically:

[0088] 1) Our drone’s sight angle Sight angle with enemy drone

[0089] 2) The distance R between the enemy and friendly drones;

[0090] 3) Our drone speed v U Speed ​​v of enemy drone T ;

[0091] 4) The height difference Δh between the enemy and friendly drones.

[0092] Select As the state space of air combat maneuver decision reinforcement learning, it describes the air combat situation of the UAV at the current moment.

[0093] (3.2) The final motion trajectory of a drone in an air combat can be viewed as a combination of maneuvering actions determined by each decision step. Here we select seven basic maneuvers of a drone in an air combat, including: 1) maintaining the original flight; 2) flying straight at maximum acceleration; 3) turning left at maximum overload; 4) turning right at maximum overload; 5) climbing at maximum overload; 6) diving at maximum overload; and 7) flying at maximum deceleration.

[0094] According to the UAV air combat motion model established by formula (1), the control quantity is designed as [η x , η z ,φ], then the control quantities corresponding to the above seven basic maneuvers are:

[0095] 1) Maintain the original flight, [η x , η z ,φ]=[0,1,0];

[0096] 2) Maximum acceleration flight,

[0097] 3) Turn left at maximum overload,

[0098] 4) Turn right at maximum overload,

[0099] 5) Maximum overload climb,

[0100] 6) Maximum overload dive,

[0101] 7) Maximum deceleration flight, here, is the maximum overload along the speed direction, i.e. the maximum thrust, It is the maximum overload along the top direction, that is, the maximum normal overload.

[0102] The control quantities of the above seven actions are respectively denoted as a i , i = 1, 2, ..., 7, constituting the action space A = {a1, a2, ..., a7} for air combat decision-making.

[0103] Step 4: Use the state set consisting of the current state vector and the state vectors of the previous two moments as the neural network input to construct a dynamic deep Q network, specifically:

[0104] (4.1) Establish a dynamic deep Q learning network with the initialization parameters shown in the following table.

[0105]

[0106] (4.2) The state set (s) consisting of the current state vector and the state vectors of the previous two moments t-2 ,s t-1 ,s t ) is used as the input of the neural network, and the output is the number of all action values ​​Q((s t-2 ,s t-1 ,s t), a; θ), where a is the action taken by the agent in this state, θ is the network weight, and state s t-2 ,s t-1 and t Specifically:

[0107]

[0108] Step 5: Let r be the immediate reward obtained by taking this action in the current state, and let r = f(ξ U ,ξ T , ΔW; ω * ), where f(ξ U ,ξ T , ΔW; ω * ) is obtained by the decision model of the optimization target of formula (6). The reward r is used to train the dynamic deep Q network for UAV maneuver decision-making. The specific steps are as follows:

[0109] (5.1) Initialize the experience pool D, here we take 50000.

[0110] (5.2) Establish an action value Q estimation network and randomly initialize the network weights θ; establish an action value target network Initialize network weights θ - =θ.

[0111] (5.3) Initialize the drone state sequence The initial input of the neural network is (s1, s2, s3).

[0112] (5.4) For each step in the episode, a random action a is chosen with probability ε k , otherwise select a k = argmax a Q(s, a; θ).

[0113] (5.5) The drone performs action a k , calculate the potential function reward r at time k k , calculate the state of the drone at time k+1 The current experience ((s k-2 ,s k-1 ,s k ), a k , r k ,(s k-1 ,s k ,s k+1 )) is stored in the experience pool D.

[0114] (5.6) Randomly extract the minimum experience block D from the experience pool D min ,, calculate the target value function y k , specifically:

[0115]

[0116] Yes (y k -Q((s k-2 ,s k-1 ,s k ), a k ;θ)) 2 Execute the gradient descent method to update the Q estimation network weights θ.

[0117] (5.7) If all training of the current episode is completed, the training ends, otherwise, jump to step (5.3).

[0118] Step 6: Input the current state set of the drone into the trained dynamic deep Q network to obtain the optimal maneuver decision. The specific steps are:

[0119] (6.1) Let k = 1, randomly initialize the states of the enemy and friendly drones, and obtain the initial neural network state (s k ,s k+1 ,s k+2 ).

[0120] (6.2) This state is input into the trained dynamic deep Q network, and the network outputs the optimal action a k = argmax a Q(s, a; θ), after the drone performs the action, it obtains the next state set (s k+1 ,s k+2 ,s k+3 ).

[0121] (6.3) If our drone is in a condition to attack the enemy, then complete this maneuver decision. Otherwise, (s k+1 ,s k+2 ,s k+3 ) is input into the neural network and jumps to step (6.2).

[0122] In order to verify the feasibility and effectiveness of the method, the present invention is further described in detail below with reference to examples.

[0123] On the macOS operating system (version number: MOjava version: 10.14.5, processor: 2.5GHz intercode i7, memory: 16GB 1600MHz DDR3, graphics card: Inter Iris Pro 1536MB), use python language to build an experimental environment on pycharm to simulate the algorithm, and export the simulation results for visualization.

[0124] The trained dynamic deep Q network is used to make maneuver decisions for the UAV in the following two situations, in which both the enemy and our side use classic tactical actions.

[0125] Situation 1: Our drone is initially in a relatively favorable situation, and the enemy drone only uses "S-shaped maneuvers" for tactical evasion

[0126] Figure 3 The figure shows the trajectory of the enemy drone's air combat using the "S-shaped maneuver" strategy. The upper curve in the figure is the trajectory of our drone, and the lower curve is the trajectory of the enemy drone. In the initial state, the coordinates of our drone are (1.2km, 8.2km, 2.4km), and the coordinates of the enemy drone are (1.5km, 1.5km, 1.0km). It can be seen from the figure that at the beginning, our drone was located above the enemy and had an advantage. After our drone dived appropriately, it pulled up its nose to avoid the fighter rushing forward, and then began to adjust the sight angle, forming the attack conditions and maintaining them.

[0127] Figure 4 The graph shows the change curves of the optimized target parameters when the enemy UAV adopts the "S-shaped maneuver" strategy. The lower curve in the figure is our current attack parameter ξ U The middle curve is the enemy's current attack parameter ξ T The upper curve is the energy parameter difference ΔW=W U -W T As can be seen from the figure, in the decision-making process, our current attack parameter ξ U Gradually increases, the enemy's attack parameter ξ T Gradually decreases, and ΔW keeps decreasing in the whole decision-making process, indicating that our UAV finally achieves the attack on the enemy at the expense of a certain energy advantage.

[0128] Situation 2: We were initially in an unfavorable situation, and the enemy used the "pure tracking" tactic to try to approach and attack our drone.

[0129] Figure 5 The figure shows the trajectory of the air combat when the enemy drone adopts the "pure tracking" strategy. The upper curve in the figure is the trajectory of our drone, and the lower curve is the trajectory of the enemy drone. In the initial state, the coordinates of our drone are (8.0km, 9.5km, 8.5km), and the coordinates of the enemy drone are (1.5km, 1.6km, 0.8km). It can be seen from the figure that our drone lost its tactical advantage after diving at the beginning, but then it maneuvered to the lower right while raising its nose, preemptively meeting the attack conditions and finally achieving the attack on the enemy.

[0130] Figure 6The following are the curves of the optimization targets of the enemy UAV adopting the "pure tracking" strategy. The lower curve in the figure is our current attack parameter ξ U The middle curve is the enemy's current attack parameter ξ T The upper curve is the energy parameter difference ΔW=W U -W T As can be seen from the figure, in the decision-making process, our current attack parameter ξ U Gradually increases, the enemy's attack parameter ξ T The energy parameter difference ΔW decreases gradually during the entire decision-making process, indicating that our UAV finally achieved the attack on the enemy at the expense of a certain energy advantage.

Claims

1. A UAV maneuver decision-making method combining hesitant fuzzy and dynamic deep reinforcement learning, characterized by: include: Step 1: Establish a UAV air combat motion model based on the attack parameters of the enemy and our side and the energy parameter difference based on the situation, specifically: (1.1) Define the attack parameter ξ of missile i i for: Where k is the attack parameter adjustment factor, R is the distance between the enemy and our drones, and R g The missile is along the line of sight of our drone Maximum attack distance in direction; If the number of missiles carried by our unmanned fighter is n, and the specific missile numbers are U1, U2, ..., Un, the attack parameters of all our missiles are obtained by equation (1): U1 ,ξ U2 , ..., ξ Un }, take the maximum value as our current attack parameter ξ U , the specific formula is: x U =arg max{ξ U1 ,x U2 ,...,x Un } (2) If the number of missiles carried by the enemy unmanned fighter is m, and the specific missile numbers are T1, T2, ..., Tm, the attack parameters of all enemy missiles are obtained by formula (1): T1 ,ξ T2 , ..., ξ Tm }, take the maximum value as the enemy's current attack parameter ξ T , the specific formula is: x T =arg max{ξ T1 ,x T2 ,...,x Tm } (3) (1.2) Define the energy parameter difference based on the situation ΔW = W U -W T , here W U is our energy parameter, W T is the enemy energy parameter, which specifically satisfies: In the formula, E Up and W Uk are our gravitational potential energy and kinetic energy respectively, m U is the quality of our drone, E Tp and W Tk are the gravitational potential energy and kinetic energy of the target UAV, m T is the mass of the target UAV, k1 and k2 are the energy adjustment parameters of our UAV and the target UAV respectively; (1.3) Establish a decision model based on weighted optimization objectives, specifically: The missile attack ξ of both sides calculated based on the current enemy and friendly state U , T And the energy parameter difference ΔW based on the situation, perform weighted summation to obtain the decision model of the optimization target, which is: f(ξ U ,x T ,ΔW;ω * )=ω1ξ U +ω2ξ T +ω3ΔW (5) Where ω1, ω2 and ω3 are weights; Step 2: According to the hesitant fuzzy theory, the maximum deviation method is used to determine the optimal weight ω of the decision model of the optimization target in real time. * ; Step 3: Construct the state space and action space of air combat maneuver decision reinforcement learning; Step 4: Combine the drone states at multiple moments into a state set as the neural network input to build a dynamic deep Q network. Step 5: Let r be the immediate reward obtained by taking this action in the current state, and let r = f(ξ U ,ξ T , ΔW; ω * ), f(ξ U ,ξ T , ΔW; ω * ) is obtained by the decision model of the optimization target of formula (5), and the reward r is used to train the dynamic deep Q network for UAV maneuver decision-making; Step 6: Input the current state set of the UAV into the trained dynamic deep Q network to obtain the optimal maneuver decision.

2. The UAV maneuver decision-making method combining hesitant fuzzy and dynamic deep reinforcement learning as claimed in claim 1 is characterized in that: In step 2, according to the hesitant fuzzy theory, the maximum deviation method is used to determine the optimal weight of the decision model of the optimization target in real time, specifically: (2.1) Based on the multi-time UAV situation, the hesitant fuzzy evaluation matrix of the optimized target weight is constructed. The specific method is as follows: Let A = {A1, A2, ..., A n } is the decision set of the UAV at n consecutive moments, X = {ξ U ,ξ T , ΔW} is the optimization parameter set, then at the i-th moment, a hesitant fuzzy set A about X i for: In the formula, Indicates the optimization target x at the i-th moment j The possible membership set under ; When x j Take U hour, Indicates the optimization target ξ at the i-th moment U The possible membership set under Our current attack parameter ξ on the enemy Ui The larger the value is, the greater the air combat advantage is; otherwise, the smaller the air combat advantage is; the corresponding hesitant fuzzy set under this attribute is Specifically: Among them, ρ is the attack factor coefficient, ρ = 0.2; When x j Take T hour, Indicates the optimization target ξ at the i-th moment T The possible membership set under The enemy's current attack parameter ξ Ti The larger the value is, the smaller the air combat advantage is; otherwise, the greater the air combat advantage is; the corresponding hesitant fuzzy set under this attribute is Specifically: Among them, ρ is the attack factor coefficient, ρ = 0.2; When x j When ΔW is taken, It represents the possible membership set under the optimization target energy parameter difference ΔW at the i-th moment, which is abbreviated as Energy parameter difference ΔW at the current moment i The larger the value is, the greater the air combat advantage is; otherwise, the smaller the air combat advantage is; the corresponding hesitant fuzzy set under this attribute is Specifically: Among them, γ is the attack factor coefficient, γ = 0.3; Assume that the hesitant fuzzy decision matrix is ​​composed of n×3 hesitant elements, where n is the number of selected moments. The hesitant fuzzy decision matrix H is specifically: (2.2) The optimal weight of the objective function is determined based on the maximum deviation method, specifically: For the optimization target x j ∈X, the hesitant fuzzy set A at the i-th moment i The deviation relative to all other moments under this optimization objective is expressed as: In the formula, h ij is the i-th row and j-th column element in the hesitant fuzzy decision matrix H, h kj is the kth row and jth column element in the hesitant fuzzy decision matrix H, m is the number of optimization targets, d(h ij ,h kj ) is the hesitant element h in the fuzzy decision matrix H ij and h kj The hesitant Euclidean distance is specifically defined as: Among them, l is the number of values ​​in the hesitant element; For the optimization target x j ∈X, the deviation of all moments relative to other moments under the optimization objective is expressed as: Among them, ω j is the weight coefficient; Construct a nonlinear model for determining the weight vector ω so that the deviation value of all optimization target parameters is maximized, specifically: The solution of the above model is transformed into a constrained optimization problem, and the Lagrangian function f(ω, λ) is constructed as shown in the following formula: Where λ is the Lagrange multiplier; The partial differential of f(ω, λ) is calculated as follows: Solving the above equations gives the weight vector ω = (ω1, ω2, ..., ω j , ..., ω m ), here ω j for: Substituting formula (12) into formula (16), the above formula can be transformed into: Then ω={ω1,ω2,...,ω m }Normalize to get the standardized optimal weight vector Specifically: In the formula, 3. The unmanned aerial vehicle maneuvering decision-making method combining hesitant fuzzy and dynamic deep reinforcement learning as claimed in claim 1 is characterized in that: In step 4, the drone states at multiple moments are combined into a state set as the neural network input, specifically: First, select As the state space of air combat maneuver decision reinforcement learning, it describes the current air combat situation of the drone. Here, This is the sight angle of our drone. is the sight angle of the enemy UAV, R is the distance between the enemy and our UAVs, v U is the speed of our drone, v T is the speed of the enemy UAV, Δh is the height difference between the enemy and our UAVs; Then, the state set (s t-2 ,s t-1 ,s t ) is used as the input of the neural network, and the state s at each moment t-2 ,s t-1 and t Respectively meet:

Images