Six-degree-of-freedom UAV air combat decision-making method based on double-delay determination policy gradient
By establishing a nonlinear six-degree of freedom model of UCAV in MATLAB/SIMULINK and combining Python's TD3 algorithm, a UAV air combat decision-making framework was built, which solved the problem of simple model and limited decision output in the existing technology, and achieved closer integration Actual air combat decision-making and wider application value.
Patent Information
- Application Number
- CN202410458544.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-17
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-04-17
AI Technical Summary
The existing model of the decision-making method for air combat in the UAV is simple and far from the actual air combat. In addition, the algorithm of discrete decision-making output needs to be discrete when applied in air combat, which reduces the selectable range of decision-making output.
Using a six-degree of freedom drone air combat decision-making method based on dual delay determination strategy gradient, a nonlinear six-degree of freedom model of UCAV is established in MATLAB/SIMULINK, and a PID controller is used to design the controller, combining TD3 implemented in Python Algorithm, build a simulated vacuum combat environment and decision-making training framework, design state space, action space and reward functions, and train a strategy network to make air combat decisions.
It realizes a decision-making model that is closer to the actual situation of air combat, broadens the range of selectable decision-making instructions, improves the practical application value of air combat decision-making, and modular design facilitates the replacement of UCAV models and deep reinforcement learning algorithms.
Smart Images

Figure CN118192212B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of unmanned aerial vehicle control, and in particular relates to a six-degree-of-freedom unmanned aerial vehicle air combat decision-making method based on double-delay determination of strategy gradients. Background Art
[0002] At present, unmanned combat air vehicles (UCAVs) have been widely used in the military field. Compared with traditional manned aircraft, UCAVs have the advantages of low cost and not being restricted by human physiological limits. With the development and progress of technologies such as sensors, communications, and artificial intelligence, UCAVs will play a more important role on the battlefield in the future. Air combat decision-making is the core of air combat and plays a key role in the process of air combat. Intelligent air combat decision-making has the advantages of rapid decision-making and strong learning ability. It can assist or even replace pilots in making decisions, and will have a very broad application prospect in future air combat.
[0003] At present, the development direction of autonomous air combat decision-making research is to combine deep reinforcement learning methods. There are two types of deep reinforcement learning algorithms used, algorithms with discrete outputs and algorithms with continuous outputs. The former mainly uses deep Q-network (DQN) or its improved algorithm. Through training, a fully connected neural network is obtained for decision-making. The network input is the state of the two sides of the air combat, and the output is the score of each action in the maneuver action library. Then the action with the highest score is selected for execution. The disadvantage of this method is that it is only applicable to scenarios with discrete action space. The application of DQN in air combat decision-making requires the control instructions to be discretized into several maneuvers, which actually reduces the range of choices for decision output. Deep reinforcement learning algorithms with continuous outputs are more suitable for air combat decision-making. Twin Delayed Deep Deterministic Policy Gradient (TD3) improves the problems of overestimation and bootstrapping in the Deep Deterministic Policy Gradient (DDPG) algorithm. It is a continuous deep reinforcement learning algorithm with excellent performance. Summary of the invention
[0004] 1. Technical issues to be resolved
[0005] The technical problem to be solved by the present invention is how to provide a six-degree-of-freedom UAV air combat decision-making method based on double-delay determination of strategy gradients, so as to solve the problem that the existing technology has a simple model and is far from the actual air combat.
[0006] (II) Technical solution
[0007] In order to solve the above technical problems, the present invention proposes a six-degree-of-freedom UAV air combat decision-making method based on double-delay determination of policy gradient, which comprises the following steps:
[0008] Step S1, establishing a nonlinear six-degree-of-freedom model of UCAV (Unmanned Combat Air Vehicle), constructing the six-degree-of-freedom model in MATLAB / SIMULINK and designing a controller using the PID method;
[0009] Step S2, constructing a simulated air combat environment, in which the performance of the enemy and our UCAVs is the same, a victory and defeat determination method based on the attack zone assumption is adopted, and the enemy aircraft decision-making adopts a pure tracking method;
[0010] Step S3, realizing an air combat decision-making training framework based on a double-delay determination policy gradient algorithm, modeling the air combat decision-making problem as a Markov decision process, and designing a state space, an action space, and a reward function;
[0011] Step S4, randomly initializing the air combat situation and obtaining the policy network used for decision-making in the air combat decision-making training framework through continuous training;
[0012] Step S5: Use the obtained strategy network to conduct air combat in a simulation environment, where our aircraft uses the instructions output by the strategy network and the enemy aircraft uses the instructions generated by the pure tracking method.
[0013] Furthermore, the step S1 specifically includes the following steps:
[0014] Step S11: Establishing a nonlinear six-degree-of-freedom model of UCAV
[0015] When UCAV is flying in the air, it is subject to aerodynamics, engine thrust and gravity. Among them, aerodynamics includes: force and torque. The former is decomposed into [-D, Y, -L] in the airflow coordinate system. T , D, Y, and L are drag, side force, and lift, respectively. The latter is decomposed into M and N are rolling moment, pitching moment and yaw moment respectively, and the engine thrust is recorded as F; the UCAV ground speed is decomposed into [u, v, w] in the body coordinate system. T , u, v, w are the components of ground speed along the x, y, z axes of the body coordinate system; the angular velocity is decomposed into [p, q, r] in the body coordinate system T , p, q, r are the roll angular velocity, pitch angular velocity, and yaw angular velocity respectively;
[0016] The dynamic equation for the movement of the center of mass in the body coordinate system is as follows:
[0017]
[0018] Where θ is the pitch angle, φ is the roll angle, α is the angle of attack, β is the sideslip angle, g is the acceleration of gravity, and m is the mass of the UCAV;
[0019] The dynamic equation for rotation around the center of mass has the following form:
[0020]
[0021] Where I x ,I y ,I z is the moment of inertia of the UCAV in the body coordinate system, I xz is the inertia product in the body coordinate system;
[0022] In the ground coordinate system, the displacement of the UCAV is obtained by converting the three-axis components of the ground speed in the body coordinate system to the ground system:
[0023]
[0024] In the formula, x g ,y g , h is the three-dimensional position of the UCAV in the ground coordinate system, ψ is the yaw angle;
[0025] The relationship between angular velocities is as follows:
[0026]
[0027] The nonlinear six-degree-of-freedom model of UCAV is established by the above formulas (1)-(4). By the above formula, the state quantity derivative is calculated when the state quantity and control input at the previous moment are known, and then the state quantity at the next moment is obtained by integration. The control input of UCAV is the throttle opening and the deflection of the elevator, aileron, and rudder, which are respectively denoted as δ p , δ e , δ a , δ r ;
[0028] Step S12: After completing the construction of the six-degree-of-freedom model in MATLAB / SIMULINK, the PID controller is designed using the PID method. In the PID method, the linear combination of the proportional term, integral term, and differential term of the deviation is used for control. Here, the deviation refers to the difference between the expected instruction and the actual output.
[0029] Furthermore, the PID controller designed in step S12 includes: a speed controller, a pitch angle controller, a roll angle controller and a sideslip angle controller, the speed controller uses the proportional term and the integral term of the speed deviation; the pitch angle controller uses the proportional term and the integral term of the pitch angle deviation and introduces the angle of attack and pitch angle velocity feedback on this basis; the roll angle controller uses the proportional term of the roll angle deviation and introduces the roll angle velocity feedback on this basis; the sideslip angle controller uses the proportional term of the sideslip angle deviation and introduces the yaw deceleration feedback, which is used to control the sideslip angle to be 0.
[0030] Furthermore, the step S2 specifically includes the following steps:
[0031] Step S21: Establish two UCAV models in MATLAB / SIMULINK. Both have the same performance parameters and PID controller. To prevent the UCAV from losing control, the input command V cmd ,θ cmd and φ cmd To limit the clipping;
[0032] Step S22: Implement the simulated air combat setting based on the attack zone, and define the angle between the velocity vector of one UCAV and the line connecting the two aircraft as the attack angle When the UCAV is When the distance between the two aircraft is less than a certain value and within a certain range, the opponent's UCAV is considered to be in the attack zone of the UCAV, and the UCAV is determined to win. During the simulated air combat, the goal of the two UCAVs is to avoid being included in the opponent's attack zone and to include the opponent in their own attack zone to win.
[0033] Step S23: Implement the enemy aircraft decision-making strategy based on the pure tracking method. The idea of this method is to continuously adjust the speed direction so that the speed vector points to the opponent, so as to achieve the purpose of bringing the opponent into the attack area; define the angle between the line connecting the two aircraft and the ground plane as the relative inclination angle μ, and the angle between the projection of the line connecting the two aircraft on the ground plane and the true north direction as the relative azimuth angle σ. The control instructions of the track inclination angle γ and the track azimuth angle χ given by the pure tracking method are shown in formulas (5) and (6), where is the maximum rate of change of the track inclination angle and track azimuth angle, the subscripts "t" and "t+1" represent the current and next simulation steps respectively, and ΔT is the simulation step length:
[0034]
[0035]
[0036] Furthermore, the step S3 specifically includes the following steps:
[0037] Step S31: State space design
[0038] During air combat, the flight state quantities related to the combat situation include three-dimensional coordinates (x, y, z), speed V, track inclination angle γ, track azimuth angle χ, pitch angle θ, and roll angle φ; the state quantities selected in the state space are as follows:
[0039] [R / 1000,γ',χ',z r / 1000,γ r ,x r ,θ r ,φ r ,γ b ,x b ,θ b ,φ b ,V r / 1000,V b / 1000](8)
[0040] Where R is the distance between the two aircraft, in meters, γ' is the track inclination angle of the line connecting the two aircraft, χ' is the track azimuth of the line connecting the two aircraft, and the subscript refers to the red party r or the blue party b;
[0041] Step S32: Action space design
[0042] Define the output action of the policy network as the change in speed V, pitch angle θ, and roll angle φ, and map the policy network output value to the UCAV control command;
[0043] Step S33: Reward function design
[0044] The reward function includes four parts: angle advantage reward, distance advantage reward, height advantage reward, and winning reward. The final reward function is obtained by combining the above four rewards.
[0045] Step S34: Implement an air combat decision-making training framework based on the TD3 algorithm in Python. The TD3 algorithm is a double-delay determination policy gradient algorithm. The purpose of TD3 algorithm training is to obtain a policy network to make decisions. The policy network outputs actions according to the state quantity shown in the input formula (8), including: speed V, pitch angle θ, roll angle φ instructions; during the training process, the policy network continuously makes decisions to interact with the environment, and optimizes the policy network parameters according to the size of the reward value obtained.
[0046] Furthermore, the step S33 specifically includes:
[0047] Step S331: Angle advantage reward
[0048] In the ground coordinate system, the coordinates of the red aircraft are (x r ,y r ,z r), the blue machine coordinate is (x b ,y b ,z b ), then the line from the red machine to the blue machine is R rb (x b -x r ,y b -y r ,z b -z r ); the velocity vector of the red machine is V r (v xr ,v yr ,v zr ), the velocity vector of the blue machine is V b (v xb ,v yb ,v zb ); the angle advantage reward function is as follows:
[0049]
[0050] In the formula The angle of attack. Refers to the escape angle, which is calculated as follows:
[0051]
[0052] Step S332: Distance advantage reward
[0053] The distance advantage reward function is as follows:
[0054]
[0055] Where R lim is the critical distance. When the distance is greater than or equal to R lim When the distance increases, the reward value gradually decreases, R lim The value setting is related to the attack zone range;
[0056] Step S333: Height Advantage Reward
[0057] The height advantage reward function has the following form:
[0058]
[0059] Where z r is the height of our aircraft, in meters;
[0060] Step S334: Winning Reward
[0061] The winner is determined when one party brings the other party into the attack zone or the other party falls to the ground, and a high reward value is given:
[0062]
[0063] Step S335: The final reward function obtained by combining the above four rewards has the following form:
[0064] r=r angle +r dis +r h +r win (14).
[0065] Furthermore, the step S4 specifically includes the following steps:
[0066] Step S41: Randomly initialize the dual-machine situation;
[0067] Step S42: The self-aircraft uses the instructions output by the strategy network in the air combat decision-making training framework, and the enemy aircraft uses the pure tracking method to start simulation training. The self-aircraft uses the TD3 algorithm, and the enemy aircraft uses the pure tracking method. After each simulation step, the control instructions of the self-aircraft and the enemy aircraft are calculated and imported into the air combat environment. After a TD3 algorithm training, the next simulation step is entered until one side wins the simulated air combat or the maximum simulation time limit is exceeded;
[0068] Step S43: Repeat the above steps S41 and S42 continuously, and record the number of simulation steps, accumulated rewards and win / loss results of each round in the training process until the algorithm converges.
[0069] Furthermore, in step S41, in order to ensure that the initial situations of both parties are balanced, the initial directions of the two machines are the same or opposite, with a random horizontal spacing and a vertical spacing of 0.
[0070] Furthermore, the step S5 specifically includes the following steps:
[0071] Step S51: using the obtained strategy network to conduct air combat in a simulation environment, the hostile aircraft uses the instructions output by the strategy network, and the enemy aircraft uses the instructions generated by the pure tracking method;
[0072] Step S52: Record the speed, pitch angle, roll angle, distance, and attack angle of the two aircraft during the air combat and draw corresponding curve graphs to show the situation changes during the air combat. Import the data of the two aircraft during the air combat into the flight data playback software tacview to obtain the air combat trajectory diagram of the two aircraft.
[0073] Furthermore, a nonlinear six-degree-of-freedom model of UCAV is established in MATLAB / SIMULINK and the corresponding PID controller is designed. The TD3 algorithm is implemented in Python, and the state space, action space and reward function are designed. MATLAB is called by Python to realize the transmission of control instructions and air combat situation, and finally a policy network for decision-making is trained.
[0074] (III) Beneficial effects
[0075] The present invention proposes a six-degree-of-freedom UAV air combat decision-making method based on double-delay determination of policy gradients. The beneficial effects of the present invention are as follows:
[0076] The present invention establishes a nonlinear six-degree-of-freedom model of UCAV, which is closer to the actual air combat than the three-degree-of-freedom particle model with overload as input, and the obtained results are more valuable for reference.
[0077] The present invention adopts a dual-delay deterministic policy gradient algorithm with continuous decision output. Compared with the discrete decision output algorithm represented by the DQN algorithm, there is no need to discretize the control instructions into several maneuvers, which broadens the selectable range of decision instructions and has more practical application value.
[0078] The six-degree-of-freedom model of the UCAV in the present invention is built in MATLAB / SIMULINK, and the TD3 algorithm is implemented in Python, which fully utilizes the advantages of rapid modeling of the MATLAB / SIMULINK platform and the rich deep reinforcement learning resources in Python. At the same time, each part has a high degree of modularity, and the UCAV model and deep reinforcement learning algorithm can be easily replaced to meet different research needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0079] Figure 1 It is the operation flow chart of the present invention;
[0080] Figure 2 The speed, pitch angle, roll angle and sideslip angle PID controller block diagram designed for the present invention;
[0081] Figure 3 Schematic diagram of network parameter update in TD3 algorithm. DETAILED DESCRIPTION
[0082] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.
[0083] The present invention mainly solves the problem of UCAV air combat decision-making in a six-degree-of-freedom environment, establishes a nonlinear six-degree-of-freedom air combat environment for UCAV in MATLAB / SIMULINK, implements the TD3 algorithm in Python, calls MATLAB through Python to realize the transmission of control instructions and air combat situation, and finally trains a policy network for decision-making. The method of the present invention has the advantages of high simulation and being closer to the actual air combat.
[0084] The present invention discloses a six-degree-of-freedom UAV air combat decision method based on double-delay determination of policy gradient. Aiming at the problem that the model is simple and far from the actual air combat in the current field, a nonlinear six-degree-of-freedom full-quantity equation of UCAV is established, and on this basis, an air combat decision framework of the double-delay determination policy gradient algorithm is constructed. The method comprises the following steps: first, a nonlinear six-degree-of-freedom full-quantity model of UCAV is constructed in MATLAB / SIMULINK and a controller is designed by using the PID method; second, a simulated air combat environment is constructed, which includes two enemy and our UCAVs with the same performance, a victory and defeat determination method based on the attack zone assumption is adopted, and the enemy aircraft decision adopts a pure tracking method; then, the air combat decision problem is modeled as a Markov decision process, including state space design, action space design and reward function design; finally, the air combat situation is randomly initialized and a policy network for decision-making is obtained through continuous training, and the obtained policy network is used to conduct air combat in a simulated environment.
[0085] To achieve the above object, the technical solution adopted by the present invention is: a six-degree-of-freedom UAV air combat decision-making method based on double-delay determination of strategy gradient, the method comprising the following steps:
[0086] Step S1, establishing a nonlinear six-degree-of-freedom model of UCAV (Unmanned Combat Air Vehicle, UCAV), constructing the six-degree-of-freedom model in MATLAB / SIMULINK and designing a controller using the PID method;
[0087] Step S2, constructing a simulated air combat environment, in which the performance of the enemy and our UCAVs is the same, a victory and defeat determination method based on the attack zone assumption is adopted, and the enemy aircraft decision-making adopts a pure tracking method;
[0088] Step S3, realizing an air combat decision-making training framework based on a double-delay determination policy gradient algorithm, modeling the air combat decision-making problem as a Markov decision process, and designing a state space, an action space, and a reward function;
[0089] Step S4, randomly initializing the air combat situation and obtaining the policy network used for decision-making in the air combat decision-making training framework through continuous training;
[0090] Step S5: Use the obtained strategy network to conduct air combat in a simulation environment, where our aircraft uses the instructions output by the strategy network and the enemy aircraft uses the instructions generated by the pure tracking method.
[0091] Furthermore, the step S1 specifically includes the following steps:
[0092] Step S11: Establish a nonlinear six-degree-of-freedom model of the UCAV.
[0093] When UCAV is flying in the air, it is subject to aerodynamic force, engine thrust and gravity. Among them, aerodynamic force includes: force and moment. The former can be decomposed into [-D, Y, -L] in the airflow coordinate system. T , D, Y, and L are drag, side force, and lift, respectively. The latter can be decomposed into M and N are rolling moment, pitching moment and yaw moment respectively, and the engine thrust is recorded as F. The ground speed of the UCAV can be decomposed into [u, v, w] in the body coordinate system. T , u, v, w are the components of ground speed along the x, y, z axes of the body coordinate system; the angular velocity is decomposed into [p, q, r] in the body coordinate system T , p, q, r are rolling angular velocity, pitch angular velocity, and yaw angular velocity respectively. The dynamic equation of mass center movement in the body coordinate system is as follows:
[0094]
[0095] Where θ is the pitch angle, φ is the roll angle, α is the angle of attack, β is the sideslip angle, g is the acceleration of gravity, and m is the mass of the UCAV.
[0096] The dynamic equation for rotation around the center of mass has the following form:
[0097]
[0098] In the formula, I x ,I y ,I z is the moment of inertia of the UCAV in the body coordinate system, I xz is the product of inertia in the body coordinate system.
[0099] In the ground coordinate system, the displacement of the UCAV can be obtained by converting the three-axis components of the ground speed in the body coordinate system to the ground system:
[0100]
[0101] In the formula, x g ,y g , h is the three-dimensional position of the UCAV in the ground coordinate system, and ψ is the yaw angle.
[0102] The relationship between angular velocities is as follows:
[0103]
[0104] The nonlinear six-degree-of-freedom model of UCAV is established through the above formulas (1)-(4). The six-degree-of-freedom model describes the movement of UCAV in three translational degrees of freedom and three rotational degrees of freedom. Compared with the three-degree-of-freedom model that regards UCAV as a particle with velocity, the six-degree-of-freedom model is more complex and closer to the real UCAV. Through the above formula, when the state quantity and control input at the previous moment are known, the state quantity derivative can be calculated, and then the state quantity at the next moment can be obtained by integration. The control input of UCAV is the throttle opening and the deflection of the elevator, aileron, and rudder, which are respectively denoted as δ p ,δ e ,δ a ,δ r .
[0105] Step S12: After the six-degree-of-freedom model is built in MATLAB / SIMULINK, a PID controller is designed using the PID method. In the PID method, a linear combination of the proportional term, integral term, and differential term of the deviation is used for control, where the deviation refers to the difference between the expected command and the actual output.
[0106] The designed PID controller includes speed controller, pitch angle controller, roll angle controller and sideslip angle controller. Among them, the speed controller uses the proportional term and integral term of speed deviation; the pitch angle controller uses the proportional term and integral term of pitch angle deviation and introduces the angle of attack and pitch angle velocity feedback on this basis; the roll angle controller uses the proportional term of roll angle deviation and introduces the roll angle velocity feedback on this basis; the sideslip angle controller uses the proportional term of sideslip angle deviation and introduces the yaw deceleration feedback, which is mainly used to control the sideslip angle to 0. The controller is used to receive the decision instructions of the strategy network and output the corresponding deflection. Considering the time required for the control instruction to be realized and the needs of the air combat decision-making task, the decision step length is selected as 1s, that is, the strategy network is called once every 1s, and the decision instruction is output once for the PID controller to execute. The decision output of the strategy network is selected as speed V, pitch angle θ, and roll angle φ.
[0107] Furthermore, the step S2 specifically includes the following steps:
[0108] Step S21: Establish two UCAV models in MATLAB / SIMULINK. Both have the same performance parameters and PID controller. To prevent the UCAV from losing control, the input command V cmd ,θ cmd and φ cmd Limiting is performed.
[0109] Step S22: Implement the simulated air combat setting based on the attack zone, and define the angle between the velocity vector of one UCAV and the line connecting the two aircraft as the attack angle φ att, when the UCAV is facing the φ of the other UCAV att When the distance between the two aircraft is less than a certain value and within a certain range, the opponent's UCAV is considered to be in the attack zone of the UCAV, and the UCAV is judged to win. During the simulated air combat, the goal of the two UCAVs is to avoid being included in the opponent's attack zone and to include the opponent in their own attack zone to win.
[0110] Step S23: Implement the enemy aircraft decision-making strategy based on the pure tracking method. The idea of this method is to continuously adjust the speed direction so that the speed vector points to the opponent, so as to achieve the purpose of bringing the opponent into the attack zone. This method is an effective air combat strategy, and in actual air combat, it can also play the role of presenting the minimum frontal area to the enemy. Define the angle between the line connecting the two aircraft and the ground plane as the relative inclination angle μ, and the angle between the projection of the line connecting the two aircraft on the ground plane and the true north direction as the relative azimuth angle σ. The control instructions of the track inclination angle γ and the track azimuth angle χ given by the pure tracking method are shown in formulas (5) and (6), where is the maximum rate of change of the track inclination angle and track azimuth angle, the subscripts "t" and "t+1" represent the current and next simulation steps respectively, and ΔT is the simulation step length:
[0111]
[0112]
[0113] Furthermore, the step S3 specifically includes the following steps:
[0114] Step S31: State space design. During the air combat, the flight state quantities related to the combat situation include three-dimensional coordinates (x, y, z), speed V, track inclination angle γ, track azimuth angle χ, pitch angle θ, and roll angle φ. To describe the current situation of both sides, the most direct way is to list all the state quantities of the two aircraft:
[0115] [x r ,y r ,z r ,V r ,γ r ,x r ,θ r ,φ r ,x b ,y b ,z b ,V b ,γ b ,x b ,θ b ,φ b ](7)
[0116] The subscripts here refer to the red square r or the blue square b.
[0117] This description of the air combat situation is not intuitive enough, and the effect is very poor in practical applications. The algorithm cannot converge after training. In order to improve the effect, the present invention improves the state space design and selects a state that can better represent the air combat situation. The state quantity selected after improvement is as follows:
[0118] [R / 1000,γ',χ',z r / 1000,γ r ,x r ,θ r ,φ r ,γ b ,x b ,θ b ,φ b ,V r / 1000,V b / 1000](8)
[0119] In the formula, R is the distance between the two aircraft, in meters, γ' is the track inclination angle of the line connecting the two aircraft, and χ' is the track azimuth of the line connecting the two aircraft. Compared with the original state space design, two improvements have been made. First, the positions of the two aircraft are described in the form of the distance between the two aircraft and the direction of the line connecting the two aircraft (R / 1000, γ', χ'), rather than the original three-dimensional coordinates of the two aircraft, which more intuitively describes the relative situation of the two aircraft. Since this description does not include the absolute positions of the two aircraft, the height of our aircraft (z r / 1000) to prevent UCAV from falling to the ground; secondly, the numerical magnitude is unified. In the original design, the three-dimensional position and speed units are m, and the angle is in radians, which leads to a large difference in the magnitude of the network input, affecting the convergence effect of the algorithm. After the improvement, the three-dimensional position and speed units are changed to km.
[0120] Step S32: Action space design: Define the policy network output action as the change in speed V, pitch angle θ, and roll angle φ, and map the policy network output value to the UCAV control command.
[0121] Step S33: Reward function design. The present invention makes certain improvements and simplifications to the traditional reward function design based on the experimental results. The improved reward function includes four parts: angle advantage reward, distance advantage reward, height advantage reward, and winning reward.
[0122] Step S331: Angle advantage reward. In the ground coordinate system, the coordinates of the red machine are (x r ,y r ,z r ), the blue machine coordinate is (x b ,y b ,z b ), the line from the red machine to the blue machine is R rb(x b -x r ,y b -y r ,z b -z r ). The velocity vector of the red machine is V r (v xr ,v yr ,v zr ), the velocity vector of the blue machine is V b (v xb ,v yb ,v zb ). The angle advantage reward function in the present invention is as follows:
[0123]
[0124] In the formula The angle of attack. Refers to the escape angle, which is calculated as follows:
[0125]
[0126] Step S332: distance advantage reward. The distance advantage reward function is as follows:
[0127]
[0128] Where R lim is the critical distance. When the distance is greater than or equal to R lim When the distance increases, the reward value gradually decreases, R lim The value setting is related to the attack zone range.
[0129] Step S333: Altitude advantage reward. The altitude advantage reward function is as follows:
[0130]
[0131] Where z r is the height of our aircraft, in meters. In many studies, the height is set to be positively correlated with the reward value. This is based on the understanding that the party with a higher altitude in an air combat often occupies an advantageous position. However, in the actual training process, it was found that adopting this setting will cause the agent to continuously climb to obtain a high reward value, ignoring the air combat with the enemy aircraft. Therefore, the height advantage reward function is simplified in the present invention, and a negative reward value is assigned to a low altitude, in order to prevent the UCAV from falling to the ground. At the same time, considering the actual UCAV performance, the maximum climbing height of the two aircraft is limited to 13000m in the simulation.
[0132] Step S334: Winning reward. When one party brings the other party into the attack zone or the other party falls to the ground, it is determined to be the winner and a high reward value is given:
[0133]
[0134] Step S335: The final reward function obtained by combining the above four rewards has the following form:
[0135] r=r angle +r dis +r h +r win (14)
[0136] Step S34: Implement an air combat decision-making training framework based on the double-delayed deterministic policy gradient algorithm (TD3 algorithm) in Python. The TD3 algorithm is a deep reinforcement learning algorithm with continuous output. It is improved from the original DDPG algorithm and has better performance in practical applications. The TD3 algorithm network structure diagram is as follows Figure 3 As shown in . The purpose of algorithm training is to obtain a policy network to make decisions, and the other networks play an auxiliary training role in the process. The policy network can output actions according to the state quantity shown in the input formula (8), including: speed V, pitch angle θ, roll angle φ instructions. During the training process, the policy network continuously makes decisions to interact with the environment, and optimizes the policy network parameters according to the size of the reward value obtained.
[0137] Furthermore, the step S4 specifically includes the following steps:
[0138] Step S41: Randomly initialize the dual-machine situation. To ensure that the initial situation of both parties is balanced, the initial directions of the two machines are the same or opposite, with random horizontal spacing and zero vertical spacing.
[0139] Step S42: Our aircraft uses the instructions output by the strategy network in the air combat decision-making training framework, and the enemy aircraft uses the pure tracking method to start simulation training. After each simulation step (1s), the control instructions of our side (TD3 algorithm) and the enemy side (pure tracking method) are calculated respectively and imported into the air combat environment. After a TD3 algorithm training, the next simulation step is entered until one side wins the simulated air combat or the maximum simulation time limit is exceeded.
[0140] Step S43: Repeat the above steps S41 and S42 continuously, and record the number of simulation steps, accumulated rewards and win / loss results of each round in the training process until the algorithm converges.
[0141] Furthermore, the step S5 specifically includes the following steps:
[0142] Step S51: Use the obtained strategy network to conduct air combat in a simulation environment, where our aircraft uses the instructions output by the strategy network and the enemy aircraft uses the instructions generated by the pure tracking method.
[0143] Step S52: Record the speed, pitch angle, roll angle, distance, and attack angle of the two aircraft during the air combat and draw corresponding curves to show the situation changes during the air combat. Import the data of the two aircraft during the air combat into the flight data playback software tacview to obtain the air combat trajectory diagram of the two aircraft.
[0144] Embodiment 1:
[0145] Aiming at the air combat decision-making problem of UCAV in a six-degree-of-freedom environment, the present invention establishes a nonlinear six-degree-of-freedom model of UCAV in MATLAB / SIMULINK and designs a corresponding PID controller, implements the TD3 algorithm in Python, designs the state space, action space and reward function, calls MATLAB through Python to realize the transmission of control instructions and air combat situation, and finally trains to obtain a policy network for decision-making. The present invention provides a digital simulation verification that an intelligent agent can defeat an enemy aircraft using pure pursuit in different initial situations through training in a six-degree-of-freedom environment.
[0146] based on Figure 1 The algorithm training framework operation flow chart shown in the figure is a six-degree-of-freedom UAV air combat decision-making method based on double-delay determination of policy gradient, which mainly includes the following steps:
[0147] Step 1: Establish a nonlinear six-degree-of-freedom model of UCAV in MATLAB / SIMULINK. The process is as follows:
[0148] Step 1-1: Establish a nonlinear six-degree-of-freedom model of UCAV. When UCAV is flying in the air, it is subject to aerodynamic force, engine thrust and gravity. Aerodynamic force includes force and moment. The former can be decomposed into [-D, Y, -L] in the airflow coordinate system. T , which can be decomposed into UCAV ground speed can be decomposed into [u,v,w] in the body coordinate system. T , the angular velocity can be decomposed into [p, q, r] in the body coordinate system T The dynamic equation of the center of mass movement in the body coordinate system is as follows:
[0149]
[0150] Where θ is the pitch angle, φ is the roll angle, α is the angle of attack, β is the sideslip angle, g is the acceleration of gravity, and m is the mass of the UCAV.
[0151] The dynamic equation for rotation around the center of mass has the following form:
[0152]
[0153] In the ground coordinate system, the displacement of the UCAV can be obtained by converting the three-axis velocity in the body coordinate system to the ground system:
[0154]
[0155] In the formula, x g ,y g , h is the three-dimensional position of the UCAV in the ground coordinate system, and ψ is the yaw angle.
[0156] The relationship between angular velocities is as follows:
[0157]
[0158] The nonlinear six-degree-of-freedom total quantity equation of UCAV is established through the above formula. When the state quantity and control input at the previous moment are known, the state quantity derivative can be calculated, and then the state quantity at the next moment can be obtained by integration.
[0159] Step 1-2: Design a PID controller based on the above UCAV nonlinear six-degree-of-freedom model, including speed, pitch angle, roll angle, and sideslip angle controllers. The controller structure is as follows: Figure 2 shown.
[0160] Step 2: Build a simulated air combat environment containing two UCAVs with equal performance. The process is as follows:
[0161] Step 2-1: Create two UCAV models in MATLAB / SIMULINK. The two models have the same performance parameters and controllers. To prevent the UCAV from losing control, the input command needs to be limited. The command amplitude is as follows:
[0162]
[0163] Step 2-2: Implement the simulated air combat setting based on the attack zone, and define the angle between the velocity vector of one UCAV and the line connecting the two aircraft as the attack angle When the UCAV is engaged by an enemy UCAV When the angle is less than 20° and the distance between the two aircraft is less than 3 km, the enemy UCAV is considered to be within the attack zone of the UCAV and the UCAV is judged to win. The attack zone range is as follows: Figure 3 shown.
[0164] Step 2-3: Implement the enemy aircraft decision-making strategy based on the pure tracking method. The idea of this method is to continuously adjust the speed direction so that the speed vector points to the enemy aircraft in order to achieve the purpose of bringing the opponent into the attack zone. This method is an effective air combat strategy, and in actual air combat, it can also play the role of presenting the minimum frontal area to the enemy. Define the angle between the line connecting the two aircraft and the ground plane as the relative inclination angle μ, and the angle between the projection of the line connecting the two aircraft on the ground plane and the true north direction as the relative azimuth angle σ. The control instructions for the track inclination angle γ and track azimuth angle χ given by the pure tracking method are as follows, where is the maximum rate of change of the track inclination angle and track azimuth angle, the subscripts "t" and "t+1" represent the current and next simulation steps respectively, and ΔT is the simulation step length:
[0165]
[0166]
[0167] During the simulated air combat, the goal of the two UCAVs is to avoid being included in the attack zone of the other side while at the same time including the other side in the attack zone to achieve victory. At the same time, the maximum change rate of the track inclination angle and the track azimuth angle is as follows:
[0168]
[0169] Step 3: Model the air combat decision problem as a Markov decision process and implement the air combat decision training framework based on the TD3 algorithm. The process is as follows:
[0170] Step 3-1: State space design. During air combat, the flight state quantities related to the combat situation include three-dimensional coordinates (x, y, z), scalar velocity V, track inclination angle γ, track azimuth angle χ, pitch angle θ, and roll angle φ. To describe the current situation of both sides, the most direct way is to list all the state quantities of the two aircraft:
[0171] [x r ,y r ,z r ,V r ,γ r ,x r ,θ r ,φ r ,x b ,y b ,z b ,V b ,γ b ,x b ,θ b ,φ b ](7)
[0172] The subscripts here refer to the red square r or the blue square b.
[0173] This description of the air combat situation is not intuitive enough, and the effect is very poor in practical applications. The algorithm cannot converge after training. In order to improve the effect, the present invention improves the state space design and selects a state that can better represent the air combat situation. The state quantity selected after improvement is as follows:
[0174] [R / 1000,γ',χ',z r / 1000,γ r ,x r ,θ r ,φ r ,γ b ,x b ,θ b ,φ b ,V r / 1000,V b / 1000](8)
[0175] Where R is the distance between the two aircraft, in meters, γ' is the track inclination angle of the line connecting the two aircraft, and χ' is the track azimuth of the line connecting the two aircraft. Compared with the original state space design, two improvements have been made. First, the positions of the two aircraft are described in the form of the distance between the two aircraft and the direction of the line connecting the two aircraft (R / 1000, γ', χ'), rather than the original three-dimensional coordinates of the two aircraft, which more intuitively describes the relative situation of the two aircraft. Since this description does not include the absolute positions of the two aircraft, the height of our aircraft (z r / 1000) to prevent UCAV from falling to the ground; secondly, the numerical magnitude is unified. In the original design, the three-dimensional position and speed units are m, and the angle is in radians, which leads to a large difference in the magnitude of the network input, affecting the convergence effect of the algorithm. After the improvement, the three-dimensional position and speed units are changed to km.
[0176] Step 3-2: Action space design. Define the network output action as the change in speed V, pitch angle θ, and roll angle φ. The control instructions of our machine are as follows, where out 1 ,out 2 ,out 3 are the three output quantities of the network, with a value range of (-1,1):
[0177]
[0178] Step 3-3: Reward function design. The present invention makes certain improvements and simplifications to the traditional reward function design based on the experimental results. The improved reward function includes four parts: angle advantage reward, distance advantage reward, height advantage reward, and winning reward.
[0179] Step 3-3-1: Angle advantage bonus. In the ground coordinate system, the coordinates of the red machine are (x r ,yr ,z r ), the blue machine coordinate is (x b ,y b ,z b ), the line from the red machine to the blue machine is R rb (x b -x r ,y b -y r ,z b -z r ). The velocity vector of the red machine is V r (v xr ,v yr ,v zr ), the velocity vector of the blue machine is V b (v xb ,v yb ,v zb ). The angle advantage reward function in the present invention is as follows:
[0180]
[0181] In the formula The angle of attack. Refers to the escape angle, which is calculated as follows:
[0182]
[0183] Step 3-3-2: Distance advantage bonus. Take R from the attack zone range lim The value is 3000m. The distance advantage reward function is as follows:
[0184]
[0185] When the distance is greater than 3km, the reward value gradually decreases as the distance increases.
[0186] Step 3-3-3: High advantage reward. The high advantage reward function is as follows:
[0187]
[0188] Where z r is the height of our aircraft, in meters. In many studies, the height is set to be positively correlated with the reward value. This is based on the understanding that the party with a higher altitude in an air combat often occupies an advantageous position. However, in the actual training process, it was found that adopting this setting will cause the agent to continuously climb to obtain a high reward value, ignoring the air combat with the enemy aircraft. Therefore, the height advantage reward function is simplified in the present invention, and a negative reward value is assigned to a low altitude, in order to prevent the UCAV from falling to the ground. At the same time, considering the actual UCAV performance, the maximum climbing height of the two aircraft is limited to 13000m in the simulation.
[0189] Step 3-3-4: Winning Reward. The winner is determined when one party brings the other party into the attack zone or the other party falls to the ground, and a high reward value is given:
[0190]
[0191] Step 3-3-5: The final reward function obtained by combining the above four rewards has the following form:
[0192] r=r angle +r dis +r h +r win (14)
[0193] Step 3-4: Implement the air combat decision-making training framework based on the double delayed deterministic policy gradient (TD3) algorithm in Python. The TD3 algorithm is a deep reinforcement learning algorithm with continuous output. It is improved from the original DDPG algorithm and has better performance in practical applications. The purpose of algorithm training is to obtain a policy network for decision-making, and the remaining networks play an auxiliary training role in the process. The policy network can output actions (speed V, pitch angle θ, roll angle φ instructions) based on the input state quantity (as shown in Formula 10). During the training process, the policy network continuously makes decisions to interact with the environment, and optimizes the policy network parameters according to the size of the reward value obtained. The network structure and parameter update method of the TD3 algorithm are as follows Figure 3 as shown in .
[0194] Step 4: Train the policy network for decision making. The process is as follows:
[0195] Step 4-1: Randomly initialize the dual-machine situation. To ensure the initial situation of both parties is balanced, the initial directions of the two machines are the same or opposite, with random horizontal spacing and vertical spacing of 0. The initial three-dimensional coordinates of the red machine (our side) are (0, 0, 5000), and the state quantity of the blue machine is initialized as shown in the following formula:
[0196] x b =0,y b ∈(-9000,9000),z b =5000,x b =π / -π
[0197] Step 4-2: Start simulation training. After each simulation step (1s), calculate the control instructions of our side (TD3 algorithm) and the enemy side (pure tracking method) and import them into the air combat environment. After training the TD3 algorithm once, enter the next simulation step until one side wins the simulated air combat or the maximum simulation time limit is exceeded.
[0198] Step 4-3: Repeat the above steps and record the number of simulation steps, cumulative rewards, and win / loss results of each round during training until the algorithm converges.
[0199] Step 5: Use the obtained strategy network to conduct air combat in a simulation environment. Our aircraft uses the instructions output by the strategy network, and the enemy aircraft uses the instructions generated by the pure tracking method. The process is as follows:
[0200] Step 5-1: Use the obtained strategy network to conduct air combat in a simulation environment. Our aircraft uses the instructions output by the strategy network, and the enemy aircraft uses the instructions generated by the pure tracking method.
[0201] Step 5-2: Record the speed, pitch angle, roll angle, distance, and attack angle of the two aircraft during the air combat and draw corresponding curves to show the situation changes during the air combat. Import the data of the two aircraft during the air combat into the flight data playback software tacview to obtain the air combat trajectory diagram of the two aircraft.
[0202] The beneficial effects of the present invention are as follows:
[0203] 1. The present invention establishes a nonlinear six-degree-of-freedom model of UCAV, which is closer to the actual air combat than the three-degree-of-freedom particle model with overload as input, and the obtained results are more valuable for reference.
[0204] 2. The present invention adopts a deep reinforcement learning algorithm with continuous decision output. Compared with the discrete decision output algorithm represented by the DQN algorithm, there is no need to discretize the control instructions into several maneuvers, which broadens the selectable range of decision instructions and has more practical application value.
[0205] 3. The six-degree-of-freedom model of the UCAV of the present invention is built in MATLAB / SIMULINK, and the TD3 algorithm is implemented in Python, which fully utilizes the advantages of rapid modeling of the MATLAB / SIMULINK platform and the rich deep reinforcement learning resources in Python. At the same time, each part has a high degree of modularity, and the UCAV model and deep reinforcement learning algorithm can be easily replaced to meet different research needs.
[0206] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. An air combat decision-making method based on double-delay determination of policy gradient in a six-degree-of-freedom environment, characterized in that: The method comprises the following steps: Step S1, establishing a nonlinear six-degree-of-freedom model of UCAV (Unmanned Combat Air Vehicle), constructing the six-degree-of-freedom model in MATLAB / SIMULINK and designing a controller using the PID method; Step S2, constructing a simulated air combat environment, in which the performance of the enemy and our UCAVs is the same, a victory and defeat determination method based on the attack zone assumption is adopted, and the enemy aircraft decision-making adopts a pure tracking method; Step S3, realizing an air combat decision-making training framework based on a double-delay determination policy gradient algorithm, modeling the air combat decision-making problem as a Markov decision process, and designing a state space, an action space, and a reward function; Step S4, randomly initializing the air combat situation and obtaining the policy network used for decision-making in the air combat decision-making training framework through continuous training; Step S5, using the obtained strategy network to conduct air combat in a simulation environment, the hostile aircraft uses the instructions output by the strategy network, and the enemy aircraft uses the instructions generated by the pure tracking method; in, The step S3 specifically includes the following steps: Step S31: State space design During air combat, the flight state quantities related to the combat situation include three-dimensional coordinates (x, y, z), speed V, track inclination angle γ, track azimuth angle χ, pitch angle θ, and roll angle φ; the state quantities selected in the state space are as follows: [R / 1000,γ',χ',z r / 1000,c r ,x r ,i r ,f r ,c b ,x b ,i b ,f b ,V r / 1000,V b / 1000] (8) Where R is the distance between the two aircraft, in meters, γ' is the track inclination angle of the line connecting the two aircraft, χ' is the track azimuth of the line connecting the two aircraft, and the subscript refers to the red party r or the blue party b; Step S32: Action space design Define the output action of the policy network as the change in speed V, pitch angle θ, and roll angle φ, and map the policy network output value to the UCAV control command; Step S33: Reward function design The reward function includes four parts: angle advantage reward, distance advantage reward, height advantage reward, and winning reward. The final reward function is obtained by combining the above four rewards. Step S34: Implement an air combat decision-making training framework based on the TD3 algorithm in Python. The TD3 algorithm is a double-delay determination policy gradient algorithm. The purpose of TD3 algorithm training is to obtain a policy network to make decisions. The policy network outputs actions according to the state quantity shown in the input formula (8), including: speed V, pitch angle θ, roll angle φ instructions; during the training process, the policy network continuously makes decisions to interact with the environment, and optimizes the policy network parameters according to the size of the reward value obtained; The step S33 specifically includes: Step S331: Angle advantage reward In the ground coordinate system, the coordinates of the red aircraft are (x r ,y r ,z r ), the blue machine coordinate is (x b ,y b ,z b ), then the line from the red machine to the blue machine is R rb (x b -x r ,y b -y r ,z b -z r ); the velocity vector of the red machine is V r (v xr ,v yr ,v zr ), the velocity vector of the blue machine is V b (v xb ,v yb ,v zb ); the angle advantage reward function is as follows: In the formula The angle of attack. Refers to the escape angle, which is calculated as follows: Step S332: Distance advantage reward The distance advantage reward function is as follows: Where R lim is the critical distance. When the distance is greater than or equal to R lim When the distance increases, the reward value gradually decreases, R lim The value setting is related to the attack zone range; Step S333: Height Advantage Reward The height advantage reward function has the following form: Where z r is the height of our aircraft, in meters; Step S334: Winning Reward The winner is determined when one party brings the other party into the attack zone or the other party falls to the ground, and a high reward value is given: Step S335: The final reward function obtained by combining the above four rewards has the following form: r=r angle +r dis +r h +r win (14)。 2. The air combat decision-making method based on double-delay determination of policy gradient in a six-degree-of-freedom environment as claimed in claim 1, characterized in that: The step S1 specifically includes the following steps: Step S11: Establishing a nonlinear six-degree-of-freedom model of UCAV When UCAV is flying in the air, it is subject to aerodynamic force, engine thrust and gravity. Among them, aerodynamic force includes: force and moment. The former is decomposed into [-D, Y, -L] in the airflow coordinate system. T , D, Y, and L are drag, side force, and lift, respectively. The latter is decomposed into M and N are rolling moment, pitching moment and yaw moment respectively, and the engine thrust is recorded as F; the UCAV ground speed is decomposed into [u, v, w] in the body coordinate system. T , u, v, w are the components of ground speed along the x, y, z axes of the body coordinate system; the angular velocity is decomposed into [p, q, r] in the body coordinate system T , p, q, r are the roll angular velocity, pitch angular velocity, and yaw angular velocity respectively; The dynamic equation for the movement of the center of mass in the body coordinate system is as follows: Where θ is the pitch angle, φ is the roll angle, α is the angle of attack, β is the sideslip angle, g is the acceleration of gravity, and m is the mass of the UCAV; The dynamic equation for rotation around the center of mass has the following form: In the formula, I x ,I y ,I z is the moment of inertia of the UCAV in the body coordinate system, I xz is the inertia product in the body coordinate system; In the ground coordinate system, the displacement of the UCAV is obtained by converting the three-axis components of the ground speed in the body coordinate system to the ground system: In the formula, x g ,y g , h is the three-dimensional position of the UCAV in the ground coordinate system, ψ is the yaw angle; The relationship between angular velocities is as follows: The nonlinear six-degree-of-freedom model of UCAV is established by the above formulas (1)-(4). By the above formula, the state quantity derivative is calculated when the state quantity and control input at the previous moment are known, and then the state quantity at the next moment is obtained by integration. The control input of UCAV is the throttle opening and the deflection of the elevator, aileron, and rudder, which are respectively denoted as δ p ,δ e ,δ a ,δ r ; Step S12: After completing the construction of the six-degree-of-freedom model in MATLAB / SIMULINK, the PID controller is designed using the PID method. In the PID method, the linear combination of the proportional term, integral term, and differential term of the deviation is used for control. Here, the deviation refers to the difference between the expected instruction and the actual output.
3. The air combat decision-making method based on double-delay determination of policy gradient in a six-degree-of-freedom environment as claimed in claim 2, characterized in that: The PID controller designed in step S12 includes: a speed controller, a pitch angle controller, a roll angle controller and a sideslip angle controller. The speed controller uses the proportional term and the integral term of the speed deviation; the pitch angle controller uses the proportional term and the integral term of the pitch angle deviation and introduces the angle of attack and pitch angle velocity feedback on this basis; the roll angle controller uses the proportional term of the roll angle deviation and introduces the roll angle velocity feedback on this basis; the sideslip angle controller uses the proportional term of the sideslip angle deviation and introduces the yaw deceleration feedback, so as to control the sideslip angle to be 0.
4. The air combat decision-making method based on double-delay determination of policy gradient in a six-degree-of-freedom environment as claimed in claim 2 or 3, characterized in that: The step S2 specifically includes the following steps: Step S21: Establish two UCAV models in MATLAB / SIMULINK. Both have the same performance parameters and PID controller. To prevent the UCAV from losing control, the input command V cmd ,θ cmd and φ cmd To limit the clipping; Step S22: Implement the simulated air combat setting based on the attack zone, and define the angle between the velocity vector of one UCAV and the line connecting the two aircraft as the attack angle When the UCAV is When the distance between the two aircraft is less than a certain value and within a certain range, the opponent's UCAV is considered to be in the attack zone of the UCAV, and the UCAV is determined to win. During the simulated air combat, the goal of the two UCAVs is to avoid being included in the opponent's attack zone and to include the opponent in their own attack zone to win. Step S23: Implement the enemy aircraft decision-making strategy based on the pure tracking method. The idea of this method is to continuously adjust the speed direction so that the speed vector points to the opponent, so as to achieve the purpose of bringing the opponent into the attack area; define the angle between the line connecting the two aircraft and the ground plane as the relative inclination angle μ, and the angle between the projection of the line connecting the two aircraft on the ground plane and the true north direction as the relative azimuth angle σ. The control instructions of the track inclination angle γ and the track azimuth angle χ given by the pure tracking method are shown in formulas (5) and (6), where is the maximum rate of change of the track inclination angle and track azimuth angle, the subscripts "t" and "t+1" represent the current and next simulation steps respectively, and ΔT is the simulation step length:
5. The air combat decision-making method based on double-delay determination of policy gradient in a six-degree-of-freedom environment as claimed in claim 1, characterized in that: The step S4 specifically includes the following steps: Step S41: Randomly initialize the dual-machine situation; Step S42: The self-aircraft uses the instructions output by the strategy network in the air combat decision-making training framework, and the enemy aircraft uses the pure tracking method to start simulation training. The self-aircraft uses the TD3 algorithm, and the enemy aircraft uses the pure tracking method. After each simulation step, the control instructions of the self-aircraft and the enemy aircraft are calculated and imported into the air combat environment. After a TD3 algorithm training, the next simulation step is entered until one side wins the simulated air combat or the maximum simulation time limit is exceeded; Step S43: Repeat the above steps S41 and S42 continuously, and record the number of simulation steps, accumulated rewards and win / loss results of each round in the training process until the algorithm converges.
6. The air combat decision-making method based on double-delay determination of policy gradient in a six-degree-of-freedom environment as claimed in claim 5, characterized in that: In step S41, in order to ensure that the initial situations of both parties are balanced, the initial directions of the two machines are the same or opposite, with random horizontal spacing and zero vertical spacing.
7. The air combat decision-making method based on double-delay determination of policy gradient in a six-degree-of-freedom environment as claimed in claim 5, characterized in that: The step S5 specifically includes the following steps: Step S51: using the obtained strategy network to conduct air combat in a simulation environment, the hostile aircraft uses the instructions output by the strategy network, and the enemy aircraft uses the instructions generated by the pure tracking method; Step S52: Record the speed, pitch angle, roll angle, distance, and attack angle of the two aircraft during the air combat and draw corresponding curve graphs to show the situation changes during the air combat. Import the data of the two aircraft during the air combat into the flight data playback software tacview to obtain the air combat trajectory diagram of the two aircraft.
8. The air combat decision-making method based on double-delay determination of policy gradient in a six-degree-of-freedom environment as claimed in claim 5, characterized in that: A nonlinear six-degree-of-freedom model of UCAV is established in MATLAB / SIMULINK and the corresponding PID controller is designed. The TD3 algorithm is implemented in Python, and the state space, action space and reward function are designed. MATLAB is called by Python to realize the transmission of control instructions and air combat situation, and finally a policy network for decision-making is trained.
