Autonomous decision-making method for UAV combat based on deep reinforcement learning TD3 algorithm

Abstract

The present invention discloses a method for autonomous decision-making in UAV combat based on deep reinforcement learning TD3 algorithm. The method establishes a UAV motion model, and establishes a UAV air combat model based on the Markov decision process according to the UAV motion model, wherein the UAV air combat model is represented by a quaternary including a state space, an action space, a reward function and a discount factor, and the state transfer function is represented as the UAV motion model. According to the UAV air combat model, the UAV is trained to learn a maneuvering strategy based on the TD3 algorithm. The present invention trains the UAV to learn a maneuvering strategy based on the TD3 algorithm according to the UAV air combat model. The TD3 algorithm can solve the problem of overestimation of Q value, so that the UAV learns a better maneuvering strategy and obtains a position advantage in combat.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles (UAVs), and in particular to an autonomous decision-making method for UAV combat based on a deep reinforcement learning TD3 algorithm. Background Art

[0002] Intelligent autonomous combat drones and drone swarms have great potential to change the battlefield situation. Maneuvering decision-making is the core technology of drone combat confrontation. It is of great significance to study the autonomous maneuvering of drones according to battlefield situation and mission objectives to gain combat advantages.

[0003] When studying the problem of aerial combat between drones based on traditional mathematical methods such as differential game theory, it is necessary to first accurately establish a mathematical model. Regardless of whether it is a qualitative or quantitative problem, the premise is that the maneuvering strategies and performance parameters of both sides need to be known, which is impossible in reality. In future combat, we generally cannot accurately predict the enemy's strategic intentions, tactics, and the performance of the deployed equipment in advance. In addition, the interference of various uncertain factors in the combat environment and the low detectability of the target will limit the applicability of the method. The dynamic model of the drone is relatively complex, and its state equation is a nonlinear differential equation, which makes it difficult to solve and the calculation is huge. It occupies a lot of computing resources and takes a long time. When the number of drones on both sides increases further, a dimensional disaster will occur.

[0004] Although the Deep Deterministic Policy Gradient (DDPG) is suitable for solving high-dimensional continuous action space problems, using this algorithm for deep reinforcement learning of drones in a drone combat environment may cause the problem of overestimation of the Q value, making the total reward value of the drone always relatively low. In other words, when the Q value of the deep deterministic policy gradient algorithm is overestimated, the strategy selected by the drone has errors, and the errors will become larger and larger, resulting in the inability to find an effective strategy and the inability to gain a positional advantage in the combat. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a UAV combat autonomous decision-making method based on deep reinforcement learning TD3 algorithm, which can solve the problem of overestimation of Q value, enable the UAV to learn better maneuvering strategies and gain position advantage in combat.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] In a first aspect, the present invention provides a method for autonomous decision-making in UAV combat based on deep reinforcement learning TD3 algorithm, comprising the steps of:

[0008] Build a UAV motion model;

[0009] According to the UAV motion model, a UAV air combat model is established based on a Markov decision process, wherein the UAV air combat model is represented by a quaternary including a state space, an action space, a reward function and a discount factor, and the UAV motion model represents a state transfer function in the UAV air combat model;

[0010] According to the UAV air combat model, the UAV is trained to learn maneuver strategies based on the TD3 algorithm.

[0011] Compared with the prior art, the first aspect of the present invention has the following beneficial effects:

[0012] According to the UAV motion model and the Markov decision process, this method establishes a UAV air combat model for the combat between our UAV and the enemy UAV. The UAV air combat model is represented by a quaternary including state space, action space, reward function and discount factor. According to the UAV air combat model, the UAV is trained to learn maneuver strategies based on the TD3 algorithm. The TD3 algorithm can solve the problem of overestimation of Q value, so that the UAV can learn better maneuver strategies and gain position advantage in combat.

[0013] Furthermore, the UAV motion model includes a dynamic model and a kinematic model, and establishing the UAV motion model includes:

[0014] Establish the dynamic model of the UAV in the inertial coordinate system:

[0015]

[0016] Wherein, g represents the acceleration of gravity; v represents the speed of the UAV and v satisfies the constraint condition: v min ≤v≤v max ; The track inclination angle γ represents the angle between v and the horizontal plane XOY, γ∈[-π / 2,π / 2]; the track deviation angle ψ represents the angle between the projection of v on the horizontal plane XOY and the coordinate axis X axis, ψ∈(-π,π]; the n τ Indicates tangential overload; the n f represents the normal overload; μ represents the roll angle;

[0017] Establish the kinematic model of the UAV in the inertial coordinate system:

[0018]

[0019] Wherein, the x, y and z represent the coordinates of the UAV in the inertial coordinate system.

[0020] Furthermore, the state space includes: the states and relative states of the enemy UAV and our UAV.

[0021] Further, the state space is constructed in the following manner:

[0022] Set the states of the enemy drone and our drone:

[0023] S = [x r ,y r ,z r ,x b ,y b ,z b ,v r ,v b ,γ r ,γ b ,ψ r ,ψ b ,μ r ,μ b ]

[0024] Based on the states of the enemy UAV and the friendly UAV, the relative states of the enemy UAV and the friendly UAV are set:

[0025] S rb =[D,α,β,v r ,v b ,γ r ,γ b ,ψ r ,ψ b ,μ r ,μ b ]

[0026] Among them, x r ,y r ,z r represents the coordinate value of our drone in three-dimensional space, x b ,y b ,z b represents the coordinate value of the enemy UAV in the three-dimensional space, and the v r represents the speed of our drone, and v b represents the speed of the enemy drone, the γ r represents the track inclination of our UAV, and the γ b represents the track inclination of the enemy UAV, the ψ r represents the track deviation angle of our UAV, and the ψ b represents the track deviation angle of the enemy UAV, and μ r represents the roll angle of our drone, and μ brepresents the roll angle of the enemy UAV, D represents the relative distance between the enemy UAV and the friendly UAV, the horizontal sight line angle α represents the angle between the projection of the sight lines of the enemy UAV and the friendly UAV on the horizontal plane and the X-axis, and the longitudinal sight line angle β represents the angle between the sight lines of the enemy UAV and the friendly UAV and the horizontal plane.

[0027] Furthermore, the action space is constructed by the following formula:

[0028] A=[n τ ,n f ,ω]

[0029] Among them, the n τ Indicates tangential overload, n f represents the normal overload, and ω represents the roll angular velocity of the aircraft.

[0030] Further, the reward function includes a locking reward function, an angle advantage function, a distance advantage function, a height advantage function and a speed advantage function, wherein the locking reward function is:

[0031]

[0032] Among them, the D * Indicates the minimum distance between our drone and the enemy drone when they successfully lock onto each other. * It indicates the maximum angle at which the speed direction of our drone deviates from the center of mass of the enemy drone when the lock is satisfied. * represents the maximum angle at which the speed direction of the enemy UAV deviates from the direction of the hostile UAV pointing to the center of mass of the enemy UAV when the locking condition is satisfied, D represents the relative distance between the hostile UAV and the hostile UAV, p represents the angle at which the speed direction of the hostile UAV deviates from the direction of the center of mass of the hostile UAV, and e represents the angle at which the speed direction of the hostile UAV deviates from the sight vector of the hostile UAV pointing to its center of mass;

[0033] The angle advantage function is:

[0034]

[0035] The distance advantage function is:

[0036]

[0037] The height advantage function is:

[0038]

[0039] The speed advantage function is:

[0040]

[0041] Among them, the v r represents the speed of our drone, and v b represents the speed of the enemy drone, and the D max represents the maximum detection distance of the UAV, Δh represents the height difference between the enemy UAV and our UAV, and v max represents the maximum value of the UAV flight speed, the v min Indicates the minimum value of the flight speed of the UAV;

[0042] The reward function of the drone's single step is:

[0043] R=r lock +k 1 r 1 +k 2 r 2 +k 3 r 3 +k 4 r 4

[0044] Among them, the k 1 to k 4 represents the weight, and the k 1 to k 4 The sum is 1.

[0045] Furthermore, the TD3 algorithm-based training of the UAV learning maneuver strategy includes:

[0046] Step S1: Initialize two evaluator networks Q 1 , Q 2 , parameters of the evaluator network The parameters of the actuator network θ μ , target network parameters θ' μ and experience pool;

[0047] Step S2: preset the number of rounds, and perform the following steps in each round:

[0048] Step S2-1, presetting the maximum number of steps of the drone in each round;

[0049] Step S2-2, the drone selects an action according to the current state and strategy, and adds random noise;

[0050] Step S2-3, the drone performs an action, obtains the next state by the state transfer function and obtains a reward;

[0051] Step S2-4, transferring the current state, the strategy selection action, the reward and the next state obtained by the state transfer function obtained in the steps S2-2 and S2-3 to the experience pool;

[0052] Step S2-5, randomly extracting N samples from the experience pool;

[0053] Step S2-6, calculating the expected reward of the action through the two evaluator target networks, selecting the smaller Q value, and updating the parameters of the evaluator network;

[0054] Step S2-7, updating the parameters of the actuator network by deterministic policy gradient;

[0055] Step S2-8, after updating the parameters of the evaluator network and the actuator network, updating the parameters of the target network;

[0056] Step S2-9, until the number of steps reaches the maximum limit, a round ends;

[0057] Step S3: After all rounds are completed, the training of the drone learning the maneuvering strategy is completed.

[0058] Further, calculating the expected return of the action through the two evaluator target networks, selecting a smaller Q value, and updating the parameters of the evaluator network include:

[0059] Learn to update the evaluator network parameters, where the formula for the loss function L is:

[0060]

[0061] Among them, the s i Indicates the current state, the a i Represents the current action, the target expected value y i According to the current real reward value r i The target expected value y is obtained by multiplying the next output value by the discount factor λ. i The formula is:

[0062]

[0063] Among them, s i+1 Indicates the next step status.

[0064] Further, the updating of the parameters of the actuator network by the deterministic policy gradient includes:

[0065] Learning to update the parameters of the actuator network, the deterministic policy gradient formula of the actuator network is:

[0066]

[0067] Wherein, N represents the number of samples randomly drawn from the experience pool, and Q(s,a|θ Q ) represents the evaluator network, the θ Q represents the parameters of the evaluator network, the μ(sθ μ ) represents the actuator network, the θ μ represents the parameters of the actuator network.

[0068] In a second aspect, the present invention provides a UAV combat autonomous decision-making system based on deep reinforcement learning TD3 algorithm, comprising:

[0069] A UAV motion model building unit is used to build a UAV motion model;

[0070] A drone air combat model establishment unit is used to establish a drone air combat model based on the drone motion model and the Markov decision process, wherein the drone air combat model is represented by a four-tuple including a state space, an action space, a reward function and a discount factor;

[0071] The UAV learning maneuvering strategy training unit is used to train the UAV learning maneuvering strategy based on the TD3 algorithm according to the UAV air combat model.

[0072] Compared with the prior art, the second aspect of the present invention has the following beneficial effects:

[0073] The drone air combat model establishment unit of this system establishes a drone air combat model for the combat between our drone and the enemy drone based on the drone motion model and the Markov decision process. The drone air combat model is represented by a four-tuple including state space, action space, reward function and discount factor; the drone learning maneuver strategy training unit trains the drone learning maneuver strategy based on the TD3 algorithm according to the drone air combat model. The TD3 algorithm can solve the problem of overestimation of Q value, so that the drone can learn a better maneuver strategy and gain position advantage in combat. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0075] Figure 1 A flowchart of a method for autonomous decision-making in UAV combat based on deep reinforcement learning TD3 algorithm provided by one embodiment of the present invention;

[0076] Figure 2 A schematic diagram of an inertial coordinate system provided for one embodiment of the present invention;

[0077] Figure 3 A schematic diagram of the structure of the TD3 algorithm provided by one embodiment of the present invention;

[0078] Figure 4 A simulation result diagram based on TD3 algorithm training provided in one embodiment of the present invention;

[0079] Figure 5 A simulation result diagram of the training comparison between the TD3 algorithm and the DDPG algorithm provided in one embodiment of the present invention;

[0080] Figure 6 A simulation result diagram of the running trajectory of the TD3 algorithm when it converges provided by one embodiment of the present invention;

[0081] Figure 7 A simulation result diagram of an advantageous situation provided by an embodiment of the present invention;

[0082] Figure 8 A simulation result diagram of a disadvantageous situation provided by an embodiment of the present invention;

[0083] Fig. 9 A diagram showing simulation results of a balance of power situation provided by an embodiment of the present invention;

[0084] Fig.10 A structural diagram of a drone combat autonomous decision-making system based on deep reinforcement learning TD3 algorithm provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0085] The technical solutions of the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other in the absence of conflict. In addition, the purpose of the drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present disclosure, but it cannot be understood as a limitation on the scope of protection of the present disclosure.

[0086] In the description of the present invention, features defined as "first" or "second" may include one or more of the features explicitly or implicitly. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0087] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0088] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] When studying the problem of aerial combat between drones based on traditional mathematical methods such as differential game theory, it is necessary to first accurately establish a mathematical model. Regardless of whether it is a qualitative or quantitative problem, the premise is that the maneuvering strategies and performance parameters of both sides need to be known, which is impossible in reality. In future combat, we generally cannot accurately predict the enemy's strategic intentions, tactics, and the performance of the deployed equipment in advance. In addition, the interference of various uncertain factors in the combat environment and the low detectability of the target will limit the applicability of the method. The dynamic model of the drone is relatively complex, and its state equation is a nonlinear differential equation, which makes it difficult to solve and the calculation is huge. It occupies a lot of computing resources and takes a long time. When the number of drones on both sides increases further, a dimensional disaster will occur.

[0090] Although the Deep Deterministic Policy Gradient (DDPG) is suitable for solving high-dimensional continuous action space problems, using this algorithm for deep reinforcement learning of drones in a drone combat environment may cause the problem of overestimation of the Q value, making the total reward value of the drone always relatively low. In other words, when the Q value of the deep deterministic policy gradient algorithm is overestimated, the strategy selected by the drone has errors, and the errors will become larger and larger, resulting in the inability to find an effective strategy and the inability to gain a positional advantage in the combat.

[0091] To solve the above problems, this application establishes a drone air combat model for combat between our drone and enemy drone based on the drone motion model and the Markov decision process. The drone air combat model is represented by a four-tuple including a state space, an action space, a reward function and a discount factor. According to the drone air combat model, the drone is trained to learn maneuver strategies based on the Twin Delayed Deep Deterministic Policy Gradient Algorithm (TD3). The TD3 algorithm can solve the problem of overestimation of Q values, so that the drone can learn better maneuver strategies and gain positional advantages in combat.

[0092] Reference Figures 1 to 9 An embodiment of the present invention provides a method for autonomous decision-making in UAV combat based on deep reinforcement learning TD3 algorithm, comprising the steps of:

[0093] Step S100: Establish a UAV motion model.

[0094] Specifically, establish the "North Celestial East" inertial coordinate system, refer to Figure 2 , the positive direction of the X-axis points to the east, the positive direction of the Y-axis points to the north, and the positive direction of the Z-axis points vertically upward. The UAV is mainly affected by engine thrust, gravity and aerodynamic force during its flight. The UAV motion model includes a dynamic model and a kinematic model. Therefore, the establishment of a UAV motion model includes:

[0095] Establish the dynamic model of the UAV in the inertial coordinate system:

[0096]

[0097] Where g represents the acceleration due to gravity; v represents the speed of the drone and v satisfies the constraint condition: v min ≤v≤v max ; The track inclination angle γ represents the angle between v and the horizontal plane XOY, γ∈[-π / 2,π / 2], pointing to the north is 0°, and turning to the west is positive; the track deviation angle ψ represents the angle between the projection of v on the horizontal plane XOY and the coordinate axis X, ψ∈(-π,π], when it is horizontal it is 0°, and when it is upward it is positive; n τ Indicates tangential overload, tangential overload n τ It expresses the effect of the combined force of thrust and drag on the speed through the tangential overload n τ Change the speed of the drone, tangential overload n τ The direction is the speed direction of the UAV, where the resistance includes the component force of gravity; n f represents the normal overload, μ represents the roll angle, and n represents the normal overload. f The direction is the direction of the top of the drone, the roll angle μ represents the rotation angle of the drone around its longitudinal axis, and the normal overload n f The roll angle μ can change the flight direction and altitude of the drone.

[0098] Establish the kinematic model of the drone in the inertial coordinate system:

[0099]

[0100] Among them, x, y and z represent the coordinates of the drone in the inertial coordinate system.

[0101] Step S200: According to the UAV motion model, a UAV air combat model is established based on the Markov decision process, wherein the UAV air combat model is represented by a four-tuple including a state space, an action space, a reward function and a discount factor, and the UAV motion model is represented as a state transfer function in the UAV air combat model.

[0102] Specifically, the reinforcement learning process is a "trial and error" process, and the Markov decision process is usually used as a model framework to describe reinforcement learning tasks. According to the UAV motion model, based on the Markov decision process, a UAV air combat model is established. The UAV air combat model is represented by a quaternion (S, A, R, λ), where S represents the state space, A represents the action space, R represents the reward function, and λ represents the discount rate. Assume that the immediate reward function that the environment gives to the UAV is r t =r t (s t ,a t ), then define the long-term reward of the drone in the current state λ represents the discount factor. The larger the discount factor, the more "farsighted" the drone is.

[0103] The state space is constructed as follows:

[0104] Set the status of the enemy drone and our drone:

[0105] S=[x r ,y r ,z r ,x b ,y b ,z b ,v r ,v b ,γ r ,γ b ,ψ r ,ψ b ,μ r ,μ b ]

[0106] Based on the states of the enemy drone and our drone, set the relative states of the enemy drone and our drone:

[0107] S rb =[D,α,β,v r ,v b ,γ r ,γ b ,ψ r ,ψ b ,μ r ,μ b ]

[0108] Among them, xr ,y r ,z r represents the coordinate value of our drone in three-dimensional space, x b ,y b ,z b represents the coordinate value of the enemy drone in three-dimensional space, v r Indicates the speed of our drone, v b represents the speed of the enemy drone, γ r represents the track inclination of our UAV, γ b represents the track inclination of the enemy UAV, ψ r represents the track deviation angle of our UAV, ψ b represents the track deviation angle of the enemy UAV, μ r represents the roll angle of our drone, μ b represents the roll angle of the enemy UAV, D represents the relative distance between the enemy UAV and our UAV, the horizontal line of sight deflection angle α represents the angle between the projection of the line of sight of the enemy UAV and our UAV on the horizontal plane and the X-axis, and the longitudinal line of sight deflection angle β represents the angle between the line of sight of the enemy UAV and our UAV and the horizontal plane.

[0109] The state space in this embodiment can not only describe the battlefield situation more intuitively, but also reduce the dimension of the state space.

[0110] The state transfer function is set as: in the current input state s i And the action taken is a i Under the condition of i+1 probability.

[0111] The action space is constructed by the following formula:

[0112] A=[n τ ,n f ,ω]

[0113] Among them, n τ Indicates tangential overload, n f represents the normal overload, and ω represents the roll angular velocity of the aircraft.

[0114] The reward function includes a locking reward function, an angle advantage function, a distance advantage function, a height advantage function, and a speed advantage function, wherein the locking reward function is:

[0115]

[0116] Among them, D * Indicates the minimum distance between our drone and the enemy drone when they successfully lock onto each other, p *It indicates the maximum angle that the speed direction of our drone deviates from the center of mass of the enemy drone when the lock is satisfied, e * It represents the maximum angle at which the enemy UAV's speed direction deviates from the direction of our UAV pointing to the enemy UAV's center of mass when the lock is satisfied, D represents the relative distance between our UAV and the enemy UAV, p represents the angle at which our UAV's speed direction deviates from the direction of the enemy UAV's center of mass, and e represents the angle at which the enemy UAV's speed direction deviates from the sight vector of our UAV pointing to its center of mass;

[0117] The angular advantage function is:

[0118]

[0119] The distance advantage function is:

[0120]

[0121] The height advantage function is:

[0122]

[0123] The speed advantage function is:

[0124]

[0125] Where p represents the angle between the speed direction of our drone and the center of mass of the enemy drone, e represents the angle between the speed direction of the enemy drone and the sight vector of our drone pointing to its center of mass, D represents the relative distance between our drone and the enemy drone, and v r Indicates the speed of our drone, v b represents the speed of the enemy drone, D max represents the maximum detection distance of the UAV, Δh represents the height difference between the enemy UAV and our UAV, and v max Indicates the maximum flight speed of the drone, v min Indicates the minimum flight speed of the drone;

[0126] The reward function of the drone's single step is:

[0127] R=r lock +k 1 r 1 +k 2 r 2 +k 3 r 3 +k 4 r 4

[0128] Among them, k 1 to k 4 represents the weight, and k1 to k 4 The sum is 1.

[0129] In this embodiment, the reward function of a single step of the drone includes a lock reward function, an angle advantage function, a distance advantage function, a height advantage function, and a speed advantage function. This reward function can solve the problem that the algorithm is difficult to converge due to sparse rewards.

[0130] Step S300: According to the drone air combat model, the drone is trained to learn maneuver strategies based on the TD3 algorithm.

[0131] Reference Figure 3 To solve the problem of overestimation of Q value, the TD3 algorithm is used. The TD3 algorithm includes an Actor network and two Critic networks. The two Critic networks are used to estimate the Q value, and the network with a relatively small Q value is selected as the target for update. Based on the TD3 algorithm, a neural network is used to train our drone. The process of using a neural network for training is to perform gradient descent on the constructed policy gradient function, and obtain the optimal neural network parameters after iterative convergence. For example, at step i, our drone inputs the current state space s of our drone i To Actor network μ(s|θ μ ), the Actor network is based on the current state space s i Output the current maneuvering action a of our drone i In order to increase the exploratory power of the drone, random noise N is introduced into the Actor network. i Get the current action of our drone a i =μ(s|θ μ )+N i . Change the current s i and a i Input into the interactive environment and obtain the reward value r through the state transfer function i and next state s i+1 , the current s i 、a i 、r i and i+1 Store them in the experience pool, and randomly extract N sample data (minibatch sampling) from the experience pool to learn and update the network parameters.

[0132] Training drones to learn maneuver strategies based on the TD3 algorithm includes:

[0133] Step S1: Initialize two evaluator networks Q 1 , Q 2 , parameters of the evaluator network The parameters of the actuator network θ μ , target network parameters θ' μ and experience pool;

[0134] Step S2: preset the number of rounds, and perform the following steps in each round:

[0135] Step S2-1, presetting the maximum number of steps of the drone in each round;

[0136] Step S2-2, the drone selects an action based on the current state and strategy, and adds random noise;

[0137] Step S2-3, the drone performs the action, obtains the next state through the state transfer function and obtains the reward;

[0138] Step S2-4, transferring the current state, strategy selection action, reward and next state obtained by the state transfer function obtained in step S2-2 and step S2-3 to the experience pool;

[0139] Step S2-5, randomly select N samples from the experience pool;

[0140] Step S2-6, calculate the expected return of the action through the two evaluator target networks, select the smaller Q value, and update the parameters of the evaluator network;

[0141] Step S2-7, updating the parameters of the actuator network through deterministic policy gradient;

[0142] Step S2-8, after updating the parameters of the evaluator network and the actuator network, update the parameters of the target network;

[0143] Step S2-9, until the number of steps reaches the maximum limit, a round ends;

[0144] Step S3: After all rounds are completed, the training of the drone learning the maneuvering strategy is completed.

[0145] Among them, the expected return of the action is calculated through two evaluator target networks, the smaller Q value is selected, and the parameters of the evaluator network are updated, including:

[0146] Learn to update the evaluator network parameters, where the formula for the loss function L is:

[0147]

[0148] Among them, the target expected value y i According to the current real reward value r i The target expected value y is obtained by multiplying the next output value by the discount factor λ. i The formula is:

[0149]

[0150] Update the parameters of the actuator network through deterministic policy gradients, including:

[0151] Learning to update the parameters of the actuator network, the deterministic policy gradient formula of the actuator network is:

[0152]

[0153] Where N represents the number of samples randomly drawn from the experience pool, Q(s,a|θ Q ) represents the evaluator network, θ Q represents the parameters of the evaluator network, μ(s|θ μ ) represents the actuator network, θ μ Represents the parameters of the actuator network.

[0154] This embodiment trains the drone to learn maneuvering strategies based on the TD3 algorithm. The TD3 algorithm uses two evaluator networks to calculate the Q value, and selects a smaller Q value to calculate the evaluator network parameters, which can prevent the problem of overestimation of the Q value, allowing the drone to learn better maneuvering strategies and gain positional advantages in combat.

[0155] For better explanation, this embodiment carried out a simulation experiment. The training simulation environment and algorithm program were programmed in Python. A deep reinforcement learning framework was built based on PyTorch. The neural networks in the algorithm all adopted a fully connected network architecture, and the activation function was a linear rectifier function (ReLU).

[0156] The red and blue drones have the same performance, the same altitude at the initial moment, a fixed initial horizontal distance, the same initial speed, a climbing angle of 0, and a random heading angle. At the next moment, the red drone makes a maneuver according to the reinforcement learning algorithm strategy, and the blue drone uses the Minmax Strategy to choose from 7 basic maneuvers. Each step is rewarded by the environment according to the status of both parties until the number of steps in a single round reaches the upper limit or one party successfully locks the opponent, and the round ends.

[0157] The blue drone has 7 basic actions including uniform level flight, acceleration, deceleration, climb, descent, left turn and right turn. The parameter selection is as follows:

[0158]

[0159] Table 1

[0160] The specific experimental parameters of the drone air combat simulation scene are shown in the following table:

[0161]

[0162] Table 2

[0163] The relevant neural network parameters and training learning parameters are shown in the following table:

[0164]

[0165] Table 3

[0166] According to the above parameter design, the UAV is trained for 100,000 rounds based on the TD3 algorithm to learn the maneuver strategy, record the cumulative reward of each round, and calculate the average reward return every 200 rounds to obtain the change of the average reward during the training process as the number of training rounds increases. Figure 4 ,The experimental results show that as the number of training rounds increases, the drone keeps trying to explore and learn ,experience strategies, and tends to converge in about 20,000 rounds.

[0167] Since the algorithm selected in this embodiment is the double-delayed deep deterministic policy gradient (TD3 algorithm), which is an optimization of the deep deterministic policy gradient (DDPG algorithm), the training process of the two algorithms is compared without changing the network structure and training parameters. The experimental results are as follows: Figure 5 As shown in the figure, under the same training environment, the TD3 algorithm converges faster than the DDPG algorithm, and the stable reward value obtained after convergence is also slightly higher than that of the DDPG algorithm.

[0168] The simulation experiment also recorded the running trajectories of the two drones when the TD3 algorithm converged. Figure 6 After training our UAV based on the TD3 algorithm, our UAV can make maneuvering decisions autonomously and gain an advantageous position in the game with enemy UAVs.

[0169] In order to verify the adaptability of the TD3 algorithm, the enemy UAV adopted a single-step MINMAX maneuver strategy and carried out 1000 Monte Carlo simulation tests under random initial conditions. The simulation results showed that the cumulative return value of our UAV in these 1000 tests could maintain a high advantage, with a winning rate of more than 80%, indicating that the TD3 algorithm has strong adaptability.

[0170] This embodiment also simulates the confrontation between our UAV and the enemy UAV under different situations. For example, the confrontation simulation results are obtained when our UAV is in an advantageous situation of chasing the enemy UAV, a disadvantageous situation of being chased by the enemy UAV, and a balanced situation where the two sides are approaching each other.

[0171] When our drone is in an advantageous position, the initial status information of our drone and the enemy drone is shown in the following table:

[0172]

[0173] Table 4

[0174] Simulation results refer to Figure 7 When our drone is in an advantageous position, it will maintain its angle advantage while looking for opportunities to gradually reduce the distance with the enemy drone. When the enemy drone makes a turning and descending maneuver to get rid of the lock, our drone will use more reasonable turning timing and speed control to autonomously maneuver to achieve the attack conditions.

[0175] When our drone is at a disadvantage, the initial status information of our drone and the enemy drone is shown in the following table:

[0176]

[0177] Table 5

[0178] Simulation results refer to Figure 8 , when our drone is at a disadvantage, it cannot get rid of the opponent by speed, so it chooses to make an "S"-shaped maneuver, constantly changing its course to get rid of the opponent's tracking, and changing its unfavorable situation by slowing down and letting the enemy drone surpass itself. At the initial moment, our drone is in a situation of being chased by the enemy drone, and the enemy drone maintains its course and accelerates to approach our drone. Our drone has been constantly changing its course since the initial moment, trying to get rid of the attack angle of the enemy drone's weapon. During the process of our drone and the enemy drone approaching each other, the enemy drone has never been able to successfully lock our drone, and was eventually reversed by our drone to occupy a more advantageous position.

[0179] When our drone and the enemy drone are in a state of equilibrium, the initial status information of our drone and the enemy drone is shown in the following table:

[0180]

[0181] Table 6

[0182] Simulation results refer to Fig. 9 When our UAV and the enemy UAV are in a balanced situation of flying towards each other, our UAV can still gain battlefield position advantage through more reasonable trajectory planning and speed control and autonomous maneuvers.

[0183] Reference Fig.10 An embodiment of the present invention provides a UAV combat autonomous decision-making system based on deep reinforcement learning TD3 algorithm, including:

[0184] A UAV motion model building unit is used to build a UAV motion model;

[0185] A drone air combat model building unit is used to build a drone air combat model based on a Markov decision process according to a drone motion model, wherein the drone air combat model is represented by a four-tuple including a state space, an action space, a reward function and a discount factor;

[0186] The UAV learning maneuver strategy training unit is used to train the UAV learning maneuver strategy based on the UAV air combat model and the TD3 algorithm.

[0187] It should be noted that since the drone combat decision system based on deep reinforcement learning TD3 algorithm in this embodiment and the above-mentioned drone combat autonomous decision-making method based on deep reinforcement learning TD3 algorithm are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0188] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0189] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A UAV combat autonomous decision-making method based on deep reinforcement learning TD3 algorithm, characterized in that: Includes steps: Establishing a UAV motion model; the UAV motion model includes a dynamic model and a kinematic model, wherein: Establish the dynamic model of the UAV in the inertial coordinate system: Wherein, g represents the acceleration of gravity; v represents the speed of the UAV and v satisfies the constraint condition: v min ≤v≤v max ; The track inclination angle γ represents the angle between v and the horizontal plane, γ∈[-π / 2,π / 2]; the track deviation angle ψ represents the angle between the projection of v on the horizontal plane and the coordinate axis X axis, ψ∈(-π,π]; the n τ Indicates tangential overload; the n f represents the normal overload; μ represents the roll angle; Establish the kinematic model of the UAV in the inertial coordinate system: Wherein, the x, y and z represent the coordinates of the UAV in the inertial coordinate system; According to the UAV motion model, a UAV air combat model is established based on a Markov decision process, wherein the UAV air combat model is represented by a quaternary including a state space, an action space, a reward function and a discount factor, and the UAV motion model represents a state transfer function in the UAV air combat model; According to the UAV air combat model, the UAV is trained to learn maneuver strategies based on the TD3 algorithm, specifically including: Step S1, initialize two evaluator networks Q1, Q2, and the parameters θ of the evaluator network Q1 ,θ Q2 , the parameters of the actuator network θ μ , target network parameters θ' μ and experience pool; Step S2: preset the number of rounds, and perform the following steps in each round: Step S2-1, presetting the maximum number of steps of the drone in each round; Step S2-2, the drone selects an action according to the current state and strategy, and adds random noise; Step S2-3, the drone performs an action, obtains the next state by the state transfer function and obtains a reward; Step S2-4, transferring the current state, the strategy selection action, the reward and the next state obtained by the state transfer function obtained in the steps S2-2 and S2-3 to the experience pool; Step S2-5, randomly extracting N samples from the experience pool; Step S2-6, calculating the expected reward of the action through the two evaluator target networks, selecting the smaller Q value, and updating the parameters of the evaluator network; Step S2-7, updating the parameters of the actuator network by deterministic policy gradient; Step S2-8, after updating the parameters of the evaluator network and the actuator network, updating the parameters of the target network; Step S2-9, until the number of steps reaches the maximum number of steps, a round ends; Step S3: After all rounds are completed, the training of the drone learning the maneuvering strategy is completed.

2. The method for autonomous decision-making in UAV combat based on deep reinforcement learning TD3 algorithm according to claim 1, characterized in that: The state space includes: the states and relative states of the enemy UAV and our UAV.

3. The method for autonomous decision-making in UAV combat based on deep reinforcement learning TD3 algorithm according to claim 2, characterized in that: The state space is constructed as follows: Set the states of the enemy drone and our drone: S=[x r ,y r ,z r ,x b ,y b ,z b ,v r ,v b ,c r ,c b ,ψ r ,ψ b ,m r ,m b ] Based on the states of the enemy UAV and the friendly UAV, the relative states of the enemy UAV and the friendly UAV are set: S rb =[D,a,b,v r ,v b ,c r ,c b ,ψ r ,ψ b ,m r ,m b ] Among them, x r ,y r ,z r represents the coordinate value of our drone in three-dimensional space, x b ,y b ,z b represents the coordinate value of the enemy UAV in the three-dimensional space, and the v r represents the speed of our drone, and v b represents the speed of the enemy drone, the γ r represents the track inclination of our UAV, and the γ b represents the track inclination of the enemy UAV, the ψ r represents the track deviation angle of our UAV, and the ψ b represents the track deviation angle of the enemy UAV, and μ r represents the roll angle of our drone, and μ b represents the roll angle of the enemy UAV, D represents the relative distance between the enemy UAV and the friendly UAV, the horizontal sight line angle α represents the angle between the projection of the sight lines of the enemy UAV and the friendly UAV on the horizontal plane and the X-axis, and the longitudinal sight line angle β represents the angle between the sight lines of the enemy UAV and the friendly UAV and the horizontal plane.

4. The method for autonomous decision-making in UAV combat based on deep reinforcement learning TD3 algorithm according to claim 1, characterized in that: The action space is constructed by the following formula: A=[n τ ,n f ,oh] Among them, the n τ Indicates tangential overload, n f represents the normal overload, and ω represents the roll angular velocity of the aircraft.

5. The method for autonomous decision-making in UAV combat based on deep reinforcement learning TD3 algorithm according to claim 3, characterized in that: The reward function includes a locking reward function, an angle advantage function, a distance advantage function, a height advantage function, and a speed advantage function, wherein the locking reward function is: Among them, the D * Indicates the minimum distance between our drone and the enemy drone when they successfully lock onto each other. * It indicates the maximum angle at which the speed direction of our drone deviates from the center of mass of the enemy drone when the lock is satisfied. * represents the maximum angle at which the speed direction of the enemy UAV deviates from the direction of the hostile UAV pointing to the center of mass of the enemy UAV when the locking condition is satisfied, D represents the relative distance between the hostile UAV and the hostile UAV, p represents the angle at which the speed direction of the hostile UAV deviates from the direction of the center of mass of the hostile UAV, and e represents the angle at which the speed direction of the hostile UAV deviates from the sight vector of the hostile UAV pointing to its center of mass; The angle advantage function is: The distance advantage function is: The height advantage function is: The speed advantage function is: Among them, the v r represents the speed of our drone, and v b represents the speed of the enemy drone, and the D max represents the maximum detection distance of the UAV, Δh represents the height difference between the enemy UAV and our UAV, and v max represents the maximum value of the UAV flight speed, the v min Indicates the minimum value of the flight speed of the drone; The reward function of the drone's single step is: R=r lock +k1r1+k2r2+k3r3+k4r4 Wherein, k1 to k4 represent weights, and the sum of k1 to k4 is 1.

6. The method for autonomous decision-making in UAV combat based on deep reinforcement learning TD3 algorithm according to claim 1, characterized in that: The step of calculating the expected return of the action through the two evaluator target networks, selecting a smaller Q value, and updating the parameters of the evaluator network includes: Learn to update the evaluator network parameters, where the formula for the loss function L is: Among them, the s i Indicates the current state, the a i Represents the current action, the target expected value y i According to the current real reward value r i The target expected value y is obtained by multiplying the next output value by the discount factor λ. i The formula is: Among them, s i+1 Indicates the next step status.

7. The method for autonomous decision-making in UAV combat based on deep reinforcement learning TD3 algorithm according to claim 6, characterized in that: The updating of the parameters of the actuator network by deterministic policy gradient includes: Learning to update the parameters of the actuator network, the deterministic policy gradient formula of the actuator network is: Wherein, N represents the number of samples randomly drawn from the experience pool, and Q(s,a|θ Q ) represents the evaluator network, the θ Q represents the parameters of the evaluator network, the μ(sθ μ ) represents the actuator network, the θ μ represents the parameters of the actuator network.

8. A UAV combat autonomous decision-making system based on deep reinforcement learning TD3 algorithm, characterized in that: include: The UAV motion model establishment unit is used to establish the UAV motion model; the UAV motion model includes a dynamic model and a kinematic model, wherein: Establish the dynamic model of the UAV in the inertial coordinate system: Wherein, g represents the acceleration of gravity; v represents the speed of the UAV and v satisfies the constraint condition: v min ≤v≤v max ; The track inclination angle γ represents the angle between v and the horizontal plane, γ∈[-π / 2,π / 2]; the track deviation angle ψ represents the angle between the projection of v on the horizontal plane and the coordinate axis X axis, ψ∈(-π,π]; the n τ Indicates tangential overload; the n f represents the normal overload; μ represents the roll angle; Establish the kinematic model of the UAV in the inertial coordinate system: Wherein, the x, y and z represent the coordinates of the UAV in the inertial coordinate system; A drone air combat model establishment unit is used to establish a drone air combat model based on the drone motion model and the Markov decision process, wherein the drone air combat model is represented by a four-tuple including a state space, an action space, a reward function and a discount factor; The UAV learning maneuvering strategy training unit is used to train the UAV learning maneuvering strategy based on the UAV air combat model and the TD3 algorithm, specifically including: Step S1, initialize two evaluator networks Q1, Q2, and the parameters θ of the evaluator network Q1 ,θ Q2 , the parameters of the actuator network θ μ 、Target network parameter θ' Q1 ,θ' Q2 ,θ' μ and experience pool; Step S2: preset the number of rounds, and perform the following steps in each round: Step S2-1, presetting the maximum number of steps of the drone in each round; Step S2-2, the drone selects an action according to the current state and strategy, and adds random noise; Step S2-3, the drone performs an action, obtains the next state by the state transfer function and obtains a reward; Step S2-4, transferring the current state, the strategy selection action, the reward and the next state obtained by the state transfer function obtained in the steps S2-2 and S2-3 to the experience pool; Step S2-5, randomly extracting N samples from the experience pool; Step S2-6, calculating the expected reward of the action through the two evaluator target networks, selecting the smaller Q value, and updating the parameters of the evaluator network; Step S2-7, updating the parameters of the actuator network by deterministic policy gradient; Step S2-8, after updating the parameters of the evaluator network and the actuator network, updating the parameters of the target network; Step S2-9, until the number of steps reaches the maximum number of steps, a round ends; Step S3: After all rounds are completed, the training of the drone learning the maneuvering strategy is completed.

Images