A deep reinforcement learning guidance method based on TD3 algorithm

Abstract

This invention discloses a deep reinforcement learning guidance method based on the TD3 algorithm. It includes the following steps: (1) establishing a relative motion model between the missile and the target; (2) designing a Markov decision process; (3) constructing a network model for the deep reinforcement learning guidance method; training the constructed model offline, and then using the trained network model to output the normal acceleration; designing a Markov decision process, using the line-of-sight angular velocity as the state space and the normal acceleration as the action space, and then using the deterministic policy gradient principle to find the optimal neural network parameters, and then using the trained neural parameters to achieve a direct mapping from the line-of-sight angular velocity to the normal acceleration.

Description

Technical Field

[0001] This invention belongs to the field of guidance methods, specifically relating to a deep reinforcement learning guidance method based on the TD3 algorithm. Background Technology

[0002] In modern missile missions, improving the accuracy of the guidance system is the most important and also the most difficult process. This paper mainly studies the design of the guidance method, which plays a crucial role in the missile guidance system, directly affecting the relative motion between the missile and the target, and also significantly influencing the final miss distance. For a long time, proportional navigation guidance has been widely used on various aircraft models. Although the proportional guidance (PNG) algorithm has achieved excellent performance in many studies, it still suffers from insufficient terminal guidance capability due to its inherent characteristics. Furthermore, since acceleration is proportional to the line-of-sight (LOS) rate, PNG is prone to generating divergent acceleration commands during the terminal guidance phase.

[0003] To address the problems of proportional guidance, some scholars have attempted to use reinforcement learning algorithms to solve these problems and design better guidance strategies to further improve performance. Chethan Chithapuram et al. proposed a reinforcement learning guidance method based on Q-learning algorithm in "Artificial Intelligence Learning based on Proportional Navigation Guidance". 2013 International Conference on Advances in Computing, Communications and Informatics: 1140–1145. However, because the state space and action space of this algorithm are discrete, the normal acceleration command obtained by this guidance method is discrete and does not conform to the real meaning. Zhang Qinhao et al. discretized the proportional coefficient as the action space in "Q-learning Reinforcement Learning Guidance Method". Systems Engineering and Electronics Technology, 2020, 42(2): 414-19. This solved the problem of discontinuity of normal acceleration well. However, this guidance method is different from PNG. Having the same drawback, it cannot solve the problem of LOS rate divergence under certain conditions; Liu Yang et al. proposed a design method for terminal guidance based on DDPG algorithm in "Design and Research of Terminal Guidance Method Based on DDPG Algorithm" Chinese Journal of Computers. 2021, 44(9):1854-1865. This guidance method realizes the output of continuous action, but since the DDPG algorithm adopts a deterministic strategy gradient descent to find the optimal parameters of the neural network, although this exploration method speeds up the convergence speed of the algorithm, it has the problem of overestimation, which leads to poor robustness of the algorithm.

[0004] In summary, existing reinforcement learning guidance methods still have the following problems: (1) insufficient output capability of continuous actions; (2) poor robustness of the algorithm. Summary of the Invention

[0005] The purpose of this invention is to provide a deep reinforcement learning guidance method based on TD3, which aims to use neural networks to directly map the missile's LOS rate to the missile's normal acceleration, thereby generating guidance commands in real time and providing a new approach to obtaining the optimal guidance strategy.

[0006] The technical solution to achieve the objective of this invention is: a deep reinforcement learning guidance method based on TD3, comprising the following steps:

[0007] Step (1): Establish a relative motion model between the missile and the target;

[0008] Step (2): Design a Markov decision process and design a new reward function;

[0009] Step (3): Construct a network model for the deep reinforcement learning guidance method; adopt a dual Critic network module to delay the update of network parameters, and then train the constructed model offline, using the trained network model to output the normal acceleration.

[0010] Furthermore, step (1) "establishing a relative motion model between the missile and the target" specifically involves: establishing a two-degree-of-freedom dynamic model in the natural coordinate system, where V and V0 are... T Let q be the linear velocity of the missile and the target, respectively; q is the line-of-sight angle between the missile and the target. Let σ and σ be the line-of-sight angular velocity. T These are the trajectory inclination angles of the missile and the target, η and η', respectively. T These are the leading angles of the missile and the target, respectively, a M and a T Indicates the normal acceleration of the missile and the target;

[0011]

[0012] Furthermore, step (2) specifically includes the following steps:

[0013] Step (21): Design a Markov decision process

[0014] We used a simplified MDP, which consists of a quadruple: M =<S,A,R,γ> Step (22): State space setting

[0015] We choose the line-of-sight angular rate as the state space, which can cover the entire guidance process. The range for the line-of-sight angular rate is set to [-0.5, 0.5] rad / s.

[0016] Step (23): Motion space selection

[0017] We choose the normal acceleration as the motion space, A: <n>g, where g is the acceleration due to gravity, and N is the normal overload, which is designed to be 10 times the acceleration due to gravity.

[0018] Step (24): Design of the reward function

[0019] In order for the agent to explore in the direction of decreasing line-of-sight angular velocity, we set the reward function to the following form:

[0020]

[0021] Hit represents the termination condition.

[0022] Furthermore, step (3) specifically includes the following steps:

[0023] Step (31): Establish an experience pool;

[0024] Establish an experience pool to store the data structure group (s, a, r, s′).

[0025] Step (32): Create the Actor module;

[0026] A batch of data (s0, a0, s1, r1) is taken from the experience pool, and s0 is input into the Actor network to obtain the predicted action a. 0_predict Here, without adding noise, we directly use s0 and a 0_predict The Q-value is obtained by inputting it into the Critic1 network, and then -Q is used as the loss function to adjust the Actor network; the smaller -Q is, the better. This -Q needs to be obtained from the Critic1 network. The loss function formula is as follows:

[0027] L=-Q(s,π(s)) (14)

[0028] The network parameters φ of the Actor network are updated through deterministic gradients:

[0029]

[0030] In the formula, θ is the parameter of the estimation function Q().

[0031] Step (32): Create the Critic module

[0032] The algorithm uses two Target Critic networks because in practical applications, the Critic network always overestimates the Q value. By using two networks to estimate the Q value and then selecting the smaller one, the overestimation of the Q value can be avoided as much as possible.

[0033] Because two Target Critic networks are used, two Critic networks are also needed for frequent updates. Finally, the smaller of the two Q values ​​is used to update the Critic network.

[0034] y=r+γmin{Q1,Q2} (16)

[0035] The mean squared errors of the above equation with Q1(s0,a0) and Q2(s0,a0) are then used as the loss function for gradient descent. The loss function is as follows:

[0036]

[0037] In addition, it is important to note the predicted action a given to the Target Actor network. 1_predict Adding a noise N transforms it into action a. 1N It is then used as input to two Target Critic networks, thus making the Q-value more accurate in the next step.

[0038] Step (32): Configure delayed soft update method

[0039] During the network model weight update process, it is necessary to update the parameters of the Critic network to the parameters of the Target Critic network and the parameters of the Actor network to the parameters of the Target Actor network. We adopt a soft update method, namely delayed soft update.

[0040] The update method is as follows:

[0041]

[0042] Compared with the prior art, the significant advantages of this invention are:

[0043] (1) The method of this invention proposes a novel missile guidance training framework using a dual-Critic neural network module. To address the overestimation bias that may occur in previous reinforcement learning guidance methods, the minimum value between the two Critics is used to reduce overestimation and improve the robustness and accuracy of the action output.

[0044] (2) Compared with the reinforcement learning guidance method based on the DDPG algorithm, the guidance method based on the TD3 algorithm has better convergence speed and higher returns under the same known environment, and the output guidance command is more robust.

[0045] (3) Compared with the traditional proportional guidance method, the accuracy of the method of the present invention is greatly improved, and when it is used for intercepting maneuvering targets, the normal acceleration of the method in the terminal guidance stage shows better convergence.

[0046] (4) Compared with previous reinforcement learning guidance methods, the method of the present invention has good generalization ability for new combat scenarios that have not been experienced during training.

[0047] (5) The method of the present invention is based on a model-free reinforcement learning guidance method, which can optimize the guidance method in a more complex environment framework.

[0048] (6) The method of the present invention proposes a novel and simple reward function that enables the agent to explore in the direction of decreasing line-of-sight angular velocity. Attached Figure Description

[0049] Figure 1 This is a diagram illustrating the algorithm framework of the guidance method based on the TD3 algorithm of the present invention.

[0050] Figure 2 Generate graphs for experience

[0051] Figure 3 Actor network module diagram

[0052] Figure 4 Critic network module diagram

[0053] Figure 5 The graph shows the learning curves of two reinforcement learning guidance methods based on the TD3 and DDPG algorithms.

[0054] Figure 6 A comparison chart of off-target amounts

[0055] Figure 7 Ballistic curve comparison chart

[0056] Figure 8 Action output diagram of the guidance method based on DDPG algorithm

[0057] Figure 9 This is a graph showing the variation of the normal acceleration curves of the present invention and the PNG algorithm. Detailed Implementation

[0058] The present invention will now be described in further detail with reference to the accompanying drawings.

[0059] A deep reinforcement learning guidance method based on the TD3 algorithm includes the following steps:

[0060] Step 1: Establish a relative motion model between the missile and the target:

[0061] Specifically as follows:

[0062] The two-degree-of-freedom dynamic model is established in the natural coordinate system, where V and V0 are... T Let q be the linear velocity of the missile and the target, respectively; q is the line-of-sight angle between the missile and the target. Let σ and σ be the line-of-sight angular velocity. T These are the trajectory inclination angles of the missile and the target, η and η', respectively. T These are the leading angles of the missile and the target, respectively, a M and a T Indicates the normal acceleration of the missile and the target;

[0063]

[0064] Step 2: Design a Markov decision process:

[0065] Specifically as follows:

[0066] Step (21): Design a Markov decision process

[0067] We used a simplified MDP, which consists of a quadruple: M =<S,A,R,γ> Step (22): State space setting

[0068] We choose the line-of-sight angular rate as the state space, which can cover the entire guidance process. The range for the line-of-sight angular rate is set to [-0.5, 0.5] rad / s.

[0069] Step (23): Motion space selection

[0070] We choose the normal acceleration as the motion space, A: <n>g, where g is the acceleration due to gravity and N is the normal overload, and the normal overload is designed to be 10 times the acceleration due to gravity.

[0071] Step (24): Design of the reward function

[0072] To make the agent explore in the direction of decreasing line-of-sight angular rate, we set the reward function in the following form:

[0073]

[0074]

[0075] The first part is shown in Equation (2). t represents the current time, and R t is the reward generated at the current time. The smaller it is, the higher the reward obtained, and the highest is 100. The second part, R1, is a terminal reward. A reward can only be generated when a collision occurs when the relative distance r < 10 m. It is expected that the minimum miss distance can reach 0.01, and the reward upper limit is controlled. When r < 0.1 m, the magnitude of R1 will affect the overall reward function. However, when 0.1 m < r < 10 m, the value of the R2 reward is still relatively small and has little impact on the overall reward. The reward function designed in this way can make the agent explore with higher precision.

[0076] So the final reward function

[0077]

[0078] where Hit represents the termination condition.

[0079] Step 3: Construct the network model of the deep reinforcement learning guidance method; perform offline training on the constructed model, and then use the trained network model to output the normal acceleration;

[0080] After determining the Markov decision process, we can construct the network model of the deep reinforcement learning guidance method. Compared with the form of a traditional guidance method which is an equation in a line, our reinforcement learning guidance method is a neural network model in form. Through the trained neural network, the state variables of the missile can be directly mapped to the corresponding guidance commands through the network.

[0081] Step (31): Establish an experience pool;

[0082] Establish an experience pool, as shown in the appendix Figure 2 All the data structure groups (s, a, r, s′) generated by the actor network will be stored in this experience pool.

[0083] Step (32): Establish an Actor module;

[0084] As attached Figure 3 As shown, a batch of data is taken from the experience pool. Here, we take a data set (s0, a0, s1, r1) as an example to illustrate the process of training a neural network.

[0085] The loss function of the Actor network is -Q, and the smaller -Q is, the better. This -Q needs to be obtained from the Critic1 network, as shown in the figure above.

[0086] Inputting s0 from experience into the Actor network yields the predicted action a. 0_predict Here, without adding noise, we directly use s0 and a 0_predict The Q-value is obtained by inputting it into the Critic1 network, and then -Q is used as the loss function to adjust the Actor network. The loss function formula is as follows:

[0087] L=-Q(s,π(s)) (5)

[0088] The network parameters φ of the Actor network are updated through deterministic gradients:

[0089]

[0090] In the formula, θ is the parameter of the estimation function Q().

[0091] Step (32): Create the Critic module

[0092] As attached Figure 4 As shown, this algorithm uses two Target Critic networks because in practical applications, the Critic network always overestimates the Q value. By using two networks to estimate the Q value and then selecting the smaller one, we can avoid overestimating the Q value as much as possible.

[0093] Because two Target Critic networks are used, two Critic networks are also needed for frequent updates. Finally, the smaller of the two Q values ​​is used to update the Critic network.

[0094] y=r+γmin{Q1,Q2} (7)

[0095] The mean squared errors of the above equation with Q1(s0,a0) and Q2(s0,a0) are then used as the loss function for gradient descent. The loss function is as follows:

[0096]

[0097] In addition, it is important to note the predicted action a given to the Target Actor network. 1_predict Adding a noise N transforms it into action a. 1N It is then used as input to two Target Critic networks, thus making the Q-value more accurate in the next step.

[0098] Step (32): Configure delayed soft update method

[0099] During the network model weight update process, it is necessary to update the parameters of the Critic network to the parameters of the Target Critic network and the parameters of the Actor network to the parameters of the Target Actor network. We adopt a soft update method, namely delayed soft update.

[0100] The update method is as follows:

[0101]

[0102] Example 1

[0103] The following is a more detailed description with reference to the embodiments.

[0104] The initial simulation parameters of the dynamic model are shown in the table below:

[0105] Table 1 Initial Simulation Parameters of the Dynamic Model

[0106]

[0107] First, when the target is in a sinusoidal maneuver state, the TD3 and DDPG algorithms are used, and the neural network of the algorithms is trained 10,000 times according to the random initial conditions in Table 1. (See attached...) Figure 5 As shown, the TD3 algorithm converges at around 4000 training iterations, while the DDPG algorithm converges at around 8000 training iterations. The reward obtained by the TD3 algorithm after stabilization is higher than that of the DDPG algorithm.

[0108] As attached Figure 6 Then, we compared the two trained reinforcement learning network models with the PNG algorithm under 2000 random conditions, and obtained the statistical comparison shown in Table 2. We defined a miss of <10m as a valid hit and a miss of <2m as a direct hit. In the number of misses less than 10m, the reinforcement learning guidance method slightly outperformed PNG. However, if the miss requirement was reduced to less than 2m, the difference became significant; the reinforcement learning guidance method achieved almost 100% success, while PNG only reached 50%.

[0109] Table 2 Initial Launch Conditions for a Certain Type of Projectile

[0110]

[0111] Subsequently, we selected 6 tests from the 2,000 results for normal acceleration analysis.

[0112]

[0113] Appendix Figure 7 The figure shows a comparison of the ballistic curves from six tests. The hit time using the reinforcement learning guidance method was faster than that of PNG in all six trials. In the first test, it can be seen that PNG's guidance accuracy error was relatively large. In the second and fifth tests, magnified images clearly show that the PNG algorithm failed to hit the target.

[0114] We will compare the normal acceleration curves from the fifth test. The reinforcement learning guidance method based on the DDPG algorithm is attached. Figure 8 As shown, guidance command oscillations occurred at the guidance terminal, while conversely, the attached... Figure 9 No oscillations were observed in the reinforcement learning guidance method based on the TD3 algorithm.< / n> < / n>

Claims

1. A deep reinforcement learning guidance method based on TD3, characterized in that, Includes the following steps: Step (1): Establish a relative motion model between the missile and the target; Step (2): Design a Markov decision process; the reward function in the process is designed as follows: To encourage the agent to explore in the direction of decreasing line-of-sight angular velocity, the reward function is set to the following form: The first part is as shown in Equation (2), where t represents the current moment, and R t represents the reward generated at the current moment. The smaller it is, the higher the reward obtained, and the highest is 100 at a time. The second part, R1, is a terminal reward. A reward can only be generated when a collision occurs when the relative distance r < 10m. It is expected that the minimum miss distance can reach 0.01, and the reward upper limit is controlled. When r < 0.1m, the magnitude of R1 will affect the overall reward function. However, when 0.1m < r < 10m, the reward value of R1 is still relatively small and has little impact on the overall reward. The reward function designed in this way enables the agent to explore towards higher precision; Final reward function Where Hit represents the termination condition; Step (3): Construct a network model for the deep reinforcement learning guidance method; adopt a dual Critic network module to delay the update of network parameters, and then train the constructed model offline, using the trained network model to output the normal acceleration.

2. The guidance method according to claim 1, characterized in that, The step (1) "establishing a relative motion model between the missile and the target" specifically involves: establishing a two-degree-of-freedom dynamic model in the natural coordinate system, where V and V0 are... T Let q be the linear velocity of the missile and the target, respectively; q is the line-of-sight angle between the missile and the target. Let σ and σ be the line-of-sight angular velocity. T These are the trajectory inclination angles of the missile and the target, η and η', respectively. T These are the leading angles of the missile and the target, respectively, a M and a T Indicates the normal acceleration of the missile and the target; 3. The guidance method according to claim 2, characterized in that, Step (2) specifically includes the following steps: Step (21): Design a Markov decision process A simplified MDP is used, consisting of a quadruple: M =<S,A,R,γ> , Step (22): State space setup Choosing the line-of-sight angular rate as the state space allows it to cover the entire guidance process. S: The range for the line-of-sight angular velocity is set to [-0.5, 0.5] rad / s; Step (23): Motion space selection Choosing normal acceleration as the motion space, A: <n> g, where g is the acceleration due to gravity, and n is the normal overload, which is designed to be 10 times the acceleration due to gravity.< / n> Step (24): Design the reward function.

4. The guidance method according to claim 3, characterized in that, Step (3) specifically includes the following steps: Step (31): Establish an experience pool; Establish an experience pool to store the data structure group (s,a,r,s′); Step (32): Create the Actor module; A batch of data (s0, a0, s1, r1) is taken from the experience pool, and s0 is input into the Actor network to obtain the predicted action a. 0_predict Here, without adding noise, we directly use s0 and a 0_predict The Q-value is obtained by inputting it into the Critic1 network. Then, -Q is used as the loss function to adjust the Actor network; the smaller -Q is, the better. This -Q needs to be obtained from the Critic1 network. The loss function formula is as follows: L=-Q(s,π(s)) (5) The network parameters φ of the Actor network are updated through deterministic gradients: In the formula, θ is the parameter of the estimation function Q(); Step (33): Create the Critic module Two networks are used to estimate the Q value, and then the smaller one is selected to avoid overestimating the Q value; Because two Target Critic networks are used, two Critic networks are also needed. Finally, the smaller of the two Q values ​​is used to update the Critic network. y=r+γmin{Q1,Q2} (7) The mean squared errors of the above equation with Q1(s0,a0) and Q2(s0,a0) are then used as the loss function for gradient descent. The loss function is as follows: Predicting action a for the Target Actor network 1_predict Adding a noise transforms it into action a. 1N It is then used as input to the two Target Critic networks, thus making the Q value more accurate in the next step; Step (34): Configure delayed soft update method During the network model weight update process, the parameters of the Critic network are updated to the parameters of the Target Critic network, and the parameters of the Actor network are updated to the parameters of the Target Actor network. A soft update method is used, namely delayed soft update. The update method is as follows:

Images