A method for generating a deep reinforcement learning guidance law based on DDPG

Abstract

The application provides a generation method of a deep reinforcement learning guidance law based on DDPG, and the method comprises the following steps: establishing a relative motion model of a missile and a target; designing a Markov decision process based on the relative motion model of the missile and the target; establishing an experience pool for storing a data structure group based on the Markov decision process; establishing an Actor network, a Critic network, a target network of the Actor network and a target network of the Critic network; training the Actor network in real time to obtain a four-tuple at each moment, and sequentially storing the four-tuple at each moment into the experience pool; continuously updating network parameters of the Actor network and the Critic network; updating network parameters of the target network of the Actor network and the target network of the Critic network by using the updated network parameters of the Actor network and the Critic network, and assigning the network parameters to network parameters of the Actor network at the current moment and network parameters of the Critic network at the current moment; and under the condition that the Actor network and the Critic network are all converged, taking the Actor network at the current moment as the guidance law.

Description

Technical Field

[0001] This invention relates to the field of guidance law technology, and in particular to a method for generating guidance laws based on deep reinforcement learning using DDPG. Background Technology

[0002] In modern flight missions, the high maneuverability of target motion and the nonlinearity, strong coupling, and multiple time-varying factors of aircraft motion present numerous challenges to the research of guidance laws. Furthermore, the new combat environment has placed new demands on guidance law research; simply achieving low miss distances is no longer sufficient to complete combat missions. Guidance laws must also satisfy multiple terminal constraints, such as terminal angle constraints. Enabling the aircraft to hit the target at a specific angle can significantly improve the destructive effectiveness of the warhead, while terminal angle control is also a necessary requirement for coordinated aircraft attacks.

[0003] Currently, traditional guidance laws cannot achieve high maneuverability of target motion as well as nonlinearity, strong coupling, and time-varying characteristics of aircraft motion. Summary of the Invention

[0004] This invention provides a method for generating guidance laws based on DDPG deep reinforcement learning, which can solve the technical problems in the prior art where guidance laws cannot achieve high maneuverability of target motion and nonlinearity, strong coupling and time-varying nature of aircraft motion.

[0005] According to one aspect of the present invention, a method for generating deep reinforcement learning guidance laws based on DDPG is provided, the method comprising:

[0006] Establish a relative motion model between the missile and the target;

[0007] Design a Markov decision process based on a model of the relative motion between the missile and the target;

[0008] An experience pool for storing data structure groups is established based on Markov decision processes;

[0009] Establish the Actor network, Critic network, the target network of the Actor network, and the target network of the Critic network;

[0010] The Actor network is trained in real time based on the relative motion model between the missile and the target and the Markov decision process to obtain the quadruplets at each time step, and the quadruplets at each time step are stored in the experience pool in sequence.

[0011] When the number of quadruplets in the experience pool reaches a preset number, a quadruplet is selected from the experience pool at preset intervals to continuously update the network parameters of the Actor network and the Critic network.

[0012] The network parameters of the target network of the Actor network and the target network of the Critic network are updated using the updated network parameters of the Actor network and the Critic network at each preset interval. The updated network parameters of the target network of the Actor network are used as the network parameters of the Actor network at the current moment, and the updated network parameters of the target network of the Critic network are used as the network parameters of the Critic network at the current moment.

[0013] If both the Actor network and the Critic network converge, the Actor network at the current moment is used as the guidance law.

[0014] Preferably, the relative motion model between the missile and the target is established using the following set of formulas:

[0015]

[0016] In the formula, m represents the missile, t represents the target, and V represents the target. m and V t denoted by , and denoted by r, respectively, where r is the relative distance between the missile and the target. q represents the rate of change of the relative distance between the missile and the target; q represents the line-of-sight angle between the missile and the target. σ is the line-of-sight angular velocity between the missile and the target. m and σ t η represents the trajectory inclination angles of the missile and the target, respectively. m and η t θ represents the leading angle of the missile and the target, respectively. m and θ t These are the missile's velocity inclination angle and the target's trajectory inclination angle, respectively. and These are the rate of change of the missile's velocity tilt angle and the rate of change of the target's trajectory tilt angle, respectively; a m and a t x represents the normal acceleration of the missile and the target, respectively. m and x t Here are the x-axis coordinates of the missile and the target, respectively, and the y-axis coordinates of the target. m and y t These are the y-axis coordinates of the missile and the target, respectively, and v m and v t These are the normal velocities of the missile and the target, respectively.

[0017] Preferably, the Markov decision process designed based on the relative motion model between the missile and the target includes:

[0018] The relative positions of the missile and the target in the x-axis and y-axis directions, the relative velocities of the missile and the target in the x-axis and y-axis directions, the current line-of-sight angle between the missile and the target, and the expected terminal line-of-sight angle are used as state variables in the state space.

[0019] The missile's normal acceleration is used as a state variable in the action space;

[0020] Hit bonuses are awarded based on the relative distance between the missile and the target.

[0021] The terminal angle reward is obtained based on the current line-of-sight angle between the missile and the target and the expected terminal line-of-sight angle.

[0022] A reward function is derived based on hit reward and terminal angle reward.

[0023] A Markov decision process is constructed based on a state space, action space, reward function, and decay rate.

[0024] Preferably, the state space is obtained using the following formula:

[0025] S=[Δx,Δy,Δv x ,Δv y ,q,q e ];

[0026] Δx=x t -x m Δy=y t -y m ;

[0027] Δv x =v xt -v xm Δv y =v yt -v ym ;

[0028] The action space is obtained using the following formula:

[0029] A = [a m ];

[0030] The hit bonus is obtained using the following formula:

[0031]

[0032] Obtain the terminal corner reward using the following formula:

[0033]

[0034] Δq=|qq e |;

[0035] The reward function is obtained using the following formula:

[0036]

[0037] In the formula, S represents the state space, Δx and Δy represent the relative positions of the missile and the target along the x-axis and y-axis, respectively, and Δv... x and Δv y Let q and q' be the relative velocities of the missile and the target along the x-axis and y-axis, respectively. e These represent the current line-of-sight angle and the desired terminal line-of-sight angle between the missile and the target, respectively. m and x t Here are the x-axis coordinates of the missile and the target, respectively, and the y-axis coordinates of the target. m and y t These are the y-axis coordinates of the missile and the target, respectively, and v xm and v xt Let v be the normal velocity of the missile and the target along the x-axis, respectively. ym and v yt Let A and a be the normal velocities of the missile and the target along the y-axis, respectively, where A is the action space and a is the velocity of the target. m R is the missile's normal acceleration, R1 is the hit reward, r is the relative distance between the missile and the target, k1 and k2 are the first and second hit reward coefficients, respectively, R2 is the terminal angle reward, Δq is the error between the current line-of-sight angle and the expected terminal line-of-sight angle between the missile and the target, k3 and k4 are the first and second line-of-sight angle reward coefficients, R is the reward function, and k5 is the terminal angle coordination coefficient.

[0038] Preferably, the Actor network has a 6-dimensional input and a 1-dimensional output, corresponding to the state space and action space, respectively; the Critic network has two 6+1-dimensional inputs, corresponding to the state space and action space, respectively, and a 1-dimensional output.

[0039] Preferably, the activation functions of the first two layers of the Actor network are ReLU functions, and the activation function of the third layer is a tanh function; the activation functions of all layers of the Critic network are ReLU functions.

[0040] Preferably, the parameters of the Actor network and the target network of the Actor network are the same, and the parameters of the Critic network and the target network of the Critic network are the same.

[0041] Preferably, the Actor network is trained in real time based on the relative motion model between the missile and the target and the Markov decision process to obtain the quadruplets at each time step, and the quadruplets at each time step are stored sequentially into the experience pool, including:

[0042] The current state is obtained in real time based on the relative motion model between the missile and the target;

[0043] The reward value at the current moment is obtained in real time based on the Markov decision process;

[0044] Input the current state into the Actor network to obtain the action at the current moment;

[0045] The quadruple consisting of the current state, reward value, action, and the state at the next moment is stored in the experience pool in sequence.

[0046] Preferably, the Actor network is updated using the following formula:

[0047]

[0048] The Critic network is updated using the following formula:

[0049]

[0050] Where, δ i =R(s) i ,a i )+γQ(s i+1 ,a(s i+1 ,θ),η)-Q(s i ,a i ,η);

[0051] In the formula, θ and η are the network parameters to be updated for the Actor network and Critic network, respectively; θ' and η' are the updated network parameters for the Actor network and Critic network, respectively; β and α are the learning rates for the Actor network and Critic network, respectively; and N is the number of empirical data points updated in one step. Let s be the partial derivatives with respect to θ, a, and η, respectively. i Let a be the state at time i. i Let a(s) be the action at time i. i Q(s,θ) is the state-action value function fitted by the Actor network at time i. i ,a i ,η) represents the Critic network at time i based on s i ,a i The fitted state-action value function, Q(s) i ,a(s i ,θ),η) are the Critic network at time i based on s i ,a(s i The state-action value function fitted by θ, δ i For the updated loss function, R(s) i ,a i Let be the reward function at time i, γ be the decay rate, and Q(s) be the reward function at time i. i+1 ,a(s i+1 ,θ),η) are the Critic network at time i+1 based on s i+1,a(s i+1 The state action value function fitted by θ).

[0052] Preferably, the target network of the Actor network is updated using the following formula:

[0053] θ”'=τθ+(1-τ)θ”;

[0054] The target network of the Critic network is updated using the following formula:

[0055] η”'=τη+(1-τ)η”;

[0056] In the formula, τ is the learning rate of the target network of the Actor network and the target network of the Critic network, θ” and η” are the network parameters to be updated of the target network of the Actor network and the target network of the Critic network, respectively, θ”' and η”' are the updated network parameters of the target network of the Actor network and the target network of the Critic network, respectively, and θ and η are the network parameters to be updated of the Actor network and the Critic network, respectively.

[0057] According to another aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the methods described above.

[0058] Compared with the prior art, the application of the technical solution of the present invention has the following beneficial effects:

[0059] 1. It implements a guidance law with angle constraints, which can attack targets at a set attack angle;

[0060] 2. Compared with the traditional proportional guidance method, it has higher accuracy and the ability to intercept maneuvering targets, and has better convergence in the terminal normal acceleration.

[0061] 3. It has good generalization ability for complex combat scenarios;

[0062] 4. It is a model-free reinforcement learning guidance law that can optimize guidance laws in more complex environmental frameworks. Attached Figure Description

[0063] The accompanying drawings, which form part of this specification, are provided to further illustrate embodiments of the invention and, together with the textual description, explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0064] Figure 1 A flowchart is shown for a method for generating deep reinforcement learning guidance laws based on DDPG according to an embodiment of the present invention;

[0065] Figure 2 A schematic diagram of the structure of an Actor network according to an embodiment of the present invention is shown;

[0066] Figure 3 A schematic diagram of the structure of a Critic network according to an embodiment of the present invention is shown;

[0067] Figure 4 The diagram shows the trajectory of a missile with DDPG guidance law and proportional guidance law according to an embodiment of the present invention.

[0068] Figure 5 The diagram shows the distance variation curves between a missile and a target provided by a DDPG guidance law and a proportional guidance law according to an embodiment of the present invention.

[0069] Figure 6 The diagram shows the line-of-sight angle variation curves of missiles with DDPG guidance law and proportional guidance law according to an embodiment of the present invention;

[0070] Figure 7 The diagram shows the normal acceleration variation curves of missiles with DDPG guidance law and proportional guidance law provided according to an embodiment of the present invention. Detailed Implementation

[0071] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0073] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0074] like Figure 1 As shown, this invention provides a method for generating deep reinforcement learning guidance laws based on DDPG, the method comprising:

[0075] Establish a relative motion model between the missile and the target;

[0076] Design a Markov decision process based on a model of the relative motion between the missile and the target;

[0077] An experience pool for storing data structure groups is established based on Markov decision processes;

[0078] Establish the Actor network, Critic network, the target network of the Actor network, and the target network of the Critic network;

[0079] The Actor network is trained in real time based on the relative motion model between the missile and the target and the Markov decision process to obtain the quadruplets at each time step, and the quadruplets at each time step are stored in the experience pool in sequence.

[0080] When the number of quadruplets in the experience pool reaches a preset number, a quadruplet is selected from the experience pool at preset intervals to continuously update the network parameters of the Actor network and the Critic network.

[0081] The network parameters of the target network of the Actor network and the target network of the Critic network are updated using the updated network parameters of the Actor network and the Critic network at each preset interval. The updated network parameters of the target network of the Actor network are used as the network parameters of the Actor network at the current moment, and the updated network parameters of the target network of the Critic network are used as the network parameters of the Critic network at the current moment.

[0082] If both the Actor network and the Critic network converge, the Actor network at the current moment is used as the guidance law.

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] 1. It implements a guidance law with angle constraints, which can attack targets at a set attack angle;

[0085] 2. Compared with the traditional proportional guidance method, it has higher accuracy and the ability to intercept maneuvering targets, and has better convergence in the terminal normal acceleration.

[0086] 3. It has good generalization ability for complex combat scenarios;

[0087] 4. It is a model-free reinforcement learning guidance law that can optimize guidance laws in more complex environmental frameworks.

[0088] In this invention, if the gradient of the change in the network parameters of the Actor network is less than a preset value, the Actor network is determined to be converged. Similarly, if the gradient of the change in the network parameters of the Critic network is less than a preset value, the Critic network is determined to be converged.

[0089] The gradient of the network parameters is the difference between the network parameters at two adjacent time points.

[0090] According to one embodiment of the present invention, a relative motion model between the missile and the target is established using the following set of formulas:

[0091]

[0092] In the formula, m represents the missile, t represents the target, and V represents the target. m and V t denoted by , and denoted by r, respectively, where r is the relative distance between the missile and the target. q represents the rate of change of the relative distance between the missile and the target; q represents the line-of-sight angle between the missile and the target. σ is the line-of-sight angular velocity between the missile and the target. m and σ t η represents the trajectory inclination angles of the missile and the target, respectively. m and η t θ represents the leading angle of the missile and the target, respectively. m and θ t These are the missile's velocity inclination angle and the target's trajectory inclination angle, respectively. and These are the rate of change of the missile's velocity tilt angle and the rate of change of the target's trajectory tilt angle, respectively; a m and a t x represents the normal acceleration of the missile and the target, respectively.m and x t Here are the x-axis coordinates of the missile and the target, respectively, and the y-axis coordinates of the target. m and y t These are the y-axis coordinates of the missile and the target, respectively, and v m and v t These are the normal velocities of the missile and the target, respectively.

[0093] According to one embodiment of the present invention, designing a Markov decision process based on a missile-target relative motion model includes:

[0094] The relative positions of the missile and the target along the x and y axes, their relative velocities along the x and y axes, the current line-of-sight angle between the missile and the target, and the desired terminal line-of-sight angle are used as state variables in the state space, i.e.:

[0095] S=[Δx,Δy,Δv x ,Δv y ,q,q e ];

[0096] Δx=x t -x m Δy=y t -y m ;

[0097] Δv x =v xt -v xm Δv y =v yt -v ym ;

[0098] The missile's normal acceleration is used as a state variable in the action space, i.e.:

[0099] A = [a m ];

[0100] To ensure the guidance law meets the requirement of low miss distance, i.e., the smaller the expected distance between the missile and the target, the higher the reward, the design ensures that a large distance r between the missile and the target yields a negative reward, while a small distance r yields a positive reward, thus obtaining the hit reward.

[0101]

[0102] The selection of k1 and k2 should be based on the principle of balancing positive and negative feedback. The so-called principle of positive and negative feedback means that when the agent performs behavior that is close to the goal, we should give positive rewards to encourage it; when the agent performs exploratory behavior that does not seriously deviate from the goal, we should give mild punishment; when the agent performs behavior that seriously deviates from the designed reward, we need to give the agent a huge punishment so that it will not dare to do it again.

[0103] In addition to meeting the low miss distance requirement, the terminal line-of-sight angle requirement should also be met to obtain a terminal angle bonus, namely:

[0104]

[0105] Δq=|qq e |;

[0106] To coordinate the in-mission reward and the terminal angle reward, when the missile is at a large target distance, the weight of the terminal angle reward can be appropriately reduced, so that the initial trajectory is not affected by angle constraints, thus obtaining the final reward function, i.e.:

[0107]

[0108] A Markov decision process is constructed based on the state space, action space, reward function, and decay rate, namely:

[0109] M = <S,A,R,γ>;

[0110] In the formula, S represents the state space, Δx and Δy represent the relative positions of the missile and the target along the x-axis and y-axis, respectively, and Δv... x and Δv y Let q and q' be the relative velocities of the missile and the target along the x-axis and y-axis, respectively. e These represent the current line-of-sight angle and the desired terminal line-of-sight angle between the missile and the target, respectively. m and x t Here are the x-axis coordinates of the missile and the target, respectively, and the y-axis coordinates of the target. m and y t These are the y-axis coordinates of the missile and the target, respectively, and v xm and v xt Let v be the normal velocity of the missile and the target along the x-axis, respectively. ym and v yt Let A and a be the normal velocities of the missile and the target along the y-axis, respectively, where A is the action space and a is the velocity of the target. m R is the missile's normal acceleration, R1 is the hit reward, r is the relative distance between the missile and the target, k1 and k2 are the first and second hit reward coefficients, respectively, R2 is the terminal angle reward, Δq is the error between the current line-of-sight angle and the expected terminal line-of-sight angle between the missile and the target, k3 and k4 are the first and second line-of-sight angle reward coefficients, R is the reward function, k5 is the terminal angle coordination coefficient, and M is the Markov decision process.

[0111] According to one embodiment of the present invention, the Actor network has an input dimension of 6 dimensions and an output dimension of 1 dimension, corresponding to the state space and the action space, respectively; the Critic network has an input dimension of two paths of 6+1 dimensions, corresponding to the state space and the action space, respectively, and an output dimension of 1 dimension.

[0112] In other words, the input of an Actor network is the state space, and the output is the action space; the input of a Critic network is the state space, the input is the action space, and the output is the state-action value function.

[0113] Specifically, such as Figure 2 As shown, the Actor network consists of three hidden layers connected in sequence. Figure 2 (H1, H2, H3), the first two layers ( Figure 2 The activation functions for H1 and H2 are ReLU functions, and the third layer ( Figure 2 The activation function for H3 is the tanh function. The number of neurons in hidden layers H1, H2, and H3 are 50, 30, and 20, respectively, and the learning rate of the Actor network is 1×10⁻⁶. -4 The target network of the Actor network has the same parameters as the Actor network.

[0114] like Figure 3 As shown, the Critic network consists of four hidden layers ( Figure 3 (S1, S2, A1, H1), the first path includes two hidden layers connected in sequence ( Figure 3 The second path includes a hidden layer (S1, S2). Figure 3 The outputs of hidden layer A1 and hidden layer S2 are added together and then input to another hidden layer (A1). Figure 3 The Critic network uses ReLU activation functions for all layers (H1). The hidden layers S1, S2, A1, and H1 have 50, 30, 20, and 20 neurons respectively, and the learning rate is 1×10⁻⁶. -3 The target network of the Critic network has the same parameters as the Critic network.

[0115] According to one embodiment of the present invention, the Actor network is trained in real time based on the relative motion model between the missile and the target and the Markov decision process to obtain quadruplets at each time step, and the quadruplets at each time step are stored sequentially into the experience pool, including:

[0116] The current state is obtained in real time based on the relative motion model between the missile and the target;

[0117] The reward value at the current moment is obtained in real time based on the Markov decision process;

[0118] Input the current state into the Actor network to obtain the action at the current moment;

[0119] The quadruple consisting of the current state, reward value, action, and the state at the next moment is stored in the experience pool in sequence.

[0120] The quadruple can be represented as (s, a, r', s'), where s is the current state, a is the current action, r' is the current reward, and s' is the next state. The experience pool size is 10. 7 .

[0121] According to one embodiment of the present invention, the Actor network is updated by the following formula:

[0122]

[0123] The Critic network is updated using the following formula:

[0124]

[0125] Where, δ i =R(s) i ,a i )+γQ(s i+1 ,a(s i+1 ,θ),η)-Q(s i ,a i ,η);

[0126] In the formula, θ and η are the network parameters to be updated for the Actor network and Critic network, respectively; θ' and η' are the updated network parameters for the Actor network and Critic network, respectively; β and α are the learning rates for the Actor network and Critic network, respectively; and N is the number of empirical data points updated in one step. Let s be the partial derivatives with respect to θ, a, and η, respectively. i Let a be the state at time i. i Let a(s) be the action at time i. i Q(s,θ) is the state-action value function fitted by the Actor network at time i. i ,a i ,η) represents the Critic network at time i based on s i ,a i The fitted state-action value function, Q(s) i ,a(s i ,θ),η) are the Critic network at time i based on s i ,a(s i The state-action value function fitted by θ, δ i For the updated loss function, R(s) i ,a i Let be the reward function at time i, γ be the decay rate, and Q(s) be the reward function at time i. i+1 ,a(s i+1 ,θ),η) are the Critic network at time i+1 based on s i+1,a(s i+1 The state action value function fitted by θ).

[0127] According to one embodiment of the present invention, the target network of the Actor network is updated using the following formula:

[0128] θ”'=τθ+(1-τ)θ”;

[0129] The target network of the Critic network is updated using the following formula:

[0130] η”'=τη+(1-τ)η”;

[0131] In the formula, τ is the learning rate of the target network of the Actor network and the target network of the Critic network, θ” and η” are the network parameters to be updated of the target network of the Actor network and the target network of the Critic network, respectively, θ”' and η”' are the updated network parameters of the target network of the Actor network and the target network of the Critic network, respectively, and θ and η are the network parameters to be updated of the Actor network and the Critic network, respectively.

[0132] To gain a further understanding of the present invention, the following description is provided in conjunction with... Figures 4-7 The method for generating guidance laws based on DDPG according to the present invention will be described in detail.

[0133] In this embodiment, the initial simulation parameters for the relative motion model between the missile and the target are as follows: initial missile coordinates (0m, 0m), initial target coordinates (4000m, 4000m), and missile normal velocity v. m =200m / s, missile velocity tilt angle θ m =45°, the horizontal velocity of the target in the x direction is a random number uniformly distributed within [-60m / s, 60m / s], and the target trajectory is a sinusoidal maneuver.

[0134] The following analysis compares and contrasts the performance of the constrained proportional guidance law in the prior art and the DDPG guidance law of this invention, taking the desired terminal line-of-sight angle as 30° and the target horizontal movement speed as -10m / s.

[0135] Figure 4 The diagrams show the trajectories of the DDPG and proportional guidance missiles with respect to their targets. The trajectories reveal that the DDPGNG initially moves in a straight line, gradually adjusting its direction as it approaches the target. Its trajectory is relatively straight in the initial and later stages, with a significant curvature during the middle stage where it adjusts its line-of-sight angle. In contrast, the PNG's trajectory exhibits a more uniform overall curvature than the DDPGNG. Ultimately, both missiles reach the vicinity of the target at the designated line-of-sight angle.

[0136] Figure 5 , Figure 6 The figures show the relative distance change curve and the line-of-sight angle change curve, respectively. The relative distance between the missile and the target continuously decreases at a relatively uniform rate. There is no significant difference between the two. Regarding the line-of-sight angle change, both algorithms eventually stabilize at a line-of-sight angle near the target's line-of-sight angle. The DDPGNG algorithm has a slightly smaller error than the PNG algorithm.

[0137] Figure 6 These are comparison curves of the normal acceleration of the two guidance laws. The normal acceleration graphs show that the DDPGNG experiences a significant overload around 20 seconds. This is because the hit reward and terminal angle reward are coordinated using relative distance in the reward function design. After a certain critical relative distance, the terminal angle reward becomes greater than the hit reward, causing the entire guidance law to adjust the line-of-sight angle, resulting in a surge in overload commands. However, the normal acceleration does not change significantly just before hitting the target. The PNG, on the other hand, shows a larger fluctuation in the amplitude of its normal acceleration just before hitting the target, and this amplitude is slightly larger than that of the DDPGNG. This is because the PNG is affected by the line-of-sight distance; as it approaches a dynamic target, the adjustment of the line of sight becomes increasingly drastic, resulting in a more dramatic change in normal acceleration.

[0138] The table below compares the missile's performance in terms of miss distance and maximum line-of-sight error under the two guidance laws. It can be seen that in this embodiment, the miss distances of the two are not significantly different, with the DDPGNG exhibiting a smaller miss distance.

[0139] In terms of viewing angle constraints, PNG is slightly lower than DDPGNG, but both satisfy the terminal viewing angle constraints.

[0140]

[0141] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the methods described above.

[0142] In summary, this invention provides a method for generating guidance laws based on deep reinforcement learning (DDPG). This method utilizes a neural network to directly map the relative motion state of the missile and the target and the desired line-of-sight angle to the missile's normal acceleration, thereby generating guidance commands in real time and providing a new approach to obtaining the optimal guidance strategy.

[0143] The parts of this invention not described in detail are techniques known to those skilled in the art.

[0144] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention; the directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0145] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0146] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0147] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating guidance laws based on DDPG deep reinforcement learning, characterized in that, The method includes: Establish a relative motion model between the missile and the target; Design a Markov decision process based on a model of the relative motion between the missile and the target; An experience pool for storing data structure groups is established based on Markov decision processes; Establish the Actor network, Critic network, the target network of the Actor network, and the target network of the Critic network; The Actor network is trained in real time based on the relative motion model between the missile and the target and the Markov decision process to obtain the quadruplets at each time step, and the quadruplets at each time step are stored in the experience pool in sequence. When the number of quadruplets in the experience pool reaches a preset number, a quadruplet is selected from the experience pool at preset intervals to continuously update the network parameters of the Actor network and the Critic network. The network parameters of the target network of the Actor network and the target network of the Critic network are updated using the updated network parameters of the Actor network and the Critic network at each preset interval. The updated network parameters of the target network of the Actor network are used as the network parameters of the Actor network at the current moment, and the updated network parameters of the target network of the Critic network are used as the network parameters of the Critic network at the current moment. If both the Actor network and the Critic network converge, the Actor network at the current moment is used as the guidance law.

2. The method according to claim 1, characterized in that, A relative motion model between the missile and the target is established using the following set of formulas: In the formula, m represents the missile, t represents the target, and V represents the target. m and V t denoted by , and denoted by r, respectively, where r is the relative distance between the missile and the target. q represents the rate of change of the relative distance between the missile and the target; q represents the line-of-sight angle between the missile and the target. σ is the line-of-sight angular velocity between the missile and the target. m and σ t η represents the trajectory inclination angles of the missile and the target, respectively. m and η t θ represents the leading angle of the missile and the target, respectively. m and θ t These are the missile's velocity inclination angle and the target's trajectory inclination angle, respectively. and These are the rate of change of the missile's velocity tilt angle and the rate of change of the target's trajectory tilt angle, respectively; a m and a t x represents the normal acceleration of the missile and the target, respectively. m and x t Here are the x-axis coordinates of the missile and the target, respectively, and the y-axis coordinates of the target. m and y t These are the y-axis coordinates of the missile and the target, respectively, and v m and v t These are the normal velocities of the missile and the target, respectively.

3. The method according to claim 1, characterized in that, The design of a Markov decision process based on a missile-target relative motion model includes: The relative positions of the missile and the target in the x-axis and y-axis directions, the relative velocities of the missile and the target in the x-axis and y-axis directions, the current line-of-sight angle between the missile and the target, and the expected terminal line-of-sight angle are used as state variables in the state space. The missile's normal acceleration is used as a state variable in the action space; Hit bonuses are awarded based on the relative distance between the missile and the target. The terminal angle reward is obtained based on the current line-of-sight angle between the missile and the target and the expected terminal line-of-sight angle. A reward function is derived based on hit reward and terminal angle reward. A Markov decision process is constructed based on a state space, action space, reward function, and decay rate.

4. The method according to claim 3, characterized in that, The state space is obtained using the following formula: S=[Δx,Δy,Δv x ,Δv y ,q,q e ]; Δx=x t -x m ,Δy=y t -y m ; Δv x =v xt -v xm ,Δv y =v yt -v ym ; The action space is obtained using the following formula: A=[a m ]; The hit bonus is obtained using the following formula: Obtain the terminal corner reward using the following formula: Δq=|q-q e |; The reward function is obtained using the following formula: In the formula, S represents the state space, Δx and Δy represent the relative positions of the missile and the target along the x-axis and y-axis, respectively, and Δv... x and Δv y Let q and q' be the relative velocities of the missile and the target along the x-axis and y-axis, respectively. e These represent the current line-of-sight angle and the desired terminal line-of-sight angle between the missile and the target, respectively. m and x t Here are the x-axis coordinates of the missile and the target, respectively, and the y-axis coordinates of the target. m and y t These are the y-axis coordinates of the missile and the target, respectively, and v xm and v xt Let v be the normal velocity of the missile and the target along the x-axis, respectively. ym and v yt Let A and a be the normal velocities of the missile and the target along the y-axis, respectively, where A is the action space and a is the velocity of the target. m R is the missile's normal acceleration, R1 is the hit reward, r is the relative distance between the missile and the target, k1 and k2 are the first and second hit reward coefficients, respectively, R2 is the terminal angle reward, Δq is the error between the current line-of-sight angle and the expected terminal line-of-sight angle between the missile and the target, k3 and k4 are the first and second line-of-sight angle reward coefficients, R is the reward function, and k5 is the terminal angle coordination coefficient.

5. The method according to claim 1, characterized in that, The Actor network has a 6-dimensional input and a 1-dimensional output, corresponding to the state space and action space, respectively; the Critic network has two 6+1-dimensional inputs, corresponding to the state space and action space, respectively, and a 1-dimensional output.

6. The method according to claim 1, characterized in that, The first two layers of the Actor network use ReLU activation functions, and the third layer uses the tanh function; all layers of the Critic network use ReLU activation functions.

7. The method according to claim 1, characterized in that, The parameters of the Actor network and the target network of the Actor network are the same, and the parameters of the Critic network and the target network of the Critic network are the same.

8. The method according to claim 1, characterized in that, The Actor network is trained in real time based on the relative motion model between the missile and the target and the Markov decision process to obtain quadruplets at each time step. These quadruplets are then stored sequentially in the experience pool, including: The current state is obtained in real time based on the relative motion model between the missile and the target; The reward value at the current moment is obtained in real time based on the Markov decision process; Input the current state into the Actor network to obtain the action at the current moment; The quadruple consisting of the current state, reward value, action, and the state at the next moment is stored in the experience pool in sequence.

9. The method according to claim 1, characterized in that, The Actor network is updated using the following formula: The Critic network is updated using the following formula: among them,d i =R(s i ,a i )+γQ(s i+1 ,a(s i+1 ,θ),η)-Q(s i ,a i ,or); In the formula, θ and η are the network parameters to be updated for the Actor network and Critic network, respectively; θ' and η' are the updated network parameters for the Actor network and Critic network, respectively; β and α are the learning rates for the Actor network and Critic network, respectively; and N is the number of empirical data points updated in one step. Let s be the partial derivatives with respect to θ, a, and η, respectively. i Let a be the state at time i. i Let a(s) be the action at time i. i Q(s,θ) is the state-action value function fitted by the Actor network at time i. i ,a i ,η) represents the Critic network at time i based on s i ,a i The fitted state-action value function, Q(s) i ,a(s i ,θ),η) are the Critic network at time i based on s i ,a(s i The state-action value function fitted by θ, δ i For the updated loss function, R(s) i ,a i Let be the reward function at time i, γ be the decay rate, and Q(s) be the reward function at time i. i+1 ,a(s i+1 ,θ),η) are the Critic network at time i+1 based on s i+1 ,a(s i+1 The state action value function fitted by θ).

10. The method according to claim 1, characterized in that, The target network of the Actor network is updated using the following formula: θ”'=τθ+(1-τ)θ”; The target network of the Critic network is updated using the following formula: η”'=τη+(1-τ)η”; In the formula, τ is the learning rate of the target network of the Actor network and the target network of the Critic network, θ” and η” are the network parameters to be updated of the target network of the Actor network and the target network of the Critic network, respectively, θ”' and η”' are the updated network parameters of the target network of the Actor network and the target network of the Critic network, respectively, and θ and η are the network parameters to be updated of the Actor network and the Critic network, respectively.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 10.

Images