Method for generating deep reinforcement learning guidance law based on DDPG

Abstract

The invention provides a method for generating a deep reinforcement learning guidance law based on DDPG. The method comprises the following steps: establishing a relative motion model of a missile and a target; designing a Markov decision process based on the relative motion model of the missile and the target; establishing an experience pool for storing the data structure group based on a Markov decision process; the method comprises the following steps: establishing an Actor network, a Critic network, a target network of the Actor network and a target network of the Critic network; performing real-time training on the Actor network to obtain a tetrad at each moment, and sequentially storing the tetrad at each moment into an experience pool; continuously updating the network parameters of the Actor network and the Critic network; updating the network parameters of the target network of the Actor network and the target network of the Critic network by using the updated network parameters of the Actor network and the Critic network, and assigning the network parameters of the Actor network at the current moment and the network parameters of the Critic network at the current moment; and under the condition that both the Actor network and the Critic network converge, taking the Actor network at the current moment as a guidance law.

Description

Technical Field

[0001] The present invention relates to the technical field of guidance law, and in particular to a method for generating a deep reinforcement learning guidance law based on DDPG. Background Art

[0002] In modern flight missions, whether it is the high maneuverability of target motion or the nonlinear, strong coupling and multi-time-varying factors of aircraft motion, they all bring many challenges to the research of guidance laws. In addition, the new combat environment has also put forward new requirements for the research of guidance laws. Simply low miss margins are no longer sufficient to complete combat missions. Guidance laws should also meet various terminal constraints, such as terminal angle constraints. Allowing aircraft to hit the target at a specific angle can fully improve the damage efficiency of the warhead. At the same time, terminal angle control is also a necessary requirement for coordinated attack of aircraft.

[0003] Currently, traditional guidance laws cannot achieve the high maneuverability of target motion and the nonlinearity, strong coupling and multi-time variation of aircraft motion. Summary of the invention

[0004] The present invention provides a method for generating a deep reinforcement learning guidance law based on DDPG, which can solve the technical problems in the prior art that the guidance law cannot achieve high maneuverability of target motion and the nonlinearity, strong coupling and multi-time variation of aircraft motion.

[0005] According to one aspect of the present invention, a method for generating a deep reinforcement learning guidance law based on DDPG is provided, the method comprising:

[0006] Establish the relative motion model between missile and target;

[0007] Design a Markov decision process based on the relative motion model between missile and target;

[0008] Establishing an experience pool for storing data structure groups based on Markov decision process;

[0009] Establish the Actor network, the Critic network, the target network of the Actor network, and the target network of the Critic network;

[0010] Based on the relative motion model between the missile and the target and the Markov decision process, the Actor network is trained in real time to obtain the quadruple at each moment, and the quadruple at each moment is stored in the experience pool in sequence;

[0011] When the number of quadruple groups in the experience pool reaches the preset number, a quadruple group is selected from the experience pool at a preset interval to continuously update the network parameters of the Actor network and the Critic network.

[0012] At each preset update number, the network parameters of the target network of the Actor network and the target network of the Critic network are updated using the updated network parameters of the Actor network and the Critic network, and the updated network parameters of the target network of the Actor network are used as the network parameters of the Actor network at the current moment, and the updated network parameters of the target network of the Critic network are used as the network parameters of the Critic network at the current moment;

[0013] When the Actor network and the Critic network are fully converged, the Actor network at the current moment is used as the guidance law.

[0014] Preferably, the relative motion model between the missile and the target is established by the following formula group:

[0015]

[0016] In the formula, m is the missile, t is the target, V m and V t are the linear velocities of the missile and the target respectively, r is the relative distance between the missile and the target, is the rate of change of the relative distance between the missile and the target; q is the line of sight angle between the missile and the target, is the line-of-sight angular velocity between the missile and the target, σ m and σ t are the ballistic inclination angles of the missile and the target, respectively, m and η t are the lead angles of the missile and the target, θ m and θ t are respectively the velocity inclination of the missile and the track inclination of the target, and are the missile's velocity angle change rate and the target's track angle change rate respectively; a m and a t are the normal accelerations of the missile and target, respectively, m and x t are the x-axis coordinates of the missile and the target, m and t are the y-axis coordinates of the missile and the target, respectively, m and v t are the normal velocities of the missile and target, respectively.

[0017] Preferably, designing a Markov decision process based on a relative motion model between the missile and the target includes:

[0018] The relative positions of the missile and the target in the x-axis and y-axis directions, the relative speeds of the missile and the target in the x-axis and y-axis directions, the current sight angle between the missile and the target, and the expected terminal sight angle are used as state quantities in the state space;

[0019] The normal acceleration of the missile is taken as the state quantity in the action space;

[0020] Get a hit bonus based on the relative distance between the missile and the target;

[0021] Obtaining a terminal angle bonus based on the current line of sight angle and the desired terminal line of sight angle between the missile and the target;

[0022] Obtain a reward function based on the hit reward and the terminal angle reward;

[0023] A Markov decision process is constructed based on state space, action space, reward function and decay rate.

[0024] Preferably, the state space is obtained by the following formula:

[0025] S=[Δx,Δy,Δv x ,Δv y ,q,q e ];

[0026] Δx=x t -x m , Δy=y t -y m ;

[0027] Δv x =v xt -v xm , Δv y =v yt -v ym ;

[0028] The action space is obtained by the following formula:

[0029] A=[a m ];

[0030] Obtain the hit bonus by:

[0031]

[0032] Get the terminal angle reward through the following formula:

[0033]

[0034] Δq=|qq e |;

[0035] The reward function is obtained by the following formula:

[0036]

[0037] Where S is the state space, Δx and Δy are the relative positions of the missile and the target in the x-axis and y-axis directions, respectively, and Δv x and Δv y are the relative speeds of the missile and the target in the x-axis and y-axis directions, q and q e are the current sight angle and the expected terminal sight angle between the missile and the target, respectively, and x m and x t are the x-axis coordinates of the missile and the target, m and t are the y-axis coordinates of the missile and the target, respectively, xm and v xt are the normal velocities of the missile and the target in the x-axis direction, v ym and v yt are the normal velocities of the missile and the target in the y-axis direction, A is the action space, a m is the normal acceleration of the missile, R1 is the hit reward, r is the relative distance between the missile and the target, k1 and k2 are the first and second hit reward coefficients respectively, R2 is the terminal angle reward, Δq is the current line of sight angle and the expected terminal line of sight angle error between the missile and the target, k3 and k4 are the first and second line of sight angle reward coefficients respectively, R is the reward function, and k5 is the terminal angle coordination coefficient.

[0038] Preferably, the input dimension of the Actor network is 6 dimensions, and the output dimension is 1 dimension, corresponding to the state space and action space respectively; the input dimension of the Critic network is two-way 6+1 dimensions, corresponding to the state space and action space respectively, and the output dimension is 1 dimension.

[0039] Preferably, the activation functions of the first two layers of the Actor network are Relu functions, and the activation function of the third layer is tanh function; the activation functions of all layers of the Critic network are Relu functions.

[0040] Preferably, the parameters of the Actor network and the target network of the Actor network are the same, and the parameters of the Critic network and the target network of the Critic network are the same.

[0041] Preferably, the Actor network is trained in real time based on the relative motion model between the missile and the target and the Markov decision process to obtain the quadruple at each moment, and the quadruple at each moment is sequentially stored in the experience pool, including:

[0042] Based on the relative motion model between the missile and the target, the current state is obtained in real time;

[0043] Obtain the current reward value in real time based on the Markov decision process;

[0044] Input the current state into the Actor network to get the current action;

[0045] The four-tuple consisting of the current state, reward value, action and the state at the next moment is stored in the experience pool in sequence.

[0046] Preferably, the Actor network is updated by the following formula:

[0047]

[0048] Update the Critic network by the following formula:

[0049]

[0050] Among them, δ i =R(s i ,a i )+γQ(s i+1 ,a(s i+1 ,θ),η)-Q(s i ,a i ,η);

[0051] In the formula, θ and η are the network parameters to be updated of the Actor network and the Critic network, θ' and η' are the updated network parameters of the Actor network and the Critic network, β and α are the learning rates of the Actor network and the Critic network, and N is the number of experience data updated at one time. are the partial derivatives of θ, a, and η respectively, s i is the state at the i-th moment, a i is the action at the i-th moment, a(s i ,θ) is the state action value function fitted by the Actor network at the i-th moment, Q(s i ,a i ,η) is the Critic network based on s at the i-th moment i ,a i The fitted state action value function, Q(s i ,a(s i ,θ),η) is the Critic network based on s at the i-th moment i ,a(s i ,θ) fitted state action value function,δ i is the updated loss function, R(s i ,a i ) is the reward function at the i-th moment, γ is the decay rate, Q(s i+1 ,a(s i+1 ,θ),η) is the Critic network based on s at the i+1th moment i+1,a(s i+1 ,θ) is the state action value function fitted.

[0052] Preferably, the target network of the Actor network is updated by the following formula:

[0053] θ”′=τθ+(1-τ)θ”;

[0054] The target network of the Critic network is updated by the following formula:

[0055] η"'=τη+(1-τ)η";

[0056] Where τ is the learning rate of the target network of the Actor network and the target network of the Critic network, θ” and η” are the network parameters to be updated of the target network of the Actor network and the target network of the Critic network, θ”’ and η”’ are the updated network parameters of the target network of the Actor network and the target network of the Critic network, and θ and η are the network parameters to be updated of the Actor network and the Critic network, respectively.

[0057] According to another aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the computer program.

[0058] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0059] 1. A guidance law with angle constraints is implemented, which can attack the target at a set attack angle;

[0060] 2. Compared with the traditional proportional guidance method, it has higher accuracy and the ability to intercept maneuvering targets, and has better convergence of normal acceleration in the terminal stage;

[0061] 3. Good generalization ability for complex combat scenarios;

[0062] 4. It is a reinforcement learning guidance law in a model-free framework, which can optimize the guidance law in a more complex environment framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 A flowchart of a method for generating a deep reinforcement learning guidance law based on DDPG is shown according to an embodiment of the present invention;

[0065] Figure 2 A schematic diagram of the structure of an Actor network provided according to an embodiment of the present invention is shown;

[0066] Figure 3 A schematic diagram of the structure of a Critic network provided according to an embodiment of the present invention is shown;

[0067] Figure 4 A motion trajectory diagram of a missile and a target according to a DDPG guidance law and a proportional guidance law provided in accordance with an embodiment of the present invention is shown;

[0068] Figure 5 A curve diagram showing the change of the distance between the missile and the target according to the DDPG guidance law and the proportional guidance law provided in one embodiment of the present invention is shown;

[0069] Figure 6 A curve diagram of the change of the sight angle of a missile according to the DDPG guidance law and the proportional guidance law provided in accordance with an embodiment of the present invention is shown;

[0070] Figure 7 A curve diagram of the change in normal acceleration of a missile using a DDPG guidance law and a proportional guidance law provided according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0071] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0073] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0074] like Figure 1 As shown, the present invention provides a method for generating a deep reinforcement learning guidance law based on DDPG, the method comprising:

[0075] Establish the relative motion model between missile and target;

[0076] Design a Markov decision process based on the relative motion model between missile and target;

[0077] Establishing an experience pool for storing data structure groups based on Markov decision process;

[0078] Establish the Actor network, the Critic network, the target network of the Actor network, and the target network of the Critic network;

[0079] Based on the relative motion model between the missile and the target and the Markov decision process, the Actor network is trained in real time to obtain the quadruple at each moment, and the quadruple at each moment is stored in the experience pool in sequence;

[0080] When the number of quadruple groups in the experience pool reaches the preset number, a quadruple group is selected from the experience pool at a preset interval to continuously update the network parameters of the Actor network and the Critic network.

[0081] At each preset update number, the network parameters of the target network of the Actor network and the target network of the Critic network are updated using the updated network parameters of the Actor network and the Critic network, and the updated network parameters of the target network of the Actor network are used as the network parameters of the Actor network at the current moment, and the updated network parameters of the target network of the Critic network are used as the network parameters of the Critic network at the current moment;

[0082] When the Actor network and the Critic network are fully converged, the Actor network at the current moment is used as the guidance law.

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] 1. A guidance law with angle constraints is implemented, which can attack the target at a set attack angle;

[0085] 2. Compared with the traditional proportional guidance method, it has higher accuracy and the ability to intercept maneuvering targets, and has better convergence of normal acceleration in the terminal stage;

[0086] 3. Good generalization ability for complex combat scenarios;

[0087] 4. It is a reinforcement learning guidance law in a model-free framework, which can optimize the guidance law in a more complex environment framework.

[0088] In the present invention, if the change gradient of the network parameters of the Actor network is less than the preset value, the Actor network is determined to be converged. Similarly, if the change gradient of the network parameters of the Critic network is less than the preset value, the Critic network is determined to be converged.

[0089] Among them, the change gradient of the network parameters is the difference between two network parameters at adjacent moments.

[0090] According to one embodiment of the present invention, the relative motion model between the missile and the target is established by the following formula group:

[0091]

[0092] In the formula, m is the missile, t is the target, V m and V t are the linear velocities of the missile and the target respectively, r is the relative distance between the missile and the target, is the rate of change of the relative distance between the missile and the target; q is the line of sight angle between the missile and the target, is the line-of-sight angular velocity between the missile and the target, σ m and σ t are the ballistic inclination angles of the missile and the target, respectively, m and η t are the lead angles of the missile and the target, θ m and θ t are respectively the velocity inclination of the missile and the track inclination of the target, and are the missile's velocity angle change rate and the target's track angle change rate respectively; a m and a t are the normal accelerations of the missile and target, respectively,m and x t are the x-axis coordinates of the missile and the target, m and t are the y-axis coordinates of the missile and the target, respectively, m and v t are the normal velocities of the missile and target, respectively.

[0093] According to an embodiment of the present invention, designing a Markov decision process based on a relative motion model between a missile and a target includes:

[0094] The relative positions of the missile and the target in the x-axis and y-axis directions, the relative speeds of the missile and the target in the x-axis and y-axis directions, the current sight angle between the missile and the target, and the expected terminal sight angle are used as the state quantities of the state space, that is:

[0095] S=[Δx,Δy,Δv x ,Δv y ,q,q e ];

[0096] Δx=x t -x m , Δy=y t -y m ;

[0097] Δv x =v xt -v xm , Δv y =v yt -v ym ;

[0098] The normal acceleration of the missile is taken as the state quantity of the action space, that is:

[0099] A=[a m ];

[0100] In order to make the guidance law meet the requirement of low miss amount, that is, to meet the expectation that the smaller the distance between the missile and the target, the higher the reward, the design makes it so that when the distance r between the missile and the target is large, a negative reward is obtained, and when it is small, a positive reward is obtained, so as to obtain a hit reward, that is:

[0101]

[0102] Among them, the selection of k1 and k2 should be based on the principle of positive and negative feedback balance. The so-called positive and negative feedback principle means that when the agent behaves close to the goal, we should give positive rewards to encourage it; when the agent behaves in an exploratory manner and does not deviate seriously from the goal, we should give it a mild punishment; when the agent behaves in a behavior that seriously deviates from the designed reward, we need to give the agent a huge punishment so that it dare not commit the same mistake again.

[0103] While meeting the requirement of low miss, the terminal sight angle requirement should also be met to obtain the terminal angle bonus, namely:

[0104]

[0105] Δq=|qq e |;

[0106] In order to coordinate the hit reward and the terminal angle reward, when the missile target distance is large, the weight of the terminal angle reward can be appropriately reduced so that the front trajectory is not affected by the angle constraint, thereby obtaining the final reward function, that is:

[0107]

[0108] The Markov decision process is constructed based on the state space, action space, reward function and decay rate, namely:

[0109] M=<S,A,R,γ>;

[0110] Where S is the state space, Δx and Δy are the relative positions of the missile and the target in the x-axis and y-axis directions, respectively, and Δv x and Δv y are the relative speeds of the missile and the target in the x-axis and y-axis directions, q and q e are the current sight angle and the expected terminal sight angle between the missile and the target, respectively, and x m and x t are the x-axis coordinates of the missile and the target, m and t are the y-axis coordinates of the missile and the target, respectively, xm and v xt are the normal velocities of the missile and the target in the x-axis direction, v ym and v yt are the normal velocities of the missile and the target in the y-axis direction, A is the action space, a m is the normal acceleration of the missile, R1 is the hit reward, r is the relative distance between the missile and the target, k1 and k2 are the first and second hit reward coefficients respectively, R2 is the terminal angle reward, Δq is the current line of sight angle and the expected terminal line of sight angle error between the missile and the target, k3 and k4 are the first and second line of sight angle reward coefficients respectively, R is the reward function, k5 is the terminal angle coordination coefficient, and M is the Markov decision process.

[0111] According to an embodiment of the present invention, the input dimension of the Actor network is 6 dimensions, and the output dimension is 1 dimension, which correspond to the state space and the action space respectively; the input dimension of the Critic network is two-way 6+1 dimensions, which correspond to the state space and the action space respectively, and the output dimension is 1 dimension.

[0112] That is to say, the input of the Actor network is the state space, and the output is the action space; one input of the Critic network is the state space, the other input is the action space, and the output is the state-action value function.

[0113] Specifically, Figure 2 As shown, the Actor network consists of three hidden layers connected in sequence ( Figure 2 Middle H1, H2, H3), the first two layers ( Figure 2 The activation function of H1 and H2 is Relu function, and the third layer ( Figure 2 The activation function of H3 is tanh function. The number of neurons in hidden layers H1, H2, and H3 are 50, 30, and 20 respectively, and the learning rate of Actor network is 1×10 -4 The target network of an Actor network has the same parameters as the Actor network.

[0114] like Figure 3 As shown, the Critic network consists of four hidden layers ( Figure 3 S1, S2, A1, H1), the first path includes two hidden layers connected in sequence ( Figure 3 S1, S2), the second path includes a hidden layer ( Figure 3 A1), the output of hidden layer S2 and hidden layer A1 are added and then input into another hidden layer ( Figure 3 H1), and the activation functions of all layers of the Critic network are Relu functions. The number of neurons in the hidden layers S1, S2, A1, and H1 are 50, 30, 20, and 20 respectively, and the learning rate of the Critic network is 1×10 -3 The target network of the Critic network has the same parameters as the Critic network.

[0115] According to an embodiment of the present invention, the Actor network is trained in real time based on the relative motion model between the missile and the target and the Markov decision process to obtain the quadruple at each moment, and the quadruple at each moment is sequentially stored in the experience pool, including:

[0116] Based on the relative motion model between the missile and the target, the current state is obtained in real time;

[0117] Obtain the current reward value in real time based on the Markov decision process;

[0118] Input the current state into the Actor network to get the current action;

[0119] The four-tuple consisting of the current state, reward value, action and the state at the next moment is stored in the experience pool in sequence.

[0120] The four-tuple can be expressed as (s, a, r', s'), where s is the current state, a is the action at the current moment, r' is the reward value at the current moment, s' is the state at the next moment, and the experience pool size is 10 7 .

[0121] According to an embodiment of the present invention, the Actor network is updated by the following formula:

[0122]

[0123] Update the Critic network by the following formula:

[0124]

[0125] Among them, δ i =R(s i ,a i )+γQ(s i+1 ,a(s i+1 ,θ),η)-Q(s i ,a i ,η);

[0126] In the formula, θ and η are the network parameters to be updated of the Actor network and the Critic network, θ' and η' are the updated network parameters of the Actor network and the Critic network, β and α are the learning rates of the Actor network and the Critic network, and N is the number of experience data updated at one time. are the partial derivatives of θ, a, and η respectively, s i is the state at the i-th moment, a i is the action at the i-th moment, a(s i ,θ) is the state action value function fitted by the Actor network at the i-th moment, Q(s i ,a i ,η) is the Critic network based on s at the i-th moment i ,a i The fitted state action value function, Q(s i ,a(s i ,θ),η) is the Critic network based on s at the i-th moment i ,a(s i ,θ) fitted state action value function,δ i is the updated loss function, R(s i ,a i ) is the reward function at the i-th moment, γ is the decay rate, Q(s i+1 ,a(s i+1 ,θ),η) is the Critic network based on s at the i+1th moment i+1,a(s i+1 ,θ) is the state action value function fitted.

[0127] According to an embodiment of the present invention, the target network of the Actor network is updated by the following formula:

[0128] θ”′=τθ+(1-τ)θ”;

[0129] The target network of the Critic network is updated by the following formula:

[0130] η"'=τη+(1-τ)η";

[0131] Where τ is the learning rate of the target network of the Actor network and the target network of the Critic network, θ” and η” are the network parameters to be updated of the target network of the Actor network and the target network of the Critic network, θ”’ and η”’ are the updated network parameters of the target network of the Actor network and the target network of the Critic network, and θ and η are the network parameters to be updated of the Actor network and the Critic network, respectively.

[0132] In order to further understand the present invention, the following Figure 4-Figure 7 The method for generating a deep reinforcement learning guidance law based on DDPG of the present invention is described in detail.

[0133] In this embodiment, the initial simulation parameters of the relative motion model between the missile and the target are as follows: the initial coordinates of the missile (0m, 0m), the initial coordinates of the target (4000m, 4000m), the normal velocity v of the missile m =200m / s, missile velocity angle θ m =45°, the target horizontal speed in the x direction is a random number uniformly distributed within [-60m / s, 60m / s], and the target trajectory is a sinusoidal maneuver.

[0134] The following takes the expected terminal sight angle as 30° and the target horizontal moving speed as -10m / s to compare and analyze the performance of the constrained proportional guidance law in the prior art and the DDPG guidance law of the present invention.

[0135] Figure 4 The motion trajectory diagrams of the missile and the target using the DDPG guidance law and the proportional guidance law are shown. From the motion trajectory, it can be seen that the strategy adopted by DDPGNG is to move straight first, and then adjust the direction after gradually approaching the target. When approaching the target, the front and back sections of the trajectory are relatively straight, and there is a process of adjusting the sight angle in the middle section, and the curvature of the turning is large. The overall curvature of the PNG trajectory is more uniform than that of DDPGNG. In the end, both can reach the vicinity of the target at the specified sight angle.

[0136] Figure 5 , Figure 6 They are the relative distance change curve and the sight angle change curve. The relative distance of the missile target has been decreasing and the change rate is relatively uniform. There is no obvious difference between the two. From the change of the sight angle, the final sight angle of the two algorithms is stable near the target sight angle. The error of the DDPGNG algorithm is smaller than that of the PNG algorithm.

[0137] Figure 6 It is a comparison curve of the normal acceleration of the two guidance laws. From the normal acceleration graph, it can be seen that DDPGNG has a large overload at around 20s. This is because when designing the reward function, the hit reward and the terminal angle reward are coordinated with relative distance. After a certain critical relative distance, the terminal angle reward will be greater than the hit reward, so the entire guidance law will begin to adjust the line of sight angle, resulting in a sharp increase in overload commands, while the normal acceleration does not change much when the target is about to be hit. When PNG is about to hit the target, the amplitude of the normal acceleration has a relatively large fluctuation, and the amplitude is slightly larger than that of DDPGNG. This is because PNG is affected by the line of sight, and the adjustment of the line of sight becomes more and more drastic when approaching a dynamic target, causing a more drastic change in the normal acceleration.

[0138] The following table compares the performance of the missiles in terms of miss distance and maximum line of sight angle error under the two guidance laws. It can be seen that in this embodiment, the miss distances of the two are not much different, and the miss distance of DDPGNG is smaller.

[0139] In terms of sight angle constraints, PNG is slightly lower than DDPGNG, and both meet the terminal sight angle constraints.

[0140]

[0141] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the computer program.

[0142] In summary, the present invention provides a method for generating a deep reinforcement learning guidance law based on DDPG. This method uses a neural network to directly map the relative motion state between the missile and the target and the expected line of sight angle to the normal acceleration of the missile, thereby generating guidance instructions in real time, providing a new way to obtain the optimal guidance strategy.

[0143] Parts of the present invention that are not described in detail are well known to those skilled in the art.

[0144] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the devices or elements referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention. The directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0145] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" may include both "above" and "below". The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0146] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0147] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating a deep reinforcement learning guidance law based on DDPG, characterized in that: The method comprises: Establish the relative motion model between missile and target; Design a Markov decision process based on the relative motion model between missile and target; Establishing an experience pool for storing data structure groups based on Markov decision process; Establish the Actor network, the Critic network, the target network of the Actor network, and the target network of the Critic network; Based on the relative motion model between the missile and the target and the Markov decision process, the Actor network is trained in real time to obtain the quadruple at each moment, and the quadruple at each moment is stored in the experience pool in sequence; When the number of quadruple groups in the experience pool reaches the preset number, a quadruple group is selected from the experience pool at a preset interval to continuously update the network parameters of the Actor network and the Critic network. At each preset update number, the network parameters of the target network of the Actor network and the target network of the Critic network are updated using the updated network parameters of the Actor network and the Critic network, and the updated network parameters of the target network of the Actor network are used as the network parameters of the Actor network at the current moment, and the updated network parameters of the target network of the Critic network are used as the network parameters of the Critic network at the current moment; When the Actor network and the Critic network are fully converged, the Actor network at the current moment is used as the guidance law.

2. The method according to claim 1, characterized in that The relative motion model between the missile and the target is established by the following formula group: In the formula, m is the missile, t is the target, V m and V t are the linear velocities of the missile and the target respectively, r is the relative distance between the missile and the target, is the rate of change of the relative distance between the missile and the target; q is the line of sight angle between the missile and the target, is the line-of-sight angular velocity between the missile and the target, σ m and σ t are the ballistic inclination angles of the missile and the target, respectively, m and η t are the lead angles of the missile and the target, θ m and θ t are respectively the velocity inclination of the missile and the track inclination of the target, and are the missile's velocity angle change rate and the target's track angle change rate respectively; a m and a t are the normal accelerations of the missile and target, respectively, m and x t are the x-axis coordinates of the missile and the target, m and t are the y-axis coordinates of the missile and the target, respectively, m and v t are the normal velocities of the missile and target, respectively.

3. The method according to claim 1, characterized in that The Markov decision process based on the relative motion model between the missile and the target includes: The relative positions of the missile and the target in the x-axis and y-axis directions, the relative speeds of the missile and the target in the x-axis and y-axis directions, the current sight angle between the missile and the target, and the expected terminal sight angle are used as state quantities in the state space; The normal acceleration of the missile is taken as the state quantity in the action space; Get a hit bonus based on the relative distance between the missile and the target; Obtaining a terminal angle bonus based on the current line of sight angle and the desired terminal line of sight angle between the missile and the target; Obtain a reward function based on the hit reward and the terminal angle reward; A Markov decision process is constructed based on state space, action space, reward function and decay rate.

4. The method according to claim 3, characterized in that The state space is obtained by: S=[Δx,Δy,Δv x ,Δv y ,q,q e ]; Δx=x t -x m ,Δy=y t -y m ; Δv x =v xt -v xm ,Δv y =v yt -v ym ; The action space is obtained by the following formula: A=[a m ]; Obtain the hit bonus by: Get the terminal angle reward through the following formula: Δq=|q-q e |; The reward function is obtained by the following formula: Where S is the state space, Δx and Δy are the relative positions of the missile and the target in the x-axis and y-axis directions, respectively, and Δv x and Δv y are the relative speeds of the missile and the target in the x-axis and y-axis directions, q and q e are the current sight angle and the expected terminal sight angle between the missile and the target, respectively, and x m and x t are the x-axis coordinates of the missile and the target, m and t are the y-axis coordinates of the missile and the target, respectively, xm and v xt are the normal velocities of the missile and the target in the x-axis direction, v ym and v yt are the normal velocities of the missile and the target in the y-axis direction, A is the action space, a m is the normal acceleration of the missile, R1 is the hit reward, r is the relative distance between the missile and the target, k1 and k2 are the first and second hit reward coefficients respectively, R2 is the terminal angle reward, Δq is the current line of sight angle and the expected terminal line of sight angle error between the missile and the target, k3 and k4 are the first and second line of sight angle reward coefficients respectively, R is the reward function, and k5 is the terminal angle coordination coefficient.

5. The method according to claim 1, characterized in that The input dimension of the Actor network is 6 dimensions, and the output dimension is 1 dimension, corresponding to the state space and action space respectively; the input dimension of the Critic network is two-way 6+1 dimensions, corresponding to the state space and action space respectively, and the output dimension is 1 dimension.

6. The method according to claim 1, characterized in that The activation functions of the first two layers of the Actor network are Relu functions, and the activation function of the third layer is tanh function; the activation functions of all layers of the Critic network are Relu functions.

7. The method according to claim 1, characterized in that The parameters of the Actor network and the target network of the Actor network are the same, and the parameters of the Critic network and the target network of the Critic network are the same.

8. The method according to claim 1, characterized in that Based on the relative motion model between the missile and the target and the Markov decision process, the Actor network is trained in real time to obtain the quadruple at each moment, and the quadruple at each moment is stored in the experience pool in sequence, including: Based on the relative motion model between the missile and the target, the current state is obtained in real time; Obtain the current reward value in real time based on the Markov decision process; Input the current state into the Actor network to get the current action; The four-tuple consisting of the current state, reward value, action and the state at the next moment is stored in the experience pool in sequence.

9. The method according to claim 1, characterized in that: Update the Actor network as follows: Update the Critic network by the following formula: among them,d i =R(s i ,a i )+γQ(s i+1 ,a(s i+1 ,θ),η)-Q(s i ,a i ,or); In the formula, θ and η are the network parameters to be updated of the Actor network and the Critic network, θ' and η' are the updated network parameters of the Actor network and the Critic network, β and α are the learning rates of the Actor network and the Critic network, and N is the number of experience data updated at one time. are the partial derivatives of θ, a, and η respectively, s i is the state at the i-th moment, a i is the action at the i-th moment, a(s i ,θ) is the state action value function fitted by the Actor network at the i-th moment, Q(s i ,a i ,η) is the Critic network based on s at the i-th moment i ,a i The fitted state action value function, Q(s i ,a(s i ,θ),η) is the Critic network based on s at the i-th moment i ,a(s i ,θ) fitted state action value function,δ i is the updated loss function, R(s i ,a i ) is the reward function at the i-th moment, γ is the decay rate, Q(s i+1 ,a(s i+1 ,θ),η) is the Critic network based on s at the i+1th moment i+1 ,a(s i+1 ,θ) is the state action value function fitted.

10. The method according to claim 1, characterized in that Update the target network of the Actor network by the following formula: θ”′=τθ+(1-τ)θ”; The target network of the Critic network is updated by the following formula: η"'=τη+(1-τ)η"; Where τ is the learning rate of the target network of the Actor network and the target network of the Critic network, θ” and η” are the network parameters to be updated of the target network of the Actor network and the target network of the Critic network, θ”’ and η”’ are the updated network parameters of the target network of the Actor network and the target network of the Critic network, and θ and η are the network parameters to be updated of the Actor network and the Critic network, respectively.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 10 is implemented.

Images