Depth deterministic strategy gradient driven multi-constraint guidance law design method

Abstract

The invention discloses a design method of a multi-constraint guidance law driven by a depth deterministic strategy gradient, and belongs to the technical field of missile guidance. The method comprises the following steps: S1, establishing a three-dimensional relative kinematics model of a missile interception target; s2, based on the S1, designing a fractional order sliding mode control law and analyzing the stability of the fractional order sliding mode control law; s3, designing and optimizing an A3DPG algorithm; and S4, inducing a strategy network to learn an action strategy through a reward mechanism in a dynamic environment, and realizing a multi-constraint guidance law with shortest time, small miss distance, field of view FOV constraint, acceleration change suppression, sliding mode surface convergence reward, end attack angle constraint and total reward. According to the method, the guidance law with the convergent line-of-sight angle is designed by using the fractional order sliding mode theory, and then the sliding mode surface parameters and the reaching law parameters are dynamically learned by using the reinforcement learning algorithm, so that the guidance law with dynamic adjustment is realized. And inducing a strategy network to learn an optimal guidance law under the evaluation of the indexes through factors such as a reward mechanism in a dynamic environment.

Description

Technical Field

[0001] The invention belongs to the technical field of missile guidance, and in particular relates to a design method for a multi-constraint guidance law driven by a deep deterministic policy gradient. Background Art

[0002] In order to accurately and effectively intercept targets and improve the performance and accuracy of the missile's terminal guidance, it is necessary to consider realistic constraints such as attack angle constraints, overload change constraints, and the seeker's field of view angle. Traditional guidance algorithms, such as biased proportional guidance and sliding mode control with attack angle constraints, can achieve attack angle constraints, but it is difficult to ensure that the target is within the seeker's field of view. With the development trend of intelligence, data-driven aircraft guidance laws have become a hot research topic today, thanks to their flexible design and easy implementation of multiple constraints. However, pure data-driven methods are inferior to traditional methods in terms of stability and reliability.

[0003] Reference [1] designed a deep reinforcement learning guidance method under field angle constraints. It uses a two-dimensional dynamic model as an example to learn the bias term of the biased proportional guidance to adjust the guidance law. It achieves quasi-parallel approach interception by suppressing the angular velocity of the missile-target line of sight, and indirectly realizes the constraint of the missile field of view angle. Since the designed guidance law is not implemented with a three-dimensional dynamic model, it has a certain simplification, and its guidance law has a huge overload when approaching the target at the end. This sharply increased overload puts a lot of pressure on the missile controller, so it has certain limitations in practical significance.

[0004] Reference [2] designed a guidance method with attack angle constraints based on distributed reinforcement learning, which not only considered the convergence of the sight angle but also directly set up a reward mechanism for the field of view angle constraint. For moving targets, it can be seen that the terminal acceleration is still too large, and the field of view angle of the set scheme itself is within the set range, and the maximum value of the field of view angle is larger than the maximum value of the field of view angle of the sliding mode guidance law, which does not reflect the advantages of sliding mode control.

[0005] The terminal constraint angle guidance law proposed in the literature [3] is a three-dimensional dynamic model and is also a bias term used to learn proportional guidance. However, the overload generated by the guidance law varies dramatically, and the reward mechanism only uses angle convergence as a dense reward. The design of the reward mechanism is too simplified.

[0006] Reference [4] designs a three-dimensional guidance law based on deep reinforcement meta-learning and attack angle constraint to solve the problem of intercepting maneuvering targets with time-varying speed missiles with partial actuator failure. The terminal overload does not produce a sudden change, but the terminal line of sight rate suddenly changes, which may cause the target to fly out of the field of view, resulting in insufficient actual terminal accuracy.

[0007] Reference [5] proposed a deep reinforcement learning guidance law based on the double-delayed deep deterministic policy gradient (TD3) algorithm to solve the problem of high-speed maneuvering target interception. It directly used the overload command as the action and verified the advantages of its algorithm. However, it did not consider the limitation of the field of view angle range during the interception of high-speed maneuvering targets, which has certain practical limitations.

[0008] The present invention improves the traditional reinforcement learning DDPG algorithm based on the guidance law designed based on the three-dimensional relative kinematics model, adds a self-attention mechanism as the feature extraction layer of the state, and uses LSTM as the first layer of the policy network and the value network, so that it is more suitable for the learning of sequence data with a longer round period; introduces an exponential decay exploration strategy (step S323), and changes the training strategy, so that the guidance purpose can be accurately achieved under realistic constraints such as attack angle constraints, field of view angle constraints, and suppression of terminal overload transformation.

[0009] [1] Zhang Qinglong, Zhao Bin, Xu Xinpeng. Deep reinforcement learning guidance method under the constraint of field of view for passive detection[J]. Journal of Astronautics, 2024, Vol. 45(8): 1281-1289;

[0010] [2] Li Bohao, An Xuman, Yang Xiaofei, Wu Yunjie, Li Guofei. Distributed reinforcement learning guidance method under attack angle constraint[J]. Journal of Astronautics, 2022, Vol. 43(8): 1061-1069;

[0011] [3] Zheng Chengchen, Li Hui, Tao Wei, Liu Sicheng, Wu Fengguo, He Li. Missile terminal constraint angle guidance law based on deep reinforcement learning[J]. Tactical Missile Technology, 2022, (6): 93-102, 110;

[0012] [4] Liang Chen, Wang Weihong, Lai Chao. Deep reinforcement meta-learning guidance law with attack angle constraint[J]. Journal of Astronautics, 2021, Vol. 42(5): 611-620.

[0013] [5]Qiu Xiaoqi, Gao Changsheng, Jing Wuxing. Deep reinforcement learning guidance law for intercepting maneuvering targets in the atmosphere[J]. Journal of Astronautics, 2022, Vol. 43(5): 685-695 Summary of the invention

[0014] The present invention proposes a design method for a multi-constraint guidance law driven by a deep deterministic policy gradient. This method combines the self-attention mechanism and the LSTM algorithm, improves the policy network and the value network in the traditional DDPG deep reinforcement learning algorithm, and proposes the A3DPG algorithm, which is an attention-driven deterministic policy gradient algorithm reinforcement learning algorithm. The attention layer is used as a feature extraction layer and is shared by the policy network and the value network. In the problem of three-dimensional high-precision interception of targets, the attack angle constraint, overload change constraint, and seeker field of view angle constraint are considered to realize the guidance law design.

[0015] To achieve the above object, the technical solution adopted by the present invention is:

[0016] A method for designing a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm, the method comprising the following steps:

[0017] S1: Establish a three-dimensional relative kinematic model of the missile interception target;

[0018] S2: Based on S1, the fractional-order sliding mode control law is designed and its stability is analyzed;

[0019] S3: Design and optimization of A3DPG algorithm;

[0020] S4: In a dynamic environment, the reward mechanism is used to induce the strategy network to learn action strategies, achieving a multi-constraint guidance law with shortest time, small miss distance, field of view angle FOV constraint, acceleration change suppression, sliding surface convergence reward, terminal attack angle constraint, and total reward.

[0021] Furthermore, in S1, the step of establishing a three-dimensional relative kinematic model of a missile interception target comprises the following steps:

[0022] S11: Establish a kinematic mathematical model for missile interception targets;

[0023]

[0024] Where: r is the relative distance between the projectile and the target; V is the relative velocity between the projectile and the target; m is the speed of the missile; θ m is the ballistic inclination angle of the missile; V is the ballistic deflection angle of the missile; t is the speed of the missile; θ t is the ballistic inclination of the target; is the ballistic deviation angle of the target; θ los is the high and low angle; is the azimuth; is the rate of change of high and low angles; is the azimuth angle change rate;

[0025] S12: Establish the missile's center of mass kinematic model;

[0026]

[0027] Where: a x ,a y ,a z are the acceleration components in the x-axis, y-axis, and z-axis directions in the ballistic coordinate system; x, y, z are the positions of the missile in the launch inertial coordinate system; V is the velocity of the missile's center of mass; θ is the velocity inclination angle of the missile; is the velocity deflection angle of the missile.

[0028] Furthermore, in S2, the design of the fractional-order sliding mode control law and its stability analysis include:

[0029] S21: Derivation of fractional-order sliding mode control law;

[0030] The Caputo type fractional integral is defined as:

[0031]

[0032] The Caputo-type fractional derivative is defined as:

[0033]

[0034] Design sliding surface;

[0035]

[0036] Where: S is the sliding surface; D α is the fractional derivative of the α-th power; sign is the sign function; k 1 ,k 2 ,p,q are sliding surface parameters;

[0037] Design convergence law;

[0038]

[0039] Where: S is the sliding surface; v is the reaching law; η 1 ,η 2 ,m,n are reaching law parameters; sign is the sign function;

[0040] S22: Proof of stability;

[0041] S221: Proof of stability of approaching modes;

[0042] For S, consider choosing the following Lyapunov function The designed composite sliding surface, if the control law satisfies the conditions, makes When the reaching law is satisfied:

[0043]

[0044] Ability to demonstrate stability;

[0045] Where: V 1 is the Lyapunov energy function; S is the sliding surface; D α is the fractional derivative of the α-th power; sign is the sign function; k 1 ,k 2 ,p,q are sliding surface parameters; η 1 ,η 2 ,m,n are reaching law parameters;

[0046] S222: Proof of finite-time convergence of approaching modes;

[0047]

[0048] Let y = ln|s|, then s = e y

[0049]

[0050]

[0051] Where: S is the designed sliding surface; η 1 ,η 2 ,m,n are the reaching law parameters; sign is the sign function; t 0 is the initial time; t s is the sliding surface convergence time;

[0052] S223: Convergence proof of sliding mode;

[0053]

[0054] When the sliding mode variables converge, the system state satisfies the following sliding surface equation:

[0055]

[0056] Further analyzing the global asymptotic stability of the above equation, according to the fractional-order integral operator distribution model, the above equation is equivalent to the following infinite-dimensional integer-order differential equation:

[0057]

[0058] Where:

[0059]

[0060] In the above frequency distribution model, ω is the frequency; z(ω,t) is the real state variable of the fractional-order system; x(t) is the pseudo-state variable; the following two Lyapunov functions are constructed;

[0061] Select the Lyapunov function V 2 Taking the derivative and simplifying we get:

[0062]

[0063] Where: V 2 is the Lyapunov energy function;

[0064] S23: Sliding surface design;

[0065]

[0066]

[0067] Where: θ los is the high and low angle; is the desired terminal attack angle; is the azimuth; is the desired terminal azimuth; S y S is the sliding surface used to control the trajectory inclination; z is the sliding surface used to control the trajectory angle; D α is the fractional derivative of the αth power; k 1 ,k 2 ,k 3 ,k 4 ,p 1 ,q 1 ,p 2 ,q 2 is the sliding surface parameter, sign is the sign function;

[0068] S24: Convergence law design;

[0069]

[0070] roll out:

[0071]

[0072] Among them: S is the sliding surface; v is the reaching law; r is the relative distance between the projectile and the target; is the relative velocity between the projectile and the target; η 1 ,η 2 ,m,n are reaching law parameters; a y is the normal overload of the vertical plane; a z is the normal overload on the lateral plane.

[0073] Furthermore, in S3, the design and optimization of the A3DPG algorithm includes the following steps:

[0074] S31: Build network structure;

[0075] S311: shared feature extraction layer;

[0076] The input state passes through a self-attention layer, which is shared by the policy network and the value network. The self-attention layer generates a representation containing global information by calculating the weighted average of the state.

[0077] S312: Policy Network;

[0078] The first layer of the policy network uses an LSTM layer, which is used to capture long-term dependencies in time series;

[0079] S313: Value Network;

[0080] The first layer of the value network also uses an LSTM layer to evaluate the current state and the actions generated by the policy network;

[0081] S32: Algorithm flow;

[0082] S321: Training strategy;

[0083] After each round, all the data of that round is used for training;

[0084] S322: optimizer update mechanism;

[0085] Updates to the policy network, value network, and attention mechanism;

[0086] S323: Adaptive exploration mechanism;

[0087] According to the total number of training rounds, the variance of Gaussian noise is exponentially reduced, so that the training results tend to be stable as the number of training rounds increases;

[0088]

[0089] In the formula: Episodes is the total number of training times; episode i is the current round number;

[0090] S33: training process;

[0091] S331: Initialize the playback pool and select the required capacity to meet the sequence length of at least 5 rounds;

[0092] S332: Initialization;

[0093] S3321: Online network model;

[0094] Including the self-attention network K(s t |θ k ), policy network u(s t |θ u ), value network Q(s t ,a t |θ Q );

[0095] Among them, s t is the current state of the environment; θ k is the self-attention network parameter; θ u is the policy network parameter; θ Q is the value network parameter; a t The action generated by the policy network at the current moment;

[0096] S3322: target network model;

[0097] Including target self-attention network, target strategy network, target value network K′,u′,Q′;

[0098] Parameter update: K′←K,u′←u,Q′←Q;

[0099] S3323: Setting training parameters;

[0100] α 1 =1×10 -2 , α 2 =1×10 -2 ,γ=0.95,N~N(0,σ 2 );

[0101] Where: α 1 is the policy network learning rate; α 2 is the learning rate of the value network; γ is the discount factor; N is the random noise used to improve the exploration ability;

[0102] S333: Training Episodes rounds, the training process of each round is as follows:

[0103] The following steps are repeated from the initial moment to the end of the round until the missile is successfully intercepted and hits the target or misses the target;

[0104] S3331: Value Network Update:

[0105] During the training process, the goal of the value network is to minimize the TD error, and its objectives are as follows:

[0106] y i =r i +γQ′(si+1 ,μ′(s i+1 ∣θ μ′ )|θ Q′ ,θ K′ )

[0107] Evaluation indicators of value network:

[0108]

[0109] The derivative is:

[0110]

[0111] Update the value network by backpropagating the evaluation indicators:

[0112]

[0113] Where: y i is the expected value of the state at step i; α 2 is the learning rate of the value network; L is the evaluation performance function of the value network; is the derivative of L; θ Q is the value network parameter; θ K is the self-attention network parameter; Q(s i ,a i ∣θ Q ,θ K ) is the value of the state-action pair estimated by the value network;

[0114] S3332: Policy Network Update:

[0115] The performance index function of the policy network is:

[0116]

[0117] The derivative is:

[0118]

[0119] Update the policy network by backpropagating the evaluation indicators:

[0120]

[0121] Where: Q(s i ,a i ∣θ Q ,θ K ) is the value of the state-action pair estimated by the value network; α 1 is the learning rate of the policy network; J is the evaluation performance function of the value network; is the derivative of J; θ μ is the value network parameter; θK are the parameters of the self-attention network.

[0122] Further, in S4, the strategy network is induced to learn the action strategy through the reward mechanism in the dynamic environment; including the following steps:

[0123] S41: State space design;

[0124] S42: Action Space Design;

[0125] S43: Strategy for reward function design;

[0126] S431: Enhance reward feedback; accelerate the learning process by introducing intermediate rewards;

[0127] S432: Introduce a penalty mechanism to prevent the agent from falling into an invalid action loop;

[0128] S433: Sparsity and density; Balancing sparse and dense rewards to optimize learning efficiency;

[0129] S434: Reward scale; ensure that the range of reward values ​​is reasonable and avoid gradient problems;

[0130] S435: Reward stability: ensuring that the reward function is not affected by environmental changes or noise;

[0131] S44: Introduce self-attention mechanism;

[0132] S45: Feature extraction layer; the goal is to extract useful features from the environment state to help the agent make better decisions.

[0133] Furthermore, in S434, the reward scale includes:

[0134] S4341: Reasonable scale: ensure that the range of reward values ​​is reasonable and avoid gradient vanishing or exploding during training;

[0135] S4342: Positive reward; represents the desired behavior so that the agent learns the correct strategy.

[0136] Furthermore, in step S435, the reward stability includes:

[0137] S4351: The reward function is stable; the agent will not be unable to learn effectively due to environmental changes or noise interference;

[0138] S4352: Reward Anatomy; Break down rewards into components to simplify task understanding.

[0139] Furthermore, in S43, the strategy for designing the reward function further includes:

[0140] S436: Multi-target reward;

[0141] S437: Exploration-based rewards;

[0142] S4371: Novelty reward;

[0143] S4372: Adaptive reward mechanism.

[0144] Furthermore, in S4, the reward function expression of the field of view FOV constraint is:

[0145] R 1 = -1·GELU(fov-fov lim ,0.01,2,10)

[0146] Where: R 1 is the reward value for the field of view constraint; fov is the missile field of view; fov lim is the field of view angle constraint range of the seeker;

[0147]

[0148] The reward function expression of the acceleration change suppression is:

[0149] R 2 =0.1·GELU(Δacc,1,1,0.1)

[0150] Δacc=A max ·dt-(Δa y |+|Δa z )

[0151] Where: R 2 is the reward value for overload change suppression; |Δa y | is the absolute value of the rate of change of normal acceleration of the vertical plane; |Δa z | is the absolute value of the rate of change of normal acceleration of the horizontal plane; A max is the maximum overload value of the missile; dt is the integration step length;

[0152] The reward function expression of the sliding surface convergence reward is:

[0153]

[0154] Where: R 3 is the reward value for the sliding surface convergence; S y S is the sliding surface used to control the trajectory inclination; z is a sliding surface used to control the trajectory angle;

[0155]

[0156] The reward function expression for small miss distance is:

[0157]

[0158] Where: R 4 is the reward value for the miss amount; M dis is the off-target amount; K r is the missile killing radius;

[0159] The reward function expression with the shortest time is:

[0160] R 5 =-t f

[0161] Where: R 5 is the reward value for the guidance process time;

[0162] The reward function expression of the terminal attack angle constraint is:

[0163]

[0164] Where: R 6 is the reward value for the attack angle constraint at the end;

[0165] The reward function expression of the total reward is:

[0166] Return = λ 1 R 1 +λ 2 R 2 +λ 3 R 3 +(R 4 +R 5 +R 6 )·done

[0167] Where: 1 =0.3,λ 2 =0.4,λ 3 =0.3, which is the weight distribution of rewards and punishments for different constraints.

[0168] Further, in S41, the state space is designed;

[0169] In order to fully describe the relative kinematics information, the following state vector is selected:

[0170]

[0171] In S42, the action space is designed;

[0172] Select the fractional sliding surface variables and parameters in the reaching law as actions;

[0173] A t ={K 1 ,K 2 ,η 1 ,η 2}

[0174] Where: K 1 ∈(9.5,10.5),K 2 ∈(9.5,10.5),η 1 ∈(0.05,0.15),η 2 ∈(0.05,0.15).

[0175] The beneficial effects of the present invention relative to the prior art are: the design method of the multi-constraint guidance law proposed by the present invention combines the traditional fractional-order sliding mode guidance algorithm with deep reinforcement learning, and uses their various advantages to cleverly realize multiple constraints. The present invention uses the fractional-order sliding mode theory to design a guidance law with convergence of the sight angle (i.e., the attack angle), and then uses the reinforcement learning algorithm to dynamically learn its sliding surface parameters and convergence law parameters, thereby realizing a dynamically adjusted guidance law. And through the reward mechanism in a dynamic environment, taking into account factors such as the possible fast time, the minimum miss amount, the field of view angle FOV constraint, the acceleration change suppression, and the sliding surface convergence reward, the induction strategy network learns the optimal guidance law under the evaluation of the above indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0176] Figure 1 It is a schematic diagram of the three-dimensional relative kinematics model;

[0177] Figure 2 This is the schematic diagram of the A3DPG algorithm;

[0178] Figure 3 It is the structural diagram of the self-attention network, strategy network and value network;

[0179] Figure 4 It is a trajectory curve diagram of different guidance laws for chasing a uniform linear target;

[0180] Figure 5 It is the longitudinal overload variation curve of different guidance laws for chasing a uniform linear target;

[0181] Figure 6 It is the lateral overload variation curve of different guidance laws for chasing a uniform linear target;

[0182] Figure 7 It is the elevation and depression angle variation curve of different guidance laws;

[0183] Figure 8 It is the azimuth angle variation curve of different guidance laws;

[0184] Fig. 9 It is the field of view angle variation curve of different guidance laws;

[0185] Fig.10 It is a trajectory curve diagram of different guidance laws for chasing strong maneuvering targets;

[0186] Fig.11 It is the longitudinal overload variation curve diagram of different guidance laws for chasing strong maneuvering targets;

[0187] Fig.12 It is the lateral overload variation curve of different guidance laws when pursuing a strong maneuvering target;

[0188] Fig.13 It is the elevation and depression angle variation curve of different guidance laws;

[0189] Fig.14 It is the azimuth angle variation curve of different guidance laws;

[0190] Fig.15 It is a curve diagram of field of view angle change for different guidance laws. DETAILED DESCRIPTION

[0191] This embodiment discloses a design method of a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm, the method comprising the following steps:

[0192] S1: Establish a three-dimensional relative kinematic model of the missile interception target;

[0193] S2: Based on S1, the fractional-order sliding mode control law is designed and its stability is analyzed (this step can ensure that the missile can be effectively guided and controlled according to the relative kinematic model);

[0194] S3: Design and optimization of the A3DPG algorithm (the A3DPG algorithm optimizes the guidance law through reinforcement learning so that it can meet multiple constraints);

[0195] S4: Induce the policy network to learn action strategies through the reward mechanism in the dynamic environment, and realize the multi-constraint guidance law with the shortest time, small miss distance, field of view angle FOV constraint, acceleration change suppression, sliding surface convergence reward, terminal attack angle constraint and total reward (this step uses the A3DPG algorithm optimized in S3 to train the policy network through the reward mechanism, so that it can learn action strategies that meet multiple constraints in the dynamic environment).

[0196] The current aircraft guidance law has the following problems:

[0197] 1. Consider the sight angle / attack angle constraints;

[0198] 2. Consider the field of view constraints of the seeker;

[0199] 3. Suppression of terminal overload drastic changes on non-maneuvering targets;

[0200] 4. Limitations of traditional DDPG.

[0201] Furthermore, in S1, the step of establishing a three-dimensional relative kinematic model of a missile interception target comprises the following steps:

[0202] S11: Establish a kinematic mathematical model for missile interception targets;

[0203]

[0204] Where: r is the relative distance between the projectile and the target; V is the relative velocity between the projectile and the target; m is the speed of the missile; θ m is the ballistic inclination angle of the missile; V is the ballistic deflection angle of the missile; t is the speed of the missile; θ t is the ballistic inclination of the target; is the ballistic deviation angle of the target; θ los is the high and low angle; is the azimuth; is the rate of change of high and low angles; is the azimuth angle change rate;

[0205] S12: Establish the missile's center of mass kinematic model;

[0206]

[0207] Where: a x ,a y ,a z are the acceleration components in the x-axis, y-axis, and z-axis directions in the ballistic coordinate system; x, y, z are the positions of the missile in the launch inertial coordinate system; V is the velocity of the missile's center of mass; θ is the velocity inclination angle of the missile; is the velocity deflection angle of the missile.

[0208] Furthermore, in S2, the design of the fractional-order sliding mode control law and its stability analysis include:

[0209] S21: Derivation of fractional-order sliding mode control law;

[0210] Fractional Calculus (FOC) Theory:

[0211] Definition of Caputo-type fractional calculus:

[0212] The Grünwald-Letnikov and Riemann-Liouville definitions require that the initial values ​​of the non-zero value problem have fractional derivatives, which is difficult to obtain in practical engineering. Therefore, the present invention adopts the Caputo-type fractional definition widely used in the engineering field. Unless otherwise specified, the subsequent schemes use D λ f(t) substitution

[0213] D λ f(t) represents the derivative of f(t) of order λ. represents the fractional differential defined by Caputo type, t 0 is the initial time of integration, and t is the current time.

[0214] The Caputo type fractional integral is defined as:

[0215]

[0216] This formula is quoted from Xue Dingyu. Fractional Calculus and Fractional Order Control[M]. Beijing: Science Press, 2018.

[0217] The Caputo-type fractional derivative is defined as:

[0218]

[0219] Cited from the same source as above.

[0220] Design sliding surface;

[0221]

[0222] Where: S is the sliding surface; D α is the fractional derivative of the α-th power; sign is the sign function; k 1 ,k 2 ,p,q are sliding surface parameters;

[0223] Design convergence law;

[0224]

[0225] Where: S is the sliding surface; v is the reaching law; η 1 ,η 2 ,m,n are reaching law parameters; sign is the sign function;

[0226] S22: Proof of stability;

[0227] S221: Proof of stability of approaching modes;

[0228] For S, consider choosing the following Lyapunov function The designed composite sliding surface, if the control law satisfies the conditions, makes When the reaching law is satisfied:

[0229]

[0230] Ability to demonstrate stability;

[0231] Where: V 1 is the Lyapunov energy function; S is the sliding surface; D α is the fractional derivative of the α-th power; sign is the sign function; k 1 ,k 2 ,p,q are sliding surface parameters; η 1 ,η 2 ,m,n are reaching law parameters.

[0232] Take the pitch plane as an example:

[0233]

[0234]

[0235] Enter the control amount a y get:

[0236]

[0237] Where: θ los It is the high and low sight angle; is the desired terminal sight angle (angle of attack); S y is the designed sliding surface; a y is the acceleration component in the x-axis direction in the ballistic coordinate system;

[0238] Stability is proven.

[0239] S222: Proof of finite-time convergence of approaching modes;

[0240]

[0241] Let y = ln|s|, then s = e y

[0242]

[0243] Where: S is the designed sliding surface; η 1 ,η 2 ,m,n are the reaching law parameters; sign is the sign function; t 0 is the initial time; t s is the sliding surface convergence time;

[0244] S223: Convergence proof of sliding mode;

[0245]

[0246] When the sliding mode variables converge, the system state satisfies the following sliding surface equation:

[0247]

[0248] Further analysis of the global asymptotic stability of the above equation shows that, according to the fractional-order integral operator distribution model (the model is cited from: Trigeassou JC, Maamri N, Sabatier J, et al. A Lyapunov Ap proach to the Stability of Fractional Differential Equations[J]. Signal Processing, 2011, 91(3): 437-445), the above equation is equivalent to the following infinite-dimensional integer-order differential equation:

[0249]

[0250] Where:

[0251]

[0252] In the above frequency distribution model, ω is the frequency; z(ω,t) is the real state variable of the fractional-order system; x(t) is the pseudo-state variable; the following two Lyapunov functions are constructed;

[0253] Select the Lyapunov function V 2 Derived and simplified, we get (cited from: Tang Xiao, Ye Jikun. Fractional-order sliding mode guidance law for high-speed maneuvering targets [J]. Aviation Weapons, 2021, Vol. 28 (2): 21-26):

[0254]

[0255] Where: V 2 is the Lyapunov energy function;

[0256] According to the Lyapunov stability theorem, the system state will converge to zero in a finite time, and the convergence speed is different for different fractional orders;

[0257] S23: Sliding surface design (the sliding surface is used to ensure that the missile's ballistic trajectory can converge to the target trajectory quickly and accurately);

[0258]

[0259] Where: θ los is the high and low angle; is the desired terminal attack angle; is the azimuth; is the desired terminal azimuth; S y S is the sliding surface used to control the trajectory inclination; z is the sliding surface used to control the trajectory angle; D α is the fractional derivative of the αth power; k 1 ,k 2 ,k 3 ,k 4 ,p 1 ,q 1 ,p 2 ,q 2 is the sliding surface parameter, sign is the sign function;

[0260] S24: Convergence law design (the convergence law is used to ensure that the missile's sight angle (i.e., attack angle) converges to the designed desired sight angle at a certain convergence speed);

[0261]

[0262] roll out:

[0263]

[0264] Among them: S is the sliding surface; v is the reaching law; r is the relative distance between the projectile and the target; is the relative velocity between the projectile and the target; η 1 ,η 2 ,m,n are reaching law parameters; a y is the normal overload of the vertical plane; a z is the normal overload in the lateral plane (the other variables are the same as described above);

[0265] In order to suppress the chattering of the sliding mode variable along the sliding surface, the function is introduced To increase the stability of the control law.

[0266] Comparison algorithms:

[0267] 1. Guidance law of linear sliding surface based on exponential reaching law:

[0268] 1.1 Linear sliding surface

[0269]

[0270] 1.2 Exponential convergence law

[0271] v=-η 1 S-η2 ·sign(S)

[0272] 1.3 Guidance Law:

[0273]

[0274] 2. Use the optimal guidance law with a navigation ratio of 3:

[0275] 2.1 Proportional guidance law,

[0276]

[0277] Furthermore, in S3, the design and optimization of the A3DPG algorithm includes the following steps:

[0278] Basic principles of A3DPG algorithm:

[0279] Based on the classic Deep Deterministic Policy Gradient (DDPG) algorithm, it has been improved by introducing LSTM and self-attention mechanism to enhance the performance of strategy and value network.

[0280] A3DPG is also based on the reinforcement learning algorithm of the actor-critic framework. It is suitable for continuous action space. Its online network and target network are composed of three networks respectively:

[0281] Self-Attention Network: Extract features from the environment;

[0282] Policy network (Actor): generates actions for a given state;

[0283] Value network (Critic): evaluates the value of a given state-action pair;

[0284] The A3DPG model is improved as follows: the policy network and the value network share the feature extraction layer (self-attention mechanism network), and the first layer of each is implemented with LSTM. The exploration mechanism is improved. As the number of training times increases, the model explores enough states and the network parameters gradually stabilize, an adaptive interference noise exploration mechanism is designed. In addition, the traditional DDPG training method based on random sampling is abandoned, and a new training mechanism is proposed to use round data for training.

[0285] S31: Build network structure;

[0286] S311: shared feature extraction layer;

[0287] The input state passes through a self-attention layer, which helps the network focus on the key parts of the state and weight different features to obtain a more meaningful representation. The self-attention mechanism helps capture dependencies in the state sequence by weighted aggregation of multiple dimensions of the state, which is especially important in tasks with time-dependent dependencies (such as trajectory prediction, target tracking, etc.).

[0288] The self-attention layer is shared by the policy network and the value network (this can improve the training efficiency of the network and reduce the redundancy of parameters, ensuring that both learn similar temporal dependency information);

[0289] The self-attention layer generates a representation containing global information by calculating the weighted average of the states, so that the policy network and the value network can make better decisions and evaluations from a global perspective.

[0290] S312: Policy Network;

[0291] The first layer of the policy network (Actor) uses an LSTM layer, which is used to capture long-term dependencies in time series (since the environmental states in reinforcement learning are often temporal, LSTM can help the policy and value networks remember past states and understand the long-term associations between states, thereby making better decisions and evaluations).

[0292] The goal of the policy network is to generate a reasonable action based on the current state to maximize the long-term reward; the policy network gradually adjusts the parameters of the online self-attention network and the online policy network model through interaction with the value network to improve the quality of decision-making. After reaching the set number of training times (3 times or more), the target self-attention network and the target policy network are updated through soft updates.

[0293] S313: Value Network;

[0294] The first layer of the value network (Critic) also uses the LSTM layer to evaluate the current state and the action generated by the policy network; based on the Q value generated by the target value network and the Q value of the state-action pair estimated by the online value network, it guides the online self-attention network and the online policy network to optimize the parameters. After reaching the set number of training times (3 times or more), the target self-attention network and the target policy network are updated through soft updates.

[0295] S32: Algorithm flow;

[0296] S321: Training strategy;

[0297] After each round, all the data of that round is used for training; unlike the traditional DDPG, which samples and trains from the experience replay pool at each step, the algorithm in the present invention uses all the data of that round for training after each round. This is because in reinforcement learning, the relationship between "state-action" pairs is often temporal, and the action at a certain moment and the future reward may have a long-term dependency. Therefore, using the data of the entire round for training can better capture this temporal information and avoid the dependency between data being destroyed during random sampling.

[0298] S322: optimizer update mechanism;

[0299] The policy network and attention mechanism are updated; the policy network optimizer not only updates the policy network, but also updates the parameters of the self-attention layer. This ensures that while optimizing the policy, the network's weighting of the state (i.e., the attention mechanism) is also optimized accordingly, allowing the entire network to select and evaluate actions more accurately.

[0300] Value network and attention mechanism updates: Similarly, the optimizer of the value network not only updates the parameters of the value network, but also updates the shared self-attention mechanism part. In this way, the value network can also adjust the feature representation according to the self-attention mechanism to better estimate the value of the state-action pair.

[0301] S323: Adaptive exploration mechanism;

[0302] According to the total number of training rounds, the variance of Gaussian noise is exponentially reduced, so that the training results tend to be stable as the number of training rounds increases;

[0303]

[0304] In the formula: Episodes is the total number of training times; episode i is the current round number;

[0305] S33: training process;

[0306] S331: Initialize the playback pool and select the required capacity to meet the sequence length of at least 5 rounds;

[0307] S332: Initialization;

[0308] S3321: Online network model;

[0309] Including the self-attention network K(s t |θ k ), policy network u(s t |θ u ), value network Q(s t,a t |θ Q );

[0310] Among them, s t is the current state of the environment; θ k is the self-attention network parameter; θ u is the policy network parameter; θ Q is the value network parameter; a t The action generated by the policy network at the current moment;

[0311] S3322: target network model;

[0312] Including target self-attention network, target strategy network, target value network K′,u′,Q′;

[0313] Parameter update: K′←K,u′←u,Q′←Q;

[0314] S3323: Setting training parameters;

[0315] α 1 =1×10 -2 , α 2 =1×10 -2 ,γ=0.95,N~N(0,σ 2 );

[0316] Where: α 1 is the policy network learning rate; α 2 is the learning rate of the value network; γ is the discount factor; N is the random noise used to improve the exploration ability;

[0317] S333: Training Episodes rounds, the training process of each round is as follows:

[0318] The following steps are repeated from the initial moment to the end of the round (the policy network takes actions based on the current state. The dynamic environment enters the next state space based on the current state and the actions taken by the model) until the missile is successfully intercepted and hits the target or misses the target;

[0319] S3331: Value Network Update:

[0320] During the training process, the goal of the value network is to minimize the TD error, and its objectives are as follows:

[0321] y i =r i +γQ′(s i+1 ,μ′(s i+1 ∣θ μ′ )|θ Q′ ,θ K′ )

[0322] Evaluation indicators of value network:

[0323]

[0324] The derivative is:

[0325]

[0326] Update the value network by backpropagating the evaluation indicators:

[0327]

[0328] Where: y i is the expected value of the state at step i; α 2 is the learning rate of the value network; L is the evaluation performance function of the value network; is the derivative of L; θ Q is the value network parameter; θ K is the self-attention network parameter; Q(s i ,a i ∣θ Q ,θ K ) is the value of the state-action pair estimated by the value network;

[0329] S3332: Policy Network Update:

[0330] The purpose of the strategy network is to make the actions generated as highly valued as possible by the value network.

[0331] The performance index function of the policy network is:

[0332]

[0333] The derivative is:

[0334]

[0335] Update the policy network by backpropagating the evaluation indicators:

[0336]

[0337] Where: Q(s i ,a i ∣θ Q ,θ K ) is the value of the state-action pair estimated by the value network; α 1 is the learning rate of the policy network; J is the evaluation performance function of the value network; is the derivative of J; θ μ is the value network parameter; θ K is the self-attention network parameter;

[0338] During the training process, the optimization of the policy network and the value network is carried out alternately, and the goal of maximizing long-term rewards is ultimately achieved by gradually adjusting the network parameters.

[0339] Algorithm advantages:

[0340] 1. Capture timing dependencies;

[0341] Introducing LSTM can help reinforcement learning models remember historical information when dealing with tasks with long-term dependencies and adjust future decisions based on this information. This is especially important for tasks where the environment states have obvious temporal relationships (e.g., videos, trajectories, game rounds, etc.).

[0342] 2. Self-attention mechanism;

[0343] The self-attention mechanism allows the network to dynamically focus on important parts of the state. This mechanism can make the model more flexible in decision-making, especially in complex state spaces, and can focus on more important state features. This mechanism helps the model understand the state as a whole rather than relying solely on local features, enhancing the model's expressiveness.

[0344] 3. Round data training;

[0345] By using all the data for training after an episode, the model can better understand the temporal dependencies between states, rather than just the decisions of a single time step. This training method is closer to the actual situation of reinforcement learning tasks and can improve the performance of the model in long-term tasks.

[0346] 4. Parameter sharing;

[0347] The shared design of LSTM and self-attention mechanism reduces the number of model parameters and improves training efficiency. The shared feature extraction layer can effectively transfer timing and global information between the policy network and the value network, which helps to accelerate the training process and improve generalization ability.

[0348] 5. Robust training process;

[0349] Since the optimizers of the policy and value networks act on the policy / value network and the self-attention mechanism respectively, the training process is more stable, avoiding the instability between the policy network and the value network that is common in traditional methods. The optimization of the self-attention mechanism will gradually guide the entire model to focus and make decisions more accurately in the state space.

[0350] In summary, the A3DPG algorithm proposed in the present invention adds LSTM and self-attention mechanisms to the framework of classic DDPG, so that the model can better handle reinforcement learning tasks with long-term dependencies and complex structures. By training and sharing LSTM layers after the end of the round, the performance of the algorithm in sequential tasks is expected to be significantly improved, especially in tasks that require memory, sequence dependencies, and global context information. At the same time, by updating the optimizers of the policy network and the value network separately, not only the network update is more robust, but also the training efficiency and performance are improved.

[0351] Further, in S4, the strategy network is induced to learn the action strategy through the reward mechanism in the dynamic environment; including the following steps:

[0352] S41: State space design;

[0353] S42: Action Space Design;

[0354] S43: Strategy for reward function design;

[0355] S431: Enhance reward feedback; speed up the learning process by introducing intermediate rewards (sparse rewards often slow down the learning process of the agent. To avoid this, introduce more intermediate rewards so that the agent can get some positive feedback as it gradually approaches the goal);

[0356] S432: Introduce a penalty mechanism to prevent the agent from falling into an invalid action loop (the agent may fall into an invalid action loop, such as switching back and forth between the same states; to avoid this situation, a penalty mechanism is introduced);

[0357] S433: Sparsity and density; Balancing sparse and dense rewards to optimize learning efficiency;

[0358] S434: Reward scale; ensure that the range of reward values ​​is reasonable and avoid gradient problems;

[0359] S435: Reward stability: ensuring that the reward function is not affected by environmental changes or noise;

[0360] S44: Introduce self-attention mechanism;

[0361] It is a technology that can dynamically capture the relationship between elements in sequence data. Its application in the DDPG algorithm can effectively enhance the feature extraction capability, help the intelligent agent better understand the state relationship in a complex environment, and improve the learning effect. This makes self-attention a feature extraction layer choice worth considering in reinforcement learning;

[0362] S45: Feature extraction layer; the goal is to extract useful features (abstract features, no physical meaning) from the environmental state (including six variables: relative distance, relative speed, elevation angle, azimuth angle, ballistic inclination of the interceptor missile, and ballistic deviation of the interceptor missile) to help the agent make better decisions. Self-attention is used to process time series state inputs to help the model understand the relationship between states. The self-attention mechanism can dynamically focus on important state information, thereby generating a more effective state representation and helping the policy network learn better.

[0363] Furthermore, in S434, the reward scale includes:

[0364] S4341: Reasonable scale: ensure that the range of reward values ​​is reasonable and avoid gradient vanishing or exploding during training;

[0365] S4342: Positive reward; represents the desired behavior so that the agent learns the correct strategy.

[0366] Furthermore, in step S435, the reward stability includes:

[0367] S4351: The reward function is stable; the agent will not be unable to learn effectively due to environmental changes or noise interference;

[0368] S4352: Reward Parsing; Breaking down rewards into components to simplify task understanding (making it easier for the agent to understand the structure and requirements of the task).

[0369] Clarify the goal; set appropriate reward values ​​for the agent (based on the constraints and expected effects, avoid sparse rewards as much as possible, keep the values ​​stable, and ensure that the reward values ​​between constraints are of the same magnitude) (reward design should clarify the learning goals of the agent so as to set appropriate reward values ​​for various behaviors of the agent);

[0370] Guide learning; guide the agent to learn through positive and negative rewards (reward design should be able to guide the agent to learn, that is, provide positive rewards for the agent to perform behaviors that help achieve learning goals, and provide negative rewards or no rewards for harmful behaviors);

[0371] Balance; ensure that the reward design does not favor a certain behavior (the reward design should be balanced, that is, it should not be too biased towards a certain behavior, otherwise it will cause the agent to learn more slowly or even fail to learn).

[0372] Furthermore, in S43, the strategy for designing the reward function further includes:

[0373] S436: Multi-target reward; In complex tasks (in multi-constraint guidance laws, not only field of view constraints but also overload change suppression are considered, the time is as short as possible, and the miss distance is as small as possible), multiple targets need to be optimized at the same time. In this case, a combined reward is designed to reflect the importance of each target, and different reward signals are integrated through a weighted method.

[0374] S437: Exploration-based rewards;

[0375] S4371: Novelty reward: rewards for exploring new states or performing new actions, encouraging agents to explore unseen environments;

[0376] S4372: Adaptive reward mechanism; the reward mechanism is dynamically adjusted according to the learning progress of the agent (for example, as the performance of the agent improves, the reward is gradually reduced to maintain the challenge. The reward is increased or decreased in a timely manner according to the learning efficiency of the agent).

[0377] Furthermore, in S4, the reward function expression of the field of view FOV constraint is:

[0378] R 1 = -1·GELU(fov-fov lim ,0.01,2,10)

[0379] Where: R 1 is the reward value for the field of view constraint; fov is the missile field of view; fov lim is the field of view angle constraint range of the seeker;

[0380] This paper proposes the "generalized exponential linear unit" function, which is a generalization of the ELU function, adding parameters and providing higher flexibility.

[0381]

[0382] The reward function expression of the acceleration change suppression is:

[0383] R 2 =0.1·GELU(Δacc,1,1,0.1)

[0384] Δacc=A max ·dt-(|Δa y |+|Δa z |)

[0385] Where: R 2 is the reward value for overload change suppression; |Δa y | is the absolute value of the rate of change of normal acceleration of the vertical plane; |Δa z | is the absolute value of the rate of change of normal acceleration of the horizontal plane; Amax is the maximum overload value of the missile; dt is the integration step length;

[0386] The reward function expression of the sliding surface convergence reward is:

[0387]

[0388] Where: R 3 is the reward value for the sliding surface convergence; S y S is the sliding surface used to control the trajectory inclination; z is a sliding surface used to control the trajectory angle;

[0389]

[0390] The reward function expression for small miss distance is:

[0391]

[0392] Where: R 4 is the reward value for the miss amount; M dis is the off-target amount; K r is the missile killing radius;

[0393] The reward function expression with the shortest time is:

[0394] R 5 =-t f

[0395] Where: R 5 is the reward value for the guidance process time;

[0396] The reward function expression of the terminal attack angle constraint is:

[0397]

[0398] R 6 is the reward value for the attack angle constraint at the end.

[0399] Where:

[0400] The reward function expression of the total reward is:

[0401] Return = λ 1 R 1 +λ 2 R 2 +λ 3 R 3 +(R 4 +R 5 +R 6 )·done

[0402] Where: 1 =0.3,λ 2 =0.4,λ 3 =0.3, which is the weight distribution of rewards and punishments for different constraints.

[0403] Further, in S41, the state space is designed;

[0404] In order to fully describe the relative kinematics information, the following state vector is selected:

[0405]

[0406] In S42, the action space is designed;

[0407] Select the fractional sliding surface variables and parameters in the reaching law as actions;

[0408] A t ={K 1 ,K 2 ,η 1 ,η 2}

[0409] Where: K 1 ∈(9.5,10.5),K 2 ∈(9.5,10.5),η 1 ∈(0.05,0.15),η 2 ∈(0.05,0.15). Design simulation cases and compare them with traditional methods to verify the effectiveness of the invention.

[0410] Embodiment 1:

[0411] Objectives of uniform linear motion:

[0412]

[0413] Embodiment 2:

[0414] The goal of strong mobility:

[0415] The target performs an 'S' shape maneuver in the pitch plane and a 'C' shape maneuver in the yaw plane.

[0416]

[0417] The comparison algorithm in the embodiment:

[0418] 1. Proportional guidance law:

[0419] Take the simplified optimal guidance law with a navigation ratio of 3 as an example;

[0420]

[0421] 2. Guidance law of linear sliding surface based on exponential reaching law;

[0422]

[0423] For Example 1, the parameters are selected as follows:

[0424] k 1 =3,k 2 =3,k 3 =0.5,k 4 =0.5,ε 1 =0.001,ε 2 =0.001

[0425] For case 2, the parameters are selected as follows:

[0426] k 1 =1,k 2 =1,k 3 =0.05,k 4 =0.05,ε 1 =0.001,ε 2 =0.001.

[0427] like Figure 2 As shown in the figure, the A3DPG (attention-driven deep deterministic policy gradient algorithm) algorithm principle diagram, the online network continuously interacts with the dynamic environment, the value network in the online network performs temporal difference with the state action value generated by the value network of the target network, and then updates the parameters of the online value network, and the online policy network updates the parameters according to the evaluation value of the value network. After a certain step length, the online network soft-updates the target network.

[0428] like Figure 3 As shown in the figure, the structure diagram of the self-attention network, policy network and value network. The policy network and the value network use the same network architecture. The first layer is composed of LSTM, followed by normalization layer, linear layer, normalization layer, random drop layer, and linear layer. External information first passes through the self-attention layer, and then passes through the value network or policy network to generate output.

[0429] Simulation case 1: Tracking a uniform linear motion target;

[0430] like Figure 4 As shown, from the three-dimensional ballistic curves of the method of the present invention and the other two classical methods for striking uniform linear targets, it can be seen that the ballistic curve based on the A3DPG guidance law of the present invention is obviously smoother than that of the other two algorithms, and is faster and hits the target earlier.

[0431] like Figure 5As shown, it can be clearly seen from the acceleration curves of the three guidance methods in the y-axis direction of the ballistic coordinate system that the guidance law of the A3DPG algorithm proposed in the present invention is relatively small, that is, the fuel consumption is small; and the terminal overload is significantly smaller than that of the other two methods, which can improve the hit rate of the interceptor missile.

[0432] like Figure 6 As shown in the figure, it can be clearly seen from the acceleration curves of the three guidance methods in the z-axis direction of the ballistic coordinate system that the total amount of the A3DPG algorithm guidance law proposed by the present invention is relatively small, that is, the fuel consumption is small. And the terminal overload is significantly smaller than the other two methods, which can improve the hit rate of the interceptor missile.

[0433] like Figure 7 As shown in the figure, it can be seen from the elevation angle curves of the missile-target sight line of sight of the three guidance methods that the classic proportional guidance method cannot achieve sight line of sight elevation angle constraints, that is, attack elevation angle constraints. The A3DPG algorithm and the sliding mode guidance method proposed in the present invention can both achieve attack elevation angle constraints, but the early convergence of the sight line of sight elevation angle of the sliding mode method may cause the interceptor missile to be sensitive to target maneuvering changes, and the overload command may be unstable and abrupt.

[0434] like Figure 8 As shown in the figure, it can be seen from the missile-target line-of-sight azimuth curves of the three guidance methods that the classic proportional guidance method cannot achieve line-of-sight azimuth constraint, that is, attack azimuth constraint. The A3DPG algorithm and the sliding mode guidance method proposed in the present invention can both achieve attack azimuth constraint, but the line-of-sight azimuth of the sliding mode method converges too early, which may cause the interceptor missile to be sensitive to target maneuver changes, and the overload command may be unstable and increase suddenly.

[0435] like Fig. 9 As shown in the figure, it is obvious from the field of view constraint curves of the three guidance laws that the maximum field of view angle is set. Only the A3DPG algorithm can achieve the whole guidance process without exceeding this angle, while the other two methods have some stages exceeding this angle. In actual engineering, exceeding this angle may cause the seeker to lose the target, making it difficult to intercept the target.

[0436] Simulation Case 2: Tracking a target with strong maneuverability;

[0437] like Fig.10 As shown, from the three-dimensional ballistic curves of the method of the present invention and the other two classical methods for attacking highly maneuverable targets, it can be seen that the ballistic curve based on the A3DPG guidance law of the present invention is obviously smoother than that of the other two algorithms, and is faster and hits the target earlier.

[0438] like Fig.11As shown, it can be clearly seen from the acceleration curves of the three guidance methods in the y-axis direction of the ballistic coordinate system that the total amount of the A3DPG algorithm guidance law proposed in the present invention is relatively small, that is, the fuel consumption is small; and the terminal overload is significantly smaller than the other two methods, which can improve the hit rate of the interceptor missile.

[0439] like Fig.12 As shown, it can be clearly seen from the acceleration curves of the three guidance methods in the z-axis direction of the ballistic coordinate system that the total amount of the A3DPG algorithm guidance law proposed in the present invention is relatively small, that is, the fuel consumption is small; and the terminal overload is significantly smaller than the other two methods, which can improve the hit rate of the interceptor missile.

[0440] like Fig.13 As shown in the figure, it can be seen from the elevation angle curves of the missile-target sight line of sight of the three guidance methods that when tracking a strong maneuvering target, the classic proportional guidance and sliding mode methods cannot achieve the sight line of sight elevation angle constraint, that is, the attack elevation angle constraint. Due to the strong maneuverability of the target, the A3DPG algorithm proposed in the present invention is not completely accurate in achieving the attack elevation angle constraint, but it is significantly improved compared with the other two algorithms.

[0441] like Fig.14 As shown in the figure, it can be seen from the line-of-sight azimuth curves of the three guidance methods that when tracking a strong maneuvering target, the classic proportional guidance and sliding mode guidance methods cannot achieve line-of-sight azimuth constraint, that is, attack azimuth constraint. The A3DPG algorithm proposed in the present invention is not completely accurate in achieving attack azimuth constraint, but it is significantly improved compared to the other two algorithms.

[0442] like Fig.15 As shown in Figure 1, it can be seen from the field of view constraint curves of the three guidance laws that the maximum field of view angle is set. All three methods can achieve field of view constraints in the whole process, but this is the constraint of the other two algorithms due to the coincidence of the relative motion scene, not the constraint of the algorithm.

[0443] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A design method for a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm, characterized by: The method comprises the following steps: S1: Establish a three-dimensional relative kinematic model of the missile interception target; S2: Based on S1, the fractional-order sliding mode control law is designed and its stability is analyzed; S3: Design and optimization of A3DPG algorithm; S4: In a dynamic environment, the reward mechanism is used to induce the strategy network to learn action strategies, achieving a multi-constraint guidance law with shortest time, small miss distance, field of view angle FOV constraint, acceleration change suppression, sliding surface convergence reward, terminal attack angle constraint, and total reward.

2. The design method of a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm according to claim 1, characterized in that: In S1, the three-dimensional relative kinematic model of the missile interception target is established; comprising the following steps: S11: Establish a kinematic mathematical model for missile interception targets; Where: r is the relative distance between the projectile and the target; V is the relative velocity between the projectile and the target; m is the speed of the missile; θ m is the ballistic inclination angle of the missile; V is the ballistic deviation angle of the missile; t is the speed of the missile; θ t is the ballistic inclination of the target; is the ballistic deviation angle of the target; θ los is the high and low angle; is the azimuth; is the rate of change of high and low angles; is the azimuth angle change rate; S12: Establish the missile's center of mass kinematic model; Where: a x ,a y ,a z are the acceleration components in the x-axis, y-axis, and z-axis directions in the ballistic coordinate system; x, y, z are the positions of the missile in the launch inertial coordinate system; V is the velocity of the missile's center of mass; θ is the velocity inclination angle of the missile; is the velocity deflection angle of the missile.

3. The design method of a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm according to claim 1, characterized in that: In S2, the design of the fractional-order sliding mode control law and its stability analysis; include: S21: Derivation of fractional-order sliding mode control law; The Caputo type fractional integral is defined as: The Caputo-type fractional derivative is defined as: Design sliding surface; Where: S is the sliding surface; D α is the fractional derivative of the αth power; sign is the sign function; k1, k2, p, q are the sliding surface parameters; Design convergence law; Where: S is the sliding surface; v is the reaching law; η1, η2, m, n are the reaching law parameters; sign is the sign function; S22: Proof of stability; S221: Proof of stability of approaching modes; For S, consider choosing the following Lyapunov function The designed composite sliding surface, if the control law satisfies the conditions, makes When the reaching law is satisfied: Ability to demonstrate stability; Where: V1 is the Lyapunov energy function; S is the sliding surface; D α is the fractional derivative of the αth power; sign is the sign function; k1, k2, p, q are the sliding surface parameters; η1, η2, m, n are the reaching law parameters; S222: Proof of finite-time convergence of approaching modes; Let y = ln|s|, then s = e y Where: S is the designed sliding surface; η1, η2, m, n are reaching law parameters; sign is the sign function; t0 is the initial time; t s is the sliding surface convergence time; S223: Convergence proof of sliding mode; When the sliding mode variables converge, the system state satisfies the following sliding surface equation: Further analyzing the global asymptotic stability of the above equation, according to the fractional-order integral operator distribution model, the above equation is equivalent to the following infinite-dimensional integer-order differential equation: Where: In the above frequency distribution model, ω is the frequency; z(ω,t) is the real state variable of the fractional-order system; x(t) is the pseudo-state variable; the following two Lyapunov functions are constructed; Select the Lyapunov function Taking the derivative of V2 and simplifying it, we get: Where: V2 is the Lyapunov energy function; S23: Sliding surface design; Where: θ los is the high and low angle; is the desired terminal attack angle; is the azimuth; is the desired terminal azimuth; S y is the sliding surface used to control the trajectory inclination; S z is the sliding surface used to control the trajectory angle; D α is the fractional derivative of the αth power; k1, k2, k3, k4, p1, q1, p2, q2 are sliding surface parameters, and sign is the sign function; S24: Convergence law design; roll out: Among them: S is the sliding surface; v is the reaching law; r is the relative distance between the projectile and the target; is the relative velocity between the projectile and the target; η1, η2, m, n are the reaching law parameters; a y is the normal overload of the vertical plane; a z is the normal overload on the lateral plane.

4. The design method of a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm according to claim 1, characterized in that: In S3, the design and optimization of the A3DPG algorithm; The steps include: S31: Build network structure; S311: shared feature extraction layer; The input state passes through a self-attention layer, which is shared by the policy network and the value network. The self-attention layer generates a representation containing global information by calculating the weighted average of the state. S312: Policy Network; The first layer of the policy network uses an LSTM layer, which is used to capture long-term dependencies in time series; S313: Value Network; The first layer of the value network also uses an LSTM layer to evaluate the current state and the actions generated by the policy network; S32: Algorithm flow; S321: Training strategy; After each round, all the data of that round is used for training; S322: optimizer update mechanism; Updates to the policy network, value network, and attention mechanism; S323: Adaptive exploration mechanism; According to the total number of training rounds, the variance of Gaussian noise is exponentially reduced, so that the training results tend to be stable as the number of training rounds increases; In the formula: Episodes is the total number of training times; episode i is the current round number; S33: training process; S331: Initialize the playback pool and select the required capacity to meet the sequence length of at least 5 rounds; S332: Initialization; S3321: Online network model; Including the self-attention network K(s t |θ k ), policy network u(s t |θ u ), value network Q(s t ,a t |θ Q ); Among them, s t is the current state of the environment; θ k is the self-attention network parameter; θ u is the policy network parameter; θ Q is the value network parameter; a t The action generated by the policy network at the current moment; S3322: target network model; Including target self-attention network, target strategy network, target value network K′,u′,Q′; Parameter update: K′←K,u′←u,Q′←Q; S3323: Setting training parameters; α1=1×10 -2 ,α2=1×10 -2 ,γ=0.95,N~N(0,σ 2 ); Among them: α1 is the learning rate of the policy network; α2 is the learning rate of the value network; γ is the discount factor; N is the random noise used to improve the exploration ability; S333: Training Episodes rounds, the training process of each round is as follows: The following steps are repeated from the initial moment to the end of the round until the missile is successfully intercepted and hits the target or misses the target; S3331: Value Network Update: During the training process, the goal of the value network is to minimize the TD error, and its objectives are as follows: y i =r i +γQ′(s i+1 ,μ′(s i+1 ∣θ μ′ )∣θ Q′ ,θ K′ ) Evaluation indicators of value network: The derivative is: Update the value network by backpropagating the evaluation indicators: Where: y i is the expected value of the state at step i; α2 is the learning rate of the value network; L is the evaluation performance function of the value network; is the derivative of L; θ Q is the value network parameter; θ K is the self-attention network parameter; Q(s i ,a i ∣θ Q ,θ K ) is the value of the state-action pair estimated by the value network; S3332: Policy Network Update: The performance index function of the policy network is: The derivative is: Update the policy network by backpropagating the evaluation indicators: Where: Q(s i ,a i ∣θ Q ,θ K ) is the value of the state-action pair estimated by the value network; α1 is the learning rate of the policy network; J is the evaluation performance function of the value network; is the derivative of J; θ μ is the value network parameter; θ K are the parameters of the self-attention network.

5. The design method of a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm according to claim 1, characterized in that: In S4, the policy network is induced to learn the action strategy through the reward mechanism in the dynamic environment; the following steps are included: S41: State space design; S42: Action Space Design; S43: Strategy for reward function design; S431: Enhance reward feedback; accelerate the learning process by introducing intermediate rewards; S432: Introduce a penalty mechanism to prevent the agent from falling into an invalid action loop; S433: Sparsity and density; Balancing sparse and dense rewards to optimize learning efficiency; S434: Reward scale; ensure that the range of reward values ​​is reasonable and avoid gradient problems; S435: Reward stability: ensuring that the reward function is not affected by environmental changes or noise; S44: Introduce self-attention mechanism; S45: Feature extraction layer; the goal is to extract useful features from the environment state to help the agent make better decisions.

6. The method for designing a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm according to claim 5, characterized in that: In S434, the reward scale includes: S4341: Reasonable scale: ensure that the range of reward values ​​is reasonable and avoid gradient vanishing or exploding during training; S4342: Positive reward; represents the desired behavior so that the agent learns the correct strategy.

7. The method for designing a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm according to claim 5, characterized in that: In step S435, the reward stability includes: S4351: The reward function is stable; the agent will not be unable to learn effectively due to environmental changes or noise interference; S4352: Reward Anatomy; Break down rewards into components to simplify task understanding.

8. The method for designing a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm according to claim 5, characterized in that: In S43, the strategy for designing the reward function further includes: S436: Multi-target reward; S437: Exploration-based rewards; S4371: Novelty reward; S4372: Adaptive reward mechanism.

9. The method for designing a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm according to claim 1, characterized in that: In S4, the reward function expression of the field of view FOV constraint is: R1=-1·GELU(fov-fov lim ,0.01,2,10) Where: R1 is the reward value for the field of view constraint; fov is the missile field of view; fov lim is the field of view angle constraint range of the seeker; The reward function expression of the acceleration change suppression is: R2=0.1·GELU(Δacc,1,1,0.1) Δacc=A max ·dt-(Δa y |+|Δa z |) Where: R2 is the reward value for overload change suppression; |Δa y | is the absolute value of the rate of change of normal acceleration of the vertical plane; |Δa z | is the absolute value of the rate of change of normal acceleration of the horizontal plane; A max is the maximum overload value of the missile; dt is the integration step length; The reward function expression of the sliding surface convergence reward is: Where: R3 is the reward value for the convergence of the sliding surface; S y is the sliding surface used to control the trajectory inclination; S z It is a sliding surface used to control the trajectory angle; The reward function expression for small miss distance is: Where: R4 is the reward value for the miss amount; M dis is the off-target amount; K r is the missile killing radius; The reward function expression with the shortest time is: R5=-t f Where: R5 is the reward value for the guidance process time; The reward function expression of the terminal attack angle constraint is: Where: R6 is the reward value for the attack angle constraint at the end; The reward function expression of the total reward is: Return=λ1R1+λ2R2+λ3R3+(R4+R5+R6)·done Where: λ1=0.3,λ2=0.4,λ3=0.3, which are the weight distribution of rewards and punishments for different constraints.

10. The method for designing a multi-constraint guidance law driven by a deep deterministic policy gradient algorithm according to claim 5, characterized in that: In S41, the state space design; In order to fully describe the relative kinematics information, the following state vector is selected: In S42, the action space is designed; Select the fractional sliding surface variables and parameters in the reaching law as actions; A t ={K1,K2,η1,η2} Among them: K1∈(9.5,10.5),K2∈(9.5,10.5),η1∈(0.05,0.15),η2∈(0.05,0.15).

Images