Intercepted maneuvering target reinforcement learning guidance law design method and device based on imitation learning

Abstract

The invention discloses an intercepted maneuvering target reinforcement learning guidance law design method and device based on imitation learning, and belongs to the technical field of guidance law design. The method comprises the following steps: firstly, establishing a maneuvering target and interceptor three-dimensional uncertain confrontation model on the basis of an interception collision triangle; then taking a proportional guidance law as a strategy, generating a large number of track samples by utilizing an interaction environment, establishing an intelligent terminal guidance model, and training the intelligent terminal guidance model by adopting a supervised learning method to obtain an intelligent terminal guidance model approaching proportional guidance interception performance; and finally, setting a reward function by utilizing a reinforcement learning method, and further training the intelligent terminal guidance model obtained by imitation learning by adopting a PPO algorithm to obtain a novel guidance law exceeding proportional guidance interception performance. Simulation results show that the novel guidance law is good in robustness and high in stability, higher interception probability and lower energy consumption are achieved compared with a traditional guidance algorithm, time consumed by a single decision is only 0.32 ms, and the novel guidance law has certain engineering application value.

Description

Technical Field

[0001] The present invention relates to the technical field of guidance law design, and in particular to a method and device for designing a guidance law for intercepting a maneuvering target based on reinforcement learning based on imitation learning. Background Art

[0002] The design of guidance laws for high-speed target interception has always been a research hotspot in the field of missile defense. Proportional navigation guidance (PNG) and its improved versions have the advantages of simple structure and easy implementation, and are widely used in the interception process. PNG assumes that the target has no maneuvers or fixed maneuvers. Under this linear assumption, it has good interception performance. However, when dealing with targets that use maneuvering penetration technology, its interception performance will be significantly reduced. Augmented proportional navigation (APN) superimposes the target maneuvering acceleration onto the PNG guidance instructions, which improves the interception performance against maneuvering targets to a certain extent. However, since APN needs to obtain target acceleration information in real time, its engineering application possibility is reduced.

[0003] In order to intercept maneuvering targets, scholars have applied nonlinear control, optimal control and differential game theory to the field of guidance and proposed a series of new guidance laws. The literature "Nonsingular Terminal Sliding Mode Guidance with Impact Angle Constraints", "Optimal sliding-mode guidance with terminal velocity constraint for fixed-interval propulsive maneuvers" and "Robust terminal angle constraint guidance law with autopilot lag for intercepting maneuvering targets" applied sliding mode control theory to the design of guidance laws for intercepting maneuvering targets, and conducted research on the shortcomings of sliding film control such as singularity and buffeting. In recent years, optimal control has been widely developed in the field of guidance law design. For example, the papers “Optimal guidance and collision avoidance for docking with the rotating target spacecraft[J]” and “Time-optimal guidance for intercepting moving targets with impact-angle constraints” respectively studied the energy-optimal and time-optimal guidance laws based on optimal control. Using differential game theory, the papers “Optimal guidance against active defense ballistic missiles via differential game strategies”, “Optimal Cooperative Guidance Laws in a Multiagent Target-Missile-Defender En-gagement” and “Guidance strategies for interceptor against active defense spacecraft in two-on-two engagement” studied the interceptor and target attack and defense confrontation problems and proposed a class of optimal guidance schemes.

[0004] Although most of the aforementioned new guidance laws have good effects in the simulation of intercepting maneuvering targets, they still have certain disadvantages: (1) Most algorithms linearize the scene, so it is difficult to guarantee optimality in practical applications; (2) The guidance law based on optimal control requires estimation of the target motion trajectory, remaining flight time and interception point, which is difficult to obtain in the actual interception process; (3) The guidance law based on differential countermeasures has high computational complexity, which is difficult to meet with the on-board computing power.

[0005] As an important branch of artificial intelligence technology, deep reinforcement learning (DRL) is widely used in sequential decision-making. DRL does not require any supervisory information, but interacts with the environment through a trial-and-error mechanism to maximize the cumulative reward in a complex environment and learn the optimal strategy. Since DRL can directly map the state to guidance instructions and achieve end-to-end decision-making, it has become a research hotspot in the field of guidance and control, such as powered descent guidance, space exploration guidance, missile guidance, and rendezvous guidance. The successful application of DRL and the continuous development of related technologies have also provided new ideas for the design of interception guidance laws, and a large number of scholars have conducted research on this.

[0006] The paper "Missile Homing-Phase Guidance Law Design Using Reinforcement Learning" first studied the DRL guidance problem in a single fixed scenario, inspiring the development of related research. The paper "Guidance law based on zero effort miss and Q-learning algorithm" discretizes the action space based on the Q-learning method and obtains the reinforcement learning guidance law in a simple scenario. The paper "Computational Missile Guidance: A Deep Reinforcement Learning Approach" studies the application of DRL in missile guidance and proposes two training modes: one is that the intelligent agent learns guidance instructions from scratch; the other is based on prior knowledge learning, that is, learning the navigation ratio of the classical guidance law. At the same time, the author points out that the former cannot guarantee the interception of the target under certain initial conditions, while the latter can improve learning efficiency. The papers “Q-learning reinforcement learning guidance law” and “Deepreinforcement learning based missile guidance law design for maneuvering target interception” focus on the problem of maneuvering target interception. They also use the PNG navigation ratio as the action and study the reinforcement learning guidance law based on deep deterministic policy gradient (DDPG) and the reinforcement learning guidance law based on Q-learning respectively.

[0007] Although the aforementioned literature has comprehensively studied the application of reinforcement learning in interception guidance laws, there is still room for improvement. Reinforcement learning agents learn from scratch, and their learning efficiency is not high in complex and long-time scenarios. In particular, for the interception problem, both the target and the interceptor are in high-speed motion, and a mistake by the interceptor in one time step will increase the possibility of interception failure; at the same time, since the success of the interception can only be obtained at the end of the simulation, there is a sparse reward problem, which makes it impossible for the agent to explore an effective interception strategy. How to improve training efficiency remains a difficult point in research. Summary of the invention

[0008] In view of the above-mentioned problems, the present invention aims to provide a method and device for designing a reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning. First, a large number of expert samples are generated based on proportional guidance, and the intelligent agent learns the expert strategy using an imitation learning method; then, the intelligent agent further uses a reinforcement learning algorithm to gradually learn the strategy for intercepting maneuvering targets, ultimately achieving the minimum energy consumption and the maximum interception probability.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0010] The method for designing a reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning includes the following steps:

[0011] S1: Based on the interception collision triangle, a three-dimensional uncertain confrontation model between maneuvering targets and interceptors is established;

[0012] S2: Taking the proportional guidance law as the strategy, a large number of trajectory samples are generated in the interactive environment, and an intelligent terminal guidance model is established. The model is trained using the supervised learning method to obtain an intelligent terminal guidance model that approaches the interception performance of the proportional guidance.

[0013] S3: Using the reinforcement learning method, the reward function is set, and the PPO algorithm is used to further train the intelligent terminal guidance model learned in step S2 to obtain a new guidance law that exceeds the proportional guidance interception performance.

[0014] Furthermore, the specific operation of step S1 includes the following steps:

[0015] S101: Design an interactive environment for maneuvering target and interceptor confrontation training based on the collision triangle model;

[0016] S102: Establishing a maneuvering model of a maneuvering target;

[0017] S103: Initializing the interactive environment designed in step S101 according to the maneuvering model of the maneuvering target, and solving the initial velocity vector of the interceptor;

[0018] S104: Establishing a terminal miss distance calculation model.

[0019] Furthermore, the maneuvering model of the maneuvering target described in step S102 is expressed as

[0020]

[0021] Where N m Maneuvering overload for maneuvering targets, a M is the maneuvering acceleration vector of the maneuvering target, g represents the gravitational acceleration; and are the minimum and maximum values ​​of the maneuverable start time of the maneuverable target, respectively; e M is the unit vector of the maneuvering direction of the maneuvering target.

[0022] Furthermore, in step S103, the interceptor initial velocity vector V D，0 Expressed as

[0023]

[0024] In the formula, function T q The output of (·,·) is a rotation matrix, the first input is the rotation angle, and the second input is the rotation axis; V D，0 With plane C t The angle between plane C t For X DM,0 and V M,0 The plane formed; V D，0 In plane C t The projection and The angle of and H E,ψ The minimum and maximum values ​​of and They are The minimum and maximum values ​​of X DM,0 =X M,0 -X D,0 is the initial position vector of the maneuvering target relative to the interceptor, X M,0 is the initial position vector of the maneuvering target, V M,0 is the initial velocity vector of the maneuvering target, X D,0 is the interceptor initial position vector; To satisfy the interceptor velocity vector of zero control miss distance of 0; f unit The function of (·) is to solve the unit vector corresponding to the input vector.

[0025] Furthermore, the specific operation of step S2 includes the following steps:

[0026] S201: Generate N using proportional guidance law and interactive environment IL trajectory samples, and sample the trajectory samples at intervals of time steps to determine the imitation learning training set in, Is a tuple A collection of is the input of imitation learning, a PN =min{||a PN，0 ||,a max}·f unit(a PN，0 ) is the proportional guidance instruction, a PN，0 =NV D ×ω DM ;

[0027] In the formula, a max is the maximum overload of the interceptor, f unit The function of (·) is to solve the unit vector corresponding to the input vector, N is the navigation ratio; ω DM is the line of sight angular velocity vector, V D is the velocity vector of the interceptor;

[0028] S202: Design an imitation learning neural network model for training an intelligent terminal guidance model;

[0029] S203: Using the imitation learning neural network model, the intelligent terminal guidance model is trained by imitation learning.

[0030] Furthermore, the specific operation of step S202 includes the following steps:

[0031] S2021: Design an imitation learning neural network model with an output of In the formula, ‖a PN ‖∈[0,a max ];

[0032] S2022: Design an imitation learning neural network model with input In the formula, R DM is the distance vector between the maneuvering target and the interceptor, is the velocity vector of the maneuvering target and the interceptor approaching, ||ω DM || is the angular velocity of line of sight;

[0033] S2023: Design activation functions and loss functions for the imitation learning neural network model, where the hidden layer activation function uses the S-Sigmoid activation function and the output layer activation function is tanh.

[0034]

[0035] The loss function uses the MSE loss function.

[0036]

[0037] Where: M is the number of samples; N is the number of neural network output values; y i,j and are the predicted value and label value of the j-th output of the i-th sample respectively;

[0038] S2024: Design a network parameter update method for imitation learning neural network model

[0039]

[0040] Where: N GPU is the number of GPUs participating in learning; and is the Jacobian matrix submatrix required to be solved by the i-th GPU each time and the error vector subvector of the predicted value relative to the label value, ζ i For J i,j and E i,j number of rows; For i, under certain circumstances, J i,j The number of n =M·N is the number of all sample label values; the subscript l represents the iterative number; x∈R k×1 is a vector composed of network parameters, i.e., the weights and thresholds of each network layer, where k is the number of network parameters; I∈R k×k is the unit matrix; μ′ is the dynamically updated parameter. If updating the network parameters reduces the network prediction error, let μ′←μ′ / k μ′ , otherwise let μ′←μ′·k μ′ , calculate the network parameter update again, where k μ′ is a hyperparameter greater than 1;

[0041] The Jacobian matrix J is expressed as

[0042]

[0043] The error vector E is expressed as

[0044]

[0045] Furthermore, the specific operation of step S203 includes the following steps:

[0046] Calculate the loss function according to the loss function formula;

[0047] Compute the Jacobian matrix and the error matrix;

[0048] According to the network parameter update method, the parameters θ of the imitation learning neural network are updated. IL to update.

[0049] Furthermore, the specific operation of step S3 includes the following steps:

[0050] S301: Establishing Markov decision space model;

[0051] The state space of the Markov decision space model is

[0052]

[0053] The action space is

[0054] u t =[0,a max ]

[0055] The reward function is

[0056] r t =r s +r e

[0057] in

[0058] r s =k s exp[-(u t / g) 2 ]

[0059]

[0060] Where: r s and r e are process rewards and terminal rewards respectively; r s is the reward value that reflects the degree of energy consumption. The lower the energy consumption, the greater the positive reward. e is a reward that reflects the terminal miss distance, and a large positive reward is given when the interception is successful; k s is a hyperparameter greater than 0; R miss is the off-target amount; and is a hyperparameter related to the off-target amount, where Indicates the miss distance corresponding to the successful interception of the target; and is a hyperparameter related to the reward;

[0061] S302: Constructing a proximal strategy optimization and pruning network model;

[0062] S303: Utilize the proximal strategy to optimize the pruned network model and perform intensive training to solve the Markov decision space model in step S301.

[0063] Further, in step S302,

[0064] The optimization objective function of the proximal strategy optimization pruning network model is:

[0065]

[0066] Where: c VF and c s is a hyperparameter; θ and ω represent the parameters of the policy network and the value network respectively; L CLIP (θ) is the optimization target of the policy network; LVF (ω) is the optimization target of the value network; is the entropy of the strategy; E is the expected value;

[0067] The goal of the scaled value network is

[0068]

[0069] Where: Yes all The standard deviation of

[0070] Introduce the gradient clipping technique to solve the gradient g of θ θ and the gradient g of ω ω

[0071]

[0072] Where: g max is the gradient clipping value;

[0073] The network parameter update method is

[0074]

[0075] Where: α lr and β lr are the learning rates of the policy network and the value network respectively;

[0076] The learning rate update method for the policy network and the value network is

[0077]

[0078] Where: α lr,0 and β lr,0 are the initial learning rates of the policy network and the value network respectively; t e is the current training generation; N epoch is the maximum number of training generations.

[0079] Furthermore, the present invention also provides a device for designing a reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning, comprising an uncertain confrontation model module, an imitation learning module and a deep reinforcement learning module;

[0080] The uncertain confrontation model module builds a three-dimensional uncertain confrontation model between maneuvering targets and interceptors based on the interception collision triangle.

[0081] The imitation learning module uses the proportional guidance law as a strategy, generates a large number of trajectory samples in the interactive environment, establishes an intelligent terminal guidance model, and trains it using the supervised learning method to obtain an intelligent terminal guidance model that approximates the interception performance of proportional guidance.

[0082] The deep reinforcement learning module uses the reinforcement learning method, sets the reward function, and adopts the PPO algorithm to further train the intelligent terminal guidance model obtained by step imitation learning, and obtains a new guidance law that exceeds the proportional guidance interception performance;

[0083] The uncertain adversarial model module, imitation learning module and reinforcement learning module are implemented using the design method described above.

[0084] The beneficial effects of the present invention are:

[0085] 1. Based on the interception collision triangle, the present invention designs a method for solving the initial states of interceptors and targets in uncertain scenarios, and establishes a three-dimensional confrontation training scenario of interceptors and targets under non-ideal conditions; and compared with the prior art, the training scenario has a larger range of initial states of interceptors and maneuvering targets, which means that the intelligent agent needs to interact more with the environment to learn the interception strategy in this scenario in order to improve the generalization ability; the interceptor and the maneuvering target fly faster and the interception time is longer, which increases the length of the intelligent agent's trajectory, which means that it is more difficult for the intelligent agent to learn an effective interception strategy; and both the interceptor and the maneuvering target are established as three-dimensional simulation models, and the state space dimension is larger.

[0086] 2. The present invention proposes a hierarchical intelligent guidance framework of "terminal guidance law model based on imitation learning → terminal guidance law evolution model based on reinforcement learning". First, the PNG guidance law is mined through imitation learning to provide a good initial strategy for the reinforcement learning agent; then a Markov decision model of the interception agent is established, and a "soft" terminal reward including a "transition segment" is introduced. The PPO algorithm is used to solve the interception strategy, and techniques such as reward normalization, gradient clipping and learning rate decay are used to stabilize the training; the learning efficiency of the agent is effectively improved and the difficulty of reward function design is reduced, and the system has better performance than the traditional interception guidance law, higher interception probability and lower energy consumption; at the same time, the algorithm has low computational complexity, and the single running time on STM32F407 is about 0.32ms.

[0087] 3. The present invention designs a process reward that includes energy consumption and a "soft" terminal reward for the "transition section", and uses the PPO algorithm to solve the interception strategy, which is helpful to alleviate the convergence difficulty caused by sparse rewards to a certain extent. When the agent fails to intercept, the agent can still obtain rewards, and the rewards can provide directions for the agent to explore successful interception strategies, thereby guiding the agent to learn successful interception strategies, achieving high probability interception with low energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is a schematic diagram of the collision triangle model in the present invention.

[0089] Figure 2Schematic diagram of the training scenario in the present invention.

[0090] Figure 3 Schematic diagram of square wave maneuvering of a maneuvering target in the present invention.

[0091] Figure 4 This is a schematic diagram of the initial state error of the interceptor in the present invention.

[0092] Figure 5 It is a schematic diagram of calculating the terminal miss amount in the present invention.

[0093] Figure 6 For the present invention Schematic diagram of network structure.

[0094] Figure 7 For the present invention, the reward r e Schematic diagram of .

[0095] Figure 8 This is a schematic diagram of the PPO strategy network in the present invention.

[0096] Fig. 9 These are 500 sets of adversarial sample trajectories used for imitation learning in the simulation experiment of the present invention.

[0097] Fig.10 This is the curve of the imitation learning loss changing with the training algebra in the simulation experiment of the present invention.

[0098] Fig.11 It is a box plot of energy consumption and off-target amount of ILG and PNG in the simulation experiment of the present invention.

[0099] Fig.12 This is the IL-PPOG training curve in the simulation experiment of the present invention.

[0100] Fig.13 This is the curve of the off-target amount and energy changing with training in the simulation experiment of the present invention.

[0101] Fig.14 This is the curve of the miss amount of different algorithms changing with training in the simulation experiment of the present invention.

[0102] Fig.15 This is the curve of energy change of different algorithms with training in the simulation experiment of the present invention.

[0103] Fig.16 It is a box plot of energy consumption and off-target amount of IL-PPOG and PNG in the simulation experiment of the present invention.

[0104] Fig.17 The different H E,ψ and different Interception probability curve in this case.

[0105] Fig.18 The different H E,ψ and different Energy consumption in case.

[0106] Fig.19 These are overload instructions for different algorithms in the simulation experiment of the present invention.

[0107] Fig. 20 It is the target maneuvering acceleration instruction in the simulation experiment of the present invention.

[0108] Fig.21 It is the line of sight angular velocity of different algorithms in the simulation experiment of the present invention.

[0109] Fig. 22 It is the three-dimensional trajectory of the target and interceptor in the simulation experiment of the present invention.

[0110] Fig.23 The IL-PPOG in the simulation experiment of the present invention is different in ω s Miss target amount in case of .

[0111] Fig.24 The IL-PPOG in the simulation experiment of the present invention is different in ω s Energy consumption in case.

[0112] Fig.25 The different H E,ψ and different The interception probability in the case (the target adopts a sinusoidal maneuver ω s =π).

[0113] Fig.26 The different H E,ψ and different Energy consumption in this case (the target adopts sinusoidal maneuver ω s =π). DETAILED DESCRIPTION

[0114] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0115] Embodiment 1:

[0116] Embodiment 1 provides a method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning, which specifically includes the following steps:

[0117] Step S1: Based on the interception collision triangle, a three-dimensional uncertain confrontation model of the maneuvering target and the interceptor is established;

[0118] Specifically, S101: Design interactive environment for maneuvering target and interceptor confrontation training;

[0119] This paper mainly studies the problem of generating guidance instructions for interceptors to intercept maneuvering targets outside the atmosphere. For the convenience of research, the following assumptions are given:

[0120] Assumption 1: During the exoatmospheric flight phase, the motion of the interceptor and the maneuvering target around the center of mass and the attitude control problem are not considered, and a six-degree-of-freedom instantaneous equilibrium model is used in the simulation.

[0121] Assumption 2: The interceptor guidance system is considered to be an ideal system with no delay.

[0122] Assumption 3: Assume that the interceptor detection noise follows a normal distribution.

[0123] More specifically, the specific operations for designing a maneuvering target and interceptor confrontation training scenario include the following steps:

[0124] The design of the interactive environment should be based on the combat scenario and the performance parameters of both sides, and should include all possible flight states of the combatants as much as possible. The present invention mainly studies the terminal guidance problem of the interceptor. Therefore, the initial state of the interactive environment is the flight state of the combatants when the interceptor switches to terminal guidance, and the flight state is closely related to the standard trajectory of the interceptor. The standard trajectory of the interceptor refers to the standard trajectory corresponding to the elements of the interceptor launch, which is a trajectory that can hit the predetermined interception point. Therefore, the flight state of the combatants in the initial state of the interactive environment should fluctuate near the corresponding state of the collision triangle.

[0125] The collision triangle model is attached Figure 1 As shown in the figure, the basic condition for the zero-control miss amount to be 0 when the interceptor is in the terminal guidance state is given, that is, when the states of the interceptor and the target satisfy the collision triangle relationship, the interceptor can successfully intercept the target. Figure 1 In this paper, all variables are expressed in inertial coordinates. D is the interceptor's velocity vector; V M is the velocity vector of the maneuvering target; X DM =X M -X D is the position vector of the maneuvering target relative to the interceptor, where X M and X D Respectively represent the position vectors of the maneuvering target and the interceptor; V D,⊥ and V M,⊥ Respectively represent V D and V M Perpendicular to X DM The weight of and Respectively represent V D and V M Parallel to X DM The amount.

[0126] Assuming that the interceptor and the maneuvering target are not subject to external forces or are subject to the same external forces, the position vector of the maneuvering target relative to the interceptor at time t0 is X DM，0 , the position vector X of the maneuvering target relative to the interceptor at time t DM,t Decomposed into parallel and perpendicular to X DM，0 Quantity, available

[0127]

[0128] Where: and Respectively represent parallel and perpendicular to X DM，0 The amount.

[0129] According to the relative motion state of the interceptor and the maneuvering target, we can get and They satisfy the relationships of equations (2) and (3) respectively.

[0130]

[0131] In the formula, and They represent the external acceleration vectors of the interceptor and maneuvering target parallel to X DM，0 of weight and satisfy Δt DM Indicates the interception duration; a D,⊥ and a M,⊥ They represent the external acceleration vectors of the interceptor and maneuvering target perpendicular to X DM，0 The weight of D,⊥ =a M,⊥ .

[0132] Design training scenarios based on the collision triangle model as shown in the attached Figure 2 shown.

[0133] Figure 2 Describes the initial three-dimensional relative motion relationship between the interceptor and the maneuvering target, where the variables have the following meanings: Coordinate system x n O n z n is the north-east coordinate system established with the maneuvering target as the origin; Θ M,0 and σ M,0 are the local velocity inclination and velocity azimuth of the maneuvering target respectively; X MD,0 is the relative position vector of the interceptor relative to the maneuvering target; α MD,0 For X MD,0 With plane x n O n z n The angle between MD,0 For X MD,0 In plane xn O n z n Projection and O n x n During the training process, the initial motion states of the two are randomly generated within a certain range of initial conditions, including ||V M,0 ||、Θ M,0 , σ M,0 , α MD,0 , β MD,0 and ||V D，0 ||. Among them, V M,0 and V D，0 Represent the initial velocity vectors of the interceptor and maneuvering target respectively; || || represents the magnitude of the vector, for example, ||V D，0 || represents the magnitude of the initial velocity vector of the maneuvering target. It should be noted that ||X MD,0 || and the interceptor velocity vector V M,0 (In addition to size, it also includes direction) is not in the initial condition for the following reasons: ① Considering that the interceptor has a certain distance to detect the maneuvering target, ||X MD,0 || is a constant; ② The interceptor speed should be near the collision triangle, so the interactive environment only randomly samples the interceptor speed as the initial condition, and the speed components are generated by adding random quantities based on the speed components solved in the collision triangle.

[0134] Considering X MD,0 There are certain constraints on the flight direction of the maneuvering target, so the X MD,0 Two variables Δα that represent the degree of deviation from the maneuvering target’s flight direction MD,0 and Δβ MD,0 , and satisfy the following relationship:

[0135]

[0136] The initial state boundary range for training is set as shown in Table 1.

[0137] Table 1. Training scenario parameter boundaries

[0138]

[0139] S102: Establishing a maneuvering model of a maneuvering target;

[0140] Since the interceptor has a very short time to capture and intercept a maneuvering target, and is limited by the maneuverability of the maneuvering target, its maneuvering form should not be too complicated and it is impossible to perform overly complicated maneuvers in engineering. The single square wave maneuvering form is simple in form, easy to implement, and has good penetration performance. It is widely used in the research of aircraft maneuvering penetration. Its schematic diagram is shown in the attached figure. Figure 3 shown.

[0141] Figure 3 In, N m is the maneuvering overload of the maneuvering target, T m,0 、T m,1 and ΔT m They are the start time of maneuver, the end time of maneuver and the duration of maneuver respectively. Considering that the maneuvering target needs to achieve two goals: ① effectively avoid interception by the interceptor; ② improve the hit accuracy. However, there is a certain contradiction between these two indicators, that is, the greater the degree of maneuverability in the early stage, the better the avoidance effect, but it is not conducive to the correction of the landing point deviation in the later stage. It can be seen that, considering the relationship between the two goals comprehensively, the degree of maneuver avoidance should be limited during the training process. Therefore, according to the performance parameters of the maneuvering target orbit control engine, the maneuvering duration is determined to be a constant. In order to increase the richness of interceptor interception target samples and improve the generalization ability of the intelligent agent, the maneuvering timing and direction of the maneuvering target are randomly generated, and then the maneuvering model of the maneuvering target is obtained as follows:

[0142]

[0143] in, a M is the maneuvering acceleration vector of the maneuvering target, g represents the gravitational acceleration; and are the minimum and maximum values ​​of the maneuverable start time of the maneuverable target, respectively; e M is the unit vector of the maneuvering direction of the maneuvering target.

[0144] Since the target exerts orbital control thrust in the vertical direction of the interceptor and the maneuvering target, it has the greatest impact on the interceptor's interception effect. At the same time, considering that the target orbital control engine can generate thrust in a total of four directions along the interceptor's lateral normal, it is assumed that e M perpendicular to the line connecting the interceptor and the maneuvering target, and e M There are only 4 directions in space. During training, one of the above 4 directions is randomly selected as e M Point.

[0145] S103: Initializing the interactive environment;

[0146] The initialization of the interactive environment mainly completes the calculation of the initial state of the interceptor and the maneuvering target. M,0 , speed V M,0 and the interceptor initial position X D,0 It can be obtained by random sampling within the flight state range. Therefore, the key to initializing the interactive environment lies in solving the initial speed of the interceptor. The interceptor speed obtained by random sampling is V D,0 (V D,0 is the interceptor speed, V D，0is the interceptor velocity vector), then according to the collision triangle model, the interceptor velocity vector that satisfies the zero control miss amount of 0 can be obtained. for

[0147]

[0148] In the formula, X DM,0 =X M,0 -X D,0 is the initial position vector of the maneuvering target relative to the interceptor. It should be noted here that It is the interceptor velocity vector calculated without error and basically conforms to the zero-control interception.

[0149] Since the interceptor and the maneuvering target are at a certain distance in space at the initial moment, the interceptor and the maneuvering target are slightly different in gravity, and the speed obtained by the above formula cannot guarantee that the zero-control miss amount is 0. However, the difference in gravity between the two is very small, so the zero-control miss amount obtained by the above method is close to 0. Considering that the main purpose of the above formula is to constrain the initial speed of the interceptor to be closer to reality, the influence of the gravity difference between the two on the initialization of the environment can be ignored.

[0150] Due to the errors in the calculation of the predetermined interception point, the initial guidance, the mid-range guidance, etc., the interceptor's flight state when entering the terminal guidance is near the collision triangle, as shown in the attached figure. Figure 4 For the convenience of description, plane C is defined as t For X DM,0 and V M,0 The plane spanned, the error angle V D，0 With plane C t The angle of error V D，0 In plane C t The projection and The angle of and H E,ψ The minimum and maximum values ​​of and They are The minimum and maximum values ​​of the interceptor are obtained as follows: D，0 for

[0151]

[0152] Where: Function T q The output of (·,·) is the rotation matrix. The first input is the rotation angle, and the second input is the rotation axis. This function can be obtained through quaternion. unit The function of (·) is to solve the unit vector corresponding to the input vector. V′ D，0 is a process variable, Around f unit (X DM,0 ×V M,0 ) Axis rotation Then we get V′ D，0 , V′ D，0 Then go around f unit (X DM，0 ×V M，0 )×f unit (V′ D，0 ) Axis rotation H E,ψ Get V D，0 .

[0153] S104: Establishing a terminal miss distance calculation model;

[0154] Since the terminal miss distance is an important criterion for measuring the success of interception, one of the key issues of interceptor interception is the accurate solution of the terminal miss distance. Usually, due to the limitation of simulation step size, the distance between the interceptor and the maneuvering target at the end of the simulation cannot be used as the terminal miss distance, that is, this distance is not the minimum distance between the interceptor and the maneuvering target. The method of reducing the step size to solve the more accurate terminal miss distance has the problem of increasing the simulation time cost.

[0155] In order to solve the terminal miss distance in the present invention, the position D of the last two steps of the interceptor integration is used. f and D′ f and the position M of the maneuvering target in the last two steps of integration f and M′ f To solve, as shown in the attached Figure 5 shown. Figure 5 middle, and They represent the interceptor and the maneuvering target at the same time t * location.

[0156] For the convenience of description, let s A,B represents the vector from point A to point B. In the integration interval Δt int The speeds of the interceptor and the maneuvering target are approximately constant. Let the interceptor be at D′ f The time is t′ f , Then the problem of terminal miss distance can be converted into solving

[0157]

[0158] because

[0159]

[0160] Substitute equation (9) into equation (8) and let Available

[0161]

[0162] According to formula (10), the terminal miss distance can be obtained as

[0163]

[0164] Further, step S2: using the proportional guidance law as a strategy, using the interactive environment to generate a large number of trajectory samples, establishing an intelligent terminal guidance model, and training it using a supervised learning method to obtain an intelligent terminal guidance model that approximates the proportional guidance interception performance;

[0165] Specifically, step S201: using the proportional guidance law and the interactive environment to generate N IL trajectory samples, and sample the trajectory samples at intervals of time steps to determine the imitation learning training set

[0166] More specifically, the proportional guidance law is based on the assumption that the target is non-maneuverable. The vector form of its three-dimensional proportional guidance can be expressed as

[0167] a PN，0 =NV D ×ω DM (12)

[0168] Where: a PN，0 is the acceleration command of the proportional guidance law; N is the navigation ratio; ω DM is the line-of-sight angular velocity vector. In the actual guidance process, this vector is measured by the seeker in real time. In the simulation process, it can be calculated by formula (13):

[0169]

[0170] Where: V DM =V M -V D is the velocity vector of the target relative to the interceptor.

[0171] Considering that there is a maximum limit on the interceptor overload, the proportional guidance command is corrected to

[0172] a PN =min{||a PN，0 ||, a max}·f unit (a PN，0 ) (14)

[0173] Where: a max is the maximum overload of the interceptor, f unit The function of (·) is to solve the unit vector corresponding to the input vector.

[0174] According to the above proportional guidance law and interactive environment, N IL Trajectory samples are sampled at intervals of time steps to obtain the imitation learning training set in, Is a tuple A collection of is the input for imitation learning.

[0175] The main purpose of the imitation learning training of the intelligent terminal guidance model is to provide initial parameters for the reinforcement learning guidance law to reduce the learning difficulty and improve the learning performance. Therefore, the intelligent terminal guidance model uses the proportional guidance instruction as a label to learn the interception strategy of the proportional guidance law, which can be formally expressed as

[0176]

[0177] In the formula, f IL express To proportional guidance command a PN The imitation learning module needs to learn the mapping.

[0178] S202: Design an imitation learning neural network model for training an intelligent terminal guidance model;

[0179] The core of the imitation learning module is composed of a neural network f NN The guidance instruction generator composed of the main body needs to reasonably design the neural network input and output, activation function and network structure to improve the performance of the imitation learning module.

[0180] More specifically, S2021: Design an imitation learning neural network model output;

[0181] Considering that the line-of-sight angular velocity needs to be suppressed in the terminal guidance interception, it is beneficial to save the interceptor fuel by setting the acceleration along the direction where the line-of-sight angular velocity changes the most. Let the unit vector in this direction be e a,unit , then combined with formula (13), we can know

[0182] e a,unit = arg max||X DM ×e a,unit || (16) According to the cross multiplication rule, we can know

[0183] e a,unit =f unit (X DM ×ω DM ) (17)

[0184] Since the interception process takes place outside the atmosphere, the influence of the atmosphere on the interceptor's attitude can be ignored, which ensures that the interceptor always points to the target direction, thus generating an interception along the e a,unit acceleration.

[0185] In order to improve the real-time performance of the neural network and reduce the difficulty of learning the neural network, when designing the output of the neural network, the output dimension should be reduced as much as possible, and variables that are easy to learn should be selected as the output. a,unit direction, so the output of the neural network can be designed as the required acceleration, and formula (15) is simplified to

[0186]

[0187] In the formula, ||a PN ||∈[0,a max ].

[0188] S2022: Design the input of the imitation learning neural network model;

[0189] Input The selection of will directly affect the difficulty of model learning and needs to reflect the key information of the interception process. The input quantity designed in this invention is

[0190]

[0191] In the formula, R DM is the relative distance vector between the maneuvering target and the interceptor, is the relative velocity vector between the maneuvering target and the interceptor, ||ω DM || is the angular velocity of line of sight.

[0192] In order to overcome the problem of low training efficiency caused by the inconsistency of different dimensional scales of input and output vectors, the min-max normalization method is used. After normalization, we get

[0193] S2023: Design activation and loss functions for imitation learning neural network models;

[0194] In addition, in order to reduce the calculation complexity of the activation function, the hidden layer activation function adopts the S-Sigmoid activation function, which is a prior art and will not be described in detail in the present invention.

[0195] Considering that the imitation learning neural network mainly completes the mapping from input to guidance instructions, and the guidance instructions have a maximum value constraint, the activation function of the output layer of the imitation learning neural network is designed to be tanh, then

[0196]

[0197] In the present invention It is a fully connected neural network with two hidden layers. Its network structure is shown in the attached figure. Figure 6 As shown, Represents a network node, The present invention selects 6 nodes in hidden layer 1 and 5 nodes in hidden layer 2.

[0198] The model learning in this invention adopts MSE loss function

[0199]

[0200] Where: M is the number of samples; N is the number of neural network output values; y i,j and are the predicted value and label value of the j-th output of the ith sample, respectively.

[0201] S2024: Design a network parameter update method for imitating learning neural network models;

[0202] Since the simulation samples are obtained by sampling the flight trajectory of the PNG guidance law, the number of samples is huge. In this invention, the improved data parallel LM optimizer (Improved Data Parallel Levenberg-Marquardt, IDP-LM) is used as the training algorithm, and its network parameter update formula is:

[0203]

[0204] Where: N GPU is the number of GPUs participating in learning; and is the Jacobian matrix submatrix required to be solved by the i-th GPU each time and the error vector subvector of the predicted value relative to the label value. Its relationship with the Jacobian matrix J and the error vector E is shown in (23) and (24). i For J i,j and E i,j number of rows; For i, under certain circumstances, J i,j The number of n =M·N is the number of all sample label values; the subscript l represents the iterative number; x∈R k×1 is a vector composed of network parameters, i.e., the weights and thresholds of each network layer, where k is the number of network parameters; I∈R k×k is the unit matrix; μ′ is the dynamically updated parameter. If updating the network parameters reduces the network prediction error, let μ′←μ′ / k μ′ , otherwise let μ′←μ′·kμ′ , calculate the network parameter update again, where k μ′ is a hyperparameter greater than 1.

[0205]

[0206]

[0207] S203: using an imitation learning neural network model; performing imitation learning training on the intelligent terminal guidance model;

[0208] Specifically, the process of imitation learning training includes:

[0209] Calculate the loss function according to formula (21);

[0210] According to formula (23) and formula (24), the Jacobian matrix and error matrix are calculated respectively;

[0211] According to formula (22), the parameters θ of the imitation learning neural network are IL to update.

[0212] Furthermore, step S3: using the reinforcement learning method, setting the reward function, and using the PPO algorithm to further train the intelligent terminal guidance model learned in step S2, to obtain a new guidance law that exceeds the proportional guidance interception performance.

[0213] Specifically, S301: establishing a Markov decision space model;

[0214] The key to the application of reinforcement learning in the field of interception guidance is how to design a reasonable Markov decision process, including state space, action space and reward function, to improve learning efficiency and interception guidance law performance.

[0215] Based on imitation learning, the reinforcement learning agent further explores effective strategies for intercepting mobile targets. Therefore, the state space of the reinforcement learning agent is consistent with the input of imitation learning, that is, formula (19), and the state space is obtained as

[0216]

[0217] The action output of the reinforcement learning agent is the guidance instruction. Combining formula (18), the action space can be obtained as follows:

[0218] u t =[0,a max ] (26)

[0219] The scenario of intercepting maneuvering targets studied in this paper contains two optimization objectives, one is to maximize the interception probability, and the other is to minimize energy consumption. At the same time, since the interception strategy of the agent can only be evaluated at the end of the simulation, there is a sparse reward problem. In order to alleviate the problem of slow convergence speed and poor learning effect of the agent caused by sparse rewards, and to achieve low energy consumption and high interception probability, a reward function containing two parts is designed as follows:

[0220] r t =r s +r e (27)

[0221] in

[0222] r s =k s exp[-(u t / g) 2 ] (28)

[0223]

[0224] Where: r s and r e are process rewards and terminal rewards respectively; r s is the reward value that reflects the degree of energy consumption. The lower the energy consumption, the greater the positive reward. e is a reward that reflects the terminal miss distance, and a large positive reward is given when the interception is successful; k s is a hyperparameter greater than 0; R miss is the off-target amount; and is a hyperparameter related to the off-target amount, where Indicates the miss distance corresponding to the successful interception of the target; and is a hyperparameter related to reward; r e The schematic diagram is attached Figure 7 shown.

[0225] By the attached Figure 7 It can be seen that r e It is mainly divided into three sections: ① When , it means the interception fails and the agent reward is 0; ② When It means that although the interception failed, the miss distance is small, so the agent can still obtain a certain positive reward; ③ When , indicating that the interception is successful, and the agent obtains a large positive reward. Different from the "hard" terminal reward, that is, the agent can only obtain a large positive reward when the interception is successful, the present invention proposes a "soft" terminal reward with a "transition segment", which is helpful to alleviate the convergence difficulty caused by sparse rewards to a certain extent. In other words, when the agent fails to intercept, the agent can still obtain rewards, and the reward can provide direction for the agent to explore the strategy of successful interception, thereby guiding the agent to learn the strategy of successful interception.

[0226] S302: Constructing a proximal strategy optimization and pruning network model;

[0227] Specifically, PPO-CLIP contains a policy network and a value network, and its optimization goal is to maximize the function given by formula (30)

[0228]

[0229] Where: c VF and c s is a hyperparameter; θ and ω represent the parameters of the policy network and the value network respectively; L CLIP (θ) is the optimization target of the policy network, which can be calculated by formula (31); L VF (ω) is the optimization target of the value network, which can be calculated by formula (33); is the entropy of the strategy, reflecting the uncertainty of the strategy. E is the expected value.

[0230]

[0231] Where: π θ and π θ,old Represent the new strategy and the old strategy respectively; p(θ) represents the probability ratio of the new and old strategies, which can be expressed as p(θ) = π θ (a|s) / π θ,old (a|s); clip(,,) is the clipping function, which can be expressed as

[0232]

[0233] Where: ε is a very small hyperparameter used to ensure that the gap between the new and old strategies is not large.

[0234]

[0235] In the formula, is the target of the value network, and the calculation formula is

[0236]

[0237] Reward scaling is introduced to avoid the negative impact of too large or too small rewards on the learning of the value network. The target of the scaled value network is expressed as

[0238]

[0239] Where: Yes all The standard deviation of .

[0240] At the same time, in order to improve the stability of the training process, the gradient clipping technique is introduced to solve the gradient g of θ θ and the gradient g of ω ω

[0241]

[0242] Where: g max is the gradient clipping value.

[0243] To update the parameters of the policy network and the value network, the Adam optimizer is used to maximize the objective function given by (30), namely

[0244]

[0245] Where: α lr and β lr are the learning rates of the policy network and the value network, respectively.

[0246] Furthermore, in order to enhance the stability in the later stage of training, the learning rate decay technique (Learning Rate Decay) is introduced, namely

[0247]

[0248] Where: α lr,0 and β lr,0 are the initial learning rates of the policy network and the value network respectively; t e is the current training generation; N epoch is the maximum number of training generations.

[0249] Furthermore, the strategy network of PPO is based on the imitation learning agent network, as shown in the attached diagram. Figure 8 As shown. It can be seen that the policy network has two outputs, namely the action mean μ and the action standard deviation a std , where the network with action mean μ is given by Inherited. During the training phase, the agent action a t According to the distribution After sampling, it is obtained by tanh activation function and action scaling.

[0250] The number of network nodes and activation function of the PPO strategy network in the present invention are shown in Table 2.

[0251] Table 2 PPO network structure and parameters

[0252]

[0253] S303: Using the PPO-CLIP model in step S302, the Markov decision space model in step S301 is subjected to intensive training and solution.

[0254] More specifically, the process of intensive training includes the following steps:

[0255] Based on the imitation learning network in step S2, update the parameters of the policy network, θ←θ IL ,θ old ←θ;

[0256] Use rollouts to Generate N for the interaction between the behavior policy network and the environment rollout Tracks, stored in the experience pool in, is a tuple (s t ,a t ,r t ,s t+1 )

[0257] According to the formula

[0258]

[0259] Calculate the advantage function and update The tuple in is

[0260] Calculate the target value function according to formula (34): and update The tuple in is

[0261] From the experience pool Random sampling in batch transfer data

[0262] Calculate the PPO objective function

[0263]

[0264] Calculate the clipped gradient according to formula (36) gθ and g ω ;

[0265] Update network parameters according to formula (37);

[0266] Clear Update the behavior policy network parameters θ old ←θ;

[0267] The learning rate is updated according to formula (38), and finally the guidance law after reinforcement learning is obtained. In summary, the learning process of the intelligent agent in the present invention is shown in Table 3 below.

[0268] Table 3 Agent learning process

[0269]

[0270] Simulation experiment:

[0271] In order to verify the effectiveness of the guidance law IL-PPOG algorithm established by the present invention, the parameters given in Table 1 are used as scene parameters, an interactive environment is established, and a simulation experiment is designed. This simulation experiment first gives the hyperparameter design, then verifies the imitation learning performance, further analyzes the reinforcement learning guidance process, and finally tests the algorithm performance in an unknown scenario.

[0272] The simulation of the present invention is carried out under the support of software tools such as Python 3.7, Pytorch 1.5, Visual Studio 2015 and SWIG 4.0.1. In order to improve the running speed of the interactive environment, an interactive environment for interceptor interception of maneuvering targets is established based on C++ language under Visual Studio 2015 environment. The fourth-order Longo Kutta and Adams integrators are used (the former provides the integration results of the first 4 steps for the latter), the integration step is 0.02s, and SWIG 4.0.1 is used to establish a data interface with Python. In the imitation learning stage, the proportional guidance law navigation ratio adopted is 4.0, and in the reinforcement learning stage, the decision step is 0.1s. The performance of the simulation platform is as follows: the CPU is Intel Core 13900K, the memory is 64G, and the graphics card is NVIDIA 3080 (10G).

[0273] Since there is uncertainty in the observation and action quantities during the interception process, observation noise and action noise are added, that is,

[0274]

[0275] Where: 1 3×1 Represents a vector whose elements are all 1; in, and The observations R DM , and ||ω DM ||Standard deviation of noise; σ a is the standard deviation of the motion noise.

[0276] The IL-PPOG algorithm established in this simulation experiment contains many hyperparameters, mainly involving network training, reward function, environment setting, etc. The specific parameters are shown in Table 4.

[0277] Table 4. Hyperparameter design of simulation experiment

[0278]

[0279] Imitation learning simulation experiment:

[0280] In order to generate imitation learning training samples, PNG output instructions are used as actions to interact with the environment and generate 10,000 interception trajectories. The flight time of each trajectory is about 15 seconds. Fig. 9 500 sets of three-dimensional trajectories of adversarial samples are given. The interception trajectories are sampled at 0.1s intervals, and then 1.5×10 6 The IDP-LM algorithm is used to train f IL After 1000 generations of training, the training curve is as shown in the attached figure. Fig.10 As shown, the y-axis is in logarithmic coordinate form.

[0281] By the attached Fig.10 It can be seen that as the number of training generations increases, the training curve gradually converges, indicating that the imitation learning agent can learn the instructions output by PNG. IL The interception performance of the guidance command was tested. 1000 interception simulations were carried out in an interactive environment, and the miss distance and energy consumption were obtained as shown in the attached figure. Fig.11 At the same time, Fig.11 The miss distance and energy consumption with PNG as the instruction are given. The definition of energy consumption performance index is as follows

[0282]

[0283] Where, t f The time corresponding to the end of the interception flight simulation.

[0284] By the attached Fig.11 It can be seen that based on f IL The guidance instructions based on PNG and the guidance instructions based on PNG are similar in terms of miss distance and energy consumption, indicating that f IL The PNG instructions are well learned and can provide a good initial strategy for reinforcement learning agents.

[0285] Reinforcement learning training simulation experiment:

[0286] In the reinforcement learning agent training phase, the training process is based on f ILThe reinforcement learning agent is trained as the initial strategy. Assuming the maximum number of training generations is 300, the number of rollouts is 50, the experience pool size is 7500, and the network parameters are updated 1000 times in a small batch (batch size is 256) each time. In order to show the changing trend of the agent strategy performance with the learning process, the agent is trained with 5 different random seeds, and the network parameters are initialized using orthogonal initialization. Each training interval is 10 generations, and 1000 Monte Carlo intercept simulations are carried out with the agent strategy as the instruction (each simulation has a random initial state), and the trajectory cumulative reward average curve is obtained as shown in the attached figure. Fig.12 The average values ​​of off-target amount and energy consumption are shown in the attached Fig.13 It should be noted that the attached Fig.12 and attached Fig.13 The solid line indicates the mean and the shading indicates the standard deviation.

[0287] In order to verify the advantages of the IL-PPOG algorithm over other studies, it is compared with two reinforcement learning guidance algorithms: the reinforcement learning guidance algorithm with PNG navigation ratio as the action, denoted as LC-PPOG; ② The reinforcement learning guidance algorithm that uses reward shaping to guide the learning of the agent, denoted as Classic-PPOG. Among them, except for the process reward design and not based on IL strategy, the other reinforcement learning elements of the comparison algorithm ② are the same as IL-PPOG. Establish reward shaping as

[0288] r s ′=k s exp[-(u t / g) 2 ]+k q exp[-(||ω DM || / ||ω DM || max ) 2 ] (43)

[0289] Where: k q =0.1 represents the weight coefficient of the line of sight angular velocity; ||ω DM || max =1×10 -2 rad·s -1 is the minimum line of sight angular velocity.

[0290] Attached Fig.14 and attached Fig.15The curves of the variation of the average interception miss amount and energy consumption with the training generations in 1000 tests of the agent every 10 generations during the training process of IL-PPOG, LC-PPOG and Classic-PPOG are given. It can be seen that IL-PPOG has a smaller average interception miss amount and energy consumption than LC-PPOG and Classic-PPOG at the beginning of training; with the increase of training generations, the average miss amount of IL-PPOG gradually decreases, and the energy consumption gradually decreases. The strategy learned by the agent achieves the lowest energy consumption and a higher interception probability. Compared with LC-PPOG and Classic-PPOG, it can be seen that IL-PPOG has lower energy consumption and higher interception probability in general:

[0291] (1) The interception performance of LC-PPOG in the early stage of training is similar to that of IL-PPOG. The average miss amount at convergence is higher than that of IL-PPOG, and the average energy consumption is slightly lower than that of IL-PPOG. Considering that the miss amount is the main factor affecting the interception performance, it can be seen that the interception performance of IL-PPOG is slightly higher than that of LC-PPOG.

[0292] (2) Classic-PPOG requires careful design s and k q There are two hyperparameters to guide the agent to explore a successful interception strategy, while balancing the importance of energy consumption and interception probability. The hyperparameter adjustment faces great difficulties. Fig.16 and Fig.17 It can be seen that although the average interception miss amount of Classic-PPOG when it converges is smaller than that of IL-PPOG, its energy consumption is much greater than that of IL-PPOG, which is not conducive to improving the interception performance under limited energy constraints.

[0293] In order to test the interception performance of IL-PPOG, in an interactive environment with various errors and noises established with the parameters given in Table 1, the target adopts square wave maneuvers and 10,000 Monte Carlo simulations are carried out. The interception miss distance and energy consumption results of different algorithms are shown in the attached figure. Fig.16 It can be seen that the 3σ value and average value of the miss distance of IL-PPOG in the known scenario are 6.11m and 0.34068m, respectively, which are smaller than the miss distances of PNG and LC-PPOG. At the same time, the 3σ value and average value of the energy consumption of IL-PPOG are 344.1658m·s -1 and 189.6307 m·s -1 , which is less than the corresponding energy consumption of PNG and Classic-PPOG. This shows that IL-PPOG has better interception performance for maneuvering targets and better robustness in external interference environments.

[0294] To further test different H E,ψ and different The interception performance of different algorithms under different conditions was evaluated. 10,000 Monte Carlo simulations were performed for different conditions. The interception probability (when the miss distance is less than 1.0 m, the interception is considered successful) and energy consumption were calculated as shown in the attached figure. Fig.17 and attached Fig.18 As shown. It can be seen that IL-PPOG has higher interception performance in different situations, and the advantage of IL-PPOG is more obvious in the case of large deviation, which further shows that the algorithm has good robustness. At the same time, considering that the large deviation represents a large error in the middle and end handover, IL-PPOG can broaden the accuracy constraints of the middle and end handover.

[0295] Assuming the target maneuver strategy is to maneuver upwards at a distance of 40 km from the interceptor, and In the case of 10 interception processes, statistics are collected on interceptor missile interception overload instructions, target maneuver instruction curves, line of sight angular velocity, interceptor missile and target flight trajectories as shown in the attached figure. Figure 19-22 As shown. It can be seen that due to H E,ψ and IL-PPOG gives a large interception overload command at the early stage of terminal guidance, quickly reduces the line of sight angular velocity, and gives a large interception overload again after the target maneuvers, thereby achieving effective interception. Fig.21 It can be seen that during the interception process, the line-of-sight angular velocity corresponding to the PNG and LC-PPOG guidance laws is greater than the line-of-sight angular velocity corresponding to the IL-PPOG guidance law; Fig.19 It can be seen that when approaching the target, the PNG guidance law reaches the maximum interception overload earlier than the IL-PPOG guidance law, which leads to the failure of PNG to intercept, while IL-PPOG can intercept successfully, further indicating that IL-PPOG has better interception performance. Fig.19 and Fig.21 It can be seen that although the Classic-PPOG has the smallest line of sight angular velocity and a smaller interception miss margin, its overload instruction is always close to saturation, consuming a lot of energy. Therefore, under the condition of limited energy, its interception effect will decrease.

[0296] Algorithm performance test in unknown scenarios:

[0297] In order to verify the generalization ability of IL-PPOG for unknown interception environments, an unknown scenario was established as follows: the target maneuver form was changed from square wave maneuver form to sinusoidal maneuver form, as shown in equation (44), and other scenario parameters remained unchanged. In the target sinusoidal maneuver scenario, the initial states of the target and interceptor were randomly generated, and observation noise and action noise were added. 10,000 Monte Carlo simulations were performed, and the miss distance and energy consumption were statistically calculated as shown in the attached figure. Figure 23-Figure 26 shown.

[0298]

[0299] Where: s is the sinusoidal motor frequency.

[0300] Attached Fig.23 and Fig.24 Given ω s The interception miss distance and energy consumption of IL-PPOG under different sinusoidal maneuvering scenarios. It can be seen that ω s =π, the interception miss amount corresponding to the other cases is larger, indicating that the target adopts the s =π sinusoidal maneuvering mode can increase the difficulty of interception by the interceptor. s =π, the 3σ value and average value of the off-target amount of IL-PPOG are 1.62m and 0.047m, respectively, and the 3σ value and average value of the energy consumption are 322.83m·s -1 and 177.91 m·s -1 , indicating that the interception miss is small and the energy consumption is low. Combined with the reinforcement learning training simulation experiment, it can be seen that IL-PPOG has good interception performance for targets with different maneuvering modes and has good generalization ability. E,ψ and different In this scenario, let the sinusoidal maneuver frequency ω s =π, 10,000 Monte Carlo simulations were carried out for each case, and the interception probability and energy consumption of different algorithms are shown in the attached figure. Fig.25 and Fig.26 As shown in the figure, it can be seen that IL-PPOG has better interception performance than LC-PPOG, Classic-PPOG and PNG. It can be seen that IL-PPOG has the ability to adapt to external interference and unknown maneuvering mode scenarios, and has certain engineering application potential.

[0301] In order to analyze the computational complexity of the method in the present invention, the STM32F407 single-chip microcomputer was used as the experimental platform, and the double data type was used to carry out the algorithm time consumption experiment. When the guidance law in the present invention is applied, it is only necessary to run the strategy network. The strategy network time consumption obtained by the single-chip microcomputer timing function is about 0.32ms, which shows that the real-time performance is good and can reduce the demand for on-board computing power.

[0302] Embodiment 2:

[0303] Embodiment 2 provides a device for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning, comprising an uncertain confrontation model module, an imitation learning module and a deep reinforcement learning module;

[0304] The uncertain confrontation model module builds a three-dimensional uncertain confrontation model between maneuvering targets and interceptors based on the interception collision triangle.

[0305] The imitation learning module uses the proportional guidance law as a strategy, generates a large number of trajectory samples in the interactive environment, establishes an intelligent terminal guidance model, and trains it using the supervised learning method to obtain an intelligent terminal guidance model that approximates the interception performance of proportional guidance.

[0306] The deep reinforcement learning module uses the reinforcement learning method, sets the reward function, and adopts the PPO algorithm to further train the intelligent terminal guidance model obtained by step imitation learning, and obtains a new guidance law that exceeds the proportional guidance interception performance;

[0307] The uncertain adversarial model module, the imitation learning module and the reinforcement learning module are implemented using the design method described in Example 1.

[0308] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A reinforcement learning guidance law design method for intercepting maneuvering targets based on imitation learning, characterized in that: The following steps are included: S1: Based on the interception collision triangle, a three-dimensional uncertain confrontation model between maneuvering targets and interceptors is established; S2: Taking the proportional guidance law as the strategy, a large number of trajectory samples are generated in the interactive environment, and an intelligent terminal guidance model is established. The model is trained using the supervised learning method to obtain an intelligent terminal guidance model that approaches the interception performance of the proportional guidance. S3: Using the reinforcement learning method, the reward function is set, and the PPO algorithm is used to further train the intelligent terminal guidance model learned in step S2 to obtain a new guidance law that exceeds the proportional guidance interception performance.

2. The method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning according to claim 1 is characterized in that: The specific operation of step S1 includes the following steps: S101: Design an interactive environment for maneuvering target and interceptor confrontation training based on the collision triangle model; S102: Establishing a maneuver model of a maneuvering target; S103: Initializing the interactive environment designed in step S101 according to the maneuvering model of the maneuvering target, and solving the initial velocity vector of the interceptor; S104: Establishing a terminal miss distance calculation model.

3. The method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning according to claim 2 is characterized in that: The maneuvering model of the maneuvering target described in step S102 is expressed as Where N m Maneuvering overload for maneuvering targets, a M is the maneuvering acceleration vector of the maneuvering target, g represents the gravitational acceleration; and are the minimum and maximum values ​​of the maneuverable start time of the maneuverable target, respectively; e M is the unit vector of the maneuvering direction of the maneuvering target.

4. The method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning according to claim 3 is characterized in that: In step S103, the interceptor initial velocity vector V D，0 Expressed as In the formula, function T q The output of (·,·) is a rotation matrix, the first input is the rotation angle, and the second input is the rotation axis; V D，0 With plane C t The angle between plane C t For X DM,0 and V M,0 The plane formed; V D，0 In plane C t The projection and The angle of and H E,ψ The minimum and maximum values ​​of and They are The minimum and maximum values ​​of X DM,0 =X M,0 -X D,0 is the initial position vector of the maneuvering target relative to the interceptor, X M,0 is the initial position vector of the maneuvering target, V M,0 is the initial velocity vector of the maneuvering target, X D,0 is the interceptor initial position vector; To satisfy the interceptor velocity vector of zero control miss distance of 0; f unit The function of (·) is to solve the unit vector corresponding to the input vector.

5. The method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning according to claim 4 is characterized in that: The specific operation of step S2 includes the following steps: S201: Generate N using proportional guidance law and interactive environment IL trajectory samples, and sample the trajectory samples at intervals of time steps to determine the imitation learning training set in, is a tuple A collection of is the input of imitation learning, a PN =min{||a PN，0 ||,a max }·f unit (a PN，0 ) is the proportional guidance instruction, a PN，0 =NV D ×ω DM ; In the formula, a max is the maximum overload of the interceptor, f unit The function of (·) is to solve the unit vector corresponding to the input vector, N is the navigation ratio; ω DM is the line of sight angular velocity vector, V D is the interceptor’s velocity vector; S202: Design an imitation learning neural network model for training an intelligent terminal guidance model; S203: Using the imitation learning neural network model, the intelligent terminal guidance model is trained by imitation learning.

6. The method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning according to claim 5 is characterized in that: The specific operation of step S202 includes the following steps: S2021: Design an imitation learning neural network model with an output of In the formula, S2022: Design an imitation learning neural network model with input In the formula, R DM is the distance vector between the maneuvering target and the interceptor, is the velocity vector of the maneuvering target and the interceptor approaching, ||ω DM || is the angular velocity of line of sight; S2023: Design activation functions and loss functions for the imitation learning neural network model, where the hidden layer activation function uses the S-Sigmoid activation function and the output layer activation function is tanh. The loss function uses the MSE loss function. Where: M is the number of samples; N is the number of neural network output values; y i,j and are the predicted value and label value of the j-th output of the i-th sample respectively; S2024: Design a network parameter update method for imitation learning neural network model Where: N GPU is the number of GPUs participating in learning; and is the Jacobian matrix submatrix required to be solved by the i-th GPU each time and the error vector subvector of the predicted value relative to the label value, ζ i For J i,j and E i,j The number of rows; θ i For i, under certain circumstances, J i,j The number of n =M·N is the number of all sample label values; the subscript l represents the iterative number; x∈R k×1 is a vector composed of network parameters, i.e., the weights and thresholds of each network layer, where k is the number of network parameters; I∈R k×k is the unit matrix; μ′ is the dynamically updated parameter. If updating the network parameters reduces the network prediction error, let μ′←μ′ / k μ′ , otherwise let μ′←μ′·k μ′ , calculate the network parameter update again, where k μ′ is a hyperparameter greater than 1; The Jacobian matrix J is expressed as The error vector E is expressed as 7. The method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning according to claim 6 is characterized in that: The specific operation of step S203 includes the following steps: Calculate the loss function according to the loss function formula; Compute the Jacobian matrix and the error matrix; According to the network parameter update method, the parameters θ of the imitation learning neural network are updated. IL to update.

8. The method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning according to claim 7 is characterized in that: The specific operation of step S3 includes the following steps: S301: Establishing Markov decision space model; The state space of the Markov decision space model is The action space is u t =[0,a max ] The reward function is r t ＝r s +r e in r s =k s exp[-(u t / g) 2 ] Where: r s and r e are process rewards and terminal rewards respectively; r s is the reward value that reflects the degree of energy consumption. The lower the energy consumption, the greater the positive reward. e is a reward that reflects the terminal miss distance, and a large positive reward is given when the interception is successful; k s is a hyperparameter greater than 0; R miss is the off-target amount; and is a hyperparameter related to the off-target amount, where Indicates the miss distance corresponding to the successful interception of the target; and is a hyperparameter related to the reward; S302: Constructing a proximal strategy optimization and pruning network model; S303: Utilize the proximal strategy to optimize the pruned network model and perform intensive training to solve the Markov decision space model in step S301.

9. The method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning according to claim 8, characterized in that: In step S302, The optimization objective function of the proximal strategy optimization pruning network model is: Where: c VF and c s is a hyperparameter; θ and ω represent the parameters of the policy network and the value network respectively; L CLIP (θ) is the optimization target of the policy network; L VF (ω) is the optimization target of the value network; is the entropy of the strategy; E is the expected value; The goal of the scaled value network is Where: Yes all The standard deviation of Introduce the gradient clipping technique to solve the gradient g of θ θ and the gradient g of ω ω Where: g max is the gradient clipping value; The network parameter update method is Where: α lr and β lr are the learning rates of the policy network and the value network respectively; The learning rate update method for the policy network and the value network is Where: α lr,0 and β lr,0 are the initial learning rates of the policy network and the value network respectively; t e is the current training generation; N epoch is the maximum number of training generations.

10. A device for designing a reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning, characterized in that: Includes uncertain adversarial model module, imitation learning module and deep reinforcement learning module; The uncertain confrontation model module builds a three-dimensional uncertain confrontation model between maneuvering targets and interceptors based on the interception collision triangle. The imitation learning module uses the proportional guidance law as a strategy, generates a large number of trajectory samples in the interactive environment, establishes an intelligent terminal guidance model, and trains it using the supervised learning method to obtain an intelligent terminal guidance model that approximates the interception performance of proportional guidance. The deep reinforcement learning module uses the reinforcement learning method, sets the reward function, and adopts the PPO algorithm to further train the intelligent terminal guidance model obtained by step imitation learning, and obtains a new guidance law that exceeds the proportional guidance interception performance; The uncertain adversarial model module, the imitation learning module and the reinforcement learning module are implemented by the design method described in any one of claims 1 to 9.

Images