Method and apparatus for designing reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning

Abstract

The application discloses an interception maneuver target reinforcement learning guidance law design method and device based on imitation learning, and belongs to the technical field of guidance law design. The application firstly establishes a three-dimensional uncertain confrontation model of a maneuver target and an interceptor based on an interception collision triangle; then, a proportional guidance guidance law is taken as a strategy, a large number of trajectory samples are generated by using an interactive environment, an intelligent terminal guidance model is established, a supervised learning method is used for training, and an intelligent terminal guidance model approximating the interception performance of proportional guidance is obtained; finally, a reward function is set by using a reinforcement learning method, a PPO algorithm is used for further training of the intelligent terminal guidance model obtained through imitation learning, and a new guidance law surpassing the interception performance of proportional guidance is obtained. Simulation results show that the new guidance law has good robustness and high stability, realizes higher interception probability and lower energy consumption than traditional guidance algorithms, and the time consumption of single decision is only 0.32 ms, so the new guidance law has certain engineering application value.

Description

Technical Field

[0001] The present invention relates to the technical field of guidance law design, and in particular to a method and device for designing a guidance law for intercepting a maneuvering target based on reinforcement learning based on imitation learning. Background Art

[0002] The design of guidance laws for intercepting high-speed targets has long been a research hotspot in the field of missile defense. Proportional navigation guidance (PNG) and its improved versions have the advantages of simple structure and easy implementation, and are widely used in the interception process. PNG assumes that the target has no maneuvers or is fixed in maneuverability. Under this linear assumption, it has good interception performance. However, when dealing with targets using maneuvering penetration techniques, its interception performance will be significantly reduced. Augmented proportional navigation (APN) superimposes the target maneuvering acceleration on the PNG guidance instructions, which improves the interception performance against maneuvering targets to a certain extent. However, since APN requires real-time acquisition of target acceleration information, its engineering application possibility is reduced.

[0003] To address the problem of intercepting maneuvering targets, researchers have applied nonlinear control, optimal control, and differential game theory to the guidance field, proposing a series of novel guidance laws. The papers "Nonsingular Terminal Sliding Mode Guidance with Impact Angle Constraints," "Optimal Sliding-Mode Guidance with Terminal Velocity Constraint for Fixed-Interval Propulsive Maneuvers," and "Robust Terminal Angle Constraint Guidance Law with Autopilot Lag for Intercepting Maneuvering Targets" apply sliding-mode control theory to the design of guidance laws for intercepting maneuvering targets, and investigate the shortcomings of sliding-mode control, such as singularities and buffeting. In recent years, optimal control has gained widespread development in the field of guidance law design. For example, the papers “Optimal guidance and collision avoidance for docking with the rotating target spacecraft[J]” and “Time-optimal guidance for intercepting moving targets with impact-angle constraints” respectively studied the energy-optimal and time-optimal guidance laws based on optimal control. Using differential game theory, the papers “Optimal guidance against active defense ballistic missiles via differential game strategies”, “Optimal Cooperative Guidance Laws in a Multiagent Target-Missile-Defender Engagement” and “Guidance strategies for interceptor against active defense spacecraft in two-on-two engagement” studied the interceptor and target attack and defense confrontation problems and proposed a class of optimal guidance schemes.

[0004] Although most of the aforementioned new guidance laws have good effects in the simulation of intercepting maneuvering targets, they still have certain shortcomings: (1) Most algorithms linearize the scene, so the optimality in actual application is difficult to guarantee; (2) The guidance law based on optimal control requires estimation of the target motion trajectory, remaining flight time and interception point, which is difficult to obtain in the actual interception process; (3) The guidance law based on differential countermeasures has high computational complexity, which is difficult to meet the computing power on the missile.

[0005] Deep reinforcement learning (DRL), a key branch of artificial intelligence technology, has been widely used in sequential decision-making. DRL requires no supervisory information. Instead, it continuously interacts with the environment through a trial-and-error mechanism, maximizing cumulative rewards in complex environments and ultimately learning optimal strategies. Because DRL can directly map states to guidance commands, enabling end-to-end decision-making, it has become a research hotspot in guidance and control, particularly in areas such as powered descent guidance, space exploration guidance, missile guidance, and rendezvous guidance. The successful application of DRL and the continuous development of related technologies have also provided new insights into interception guidance law design, sparking extensive research.

[0006] The document "Missile Homing-Phase Guidance Law Design Using Reinforcement Learning" first studies the DRL guidance problem in a single fixed scenario, which inspires the development of related research. The document "Guidance law based on zero effort miss and Q-learning algorithm" is based on the Q-learning method, discretizes the action space, and obtains a reinforcement learning guidance law in a simple scenario. The document "Computational Missile Guidance: A Deep Reinforcement Learning Approach" studies the application of DRL in missile guidance and proposes two training modes: one is that the agent learns the guidance command from scratch; the other is based on prior knowledge learning, i.e., learning the navigation ratio of the classic guidance law. At the same time, the author points out that the former cannot guarantee interception of the target in some initial states, while the latter can improve the learning efficiency. The documents "Q-learning reinforcement learning guidance law" and "Deep reinforcement learning based missile guidance law design for maneuvering target interception" aim at the problem of intercepting a maneuvering target, and also take the PNG navigation ratio as the action. They respectively study the reinforcement learning guidance law based on deep deterministic policy gradient (DDPG) and the reinforcement learning guidance law based on Q-learning.

[0007] Although the foregoing documents comprehensively study the application of reinforcement learning in intercepting guidance law, there is still room for improvement. Reinforcement learning agents learn from scratch, and the learning efficiency is not high in complex and long-time scenarios. In particular, for the interception problem, both the target and the interceptor are in high-speed motion, and a mistake of the interceptor in one time step will increase the possibility of interception failure. At the same time, since the success or failure of interception can only be obtained at the end of simulation, there is a problem of sparse rewards, which leads to the fact that the agent cannot explore effective interception strategies. How to improve the training efficiency is still a difficulty in research. SUMMARY

[0008] In response to the above-mentioned problems, the present invention aims to provide a reinforcement learning guidance law design method and device for intercepting maneuvering targets based on imitation learning. First, a large number of expert samples are generated based on proportional guidance, and the intelligent agent adopts the imitation learning method to learn the expert strategy; then, the intelligent agent further uses the reinforcement learning algorithm to gradually learn the strategy for intercepting maneuvering targets, ultimately achieving the minimum energy consumption and the maximum interception probability.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0010] The method for designing a reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning includes the following steps:

[0011] S1: Based on the interception collision triangle, a three-dimensional uncertain confrontation model between the maneuvering target and the interceptor is established;

[0012] S2: Using the proportional guidance law as a strategy, a large number of trajectory samples are generated in an interactive environment to establish an intelligent terminal guidance model. This model is trained using supervised learning methods to obtain an intelligent terminal guidance model that approaches the intercept performance of proportional guidance.

[0013] S3: Using the reinforcement learning method, setting the reward function, and using the PPO algorithm to further train the intelligent terminal guidance model learned in step S2, a new guidance law that exceeds the proportional guidance interception performance is obtained.

[0014] Furthermore, the specific operation of step S1 includes the following steps:

[0015] S101: Design an interactive environment for maneuvering target and interceptor confrontation training based on the collision triangle model;

[0016] S102: Establishing a maneuvering model for a maneuvering target;

[0017] S103: Initializing the interactive environment designed in step S101 according to the maneuvering model of the maneuvering target, and solving the initial velocity vector of the interceptor;

[0018] S104: Establishing a terminal miss distance calculation model.

[0019] Furthermore, the maneuvering model of the maneuvering target described in step S102 is expressed as

[0020]

[0021] Where N m Maneuvering overload for maneuvering target, a M is the maneuvering acceleration vector of the maneuvering target, g represents the acceleration due to gravity; and are the minimum and maximum values ​​of the maneuverable target start time respectively; e M is the unit vector of the maneuvering direction of the maneuvering target.

[0022] Furthermore, in step S103, the interceptor initial velocity vector V D，0 Expressed as

[0023]

[0024] In the formula, function T q The output of (·,·) is the rotation matrix, the first input is the rotation angle, and the second input is the rotation axis; V D，0 With plane C t The angle between plane C t For X DM,0 and V M,0 The plane formed; V D，0 In plane C t The projection and The angle between and H E,ψ The minimum and maximum values ​​of and They are The minimum and maximum values ​​of X DM,0 =X M,0 -X D,0 is the initial position vector of the maneuvering target relative to the interceptor, X M,0 is the initial position vector of the maneuvering target, V M,0 is the initial velocity vector of the maneuvering target, X D,0 is the interceptor initial position vector; To satisfy the interceptor velocity vector of zero miss distance; f unit The function of (·) is to find the unit vector corresponding to the input vector.

[0025] Furthermore, the specific operation of step S2 includes the following steps:

[0026] S201: Generate N using proportional guidance law and interactive environment IL Trajectory samples are sampled at intervals of time steps to determine the imitation learning training set in, is a tuple A collection of is the input of imitation learning, a PN =min{||a PN，0 ||,a max}·f unit(a PN，0 ) is the proportional guidance instruction, a PN，0 =NV D ×ω DM ;

[0027] Where a max is the maximum overload of the interceptor, f unit The function of (·) is to solve the unit vector corresponding to the input vector, N is the navigation ratio; ω DM is the line of sight angular velocity vector, V D is the interceptor’s velocity vector;

[0028] S202: Design an imitation learning neural network model for training an intelligent terminal guidance model;

[0029] S203: Using the imitation learning neural network model, the intelligent terminal guidance model is trained through imitation learning.

[0030] Furthermore, the specific operation of step S202 includes the following steps:

[0031] S2021: Design an imitation learning neural network model with an output of Where, ‖a PN ‖∈[0,a max ];

[0032] S2022: Design an imitation learning neural network model with input Where R DM is the distance vector between the maneuvering target and the interceptor, is the velocity vector of the maneuvering target and interceptor approaching, ||ω DM || is the angular velocity of the line of sight;

[0033] S2023: Design activation functions and loss functions for the imitation learning neural network model, where the hidden layer activation function uses the S-Sigmoid activation function and the output layer activation function is tanh.

[0034]

[0035] The loss function uses the MSE loss function.

[0036]

[0037] Where: M is the number of samples; N is the number of neural network output values; y i,j and are the predicted value and label value of the j-th output of the i-th sample respectively;

[0038] S2024: Design a network parameter update method for imitation learning neural network models

[0039]

[0040] Where: N GPU is the number of GPUs participating in learning; and is the Jacobian matrix submatrix required to be solved by the i-th GPU each time and the error vector subvector of the predicted value relative to the label value, ζ i For J i,j and E i,j number of rows; For i, under certain circumstances, J i,j The number of m n =M·N is the number of all sample label values; the subscript l represents the iterative algebra; x∈R k×1 is a vector composed of network parameters, i.e., the weights and thresholds of each network layer, where k is the number of network parameters; I∈R k×k is the unit matrix; μ′ is the dynamically updated parameter. If updating the network parameters reduces the network prediction error, let μ′←μ′ / k μ′ , otherwise let μ′←μ′·k μ′ , calculate the network parameter update amount again, where k μ′ is a hyperparameter greater than 1;

[0041] The Jacobian matrix J is expressed as

[0042]

[0043] The error vector E is expressed as

[0044]

[0045] Furthermore, the specific operation of step S203 includes the following steps:

[0046] Calculate the loss function according to the loss function formula;

[0047] Compute the Jacobian matrix and error matrix;

[0048] According to the network parameter update method, the parameters θ of the imitation learning neural network are updated. IL to update.

[0049] Furthermore, the specific operation of step S3 includes the following steps:

[0050] S301: Establishing Markov decision space model;

[0051] The state space of the Markov decision space model is

[0052]

[0053] The action space is

[0054] u t =[0,a max ]

[0055] The reward function is

[0056] r t =r s +r e

[0057] in

[0058] r s =k s exp[-(u t / g) 2 ]

[0059]

[0060] Where: r s and r e are process rewards and terminal rewards respectively; r s is the reward value that reflects the degree of energy consumption. The lower the energy consumption, the greater the positive reward. e It is a reward that reflects the terminal miss distance, and a large positive reward is given when the interception is successful; k s is a hyperparameter greater than 0; R miss is the off-target amount; and is a hyperparameter related to the off-target amount, where Indicates the miss distance corresponding to the successful interception of the target; and is a hyperparameter related to the reward;

[0061] S302: Constructing a proximal strategy optimization and pruning network model;

[0062] S303: Utilize the proximal strategy to optimize the pruned network model and perform intensive training on the Markov decision space model in step S301.

[0063] Furthermore, in step S302,

[0064] The optimization objective function of the proximal strategy optimization pruning network model is

[0065]

[0066] Where: c VF and c s are hyperparameters; θ and ω represent the parameters of the policy network and the value network respectively; L CLIP (θ) is the optimization target of the policy network; LVF (ω) is the optimization target of the value network; is the entropy of the strategy; E is the expected value;

[0067] The goal of the scaled value network is

[0068]

[0069] Where: Yes all The standard deviation of

[0070] Introduce the gradient clipping technique to solve the gradient g of θ θ and the gradient g of ω ω

[0071]

[0072] Where: g max is the gradient clipping value;

[0073] The network parameter update method is

[0074]

[0075] Where: α lr and β lr are the learning rates of the policy network and the value network respectively;

[0076] The learning rate update method for the policy network and value network is

[0077]

[0078] Where: α lr,0 and β lr,0 are the initial learning rates of the policy network and the value network respectively; t e is the current training generation; N epoch is the maximum number of training generations.

[0079] Furthermore, the present invention also provides a device for designing a reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning, comprising an uncertain adversarial model module, an imitation learning module, and a deep reinforcement learning module;

[0080] The uncertain confrontation model module establishes a three-dimensional uncertain confrontation model between the maneuvering target and the interceptor based on the interception collision triangle;

[0081] The imitation learning module uses the proportional guidance law as a strategy, utilizes the interactive environment to generate a large number of trajectory samples, establishes an intelligent terminal guidance model, and trains it using supervised learning methods to obtain an intelligent terminal guidance model that approximates the interception performance of proportional guidance.

[0082] The deep reinforcement learning module uses reinforcement learning methods, sets a reward function, and adopts the PPO algorithm to further train the intelligent terminal guidance model obtained by step imitation learning, thereby obtaining a new guidance law that exceeds the interception performance of proportional guidance.

[0083] The uncertain adversarial model module, imitation learning module and reinforcement learning module are implemented using the design method described above.

[0084] The beneficial effects of the present invention are:

[0085] 1. Based on the interception collision triangle, the present invention designs a method for solving the initial states of the interceptor and target in uncertain scenarios, and establishes a three-dimensional confrontation training scenario for the interceptor and target under non-ideal conditions. Compared with the existing technology, the range of the initial states of the interceptor and the maneuvering target in this training scenario is larger, which means that the intelligent agent needs to interact more with the environment to improve the generalization ability when learning the interception strategy in this scenario. The interceptor and the maneuvering target fly faster and the interception time is longer, which increases the length of the intelligent agent's trajectory, which means that it is more difficult for the intelligent agent to learn an effective interception strategy. In addition, both the interceptor and the maneuvering target are established as three-dimensional simulation models, and the state space dimension is larger.

[0086] 2. The present invention proposes a hierarchical intelligent guidance framework: "a terminal guidance law model based on imitation learning → a terminal guidance law evolution model based on reinforcement learning". First, the PNG guidance law is mined through imitation learning to provide a good initial strategy for the reinforcement learning agent. Then, a Markov decision model for the interception agent is established, and a "soft" terminal reward including a "transition segment" is introduced. The PPO algorithm is used to solve the interception strategy. At the same time, techniques such as reward normalization, gradient clipping, and learning rate decay are used to stabilize training. This effectively improves the learning efficiency of the agent and reduces the difficulty of reward function design. It has better performance than traditional interception guidance laws, with a higher interception probability and lower energy consumption. At the same time, the algorithm has low computational complexity, and a single run time of approximately 0.32ms on the STM32F407.

[0087] 3. This invention incorporates a process reward for energy consumption and a "soft" terminal reward for the "transition phase." Using the PPO algorithm to solve the interception strategy, this approach helps alleviate the convergence difficulties caused by sparse rewards. Even when the agent fails an interception, it still receives a reward, which guides it in exploring successful interception strategies. This reward, in turn, guides the agent to learn a successful interception strategy, achieving a high probability of interception with low energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 Schematic diagram of the collision triangle model in the present invention.

[0089] Figure 2Schematic diagram of the training scenario in the present invention.

[0090] Figure 3 Schematic diagram of square wave maneuvering of a maneuvering target in the present invention.

[0091] Figure 4 This is a schematic diagram of the initial state error of the interceptor in the present invention.

[0092] Figure 5 This is a schematic diagram of the calculation of the terminal miss distance in the present invention.

[0093] Figure 6 For the present invention Schematic diagram of network structure.

[0094] Figure 7 For the present invention, reward r e Schematic diagram of .

[0095] Figure 8 This is a schematic diagram of the PPO strategy network in the present invention.

[0096] Figure 9 These are 500 sets of adversarial sample trajectories used for imitation learning in the simulation experiment of this invention.

[0097] Figure 10 This is the curve of the imitation learning loss changing with the number of training generations in the simulation experiment of the present invention.

[0098] Figure 11 This is a box plot of the energy consumption and off-target amount of ILG and PNG in the simulation experiment of the present invention.

[0099] Figure 12 This is the IL-PPOG training curve in the simulation experiment of the present invention.

[0100] Figure 13 This is the curve of the off-target amount and energy changing with training in the simulation experiment of the present invention.

[0101] Figure 14 This is the curve of the off-target amount of different algorithms changing with training in the simulation experiment of the present invention.

[0102] Figure 15 This is the curve of energy change of different algorithms with training in the simulation experiment of the present invention.

[0103] Figure 16 This is a box plot of the energy consumption and off-target amount of IL-PPOG and PNG in the simulation experiment of the present invention.

[0104] Figure 17 The different H E,ψ and different Interception probability curve under these conditions.

[0105] Figure 18 The different H E,ψ and different Energy consumption in this case.

[0106] Figure 19 These are overload instructions for different algorithms in the simulation experiment of the present invention.

[0107] Figure 20 It is the target maneuvering acceleration instruction in the simulation experiment of the present invention.

[0108] Figure 21 is the line of sight angular velocity of different algorithms in the simulation experiment of the present invention.

[0109] Figure 22 It is the three-dimensional trajectory of the target and interceptor in the simulation experiment of the present invention.

[0110] Figure 23 The IL-PPOG in the simulation experiment of the present invention is different in ω s The off-target amount in the case.

[0111] Figure 24 The IL-PPOG in the simulation experiment of the present invention is different in ω s Energy consumption in this case.

[0112] Figure 25 The different H E,ψ and different The interception probability in the case of (the target adopts sinusoidal maneuver ω s =π).

[0113] Figure 26 The different H E,ψ and different Energy consumption in this case (target adopts sinusoidal maneuver ω s =π). DETAILED DESCRIPTION

[0114] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0115] Example 1:

[0116] Embodiment 1 provides a method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning, which specifically includes the following steps:

[0117] Step S1: Based on the interception collision triangle, a three-dimensional uncertain confrontation model of the maneuvering target and the interceptor is established;

[0118] Specifically, S101: Design an interactive environment for maneuvering target and interceptor confrontation training;

[0119] The present application mainly studies the generation of guidance instructions for intercepting a maneuvering target outside the atmosphere. For the convenience of research, the following assumptions are given:

[0120] Assumption 1: In the flight stage outside the atmosphere, the motion around the mass center and the attitude control of the interceptor and the maneuvering target are not considered, and a six-degree-of-freedom instantaneous balance model is used in the simulation.

[0121] Assumption 2: The interceptor guidance system is regarded as a non-delay ideal system.

[0122] Assumption 3: It is assumed that the interceptor detection noise obeys a normal distribution.

[0123] More specifically, the specific operation of designing the maneuvering target and the interceptor confrontation training scene includes the following steps:

[0124] The design of the interactive environment should be based on the combat scene and the performance parameters of both sides, and as much as possible to include all possible flight states of both sides in combat. The present application mainly studies the terminal guidance problem of the interceptor, therefore, the initial state of the interactive environment is the flight state of both sides when the interceptor switches to terminal guidance. This flight state is closely related to the standard trajectory of the interceptor. The standard trajectory of the interceptor refers to the standard trajectory corresponding to the launch parameters of the interceptor, which is a trajectory that can hit the predetermined interception point. Therefore, the flight state of both sides in the initial state of the interactive environment should fluctuate around the corresponding state of the collision triangle.

[0125] The collision triangle model is shown in the accompanying Figure 1 The basic condition for the zero-control miss distance of the interceptor to be 0 when the interceptor is in the terminal guidance state is given, that is, when the states of the interceptor and the target satisfy the collision triangle relationship, the interceptor can successfully intercept the target. Figure 1 In the above formula, all variables are represented in the inertial coordinate system. D V is the velocity vector of the interceptor; V M is the velocity vector of the maneuvering target; X DM = X M -X D is the position vector of the maneuvering target relative to the interceptor, wherein X M and X D represent the position vectors of the maneuvering target and the interceptor, respectively; V D,⊥ and V M,⊥ represent the components of V D and V M perpendicular to X DM ; and represent the components of V D and V M parallel to X DM .

[0126] Assuming that the interceptor and the maneuvering target are not subject to external forces or are subject to the same external forces, the position vector of the maneuvering target relative to the interceptor at time t0 is X DM，0 , the position vector X of the maneuvering target relative to the interceptor at time t DM,t Decomposed into parallel and perpendicular to X DM，0 Quantity, available

[0127]

[0128] Where: and Represents parallel and perpendicular to X respectively DM，0 The weight.

[0129] According to the relative motion state of the interceptor and the maneuvering target, we can get and They satisfy the relationships of formula (2) and formula (3) respectively.

[0130]

[0131] Where, and They represent the external acceleration vectors of the interceptor and maneuvering target parallel to X DM，0 of weight and satisfies Δt DM Indicates the interception duration; a D,⊥ and a M,⊥ They represent the external acceleration vectors of the interceptor and maneuvering target perpendicular to X DM，0 The weight of , and satisfy a D,⊥ =a M,⊥ .

[0132] Design a training scenario based on the collision triangle model as shown in the attached Figure 2 shown.

[0133] Figure 2 Describes the initial three-dimensional relative motion relationship between the interceptor and the maneuvering target, where the variables have the following meanings: coordinate system x n O n z n is the northeast coordinate system established with the maneuvering target as the origin; Θ M,0 and σ M,0 are the local velocity inclination and velocity azimuth of the maneuvering target respectively; X MD,0 is the relative position vector of the interceptor relative to the maneuvering target; α MD,0 For X MD,0 With plane x n O n z n The angle between MD,0 For X MD,0 In plane xn O n z n Projection and O n x n During the training process, the initial motion states of the two are randomly generated within a certain range of initial conditions, including ||V M,0 ||、Θ M,0 , σ M,0 , α MD,0 , β MD,0 and ||V D，0 ||. Among them, V M,0 and V D，0 Represent the initial velocity vectors of the interceptor and the maneuvering target respectively; || || represents the magnitude of the vector, for example, ||V D，0 || represents the magnitude of the initial velocity vector of the maneuvering target. It should be noted that ||X MD,0 || and interceptor velocity vector V M,0 The reasons why (in addition to size, direction is also included) is not in the initial conditions are as follows: ① Considering that the interceptor has a certain distance to detect the maneuvering target, ||X MD,0 || is a constant value; ② The interceptor velocity should be near the collision triangle, so the interactive environment only randomly samples the interceptor velocity as the initial condition, and each velocity component is generated by adding random amounts based on the velocity components solved in the collision triangle.

[0134] Considering X MD,0 There are certain constraints on the flight direction of the maneuvering target, so the X MD,0 Two variables Δα that represent the degree of deviation from the maneuvering target's flight direction MD,0 and Δβ MD,0 , and satisfy the following relationship:

[0135]

[0136] The initial state boundary range for training is set as shown in Table 1.

[0137] Table 1 Training scenario parameter boundaries

[0138]

[0139] S102: Establishing a maneuvering model for a maneuvering target;

[0140] Since the interceptor has a very short time to capture and intercept a maneuvering target, and is limited by the maneuverability of the maneuvering target, its maneuvering form should not be too complex and it is impossible to perform overly complex maneuvers in engineering. The single square wave maneuvering form has the characteristics of simple form, easy implementation, and good penetration performance. It is widely used in the research of aircraft maneuver penetration. Its schematic diagram is shown in the attached figure. Figure 3 shown.

[0141] Figure 3 N m is the maneuvering acceleration vector of the maneuvering target, and g represents the gravity acceleration; m,0 , T m,1 , and ΔT m are the maneuvering start time, the maneuvering end time, and the maneuvering duration, respectively. It is considered that the maneuvering target needs to achieve two targets: ① effectively avoiding interception by the interceptor; and ② improving the hit accuracy. However, there is a certain contradiction between the two indexes, that is, the greater the early maneuvering degree, the better the avoidance effect, but it is not conducive to the later landing point deviation correction. Therefore, the relationship between the two targets is comprehensively considered, and the maneuvering avoidance degree should be limited in the training process. Therefore, according to the performance parameters of the trajectory control engine of the maneuvering target, the size of the maneuvering duration is determined as a constant value. In order to increase the richness of the target sample intercepted by the interceptor and improve the generalization ability of the agent, the maneuvering time and direction of the maneuvering target are randomly generated, and then the maneuvering model of the maneuvering target is obtained as

[0142]

[0143] wherein, a M is the maneuvering acceleration vector of the maneuvering target, and g represents the gravity acceleration; and are the minimum value and the maximum value of the maneuverable start time of the maneuvering target, respectively; e M is the unit vector of the maneuvering direction of the maneuvering target.

[0144] Since the target applies the trajectory control thrust in the vertical direction of the line connecting the interceptor and the maneuvering target, it has the greatest influence on the interception effect of the interceptor. At the same time, considering that the trajectory control engine of the target can generate thrust in 4 directions along the transverse direction of the interceptor, it is assumed that e M is perpendicular to the line connecting the interceptor and the maneuvering target, and e M points in the space and is limited to 4. In the training process, one of the above 4 directions is randomly selected as the pointing direction of e M .

[0145] S103: Perform initialization processing on the interactive environment;

[0146] The initialization of the interactive environment mainly completes the calculation of the initial states of the interceptor and the maneuvering target. Since the initial position X M,0 , the velocity V M,0 of the maneuvering target, and the initial position X D,0 of the interceptor can be randomly sampled in the flight state range, the key of the initialization of the interactive environment lies in the solution of the initial velocity of the interceptor. The size of the velocity of the interceptor is randomly sampled as V D,0 (V D,0 is the size of the velocity of the interceptor, and V D，0is the interceptor velocity vector), then according to the collision triangle model, the interceptor velocity vector that satisfies the zero control miss distance of 0 can be obtained. for

[0147]

[0148] Where, X DM,0 =X M,0 -X D,0 is the initial position vector of the maneuvering target relative to the interceptor. It should be noted that It is the interceptor velocity vector calculated without error and basically conforms to the zero-control interception.

[0149] Because the interceptor and maneuvering target are initially separated by a certain distance in space, the gravitational forces acting on them differ slightly, and the velocity obtained using the above equation cannot guarantee a zero miss margin. However, the difference in gravitational forces between the two is minimal, so the zero miss margin obtained using the above method is close to zero. Considering that the main purpose of the above equation is to constrain the interceptor's initial velocity to be more realistic, the effect of the gravitational difference between the two on the initialization of the environment can be ignored.

[0150] Due to the errors in the calculation of the predetermined interception point, the initial guidance, the mid-range guidance, etc., the interceptor's flight state when entering the terminal guidance is near the collision triangle, as shown in the attached figure. Figure 4 As shown. For the convenience of description, define plane C t For X DM,0 and V M,0 The plane spanned, the error angle V D，0 With plane C t The angle, error angle V D，0 In plane C t The projection and The angle between and H E,ψ The minimum and maximum values ​​of and They are The minimum and maximum values ​​of the interceptor are obtained. The initial velocity vector V D，0 for

[0151]

[0152] Where: function T q The output of (·,·) is the rotation matrix. The first input is the rotation angle, and the second input is the rotation axis. This function can be obtained through quaternion. unit The function of (·) is to solve the unit vector corresponding to the input vector. V′ D，0 is a process variable, Around f unit (X DM,0 ×V M,0 ) axis rotation Then we get V′ D，0 , V′ D，0 Then go around f unit (X DM，0 ×V M，0 )×f unit (V′ D，0 ) Axis rotation H E,ψ Get V D，0 .

[0153] S104: Establishing a terminal miss distance calculation model;

[0154] Because the terminal miss distance is an important metric for measuring intercept success, accurately determining it is a key issue for interceptors attempting to intercept targets. Typically, due to limitations on the simulation step size, the distance between the interceptor and the maneuvering target at the end of the simulation cannot be used as the terminal miss distance. This distance is not the minimum distance between the interceptor and the maneuvering target. Reducing the step size to achieve a more accurate terminal miss distance increases simulation time.

[0155] In order to solve the terminal miss distance in the present invention, the position D of the interceptor in the last two steps of integration is used. f and D′ f And the position M of the maneuvering target in the last two steps of integration f and M′ f To solve, as shown in the attached Figure 5 shown. Figure 5 middle, and They represent the interceptor and the maneuvering target at the same time t * location.

[0156] For the convenience of description, let s A,B Represents the vector from point A to point B. In the integration interval Δt int The speeds of the interceptor and the maneuvering target are approximately constant, and the interceptor is at D′ f The time is t′ f , Then the problem of terminal miss distance can be converted into solving

[0157]

[0158] because

[0159]

[0160] Substitute equation (9) into equation (8), and let Available

[0161]

[0162] According to formula (10), the terminal miss distance can be obtained as

[0163]

[0164] Furthermore, step S2: using the proportional guidance law as a strategy, a large number of trajectory samples are generated in an interactive environment, an intelligent terminal guidance model is established, and the model is trained using a supervised learning method to obtain an intelligent terminal guidance model that approximates the interception performance of the proportional guidance.

[0165] Specifically, step S201: Generate N using the proportional guidance law and the interactive environment IL Trajectory samples are sampled at intervals of time steps to determine the imitation learning training set

[0166] More specifically, the proportional guidance law is based on the assumption that the target is non-maneuverable. The vector form of its three-dimensional proportional guidance can be expressed as

[0167] a PN，0 =NV D ×ω DM (12)

[0168] Where: a PN，0 is the acceleration command of the proportional guidance law; N is the navigation ratio; ω DM is the line-of-sight angular velocity vector. In the actual guidance process, this vector is measured in real time by the seeker and can be calculated by formula (13) during the simulation process.

[0169]

[0170] Where: V DM =V M -V D is the velocity vector of the target relative to the interceptor.

[0171] Considering that there is a maximum limit on the interceptor overload, the proportional guidance instruction is corrected to

[0172] a PN =min{||a PN，0 ||,a max}·f unit (a PN，0 ) (14)

[0173] Where: a max is the maximum overload of the interceptor, f unit The function of (·) is to find the unit vector corresponding to the input vector.

[0174] According to the above proportional guidance law and the interactive environment, N IL Trajectory samples are sampled at intervals of time steps to obtain the imitation learning training set in, is a tuple A collection of is the input for imitation learning.

[0175] The main purpose of the imitation learning training of the intelligent terminal guidance model is to provide initial parameters for the reinforcement learning guidance law to reduce the learning difficulty and improve the learning performance. Therefore, the intelligent terminal guidance model uses the proportional guidance instruction as a label to learn the interception strategy of the proportional guidance law, which can be formally expressed as

[0176]

[0177] Where, f IL express To proportional guidance command a PN The imitation learning module needs to learn the mapping.

[0178] S202: Design an imitation learning neural network model for training an intelligent terminal guidance model;

[0179] The core of the imitation learning module is composed of a neural network f NN The guidance instruction generator composed of the main body needs to reasonably design the neural network input and output, activation function and network structure to improve the performance of the imitation learning module.

[0180] More specifically, S2021: Designing an imitation learning neural network model output;

[0181] Considering that the generation of line-of-sight angular velocity needs to be suppressed during terminal guidance interception, it is beneficial to save interceptor fuel by setting the acceleration along the direction where the line-of-sight angular velocity changes the most. The unit vector in this direction is e a,unit , then combined with formula (13) we can know

[0182] e a,unit =arg max||X DM ×e a,unit || (16) According to the cross multiplication rule, we can know

[0183] e a,unit =f unit (X DM ×ω DM ) (17)

[0184] Since the interception process occurs outside the atmosphere, the influence of the atmosphere on the interceptor's attitude can be ignored, ensuring that the interceptor's direction always points to the target, thereby generating an interception along the e a,unit acceleration.

[0185] In order to improve the real-time performance of the neural network and reduce the difficulty of learning the neural network, when designing the output of the neural network, the output dimension should be reduced as much as possible, and variables that are easy to learn should be selected as the output. Therefore, the design requires acceleration (guidance instruction) along e a,unit direction, so the output of the neural network can be designed to be the required acceleration size, and formula (15) is simplified to

[0186]

[0187] Where, ||a PN ||∈[0,a max ].

[0188] S2022: Design inputs for imitation learning neural network models;

[0189] Input The selection of will directly affect the difficulty of model learning and needs to reflect the key information of the interception process. The input quantity designed in this invention is

[0190]

[0191] Where R DM is the relative distance vector between the maneuvering target and the interceptor, is the relative velocity vector between the maneuvering target and the interceptor, ||ω DM || is the angular velocity of the line of sight.

[0192] In order to overcome the problem of low training efficiency caused by the non-uniform scale of different dimensions of input and output vectors, the min-max normalization method is used. After normalization, we get

[0193] S2023: Design activation and loss functions for imitation learning neural network models;

[0194] In addition, in order to reduce the computational complexity of the activation function, the hidden layer activation function adopts the S-Sigmoid activation function, which is a prior art and will not be described in detail in the present invention.

[0195] Considering that the imitation learning neural network mainly completes the mapping from input to guidance instructions, and the guidance instructions have a maximum value constraint, the activation function of the output layer of the imitation learning neural network is designed to be tanh, then

[0196]

[0197] In the present invention It is a fully connected neural network with two hidden layers. Its network structure is shown in the attached figure. Figure 6 As shown, Represents a network node, The present invention selects 6 nodes for the hidden layer 1 and 5 nodes for the hidden layer 2.

[0198] The model learning in this invention adopts MSE loss function

[0199]

[0200] Where: M is the number of samples; N is the number of neural network output values; y i,j and are the predicted value and label value of the j-th output of the i-th sample respectively.

[0201] S2024: Design a network parameter update method for imitation learning neural network models;

[0202] Since the simulation samples are obtained by sampling the flight trajectory of the PNG guidance law, the number of samples is huge. In this paper, the improved data parallel LM optimizer (Improved Data Parallel Levenberg-Marquardt, IDP-LM) is used as the training algorithm, and its network parameter update formula is:

[0203]

[0204] Where: N GPU is the number of GPUs participating in learning; and is the sub-matrix of the Jacobian matrix required to be solved by the i-th GPU each time and the sub-vector of the error vector of the predicted value relative to the label value. Its relationship with the Jacobian matrix J and the error vector E is shown in (23) and (24). i For J i,j and E i,j number of rows; For i, under certain circumstances, J i,j The number of m n =M·N is the number of all sample label values; the subscript l represents the iterative algebra; x∈R k×1 is a vector composed of network parameters, i.e., the weights and thresholds of each network layer, where k is the number of network parameters; I∈R k×k is the unit matrix; μ′ is the dynamically updated parameter. If updating the network parameters reduces the network prediction error, let μ′←μ′ / k μ′ , otherwise let μ′←μ′·kμ′ , calculate the network parameter update amount again, where k μ′ is a hyperparameter greater than 1.

[0205]

[0206]

[0207] S203: Using an imitation learning neural network model; performing imitation learning training on the intelligent terminal guidance model;

[0208] Specifically, the process of imitation learning training includes:

[0209] Calculate the loss function according to formula (21);

[0210] Calculate the Jacobian matrix and error matrix according to formula (23) and formula (24) respectively;

[0211] According to formula (22), the parameters θ of the imitation learning neural network are IL to update.

[0212] Furthermore, step S3: using the reinforcement learning method, setting the reward function, and using the PPO algorithm to further train the intelligent terminal guidance model learned in step S2, a new guidance law that exceeds the proportional guidance interception performance is obtained.

[0213] Specifically, S301: establishing a Markov decision space model;

[0214] The key to the application of reinforcement learning in the field of interception and guidance is how to design a reasonable Markov decision process, including state space, action space and reward function, to improve learning efficiency and interception and guidance law performance.

[0215] Based on imitation learning, the reinforcement learning agent further explores effective strategies for intercepting maneuvering targets. Therefore, the state space of the reinforcement learning agent is consistent with the input of imitation learning, that is, formula (19), and the state space is obtained as

[0216]

[0217] The action output of the reinforcement learning agent is the guidance instruction. Combining formula (18), the action space can be obtained as follows:

[0218] u t =[0,a max ] (26)

[0219] The interception scenario for maneuvering targets studied in this paper involves two optimization objectives: maximizing the interception probability and minimizing energy consumption. Furthermore, since the agent's interception strategy can only be evaluated at the end of the simulation, there is a sparse reward problem. To alleviate the slow convergence and poor learning results caused by sparse rewards, and to achieve low energy consumption and high interception probability, a two-part reward function is designed:

[0220] r t =r s +r e (27)

[0221] in

[0222] r s =k s exp[-(u t / g) 2 ] (28)

[0223]

[0224] Where: r s and r e are process rewards and terminal rewards respectively; r s is the reward value that reflects the degree of energy consumption. The lower the energy consumption, the greater the positive reward. e It is a reward that reflects the terminal miss distance, and a large positive reward is given when the interception is successful; k s is a hyperparameter greater than 0; R miss is the off-target amount; and is a hyperparameter related to the off-target amount, where Indicates the miss distance corresponding to the successful interception of the target; and is a hyperparameter related to reward; r e The schematic diagram is as attached Figure 7 shown.

[0225] By the attached Figure 7 It can be seen that r e It is mainly divided into 3 sections: ① When , it means the interception fails and the agent reward is 0; ② When Indicates that although the interception fails, the miss distance is small, so the agent can still obtain a certain positive reward; ③ When , indicating a successful interception, and the agent receives a large positive reward. Unlike a "hard" terminal reward, which only grants a large positive reward to the agent upon successful interception, this paper proposes a "soft" terminal reward that incorporates a "transition segment," which, to a certain extent, helps alleviate the convergence difficulties caused by sparse rewards. In other words, even when the agent fails to intercept, it still receives a reward, and this reward can guide the agent in exploring successful interception strategies, thereby guiding the agent to learn a successful interception strategy.

[0226] S302: Constructing a proximal strategy optimization and pruning network model;

[0227] Specifically, PPO-CLIP contains a policy network and a value network, and its optimization goal is to maximize the function given by formula (30)

[0228]

[0229] Where: c VF and c s are hyperparameters; θ and ω represent the parameters of the policy network and the value network respectively; L CLIP (θ) is the optimization target of the policy network, which can be calculated by formula (31); L VF (ω) is the optimization target of the value network, which can be calculated by formula (33); is the entropy of the strategy, reflecting the uncertainty of the strategy. E is the expected value.

[0230]

[0231] Where: π θ and π θ,old Represent the new strategy and the old strategy respectively; p(θ) represents the probability ratio of the new and old strategies, which can be expressed as p(θ) = π θ (a|s) / π θ,old (a|s); clip(,,) is the clipping function, which can be expressed as

[0232]

[0233] Where: ε is a very small hyperparameter used to ensure that the gap between the new and old strategies is not large.

[0234]

[0235] Where, is the target of the value network, and the calculation formula is

[0236]

[0237] Reward scaling is introduced to avoid the negative impact of excessive or small rewards on the learning of the value network. The target of the scaled value network can be expressed as

[0238]

[0239] Where: Yes all The standard deviation of .

[0240] At the same time, in order to improve the stability of the training process, the gradient clipping technique is introduced to solve the gradient g of θ θ and the gradient g of ω ω

[0241]

[0242] Where: g max is the gradient clipping value.

[0243] To update the parameters of the policy network and the value network, the Adam optimizer is used to maximize the objective function given by (30), that is,

[0244]

[0245] Where: α lr and β lr are the learning rates of the policy network and the value network, respectively.

[0246] Furthermore, in order to enhance the stability of the later stage of training, the learning rate decay technique (Learning Rate Decay) is introduced, namely

[0247]

[0248] Where: α lr,0 and β lr,0 are the initial learning rates of the policy network and the value network respectively; t e is the current training generation; N epoch is the maximum number of training generations.

[0249] Furthermore, the strategy network of PPO is based on the imitation learning agent network, as shown in the attached diagram. Figure 8 As shown. It can be seen that the policy network has two outputs, namely the action mean μ and the action standard deviation a std , where the network of action mean μ is given by Inherited. During the training phase, the agent action a t According to the distribution After sampling, it is obtained by tanh activation function and action scaling.

[0250] The number of network nodes and activation function of the PPO strategy network in the present invention are shown in Table 2.

[0251] Table 2 PPO network structure and parameters

[0252]

[0253] S303: Using the PPO-CLIP model in step S302, perform intensive training on the Markov decision space model in step S301 to obtain a solution.

[0254] More specifically, the process of intensive training includes the following steps:

[0255] Based on the imitation learning network in step S2, the parameters of the policy network are updated, θ←θ IL ,θ old ←θ;

[0256] Use rollouts to Generate N for the interaction between the behavior policy network and the environment rollout Trajectories, stored in the experience pool in, is a tuple (s t ,a t ,r t ,s t+1 )

[0257] According to the formula

[0258]

[0259] Calculate the advantage function and update The tuple in is

[0260] Calculate the target value function according to formula (34) and update The tuple in is

[0261] From the experience pool Random sampling to obtain batch transfer data

[0262] Calculate the PPO objective function

[0263]

[0264] Calculate the clipped gradient according to formula (36) gθ and g ω ;

[0265] Update network parameters according to formula (37);

[0266] emptying updating behavior policy network parameters theta old ← theta;

[0267] The learning rate is updated according to formula (38), and finally the guidance law after reinforcement learning is obtained. In summary, the agent learning process in the application is shown in Table 3.

[0268] Table 3 Agent learning process

[0269]

[0270] Simulation experiment:

[0271] In order to verify the effectiveness of the guidance law IL-PPOG algorithm established in the application, the simulation experiment is designed by taking the parameters in Table 1 as the scene parameters to establish an interactive environment. In the simulation experiment, the hyperparameters are first given, then the performance of imitation learning is verified, the reinforcement learning guidance process is further analyzed, and finally the performance of the algorithm is tested in an unknown scene.

[0272] The simulation of the application is carried out under the support of software tools such as Python 3.7, Pytorch 1.5, Visual Studio 2015 and SWIG 4.0.1. In order to improve the running speed of the interactive environment, the interactive environment of the interceptor intercepting the maneuvering target is established based on C++ language in the Visual Studio 2015 environment. Fourth-order Runge-Kutta and Adams integrator (the former provides the first four steps of integration results for the latter) are adopted, the integration step is 0.02s, and SWIG 4.0.1 is used to establish the data interface with Python. In the imitation learning stage, the navigation ratio of the proportional guidance law is 4.0, and in the reinforcement learning stage, the decision step is 0.1s. The performance of the simulation platform is as follows: the CPU is Intel Core 13900K, the memory is 64G, and the graphics card is Nvidia 3080 (10G).

[0273] Since there is uncertainty in the observation and action during the interception process, observation noise and action noise are added, that is,

[0274]

[0275] In the formula: 1 3×1 represents a vector with all elements being 1; wherein, and are the standard deviations of the observation R DM , and ||ω DM || noise; and a is the standard deviation of the action noise.

[0276] The IL-PPOG algorithm established in this simulation experiment contains many hyperparameters, mainly involving network training, reward function, environment settings, etc. The specific parameters are shown in Table 4.

[0277] Table 4. Hyperparameter design of simulation experiment

[0278]

[0279] Imitation learning simulation experiment:

[0280] In order to generate imitation learning training samples, PNG output instructions are used as actions to interact with the environment, generating 10,000 interception trajectories, each of which lasts about 15 seconds. Figure 9 500 sets of three-dimensional trajectories of adversarial samples are given. The interception trajectory is sampled at 0.1s intervals, and then a 1.5×10 6 The training set of samples is used. IDP-LM algorithm is used to train f IL After 1000 generations of training, the training curve is as shown in the attached figure. Figure 10 As shown, the y-axis adopts logarithmic coordinate form.

[0281] By the attached Figure 10 It can be seen that as the number of training generations increases, the training curve gradually converges, indicating that the imitation learning agent can learn the instructions output by PNG. IL The interception performance of the guidance command was tested, and 1000 interception simulations were carried out in an interactive environment. The miss distance and energy consumption were obtained as shown in the attached figure. Figure 11 At the same time, Figure 11 The miss distance and energy consumption of PNG are given. The definition of energy consumption performance index is as follows:

[0282]

[0283] Where, t f The time corresponding to the end of the interception flight simulation.

[0284] By the attached Figure 11 It can be seen that based on f IL The guidance instructions based on PNG and guidance instructions based on PNG are similar in terms of miss distance and energy consumption, indicating that f IL The PNG instructions are well learned and can provide a good initial strategy for reinforcement learning agents.

[0285] Reinforcement learning training simulation experiment:

[0286] In the reinforcement learning agent training phase, the training process is based on f ILThe reinforcement learning agent is trained for the initial strategy. Assuming the maximum training generation is 300, the number of rollouts trajectories is 50, the experience pool size is 7500, and each time the network parameters are updated, the network is updated 1000 times in a small batch manner (batch size is 256). In order to show the changing trend of the agent strategy performance with the learning process, the agent is trained with 5 different random seeds, and the network parameters are initialized using the orthogonal initialization method. Each training interval is 10 generations, and 1000 Monte Carlo intercept simulations are carried out with the agent strategy as the instruction (each simulation has a random initial state), and the trajectory cumulative reward average curve is obtained as shown in the attached figure. Figure 12 The average values ​​of off-target amount and energy consumption are shown in the appendix. Figure 13 It should be noted that the attached Figure 12 and attached Figure 13 The solid line indicates the mean, and the shading indicates the standard deviation.

[0287] To verify the advantages of the IL-PPOG algorithm over other studies, we compared it with two reinforcement learning guidance algorithms: the reinforcement learning guidance algorithm that uses the PNG navigation ratio as the action, denoted as LC-PPOG; and the reinforcement learning guidance algorithm that uses reward shaping to guide the agent's learning, denoted as Classic-PPOG. Among them, in the comparison algorithm ②, except for the process reward design and the fact that it is not based on the IL strategy, other reinforcement learning elements are the same as IL-PPOG.

[0288] r s ′=k s exp[-(u t / g) 2 ]+k q exp[-(||ω DM || / ||ω DM || max ) 2 ] (43)

[0289] Where: k q =0.1 represents the weight coefficient of the line of sight angular velocity; ||ω DM || max =1×10 -2 rad·s -1 is the minimum line of sight angular velocity.

[0290] Attachment Figure 14 and attached Figure 15The curves showing how the average miss rate and energy consumption of the agents change with the number of training generations during 1000 tests of the agents every 10 generations are presented for the training of IL-PPOG, LC-PPOG, and Classic-PPOG. It can be seen that IL-PPOG has a smaller average miss rate and energy consumption than LC-PPOG and Classic-PPOG in the early stages of training. As the number of training generations increases, the average miss rate and energy consumption of IL-PPOG gradually decrease, and the strategy learned by the agent achieves the lowest energy consumption and a higher interception probability. Compared with LC-PPOG and Classic-PPOG, it can be seen that IL-PPOG has lower energy consumption and a higher interception probability overall:

[0291] (1) The interception performance of LC-PPOG in the initial stage of training is similar to that of IL-PPOG. At the time of convergence, the average miss distance is higher than that of IL-PPOG, and the average energy consumption is slightly lower than that of IL-PPOG. Considering that the miss distance is the main factor affecting the interception performance, it can be seen that the interception performance of IL-PPOG is slightly higher than that of LC-PPOG.

[0292] (2) Classic-PPOG requires careful design s and k q There are two hyperparameters to guide the agent to explore a successful interception strategy while balancing the importance of energy consumption and interception probability. The hyperparameter adjustment faces great difficulties. Figure 16 and Figure 17 It can be seen that although the average interception miss of Classic-PPOG at convergence is smaller than that of IL-PPOG, its energy consumption is much greater than that of IL-PPOG, which is not conducive to improving the interception performance under limited energy constraints.

[0293] In order to test the interception performance of IL-PPOG, in an interactive environment with various errors and noises established with the parameters given in Table 1, the target adopts square wave maneuvers and 10,000 Monte Carlo simulations are carried out. The results of interception miss distances and energy consumption of different algorithms are shown in the attached figure. Figure 16 As shown in the figure, it can be seen that the 3σ value and average value of the miss distance of IL-PPOG in the known scenario are 6.11m and 0.34068m, respectively, which are smaller than the miss distances of PNG and LC-PPOG. At the same time, the 3σ value and average value of the energy consumption of IL-PPOG are 344.1658m·s -1 and 189.6307 m·s -1 , which is less than the corresponding energy consumption of PNG and Classic-PPOG. This shows that IL-PPOG has better interception performance against maneuvering targets and has good robustness in external interference environments.

[0294] To further test different H E,ψ and different The interception performance of different algorithms under different conditions was evaluated. 10,000 Monte Carlo simulations were conducted for different situations. The interception probability (when the miss distance is less than 1.0m, the interception is considered successful) and energy consumption were calculated as shown in the attached figure. Figure 17 and attached Figure 18 As shown in the figure, IL-PPOG has higher interception performance in all situations, and its advantage is more obvious under large deviation conditions, further demonstrating the algorithm's robustness. Furthermore, considering that large deviations represent large errors in the mid- and end-of-shift handover, IL-PPOG can broaden the mid- and end-of-shift handover accuracy constraints.

[0295] Assuming the target maneuver strategy is to maneuver upwards at a distance of 40 km from the interceptor, and In the case of 10 interception processes, statistics are collected on interceptor missile interception overload instructions, target maneuver instruction curves, line of sight angular velocity, interceptor missile and target flight trajectories as shown in the attached figure. Figure 19-22 As shown. It can be seen that due to H E,ψ and With the existence of IL-PPOG, a large interception overload instruction is given at the early stage of terminal guidance, which quickly reduces the line of sight angular velocity, and then a large interception overload is given again after the target maneuvers, thereby achieving effective interception. Figure 21 It can be seen that during the interception process, the line-of-sight angular velocity corresponding to the PNG and LC-PPOG guidance laws is greater than the line-of-sight angular velocity corresponding to the IL-PPOG guidance law; Figure 19 It can be seen that when approaching the target, the PNG guidance law reaches the maximum interception overload earlier than the IL-PPOG guidance law, which leads to the failure of PNG to intercept, while the IL-PPOG can intercept successfully, further indicating that the IL-PPOG has better interception performance. Figure 19 and Figure 21 It can be seen that although the Classic-PPOG has the smallest line-of-sight angular velocity and a smaller interception miss margin, its overload command is always close to saturation, consuming a lot of energy. Therefore, under limited energy conditions, its interception effect will decrease.

[0296] Algorithm performance test in unknown scenarios:

[0297] To verify the generalization ability of IL-PPOG for unknown interception environments, an unknown scenario was established as follows: the target maneuver form was changed from square wave maneuver form to sinusoidal maneuver form, as shown in Equation (44), and other scenario parameters remained unchanged. In the target sinusoidal maneuver scenario, the initial states of the target and interceptor were randomly generated, and observation noise and action noise were added. 10,000 Monte Carlo simulations were performed, and the miss distance and energy consumption were statistically calculated as shown in the attached figure. Figure 23-26 shown.

[0298]

[0299] Where: ω s is the sinusoidal maneuvering frequency.

[0300] Attachment Figure 23 and Figure 24 Given ω s The interception miss distance and energy consumption of IL-PPOG under different sinusoidal maneuvering scenarios. It can be seen that ω s =π, the interception miss amount corresponding to the case is larger than that of other cases, indicating that the target adopts the s =π sinusoidal maneuvering mode can increase the difficulty of interception by interceptor. s =π, the 3σ value and average value of the IL-PPOG miss distance are 1.62m and 0.047m, respectively, and the 3σ value and average value of the energy consumption are 322.83m·s -1 and 177.91 m·s -1 , indicating that the interception miss is small and the energy consumption is low. Combined with the reinforcement learning training simulation experiment, it can be seen that IL-PPOG has good interception performance for targets with different maneuvering modes and has good generalization ability. E,ψ and different In this scenario, let the sinusoidal maneuver frequency ω s =π, and 10,000 Monte Carlo simulations were carried out for each case. The interception probability and energy consumption of different algorithms are shown in the attached figure. Figure 25 and Figure 26 As shown in the figure, IL-PPOG has better interception performance than LC-PPOG, Classic-PPOG and PNG. Therefore, IL-PPOG has the ability to adapt to external interference and unknown maneuvering mode scenarios, and has certain engineering application potential.

[0301] To analyze the computational complexity of the proposed method, a timing experiment was conducted using an STM32F407 microcontroller as the experimental platform and the double data type. When applying the proposed guidance law, only the strategy network needs to be executed. The timing function of the microcontroller indicates that the strategy network consumes approximately 0.32ms, demonstrating good real-time performance and reducing the computing power required onboard the missile.

[0302] Example 2:

[0303] Embodiment 2 provides a device for designing a reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning, comprising an uncertain adversarial model module, an imitation learning module, and a deep reinforcement learning module;

[0304] The uncertain confrontation model module establishes a three-dimensional uncertain confrontation model between the maneuvering target and the interceptor based on the interception collision triangle;

[0305] The imitation learning module uses the proportional guidance law as a strategy, utilizes the interactive environment to generate a large number of trajectory samples, establishes an intelligent terminal guidance model, and trains it using supervised learning methods to obtain an intelligent terminal guidance model that approximates the interception performance of proportional guidance.

[0306] The deep reinforcement learning module uses reinforcement learning methods, sets a reward function, and adopts the PPO algorithm to further train the intelligent terminal guidance model obtained by step imitation learning, thereby obtaining a new guidance law that exceeds the interception performance of proportional guidance.

[0307] The uncertain adversarial model module, the imitation learning module and the reinforcement learning module are implemented using the design method described in Example 1.

[0308] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A reinforcement learning guidance law design method for intercepting maneuvering targets based on imitation learning, characterized by: The following steps are included: S1: Based on the interception collision triangle, a three-dimensional uncertain confrontation model between the maneuvering target and the interceptor is established; S2: Using the proportional guidance law as a strategy, a large number of trajectory samples are generated in an interactive environment to establish an intelligent terminal guidance model. This model is trained using supervised learning methods to obtain an intelligent terminal guidance model that approaches the intercept performance of proportional guidance. S3: Using reinforcement learning methods, setting a reward function, and employing the PPO algorithm to further train the intelligent terminal guidance model learned in step S2, a new guidance law is obtained that surpasses the interception performance of proportional guidance. The specific operations of step S2 include the following steps: S201: Generate N using proportional guidance law and interactive environment IL Trajectory samples are sampled at intervals of time steps to determine the imitation learning training set in, is a tuple A collection of is the input of imitation learning, a PN =min{||a PN，0 ||,a max }·f unit (a PN，0 ) is the proportional guidance instruction, a PN，0 =NV D ×ω DM ; Where a max is the maximum overload of the interceptor, f unit The function of (·) is to solve the unit vector corresponding to the input vector, N is the navigation ratio; ω DM is the line of sight angular velocity vector, V D is the interceptor's velocity vector; S202: Design an imitation learning neural network model for training an intelligent terminal guidance model; S203: Using the imitation learning neural network model, performing imitation learning training on the intelligent terminal guidance model; The specific operation of step S3 includes the following steps: S301: Establishing Markov decision space model; The state space of the Markov decision space model is The action space is u t =[0,a max ] The reward function is r t ＝r s +r e in r s =k s exp[-(u t / g) 2 ] Where: R DM is the distance vector between the maneuvering target and the interceptor, is the velocity vector of the maneuvering target and interceptor approaching, ||ω DM || is the angular velocity of the line of sight; r s and r e are process rewards and terminal rewards respectively; r s is the reward value that reflects the degree of energy consumption. The lower the energy consumption, the greater the positive reward. e It is a reward that reflects the terminal miss distance, and a large positive reward is given when the interception is successful; k s is a hyperparameter greater than 0; R miss is the off-target amount; and is a hyperparameter related to the off-target amount, where Indicates the miss distance corresponding to the successful interception of the target; and is a hyperparameter related to the reward; S302: Constructing a proximal strategy optimization and pruning network model; S303: Utilize the proximal strategy to optimize the pruned network model and perform intensive training on the Markov decision space model in step S301.

2. The method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning according to claim 1 is characterized in that: The specific operation of step S1 includes the following steps: S101: Design an interactive environment for maneuvering target and interceptor confrontation training based on the collision triangle model; S102: Establishing a maneuvering model for a maneuvering target; S103: Initializing the interactive environment designed in step S101 according to the maneuvering model of the maneuvering target, and solving the initial velocity vector of the interceptor; S104: Establishing a terminal miss distance calculation model.

3. The method for designing a reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning according to claim 2 is characterized in that: The maneuvering model of the maneuvering target described in step S102 is expressed as Where N m Maneuvering overload for maneuvering target, a M is the maneuvering acceleration vector of the maneuvering target, g represents the acceleration due to gravity; and are the minimum and maximum values ​​of the maneuverable target start time respectively; e M is the unit vector of the maneuvering direction of the maneuvering target.

4. The method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning according to claim 3 is characterized in that: In step S103, the interceptor initial velocity vector V D，0 Expressed as In the formula, function T q The output of (·,·) is the rotation matrix, the first input is the rotation angle, and the second input is the rotation axis; V D，0 With plane C t The angle between plane C t For X DM,0 and V M,0 The plane formed; V D，0 In plane C t The projection and The angle between and H E,ψ The minimum and maximum values ​​of and They are The minimum and maximum values ​​of X DM,0 =X M,0 -X D,0 is the initial position vector of the maneuvering target relative to the interceptor, X M,0 is the initial position vector of the maneuvering target, V M,0 is the initial velocity vector of the maneuvering target, X D,0 is the interceptor initial position vector; To satisfy the interceptor velocity vector of zero miss distance; f unit The function of (·) is to find the unit vector corresponding to the input vector.

5. The method for designing a reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning according to claim 4 is characterized in that: The specific operation of step S202 includes the following steps: S2021: Design an imitation learning neural network model with an output of f IL : Where, ||a PN ||∈[0,a max ]; S2022: Design an imitation learning neural network model with input S2023: Design activation functions and loss functions for the imitation learning neural network model, where the hidden layer activation function uses the S-Sigmoid activation function and the output layer activation function is tanh. The loss function uses the MSE loss function. Where: M is the number of samples; N is the number of neural network output values; y i,j and are the predicted value and label value of the j-th output of the i-th sample respectively; S2024: Design a network parameter update method for imitation learning neural network models Where: N GPU is the number of GPUs participating in learning; and is the Jacobian matrix submatrix required to be solved by the i-th GPU each time and the error vector subvector of the predicted value relative to the label value, ζ i For J i,j and E i,j number of rows; For i, under certain circumstances, J i,j The number of m n =M·N is the number of all sample label values; the subscript l represents the iterative algebra; x∈R k×1 is a vector composed of network parameters, i.e., the weights and thresholds of each network layer, where k is the number of network parameters; I∈R k×k is the unit matrix; μ′ is the dynamically updated parameter. If updating the network parameters reduces the network prediction error, let μ′←μ′ / k μ′ , otherwise let μ′←μ′·k μ′ , calculate the network parameter update amount again, where k μ′ is a hyperparameter greater than 1; The Jacobian matrix J is expressed as The error vector E is expressed as 6. The method for designing a reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning according to claim 5 is characterized in that: The specific operation of step S203 includes the following steps: Calculate the loss function according to the loss function formula; Compute the Jacobian matrix and error matrix; According to the network parameter update method, the parameters θ of the imitation learning neural network are updated. IL to update.

7. The method for designing a reinforcement learning guidance law for intercepting a maneuvering target based on imitation learning according to claim 6, characterized in that: In step S302, The optimization objective function of the proximal strategy optimization pruning network model is Where: c VF and c s are hyperparameters; θ and ω represent the parameters of the policy network and the value network respectively; L CLIP (θ) is the optimization target of the policy network; L VF (ω) is the optimization target of the value network; is the entropy of the strategy; E is the expected value; The goal of the scaled value network is Where: Yes all The standard deviation of Introduce the gradient clipping technique to solve the gradient g of θ θ and the gradient g of ω ω Where: g max is the gradient clipping value; The network parameter update method is Where: α lr and β lr are the learning rates of the policy network and the value network respectively; The learning rate update method for the policy network and value network is Where: α lr,0 and β lr,0 are the initial learning rates of the policy network and the value network respectively; t e is the current training generation; N epoch is the maximum number of training generations.

8. A device for designing a reinforcement learning guidance law for intercepting maneuvering targets based on imitation learning, characterized in that: Includes uncertain adversarial model module, imitation learning module and deep reinforcement learning module; The uncertain confrontation model module establishes a three-dimensional uncertain confrontation model between the maneuvering target and the interceptor based on the interception collision triangle; The imitation learning module uses the proportional guidance law as a strategy, utilizes the interactive environment to generate a large number of trajectory samples, establishes an intelligent terminal guidance model, and trains it using supervised learning methods to obtain an intelligent terminal guidance model that approximates the interception performance of proportional guidance. The deep reinforcement learning module uses reinforcement learning methods, sets a reward function, and adopts the PPO algorithm to further train the intelligent terminal guidance model obtained by step imitation learning, thereby obtaining a new guidance law that exceeds the interception performance of proportional guidance. The uncertain adversarial model module, the imitation learning module and the reinforcement learning module are implemented using the design method described in any one of claims 1 to 7.

Images