Guidance method with impact angle constraint based on generative adversarial imitation learning

Abstract

The application relates to a trajectory guidance method based on a generative adversarial imitation learning, establishes an adversarial learning framework, trains a discriminator by using expert data to generate a reward model, thereby solving a reward function design problem; the reward value is used to guide the exploration and learning process of an agent, the agent is prevented from exploring a poor decision space, and learning efficiency is improved; an interaction data is generated by using a generator to interact with an environment, thereby driving strategy improvement and updating; a generator and a discriminator are alternately trained, the discriminator and the generator are in dynamic game, a sample with high diversity is generated, the coverage range of data is increased, and the adaptability of the agent in a new scene is improved. The method does not depend on accurate measurement of the remaining flight time, and therefore has better generalization performance.

Description

Technical Field

[0001] This invention belongs to the field of reinforcement learning guidance technology, and relates to a guidance method with landing angle constraints based on generative adversarial imitation learning. Background Technology

[0002] The primary purpose of traditional terminal guidance technology is to generate appropriate guidance commands to minimize the missile's terminal miss distance. To enhance the destructive effect of a collision while ensuring accurate target hits, modern military applications also impose terminal requirements on the collision angle. Numerous research results exist on guidance methods constrained by the angle of impact, which can be categorized into model-based and data-based methods. Model-based methods involve establishing a dynamic model, combining modern control theory and other methods to design decision algorithms, and obtaining the optimal solution by optimizing the objective function. The traditional proportional guidance law is the most typical guidance method, but it cannot directly solve the angle-of-impact constraint problem. Some researchers have added a time-varying bias term to the traditional proportional guidance law to eliminate the angle-of-impact error; others have designed a proportional guidance system where the proportional coefficient varies with time. Furthermore, some researchers have derived the optimal angle-of-impact constraint guidance law based on energy-optimal performance indicators. However, these methods often require accurate estimation of the remaining flight time and have significant limitations in dealing with model and environmental uncertainties. In addition, there are numerical optimization methods that achieve the desired effect by optimizing the objective function. However, this type of method has the following drawbacks: it requires accurate modeling of the aircraft's dynamics; it is highly dependent on the initial values, and the choice of initial values ​​may have a significant impact on the results; it requires solving the objective function multiple times, resulting in a large amount of computation and a long running time.

[0003] In recent years, with the development of artificial intelligence technology, data-driven guidance methods have received increasing attention. These methods can be further divided into label-based and interaction-based intelligent methods based on the data source. Label-based intelligent methods mainly employ deep learning algorithms to learn the mapping relationship between flight state and guidance commands or partial parameter values ​​from existing guidance trajectory data. This allows for dynamic adjustment of guidance commands based on flight state to achieve the guidance task. For example, a functional relationship between the optimal guidance law expression and the relative state of the missile and target has been established by training a neural network. Other researchers use deep learning to calculate flight time error and design correction guidance laws to converge the flight time error. The advantage of this type of method is that it can fully utilize existing knowledge and experience to quickly complete tasks. However, this method also has some disadvantages, such as requiring a large amount of labeled data to train the model, the problem of overfitting, and difficulty in extending existing knowledge to new environments. Furthermore, the predictions obtained by deep learning models often rely only on the current state and actions, without modeling and optimizing the entire decision sequence, making it difficult to fully utilize sequence information. Therefore, errors gradually accumulate as the number of decisions increases.

[0004] Intelligent methods based on interactive data primarily employ reinforcement learning algorithms, such as Q-learning and DDPG. These methods learn guidance strategies through data generated from the interaction between the agent and its environment. The advantage of this approach is that it can continuously improve through trial and error, cope with complex and changing environments, and possess a certain degree of generalization ability. Applying reinforcement learning methods to optimize guidance is an effective means to improve guidance accuracy and robustness, applicable to various constraints and real-world scenarios. However, the challenges of applying reinforcement learning to guidance under corner constraints are as follows: traditional reinforcement learning methods typically rely on manually designed reward functions, which are difficult to design in complex tasks and are easily influenced by human experience and limitations; reinforcement learning requires a large amount of interactive data and computational resources, and needs to address the exploration-exploitation balance problem, which limits the learning efficiency and stability of the algorithm; the training process is complex, takes a long time to converge, and is prone to getting trapped in local optima.

[0005] To improve the learning efficiency of reinforcement learning tasks, a feasible approach is to leverage existing expert data to enhance prior knowledge. However, directly incorporating expert data into training may lead to an algorithm's over-reliance on expert data and specific solutions, thereby losing its ability to learn autonomously, resulting in overfitting and a lack of exploration. Therefore, Generative Adversarial Imitation Learning (GAIL) was proposed. GAIL is a hybrid data-driven reinforcement learning algorithm that trains by combining expert data and interaction data to improve the agent's performance on specific tasks. Summary of the Invention

[0006] Technical problems to be solved

[0007] To overcome the shortcomings of existing technologies, this invention proposes a landing angle-constrained guidance method based on generative adversarial imitation learning, a deep reinforcement learning guidance method that combines expert data and interactive data based on the GAIL algorithm. This method establishes an adversarial learning framework, using expert data to train a discriminator to generate a reward model, thus solving the reward function design problem; it uses reward values ​​to guide the agent's exploration and learning process, avoiding the agent from exploring undesirable decision spaces and improving learning efficiency; it utilizes the interaction between the generator and the environment to produce interactive data, thereby driving policy improvement and updates; and it employs an alternating training method for the generator and discriminator, placing them in a dynamic game, generating highly diverse samples, increasing data coverage, and improving the agent's adaptability in new scenarios. This method does not rely on precise measurement of remaining flight time and does not require additional design of the neural network structure, thus exhibiting better generalization performance.

[0008] Technical solution

[0009] A guidance method with landing angle constraints based on generative adversarial imitation learning is characterized by the following steps:

[0010] Step 1: Establish a three-dimensional dynamic model of the missile without thrust in the ground coordinate system;

[0011]

[0012] Where D is the drag force experienced by the missile during flight, which is a function of velocity, m is the mass of the missile, and a y ,a z These are the missile's accelerations in the pitch and yaw directions, respectively.

[0013] Step 2: Generate a dataset using the optimal guidance law:

[0014] The normal overload n is learned using a reinforcement learning algorithm. y A dataset is required for guidance; the dataset consists of trajectory data generated using optimal guidance law.

[0015]

[0016] Where: the velocity vector V during the flight of the aircraft i The optimal guidance law is that the rotational angular velocity is proportional to the rotational angular velocity of the target's line of sight. The remaining flight time can be determined according to... Direct calculation; the terminal angle is estimated using the following formula:

[0017]

[0018] Where: η is the initial missile lead angle, q0 is the initial line-of-sight angle, and N is the proportional guidance coefficient; the trajectory data includes a series of state and action pairs;

[0019] Step 3: Design the input states and outputs:

[0020] Step 3-1: The state-space expression represents the relative relationship between the attacking missile and the target. It not only describes the relative distance and line-of-sight angle, but the existence of the rate of change of relative distance also makes the relationship between the two sides more distinguishable. Moreover, the existence of the difference in trajectory inclination angle and the remaining flight time optimizes the missile's impact angle constraint and flight time. The expression is:

[0021] S=[R,dR,q,θ t -θ,t go (3)

[0022] These include relative distance R, relative distance change rate dR, projectile-target line-of-sight angle q, projectile-target line-of-sight angular rate dq, and the difference θ between the current trajectory inclination angles. t -θ, the time t required for the missile to hit the target while guided by proportional guidance at its current position using a set proportional coefficient. go ;

[0023] Step 3-2: Calculate the normal overload n using the reinforcement learning algorithm obtained in Step 2. y As an additional control, the normal overload n calculated by the traditional proportional guide is... yp As a conventional control measure, the normal control overload ultimately manifested in the missile is derived by adding the two, as expressed below:

[0024] n actual =n p +n RL (4)

[0025]

[0026] Where: n p Indicates the required overload of the aircraft; K represents the navigation ratio; θ i represents the trajectory tilt angle of the aircraft; q represents the line-of-sight azimuth angle between the missile and the target; the ultimate goal of learning is to narrow the gap between the reinforcement learning strategy and the expert strategy, that is, the ideal learning effect is n. actual =n*;

[0027] Step 4: Train the discriminator:

[0028] The training process uses expert trajectories and generator trajectories as input data and aims to minimize classification error.

[0029] The discriminator loss function of GAIL is expressed as:

[0030]

[0031] Where, π E It is an expert strategy, f θ is the policy network of the generator, and p is the transition probability distribution of the environment;

[0032] Step 5: Utilize the PPO learning strategy for learning:

[0033] Pre-training should be done before learning;

[0034] Interacting with the environment generates interactive data; the agent selects an action based on the current state and executes it; the environment receives the agent's action and returns a new state, and stores the state, behavior, and new state until a certain number are stored;

[0035] The reward model is used to output reward and penalty values ​​for state-action pairs. For each state-action pair, the discriminator outputs a probability value, which represents the probability that the state-action pair comes from the expert. Then, the agent reward is calculated based on this probability value. The size of the agent reward depends on the discriminator's ability to distinguish between the generator and the expert.

[0036] Update strategy: Use the PPO algorithm as the generator, with the generator loss function as follows:

[0037]

[0038] Among them, f θ It is the policy network of the generator, ρ0 is the initial state distribution, and D(s) t ,a t ) is the output of the discriminator, representing the trajectory (s) t ,a t ) is the probability derived from the expert's trajectory; This represents the desired operation, where s0 is the initial state and a t In state s t The action output by the generator;

[0039] The loss function is optimized by using the policy gradient generated by the PPO function:

[0040]

[0041] in, It's a ratio. ∈ is the advantage estimate, and ∈ is the hyperparameter of the cutoff range.

[0042] The ground coordinate system is as follows: with the projection of the current position on the ground as the origin A, the Ax axis points from the missile to the target, the Ay axis is perpendicular to the ground and points upward, and the Az axis conforms to the right-hand rule; in the movement of the missile in the ground coordinate system, x, y, z are the position coordinates of the missile, V is the speed of the missile, and θ, ψ are the trajectory inclination angle and trajectory deflection angle of the missile, respectively.

[0043] The navigation ratio K ranges from 3 to 6.

[0044] The end of each loop in step 3-2 includes the following three cases:

[0045] (1) If the accuracy requirement is not met within a given time domain length, the loop ends.

[0046] (2) If the vertical coordinate is less than 0, it means that the ground is touched in reality, and the cycle ends.

[0047] (3) If the L2 norm of the relative position vector with respect to the target is less than 10, it means that the cycle is successful and the cycle ends.

[0048] The loss function of the discriminator in step 4 includes the difference between two expected values; the first expected value represents the probability that the state-action under the expert strategy is judged as a positive sample by the discriminator, and the second expected value represents the probability that the state-action under the generator strategy is judged as a negative sample by the discriminator.

[0049] The reward model outputs reward and penalty values ​​for state behaviors. The reward function consists of two parts: a process reward function and a terminal reward function.

[0050] r(s,a)=l f (s,a)+l(s,a) (9)

[0051] The process reward is generated using the output of the discriminator model. For each state-action pair, the discriminator outputs a probability value, which represents the probability that the state-action pair comes from an expert.

[0052] The generator can generate trajectories that closely resemble expert trajectories, and the discriminator's output will be closer to 1. At this point, L... GAIL (f θ The value of L approaches 0; conversely, if the generator's trajectory differs greatly from the expert's trajectory, the discriminator's output will be even closer to 0, at which point L... GAIL (f θ The value of ) will increase; by minimizing the loss function, the generator gradually optimizes its strategy, generating trajectories that are closer to the expert's trajectory; the agent reward value depends on the discriminator's ability to distinguish between the generator and the expert, expressed as:

[0053] l(s,a)=-log(D(s,a)) (10)

[0054] l f (s,a) is designed as the terminal reward m, based on the task completion status after a certain round.

[0055] The process reward is provided by the trained discriminator; the terminal reward is designed as follows:

[0056]

[0057] The objective function of the PPO is:

[0058]

[0059] Among them, s i and a i State-action pairs from expert datasets, f θ (s i ) indicates that the state s i After inputting into the neural network and the expert dataset, a i The difference is used as the loss function; It is a ratio used to measure whether a new strategy is better than an old strategy; It is the dominant function; ∈ is the hyperparameter of the cutoff range; H(π) θ ) represents the entropy of the current strategy.

[0060] The advantage function is the difference between the expected reward of taking a certain action in the current state and the expected reward of only taking the average action, expressed as:

[0061] A(s,a)=Q(s,a)-V(s;θ) c (13)

[0062] Where Q(s,a) is the expected reward for taking a certain action in the current state, and V(s; θ) is the expected reward for taking a certain action in the current state. c The expected return of taking only average actions.

[0063] Beneficial effects

[0064] This invention proposes a landing angle-constrained guidance method based on generative adversarial imitation learning. It establishes an adversarial learning framework, using expert data to train a discriminator to generate a reward model, thus solving the reward function design problem. The reward value guides the agent's exploration and learning process, preventing the agent from exploring undesirable decision spaces and improving learning efficiency. Interactive data generated by the generator's interaction with the environment drives policy improvement and updates. Alternating training of the generator and discriminator puts them in a dynamic game, generating highly diverse samples, increasing data coverage, and improving the agent's adaptability in new scenarios. This method does not rely on precise measurement of remaining flight time, thus exhibiting better generalization performance. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of the motion of the present invention;

[0066] Figure 2 This is a schematic diagram of the expert trajectory and reinforcement learning exploration space of this invention;

[0067] Figure 3 This invention provides an adversarial learning framework;

[0068] Figure 4 Guidance performance of the present invention at different desired landing angles:

[0069] (a) Longitudinal overload command

[0070] (b) Lateral overload command

[0071] (c) Ballistic inclination angle

[0072] (d) Flight speed

[0073] (e) Flight trajectory

[0074] (f) Flight trajectory

[0075] Figure 5 The guidance performance of the method and optimal guidance law in this invention is as follows:

[0076] (g) Longitudinal overload command

[0077] (h) Lateral overload command

[0078] (i) Ballistic inclination angle

[0079] (j) Flight speed

[0080] (k) Flight trajectory

[0081] (l) Flight trajectory

[0082] Figure 6 The spatial trajectory of 100 experiments in this invention. Detailed Implementation

[0083] The present invention will now be further described in conjunction with the embodiments and accompanying drawings:

[0084] The technical problem solved by this invention is to provide a deep reinforcement learning guidance method that combines expert data and interactive data to address the shortcomings of existing technologies, thereby significantly improving the convergence ability of reinforcement learning algorithms in guidance problems with complex constraints.

[0085] The technical solution adopted by the embodiments of the present invention to solve its technical problem includes the following steps:

[0086] 1. A guidance method with landing angle constraints based on generative adversarial imitation learning, comprising the following steps:

[0087] Step 1: Establish the coordinate system and the three-degree-of-freedom dynamic equations, such as... Figure 1 As shown;

[0088] Step 1-1: Taking the projection of the missile's current position onto the ground as the origin A, the Ax axis points from the missile towards the target, the Ay axis is perpendicular to the ground and points upwards, and the Az axis follows the right-hand rule. In the missile's motion in the ground coordinate system, x, y, and z are the missile's position coordinates, V is the missile's velocity, and θ and ψ are the missile's trajectory inclination and deflection angles, respectively.

[0089] Step 1-2: The thrustless three-dimensional particle dynamics model of the missile in the ground coordinate system is as follows:

[0090]

[0091] Where D is the drag force experienced by the missile during flight, which is a function of velocity, m is the mass of the missile, and a y ,a z These represent the missile's acceleration in the pitch and yaw directions, respectively.

[0092] Drag is a function of velocity; the expression for aerodynamic drag is:

[0093]

[0094] In the formula: ρ is the atmospheric density; S m Let be the reference area of ​​the missile; C be the zero-lift drag coefficient; atmospheric density and drag coefficient are functions of altitude or Mach number, and their specific values ​​can be obtained by interpolation from tables. Under the constraint of the angle of impact, the desired missile terminal state is:

[0095] θ(t f )=θ t

[0096] x(t f )=xf

[0097] y(t f )=y f

[0098] z(t f )=z f

[0099] In the formula: t f For terminal time; θ t The expected landing angle; (x) f ,y f ,z f () is the target location.

[0100] Step 2: Generate an expert dataset using the optimal guidance law;

[0101] Proportional guidance refers to the method of guiding the velocity vector V during the flight of an aircraft. i A guidance method in which the rotational angular velocity is proportional to the rotational angular velocity of the target's line of sight, expressed as follows:

[0102]

[0103] In the above formula, n represents the required overload of the aircraft; K represents the navigation ratio, which is generally taken as 3 to 6; θ i The trajectory inclination angle of the aircraft is represented by 'θ'; 'q' represents the line-of-sight azimuth angle between the missile and the target. Missile guidance commands are calculated from the relative kinematic equations between the attacking missile and the target.

[0104] Since there is a one-to-one correspondence between the terminal line-of-sight angle and the terminal trajectory inclination angle, it can be equivalent to controlling the terminal line-of-sight angle. The missile's lead angle η will gradually converge to 0 as the missile-target distance decreases. Therefore, it can be seen that...

[0105]

[0106] For attacks on fixed targets, the collision angle is determined by the initial line-of-sight angle q0, the initial missile lead angle η, and the proportional guidance coefficient N. The optimal guidance method theoretically possesses optimality and is easy to handle various constraints, such as minimum energy and specific impact angles. Its expression is:

[0107]

[0108] The remaining flight time can be determined according to Direct computation: The expert dataset comprises trajectory data generated according to (4) guidance. The expert dataset consists of trajectories generated by expert policies. Each trajectory contains a series of state-action pairs. These trajectories are generated by an expert policy that is already capable of performing the task. The discriminator learns from the expert data to distinguish between samples generated by the generator and real expert trajectories. Inputs and outputs will be given in the next step.

[0109] Step 3: Design the input states and outputs;

[0110] Step 3-1: Generally, the relative information between the missile and the target mainly consists of four states: relative distance R, relative distance change rate dR, missile-target line-of-sight angle q, and missile-target line-of-sight angular rate dq. Considering the landing constraint, this paper, after extensive simulation verification, adds θ to the original state variables. t -θ and t go , where θ t The trajectory inclination angle θ represents the missile's angle of impact constraint. t -θ represents the difference between the landing angle constraint and the current trajectory inclination angle, t go θ represents the time required for a missile to hit its target from its current position using proportional guidance with a certain scaling factor. This has been verified. t -θ and t go The addition of this feature allows the missile to hit the target in a shorter time while satisfying the angle of impact constraint. The final state-space expression is selected as follows:

[0111] S=[R,dR,q,θ t -θ,t go (1-5)

[0112] The above formula can represent the relative relationship between the attack missile and the target. It not only describes the relative distance and line-of-sight angle, but the existence of the relative distance change rate can also make the relationship between the two sides more distinguishable. Moreover, the existence of the ballistic inclination angle difference and the remaining flight time optimizes the missile's impact angle constraint and flight time.

[0113] Step 3-2: Calculate the normal overload n using a reinforcement learning algorithm. y As an additional control, the normal overload n calculated by the traditional proportional guide is... yp As a conventional control measure, the normal control overload ultimately manifested in the missile is derived by adding the two, as expressed below:

[0114] n actual =n p +n RL (1-6)

[0115] The ultimate goal of learning is to narrow the gap between reinforcement learning strategies and expert strategies, that is, the ideal learning outcome is n. actual=n*.

[0116] The end of each loop includes the following three cases:

[0117] (1) If the accuracy requirement is not met within a given time domain length, then the loop ends.

[0118] (2) If the vertical coordinate is less than 0, it means that the ground is touched in reality, and the cycle ends.

[0119] (3) If the L2 norm of the relative position vector with respect to the target is less than 10, it means that the cycle is successful and the cycle ends.

[0120] Step 4: Design the discriminator;

[0121] The process reward is generated using a discriminator model within a generative adversarial network. During discriminator training, expert trajectories and generator trajectories are used as input data, with the optimization objective being to minimize the classification error. GAIL's discriminator loss function is expressed as follows:

[0122]

[0123] Where, π E It is an expert strategy, f θ This is the generator's policy network, where p is the transition probability distribution of the environment. The discriminator's loss function includes the difference between two expected values. The first expected value represents the probability that the state-action under the expert policy is classified as a positive sample by the discriminator, and the second expected value represents the probability that the state-action under the generator policy is classified as a negative sample by the discriminator. By minimizing the discriminator's loss function, the discriminator can better distinguish between expert samples and generator samples. A reward model is used to output reward and penalty values ​​for state-action pairs. The reward function consists of two parts: a process reward function and a terminal reward function.

[0124] r(s,a)=l f (s,a)+l(s,a) (8)

[0125] The process reward is generated using the discriminator model output. For each state-action pair, the discriminator outputs a probability value representing the probability that the state-action pair originates from an expert. For example, if the generator can generate a trajectory close to the expert's trajectory, the discriminator's output will be closer to 1. Conversely, if the generator's trajectory differs significantly from the expert's trajectory, the output will be closer to 0. The proxy reward is then calculated based on this probability value. The magnitude of the proxy reward depends on the discriminator's ability to distinguish between the generator and the expert, and can be expressed as...

[0126] l(s,a)=-log(D(s,a)) (9)

[0127] l f(s,a) serves as the terminal reward, which can be manually designed based on task completion at the end of a round. The process reward is provided by the trained discriminator, and the terminal reward is designed as follows:

[0128]

[0129] Step 5: Utilize the PPO learning strategy;

[0130] Step 5-1: Pre-training is the first step during learning. Pre-training can provide good initial parameters, speed up the model's convergence, and shorten the training time.

[0131] Step 5-2: Interact with the environment to generate interaction data. The agent selects an action based on the current state and executes it. The environment receives the agent's action and returns a new state, storing the state, action, and new state until a certain number are stored.

[0132] Step 5-3: Output reward / penalty values ​​for state-action pairs using a reward model. For each state-action pair, the discriminator outputs a probability value, representing the probability that the state-action pair comes from an expert. The agent reward is then calculated based on this probability value; the magnitude of the agent reward depends on the discriminator's ability to distinguish between the generator and the expert.

[0133] Step 5-4: Update the strategy. Use the PPO algorithm as the generator; the PPO objective function is...

[0134] L ppo (θ)=E[min(r t (θ)A t ,clip(r t (θ), 1-∈, 1+∈)A t )-βH(πθ)] (11)

[0135] in, It is a ratio used to measure whether a new strategy is better than an old strategy. The advantage function refers to the degree to which taking a certain action in the current state is better or worse than the average performance. Specifically, the advantage function can be seen as the difference between the expected reward of taking a certain action in the current state and the expected reward of only taking the average action, expressed as:

[0136] A(s,a)=Q(s,a)-V(s;θ) c (12)

[0137] ∈ is a hyperparameter representing the truncation range. By using probability ratio truncation and value function truncation to limit the range of policy updates and constrain the magnitude of policy updates, we can avoid overly drastic policy updates, thereby improving the stability and convergence speed of the algorithm. H(π) θThe entropy of the current policy is represented by , used to promote policy exploration. This method can combine pre-training to improve training performance. To balance the pre-training loss and reinforcement learning loss, this paper uses a weighted loss function. Specifically, a weight parameter is used to control the relative weights of the two loss functions. During training, both the MSE loss and the reinforcement learning loss are optimized simultaneously. The advantage of this is that during reinforcement learning, the model does not completely discard pre-trained knowledge, but combines it to achieve better performance. This is represented as...

[0138]

[0139] L = L ppo +λL MSE (14)

[0140] Among them, a i This represents the corresponding expert behavior. State s i and a i From expert dataset, f θ (s i ) represents the neural network's response to state s i The predicted output.

[0141] like Figure 2 As shown, by learning from the trajectories and behaviors of experts, the large number of trial-and-error processes in reinforcement learning are avoided, enabling the agent to have more reasonable behavioral strategies when exploring the decision space. Furthermore, this paper introduces a designed terminal reward function, which alters the value distribution of states in sequential decision-making, allowing the agent to better adapt to task requirements and still possess the ability to autonomously learn and optimize strategies even when expert data is incomplete.

[0142] In guidance missions Figure 3 As shown, the agent generates a guidance command distribution based on the current network parameters and state observation input. A guidance command is sampled and input into the missile dynamics model to adjust the missile's motion state, change the target state, and obtain the state observation for the next moment. Trajectory data of the agent, including current state, actions, and rewards, is acquired through a simulation environment and stored in an experience pool. The discriminator uses the data in the experience pool and the dataset to distinguish between the guidance command generated by the agent and the guidance command generated by the optimal guidance law, determining which is optimal. Based on the discriminator's feedback, a reward / penalty value is output using the reward model to construct interactive data. The PPO algorithm is used to adjust the network parameters so that the guidance command generated by the agent is closer to the guidance command generated by the optimal guidance law. The above steps are repeated until the network converges and the generated guidance command is consistent with the guidance command generated by the optimal guidance law, thus achieving intelligent guidance learning.

[0143] This embodiment performs a landing angle constraint braking simulation. All simulations are executed on a desktop computer equipped with an Intel i5-7200U processor with a main frequency of 2.50GHz. All programs are compiled and run in the PyCharm environment.

[0144] To verify the adaptability of this invention, a simulation experiment was conducted on missile guidance during the testing process, and the results were compared with traditional deep learning methods and expert strategies. First, a missile guidance simulation environment was built, including elements such as missile dynamics model, missile-target state, and observation state. The missile had no thrust during the terminal guidance phase, meaning its mass remained constant at 1601.78 kg, and the reference area of ​​the missile during the terminal guidance phase was 0.282743 m². 2 Then, based on the optimal guidance law, the simulation environment was solved, yielding 800,000 data points, which were used to construct an expert dataset. On this basis, the MSE loss function was used to train the gradient update of the deep neural network to achieve autonomous learning for missile guidance missions. Finally, the hyperparameters of each part of the neural network were designed.

[0145] The training efficiency using the method presented in this paper is greatly improved. Compared to some reinforcement learning methods that require up to 40,000 training iterations, this method reduces the number of training iterations to 600. To further evaluate this method, this paper tests the guidance performance of a missile under different landing angle constraints (50°, 60°, and 70°) with the same initial position [42, 26000, 0.5], and analyzes the results. Figure 4 The figure shows the missile's trajectory, guidance command diagram, angle of impact variation, and velocity variation. The results demonstrate that our method achieves good guidance performance under different angle of impact constraints, and the missile's guidance accuracy improves with increasing angle of impact constraints. Furthermore, the changes in the missile's trajectory, velocity, and angle of impact are as expected, further validating the effectiveness of our method.

[0146] To compare the effectiveness of different methods, we compared our proposed method with the optimal guidance law, designing a desired landing angle of -55°. The results are as follows: Figure 5 As shown, the two methods are largely consistent in terms of trajectory, velocity, and angle changes, indicating that the proposed method has good simulation performance. Specifically, in terms of trajectory, the trajectory obtained by the proposed method has a high degree of overlap with the trajectory obtained from the expert data, indicating that it can accurately simulate the expert's guidance strategy. In terms of velocity and angle changes, the curve trends obtained by the proposed method and the expert data are basically consistent, indicating that the proposed method can effectively control the missile's velocity and attitude angle.

[0147] To further verify the effectiveness of the proposed method, 100 simulations were performed under different initial positions and angle constraints. Finally, the guidance accuracy was statistically analyzed (the simulated trajectories of the proposed method are shown in the figure). Figure 6 Data shows that the proposed method performs well in missile guidance. It achieves similar results to traditional optimal guidance law methods, and even outperforms them in some timeframes. While the proposed method is slightly less accurate than optimal proportional guidance, with a larger difference in maximum deviation, the differences in minimum and average deviation are not significant. It is important to note that this method is a hybrid data-driven approach, capable of obtaining excellent guidance strategies through learning from expert data without requiring precise mathematical models. Furthermore, it avoids reliance on precise measurements of remaining time, making it more universal and adaptable than traditional optimal proportional guidance methods.

[0148] Finally, this paper compares the proposed method with traditional deep learning methods. The results show that, by comparing the performance of the proposed method and traditional deep learning methods in missile guidance, the proposed method outperforms the deep learning method in terms of maximum, minimum, and mean deviations. Specifically, the deviation range of the proposed method is 0.001–0.408, while the deviation of the deep learning method is 0.007–1.24, indicating that the proposed method has higher accuracy in controlling missile deviations. Furthermore, the mean deviation of the proposed method is 0.049, while the mean deviation of the deep learning method is 0.423, indicating that the proposed method has better stability and reliability in controlling missile deviations. In conclusion, the proposed method exhibits better performance than traditional deep learning methods and demonstrates higher feasibility and reliability in practical applications.

[0149] In summary, the experimental simulation results demonstrate that the method can satisfy various initial conditions and angle constraints in guidance problems. It better adapts to real-world scenarios in terms of sample efficiency and strategy stability, thereby improving the real-time performance and speed of the guidance method. Compared to deep learning methods, the proposed method exhibits better performance and generalization ability in guidance problems, and can better learn strategies from data of optimal guidance law methods. Therefore, the proposed method can be considered an effective missile guidance strategy learning method with broad application prospects.

Claims

1. A guidance method with landing angle constraints based on generative adversarial imitation learning, characterized in that... The steps are as follows: Step 1: Establish a three-dimensional dynamic model of the missile without thrust in the ground coordinate system; in, The drag force experienced by the missile during flight is a function of velocity. For missile quality, These are the missile's accelerations in the pitch and yaw directions, respectively. For the speed of the missile, These are the missile's trajectory inclination angle and trajectory deflection angle, respectively. Step 2: Generate a dataset using the optimal guidance law: Learn normal overload using reinforcement learning algorithms. A dataset is required for guidance; the dataset consists of trajectory data generated using optimal guidance law. (1) Where: the velocity vector during the flight of the aircraft The optimal guidance law is that the rotational angular velocity is proportional to the rotational angular velocity of the target's line of sight. The remaining flight time can be determined according to... Direct calculation; the terminal angle is estimated using the following formula: (2) in: This is the initial missile lead angle. The initial viewing angle, The proportional guidance coefficient; the trajectory data contains a series of state-action pairs; Step 3: Design the input states and outputs: Step 3-1: The state-space expression represents the relative relationship between the attacking missile and the target. It not only describes the relative distance and line-of-sight angle, but the existence of the rate of change of relative distance makes the relationship between the two sides more distinguishable. Moreover, the existence of the difference in trajectory inclination angle and the remaining flight time optimizes the missile's impact angle constraint and flight time. The expression is: (3) This includes relative distance Rate of change of relative distance , Bullet eye line of sight Angular velocity of bullet eye line The difference between the current trajectory inclination angle The time required for a missile to hit its target from its current position using proportional guidance with a set ratio. ; Step 3-2: Calculate the normal overload using the reinforcement learning algorithm from Step 2. As an additional control, the normal overload calculated by the traditional proportional guide is... As a control mechanism, the normal control overload ultimately manifested in the missile is derived by adding the two, as expressed below: (4) (5) in: Indicates the required overload of the aircraft; Indicates the navigation ratio; Indicates the trajectory inclination angle of the aircraft; This indicates the line-of-sight angle; the ultimate goal of learning is to narrow the gap between reinforcement learning strategies and expert strategies, that is, the ideal learning outcome is... ; Step 4: Train the discriminator: The training process uses expert trajectories and generator trajectories as input data and aims to minimize classification error. The discriminator loss function of GAIL is expressed as: (6) in, It is an expert strategy. It is the generator's policy network. It is the probability distribution of environmental transitions; Step 5: Utilize the PPO learning strategy for learning: Pre-training should be done before learning; Interacting with the environment generates interactive data; the agent selects an action based on the current state and executes it; the environment receives the agent's action and returns a new state, and stores the state, behavior, and new state until a certain number are stored; The reward model is used to output reward and penalty values ​​for state-action pairs. For each state-action pair, the discriminator outputs a probability value, which represents the probability that the state-action pair comes from the expert. Then, the agent reward is calculated based on this probability value. The size of the agent reward depends on the discriminator's ability to distinguish between the generator and the expert. Update strategy: Use the PPO algorithm as the generator, with the generator loss function as follows: (7) in, It is the generator's policy network. It is the initial state distribution. It is the output of the discriminator, representing the trajectory. It is a probability derived from the expert's trajectory; Indicates the desired operation. It is the initial state. It is in state The action output by the generator; The loss function is optimized by using the policy gradient generated by the PPO function: (8) in, It's a ratio. It is an advantage estimate. It is a hyperparameter for the cutoff range.

2. The guidance method with landing angle constraint based on generative adversarial imitation learning according to claim 1, characterized in that: The ground coordinate system is defined with the projection of the current location onto the ground as the origin. , The axis is pointed from the missile to the target. The axis is perpendicular to the ground and points upwards. The axis conforms to the right-hand rule; during the missile's motion in the ground coordinate system... These are the missile's position coordinates. For the speed of the missile, These are the missile's trajectory inclination angle and trajectory deflection angle, respectively.

3. The guidance method with landing angle constraint based on generative adversarial imitation learning according to claim 1, characterized in that: The navigation ratio Values ​​range from 3 to 6.

4. The guidance method with landing angle constraint based on generative adversarial imitation learning according to claim 1, characterized in that: The end of each loop in step 3-2 includes the following three cases: (1) If the accuracy requirement is not met within a given time domain length, the loop ends. (2) If the vertical coordinate is less than 0, it means that the ground is touched in reality, and the cycle ends; (3) If the L2 norm of the relative position vector with respect to the target is less than 10, it means that the cycle is successful and the cycle ends.

5. The guidance method with landing angle constraint based on generative adversarial imitation learning according to claim 1, characterized in that: The loss function of the discriminator in step 4 includes the difference between two expected values; the first expected value represents the probability that the state-action under the expert strategy is judged as a positive sample by the discriminator, and the second expected value represents the probability that the state-action under the generator strategy is judged as a negative sample by the discriminator.

6. The guidance method with landing angle constraint based on generative adversarial imitation learning according to claim 1, characterized in that: The reward model outputs reward and penalty values ​​for state behaviors. The reward function consists of two parts: a process reward function and a terminal reward function. (9) The process reward is generated using the output of the discriminator model. For each state-action pair, the discriminator outputs a probability value, which represents the probability that the state-action pair comes from an expert. For terminal rewards, Rewards for agents.

7. The guidance method with landing angle constraint based on generative adversarial imitation learning according to claim 1, characterized in that: The generator can generate trajectories that closely resemble expert trajectories, and the discriminator's output will be closer to 1. The value of approaches 0; conversely, if the generator's trajectory differs greatly from the expert's trajectory, the discriminator's output will be even closer to 0. The value will increase; by minimizing the loss function, the generator gradually optimizes its strategy, generating trajectories that are closer to the expert's trajectory; the agent reward value depends on the discriminator's ability to distinguish between the generator and the expert, expressed as: (10) The terminal reward m is designed to be based on the task completion status after a certain round.

8. The guidance method with landing angle constraint based on generative adversarial imitation learning according to claim 1, characterized in that: The process reward is provided by the trained discriminator; the terminal reward is designed as follows: (11)。 9. The guidance method with landing angle constraint based on generative adversarial imitation learning according to claim 1, characterized in that: The objective function of the PPO is: in, and State-action pairs from expert datasets Indicates the state After inputting into the neural network and the expert dataset The difference is used as the loss function; It is a ratio used to measure whether a new strategy is better than an old strategy; It is the dominant function; It is a hyperparameter of the cutoff range; This represents the entropy of the current strategy.

10. The guidance method with landing angle constraint based on generative adversarial imitation learning according to claim 9, characterized in that: The advantage function is the difference between the expected reward of taking a certain action in the current state and the expected reward of only taking the average action, expressed as: (13) in, The expected reward for taking a certain action in the current state. The expected return of taking only average actions.

Images