Missile avoidance and rotary enemy locking maneuvering strategy generation method

Abstract

The invention discloses a missile avoidance and rotary enemy locking maneuvering strategy generation method, which comprises the following steps: S1, establishing a kinematics model and a dynamics model of an airplane and a missile based on Newton's law of motion, and simulating flight tracks and states of the airplane and the missile; s2, based on the kinematics model and the dynamics model, establishing a 1v1 beyond-visual-range air combat environment; s3, designing a state space, an action space and a reward function of deep reinforcement learning for the quadruple of deep reinforcement learning, and integrating the three into an over-the-horizon air combat environment to complete the design of a state transfer function; s4, simulating a warplane by using an intelligent agent, and performing repeated simulation confrontation by using the deep reinforcement learning framework to optimize and adjust a strategy and improve the survivability and combat efficiency in an air combat; and through continuous iterative training of the reinforcement learning model, the intelligent agent autonomously learns an effective maneuvering strategy so as to cope with threats of enemy planes and missiles from different directions.

Description

Technical Field

[0001] The invention relates to the technical field of air combat, and in particular to a method for generating a maneuver strategy for avoiding missiles and rotating to lock onto an enemy. Background Art

[0002] Beyond visual range (BVR) air combat refers to a type of air combat in which two fighters use onboard sensors to detect each other and engage each other using medium- and long-range air-to-air missiles. The modern air combat environment is complex. With the continuous improvement of airborne equipment and radar performance of air-to-air missiles, the proportion of air combat ending in the beyond visual range stage has increased from less than 30% in the 1980s to more than 50% in the early 21st century. To gain an advantage in the BVR stage, two conditions must be met: (1) adopt strategic maneuvers based on the overload limit to avoid the threat of incoming missiles and improve survivability; (2) take favorable maneuvers based on radar ranging and aircraft engine performance to expand the attack area and achieve target lock. With the rapid development of airborne equipment and air combat weapon equipment technology, new high-overload and high-speed missiles have gradually become powerful weapons, greatly increasing the proportion of engagements in the BVR stage. Therefore, formulating intelligent maneuvering strategies within the scope of aircraft maneuverability to avoid incoming missiles and re-lock the target after evasion is the key to improving aircraft survivability and achieving victory in BVR air combat.

[0003] Air combat is an extremely complex confrontation process. At present, the research on the intelligence of airborne equipment is constantly deepening, and the intelligence level of weapons and equipment is gradually improving. The development of air force equipment is bound to move towards autonomous and intelligent decision-making. In this case, most of the current research is committed to improving the intelligence and environmental adaptability of decision-making systems in air combat, especially in combat scenarios such as beyond-visual-range air combat where the air combat situation changes rapidly and dynamically. Facing incoming missiles, if the aircraft's maneuvering escape relies solely on the pilot's experience control, the correctness of the decision and the accuracy of the execution of the maneuvering command cannot be ensured due to the extremely short time available for decision-making and aircraft execution. Therefore, how to achieve rapid autonomous maneuvering decision-making or assist the pilot in outputting maneuvering commands and completing maneuvering control is crucial to improving the aircraft's survivability.

[0004] In air combat, avoiding incoming missiles is the lowest cost and most effective survival strategy. Traditional avoidance strategy methods mainly include expert system methods, differential countermeasures methods and optimal control methods. Considering the dynamic and uncertain nature of complex air combat environments, the application effects of these traditional methods still have great limitations in practice. On the one hand, the expert system method is difficult to adapt to the rapidly changing battlefield environment; on the other hand, the computational complexity of differential countermeasures and optimal control methods is difficult to meet the timeliness requirements of tactical decision-making in highly dynamic and strong real-time air combat scenarios. With the development of artificial intelligence, deep reinforcement learning has begun to be widely used in various research fields of air combat and has achieved good results, relying on its algorithm characteristics of not requiring modeling, not overly relying on prior knowledge, and realizing direct control from raw input to output through end-to-end learning. Deep reinforcement learning algorithms can make up for the shortcomings of traditional algorithms and can be used in various complex air combat environments, with good versatility and practicality. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention provides a method for generating a maneuver strategy for evading missiles and rotating to lock on the enemy, aiming to solve the complex problems of missile evasion and rotating to lock on the enemy in beyond-visual-range air combat. In a beyond-visual-range air combat environment, the method of the present invention can provide intelligent avoidance and attack decision-making suggestions for fighters, enhance the survivability and combat effectiveness of fighters in complex battlefield environments; the relevant algorithm training and learning process can continuously optimize the decision-making performance, so that the fighter can generate the best avoidance and rotation strategies in real time when facing changeable enemy maneuvers and missile threats. Through the model training and adaptive learning framework of deep reinforcement learning, the present invention can achieve autonomous decision-making optimization in complex air combat situations.

[0006] To achieve the above objectives, the present invention provides a method for generating a missile evasion and enemy-locking maneuver strategy, which comprises the following steps:

[0007] S1: Based on Newton's laws of motion, the kinematic and dynamic models of aircraft and missiles were established to simulate the multi-degree-of-freedom motion states of fighters and missiles, thereby simulating the flight trajectory and state of aircraft and missiles;

[0008] S2: Based on the kinematic and dynamic models of aircraft and missiles, a 1v1 beyond-visual-range air combat environment is established to achieve the initialization of the air combat situation and the termination judgment of the engagement;

[0009] S3: For the quadruple of deep reinforcement learning, the state space, action space and reward function of deep reinforcement learning are designed, and the three are integrated into the beyond-visual-range air combat environment, the design of the state transfer function is completed, and the deep reinforcement learning framework for beyond-visual-range air combat is realized;

[0010] S4: Use intelligent agents to simulate fighter jets and use the deep reinforcement learning framework to repeatedly simulate confrontations to optimize and adjust strategies and improve survivability and combat effectiveness in air combat. Through continuous iterative training of the reinforcement learning model, the intelligent agent can autonomously learn effective maneuvering strategies to deal with enemy aircraft and missile threats from different directions.

[0011] A further improvement of the present invention is that in step S1, the expressions of the kinematic model and the dynamic model are:

[0012]

[0013] Among them: (x, y, z) represents the three-dimensional coordinates of the aircraft, v represents the flight speed of the aircraft, is the flight path angle of the aircraft; n represents the normal overload of the aircraft, which is used to control the change in the direction of the aircraft's flight speed. y ,n z are the components of the normal overload in the y and z directions, g is the gravitational acceleration, C D0 is the drag coefficient, ρ is the atmospheric density, and S is the wing area of ​​our aircraft. In the dynamic control process of the aircraft, the overload and roll angle are used as the control variables of the aircraft. When considering the dynamic control of the missile, the normal overload in the y and z axes are used as the control variables.

[0014] A further improvement of the present invention is that in step S2, the beyond visual range air combat environment includes an aircraft motion module, a missile motion module, a relative motion module, and a missile guidance and control module; wherein:

[0015] The input information of the aircraft motion module includes: combat altitude, initial speed, maneuver decision control quantity; the output information includes: aircraft position vector, aircraft velocity vector;

[0016] The input information of the missile motion module includes: combat altitude, initial velocity, missile track angle acceleration control signal; its output information includes: missile position vector, missile velocity vector, missile track inclination, missile track deviation angle, missile velocity vector;

[0017] The input information of the relative motion module includes: aircraft position vector, aircraft velocity vector, missile position vector, missile velocity vector; its output information includes: relative position vector, relative velocity vector;

[0018] The input information package of the missile guidance and control module includes: relative position vector, relative velocity vector, missile track inclination, missile track deviation, and missile velocity vector; its output information includes: missile track angular acceleration control signal.

[0019] A further improvement of the present invention is that in step S2, the beyond-visual-range air combat environment includes a reset function for automatically restoring the system state to a preset initialization scene before each air combat simulation; and the beyond-visual-range air combat environment also includes a termination condition for the air combat.

[0020] A further improvement of the present invention is that in step S3, the state space parameters of the state space include: the relative distance r between the enemy missile and the host aircraft, the approach rate between the enemy missile and the host aircraft, Aircraft lead angle λ t , the missile's entry angle λ m 、The aircraft's flight altitude z t 、The aircraft's flight speed v t , the flight height of the enemy missile z m , enemy missile flight speed v m , the horizontal azimuth of the enemy aircraft relative to the aircraft The vertical azimuth of the enemy aircraft relative to the aircraft

[0021] A further improvement of the present invention is that, in step S3, the action space uses normal overload and roll angle as control variables, and each control variable is discretized to obtain a discretized action space.

[0022] A further improvement of the present invention is that in step S3, the reward function includes a sparse reward and a process reward;

[0023] Sparse rewards are obtained based on the results of the air combat. When the result of the air combat is a successful turn, the sparse reward r t is 100; when the result of the air combat is aircraft stall, aircraft crash or engine failure, reaching the maximum engagement time, or the aircraft is hit, the sparse reward r t = -100; when the result of the air combat is that the missile fails, the sparse reward r t is 0;

[0024] Process Rewards s The expression is:

[0025]

[0026] Among them: c1, c2, c3, c4, c5, c6, c7, c8, c9 are respectively the distance reward weight, the closing rate reward weight, the missile speed reward weight, the missile altitude reward weight, the lead angle reward weight, the missile entry angle reward weight, the energy reward weight, the avoidance time reward weight, the radar detection target horizontal azimuth reward weight and the radar detection target vertical azimuth reward weight; t is the avoidance time of our aircraft; v′ m and h′ m They are respectively the missile speed and altitude at the previous moment;

[0027] The final reward function is:

[0028] R = γ (tmax-t) r t +r s

[0029] Where: γ is the discount factor of the process reward; t is the training time.

[0030] A further improvement of the present invention is that in step S4, the intelligent agent selects a feedforward neural network with a total number of 4 layers; wherein the state space of the beyond-visual-range air combat environment is taken as input in the form of a 10-dimensional state vector, and the intelligent agent's maneuver decision is taken as output, corresponding to the action space.

[0031] A further improvement of the present invention is that in step S4, an ∈-greedy strategy is used to select actions in the action space, a deep reinforcement learning framework is used to update the network, and the memory library is updated, thereby realizing the network parameter update of the deep reinforcement learning framework.

[0032] The advantages of the present invention are as follows: The present invention provides a 1v1 beyond-visual-range air combat maneuver decision-making method based on deep reinforcement learning, which has the effect of generating low-energy-consumption and short-time maneuver decisions by using a trained and learned network for different battlefield situations, thereby achieving evasion of incoming missiles and re-locking of enemy aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 : Schematic diagram of the relative position relationship of aircraft and missiles;

[0034] Figure 2 : Schematic diagram of the input and output relationship between modules in a beyond visual range air combat environment;

[0035] Figure 3 The trajectory results obtained from simulation test 1; where: (a) 3D trajectory diagram (b) projection of the trajectory on the xy plane (c) projection of the trajectory on the yz plane;

[0036] Figure 4 The state parameter changes during the air combat of simulation test 1 are as follows: (a) the curve of the approach rate between the enemy missile and our aircraft; (b) the curve of the energy change of the enemy missile; (c) the curve of the normal overload change of the enemy missile; (d) the curve of the azimuth change of the enemy aircraft relative to our aircraft.

[0037] Figure 5 :The trajectory results obtained from simulation test 2: (a) three-dimensional trajectory diagram (b) projection of the trajectory on the xy plane (c) projection of the trajectory on the yz plane.

[0038] Figure 6: Changes of state parameters during the air combat of simulation test 2: (a) Change curve of the approach rate between the enemy missile and our aircraft (b) Change curve of the energy of the enemy missile (c) Change curve of the normal overload of the enemy missile (d) Change curve of the azimuth angle of the enemy aircraft relative to our aircraft.

[0039] Figure 7 :The trajectory results obtained by simulation test 3: (a) three-dimensional trajectory diagram (b) projection of the trajectory on the xy plane (c) projection of the trajectory on the yz plane.

[0040] Figure 8 : Changes of state parameters during the air combat of simulation test 3: (a) Change curve of the approach rate between the enemy missile and our aircraft (b) Change curve of the energy of the enemy missile (c) Change curve of the normal overload of the enemy missile (d) Change curve of the azimuth angle of the enemy aircraft relative to our aircraft. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0042] While some exemplary embodiments of the present invention have been described for the purpose of illustration, it should be understood that the present invention may be implemented in other ways not specifically shown in the drawings.

[0043] The present invention provides a method for generating missile evasion and enemy-locking maneuver strategies based on deep reinforcement learning. It is aimed at the mission scenario of 1v1 beyond-visual-range air combat. By constructing a virtual air combat environment and an adaptive learning framework, the fighter jets can learn the optimal maneuver strategy in the process of continuous simulation and confrontation.

[0044] Deep reinforcement learning is a machine learning method that aims to learn how to take actions to maximize the accumulated rewards through interaction with the environment. Its core idea is to continuously improve the strategy through a trial and error process and eventually find the best strategy to solve the problem.

[0045] Therefore, the present invention first constructs a 1v1 beyond-visual-range air combat virtual environment to simulate the confrontation scene between a fighter and an enemy aircraft. The environment can dynamically simulate the motion state and flight trajectory of both aircraft in air combat, including the maneuvering avoidance, attack locking, guidance tracking of enemy missiles, and fixed-length level flight of enemy aircraft. By constructing a 3D simulated beyond-visual-range air combat environment, it is possible to interact with the deep reinforcement learning framework, thereby realizing a continuous trial and error process.

[0046] The deep reinforcement learning framework can usually be described as a four-tuple: <S, A, P, R>. The state set S represents the set of all possible states in the over-the-horizon controlled arid environment; the action set A represents the set of all possible actions that the agent (i.e., the local machine) can perform in each state, and the agent selects an action to perform based on the current state at each time step. The state transition probability P represents the probability distribution of transitioning from one state to another. Usually, this probability distribution is defined as the conditional probability P(s′|s,a), which represents the probability of transitioning to state s′ after performing action a in state s. The state transition probability reflects the dynamic characteristics of the environment, which depends not only on the current state, but also on the current behavior of the agent.

[0047] The reward function R represents the immediate reward obtained after executing action a in state s. The reward function can be expressed as R(s,a,s′), which represents the reward obtained after executing action a in state s and transferring to state s′. The role of the reward function is to feedback the reward value of the action based on the action of the agent and the state change caused by the action. The expected cumulative reward value during the agent training process is related to the reward function in the training environment.

[0048] In the reinforcement learning training model of the present invention, the maximum cumulative reward can be obtained by calculating the optimal value function to obtain the optimal value under each state, and then calculating the maximum cumulative reward, so as to solve the optimal maneuver strategy of avoiding missiles, that is, turning and locking the enemy.

[0049] 1.1 Establishment of Beyond Visual Range Air Combat Simulation Environment Based on Deep Reinforcement Learning

[0050] In the embodiment of the present invention, for a 1v1 beyond visual range air combat, the initial enemy-friendly distance is set to 25km, the air combat decision cycle is set to 90s, and the calculation time step is 0.2 seconds. The initial situation of both parties is fixed, see Table 1. And the following assumptions are met: (1) the enemy missile, the enemy aircraft and the host are facing each other head-on; (2) the enemy missile has completed acceleration and is in the unpowered flight stage; (3) the enemy aircraft is always in a horizontal straight flight.

[0051] Table 1 Initialization settings of the present invention environment

[0052]

[0053] Based on the above assumptions, the present invention refers to the basic knowledge of flight mechanics and creates a three-degree-of-freedom kinematic and dynamic model of the aircraft:

[0054]

[0055] In formula (1), g is the acceleration due to gravity, C is D0is the drag coefficient, ρ is the atmospheric density, S is the wing area of ​​our aircraft; the subscripts z and y represent the longitudinal and transverse plane directions of our aircraft respectively, (x, y, z) represents the three-dimensional coordinates of the aircraft, v represents the flight speed of the aircraft, is the flight path angle of the aircraft. n represents the normal overload of the aircraft, which is used to control the change in the direction of the aircraft's flight speed. y ,n z are the components of normal overload in the y and z directions, respectively. Referring to the basic knowledge of flight mechanics and combining the different characteristics of aircraft and missiles, the present invention uses overload and roll angle as the control variables of the aircraft in the process of considering the dynamic control of the aircraft. When considering the dynamic control of the missile, the normal overload in the y and z directions is used as the control variable. In addition, based on Figure 1 The relative motion relationship between the aircraft and the missile shown in the figure, the present invention constructs a three-dimensional proportional guidance model of the missile:

[0056]

[0057] In formulas (2), (3), and (4), subscripts f and m represent our aircraft and missile respectively; subscript L represents our aircraft's line of sight coordinate system; and subscripts x, y, and z represent our aircraft's three-dimensional coordinate directions.

[0058] According to the above model, the air combat simulation environment is constructed. The input and output relationships between the modules are as follows: Figure 2 As shown in the figure, we can see that: (1) The variables that the model needs external input only include the maneuver decision control variables in the aircraft motion module. The aircraft is used as the intelligent agent of deep reinforcement learning for training in subsequent research, and the maneuver decision control variables are the training targets. (2) The three modules involved in the missile are related to each other, and the tracking movement of the aircraft can be achieved without external input variables. In the study of deep reinforcement learning, the missile is regarded as a component of the environment. Therefore, the input variables of the constructed air combat simulation environment are the aircraft maneuver decision control variables: normal overload and roll angle; the output variables are the situational relationship between the aircraft and the enemy missiles and aircraft.

[0059] In order to enhance the self-starting mode of the beyond visual range air combat simulation environment, the present invention designs a fully functional reset function. As a built-in function of the air combat simulation environment, the reset function is mainly used to automatically restore the system state to the preset initialization scene before each air combat simulation (see Table 1). In addition, we also define the termination condition of the air combat. When the air combat meets the engagement condition, the environment meets the self-starting mode (see Table 2).

[0060] Table 2 Termination conditions of the present invention environment

[0061]

[0062] 1.2 State Space Design of Deep Reinforcement Learning Networks

[0063] According to the air combat simulation environment established in 1.1, the present invention selects the following 10 battlefield situation data features as the state space parameters of the deep reinforcement learning network:

[0064] (1)r

[0065] The relative distance between the enemy projectile and the aircraft.

[0066] (2)

[0067] The rate at which the enemy missile approaches our aircraft.

[0068] (3)λ t

[0069] Aircraft lead angle: the angle between the projection of the aircraft speed in the horizontal plane and the aiming line.

[0070] (4)λ m

[0071] Missile entry angle: the angle between the projection of the missile's velocity vector in the horizontal plane and the aiming line.

[0072] (5)z t

[0073] The aircraft's flight altitude.

[0074] (6)v t

[0075] The aircraft's flight speed.

[0076] (7)z m

[0077] The flight altitude of the enemy missile.

[0078] (8)v m

[0079] The speed of the enemy projectile.

[0080] (9)ESM ψe

[0081] The horizontal azimuth of the enemy aircraft relative to your aircraft.

[0082] (10)ESM θe

[0083] The vertical azimuth of the enemy aircraft relative to your aircraft.

[0084] 1.3 Design of action space for deep reinforcement learning network

[0085] At present, the research on missile avoidance algorithm based on reinforcement learning usually selects actions from a predefined maneuver library. Considering the mission scenario targeted by the present invention, after avoiding the missile, sufficient energy must be retained to perform the turning maneuver and successfully lock the enemy aircraft. Therefore, the traditional maneuver library, such as the seven basic maneuvers defined by NASA, cannot meet the mission requirements of the present invention due to its limited maneuver options and high energy consumption.

[0086] In order to explore more effective new maneuvering strategies, the present invention selects two control variables of the agent: normal overload and roll angle according to the aircraft model established in Section 1.1. In addition, the normal overload and roll angle are discretized to obtain a discretized action space, as shown in Table 3.

[0087] Table 3 Action space settings

[0088]

[0089] 1.4 Design of action space for deep reinforcement learning network

[0090] In order to achieve the best reinforcement learning strategy, the agent needs to constantly interact with its environment and update its training network through multiple iterations to find the best strategy that can maximize the reward. In the training process of deep reinforcement learning, the agent can only get rewards based on the task results after completing a training round. This reward is called a sparse reward. Considering that there are many factors that affect the results of the task before the aircraft successfully escapes the missile and relocks the enemy aircraft, relying solely on sparse rewards will make the reinforcement learning training of the aircraft very difficult. Therefore, in order to guide the aircraft to successfully evade the missile through maneuvers and relock the enemy aircraft after the evasion is successful, the present invention designs a reward function that combines process rewards with sparse rewards. The specific design is as follows:

[0091] The sparse reward is the result of each round of combat. Based on the air combat termination conditions in Table 2, we design the sparse reward:

[0092]

[0093] Before the task is completed, the process reward can be used to evaluate the quality of the agent's action selection, and then evaluate the policy function. This study takes into account factors such as the flight state of the aircraft and the relative situation between the aircraft and the missile, and designs the process reward:

[0094]

[0095] Among them: c1, c2, ... c8 are respectively the distance reward weight, the closing rate reward weight, the missile speed reward weight, the missile altitude reward weight, the lead angle reward weight of our aircraft, the missile entry angle reward weight, the energy reward weight, the avoidance time reward weight, the radar detection target horizontal azimuth reward weight and the radar detection target vertical azimuth reward weight; t is the avoidance time of our aircraft; v` m and h` m are respectively the missile speed and height at the last moment, r s Reward for the overall process.

[0096] Combining equations 5, 6, and 7, we can get the total reward expression as:

[0097] R = γ (tmax-t) r t +r s (8)

[0098] 1.5 Training, testing and optimization design of BVR space maneuvering decision network based on deep reinforcement learning

[0099] The present invention proposes a beyond-visual-range air combat maneuver decision method based on deep reinforcement learning, and uses a forward fully connected neural network to approximate the solution of the numerical function to achieve higher accuracy and reliability. The network parameters are continuously updated through the principle of error reverse propagation gradient descent, and finally the fitting of the value function is achieved. When designing the deep neural network, the present invention selects a feedforward neural network with a total number of 4 layers. Among them, the state vector of the air combat environment, that is, the 10-dimensional state space, is used as input, and the intelligent agent maneuver decision is used as output, corresponding to the action space.

[0100] The deep reinforcement learning framework is a reinforcement learning framework based on DQN. This framework is applied to the training process of beyond-visual-range air combat maneuvering decision-making. It adopts the ∈-greedy strategy for action selection, uses the DQN air combat decision-making framework for network update, and updates (s, a, r, s′) to the memory bank, thereby realizing the parameter update of the DQN network. Given the initial position, the reinforcement learning framework is used for action selection and state transfer. The air combat data is actually generated during the training process. Therefore, the DQN agent only relies on the experience data and reward function generated by the beyond-visual-range air combat environment and stored in the experience pool, and does not rely on any prior knowledge. It can find an effective maneuvering strategy through training.

[0101] The mission scenario of this method is set to an air combat decision cycle of 90s and a calculation time step of 0.2 seconds. The initial situation of both sides is fixed, as shown in Table 4. And the following assumptions are met: (a) The enemy missile, enemy aircraft and our aircraft are facing each other head-on; (b) The enemy missile has completed acceleration and is in the unpowered flight stage. (c) The enemy aircraft is always in a horizontal straight flight. Let the simulation test scenario be consistent with the mission definition, forming situation 1, and the test results are shown in Figure 3 , Figure 4 .

[0102] Table 4 Mission scenario definition

[0103]

[0104] Analysis shows that at 50s, the enemy-to-our-aircraft approach rate r = 0, indicating that the enemy missile speed gradually decreases to less than our aircraft speed, the enemy missile energy is exhausted, and the agent successfully evades the incoming missile. During the 40s-50s period, the horizontal azimuth and vertical azimuth of the enemy aircraft relative to our aircraft are both within the radar detection range of the agent, and the duration exceeds 1s, indicating that the agent has re-locked the enemy aircraft.

[0105] Combination Figure 3 and Figure 4 Analysis shows that the agent uses a large maneuver turn in the early stage of evasion to place the missile behind as soon as possible, thereby gaining an evasive advantage and consuming a large amount of missile energy. While turning, our aircraft lowers its altitude to about 5000m, realizing the mutual conversion of gravitational potential energy and kinetic energy, and reducing the enemy-to-enemy approach rate. Finally, the aircraft is pulled up and climbed to perform a maneuver turn, successfully evading the missile while also completing the re-locking of the enemy aircraft.

[0106] This maneuver strategy turns around to engage the enemy during the evasion phase. Compared with the traditional maneuver strategy of evading first and then turning around to engage the enemy, this maneuver strategy can effectively shorten the maneuvering time of our aircraft, allowing our aircraft to quickly prepare for the enemy.

[0107] In addition, the present invention performs simulation verification of other situations on the training model, and the situation initialization settings are shown in Table 5.

[0108] Table 5 Other situation verification

[0109]

[0110]

[0111] Situation 2 and Situation 3 are more threatening than the training situation: Situation 1, but the learned maneuver strategies (such as Figure 4-Figure 8The results shown in the figure above show that the evasion of incoming missiles and the re-locking of enemy aircraft are achieved, and the effect is relatively ideal. Therefore, the evasion decision obtained by training the DQN algorithm has a certain generalization ability.

[0112] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for generating a maneuver strategy for evading missiles and rotating to lock onto the enemy, characterized in that: The following steps are involved: S1: Based on Newton's laws of motion, the kinematic and dynamic models of aircraft and missiles are established to simulate the multi-degree-of-freedom motion states of aircraft and missiles, thereby simulating the flight trajectory and state of aircraft and missiles; S2: Based on the kinematic and dynamic models of aircraft and missiles, a 1v1 beyond-visual-range air combat environment is established to achieve the initialization of the air combat situation and the termination judgment of the engagement; S3: For the quadruple of deep reinforcement learning, the state space, action space and reward function of deep reinforcement learning are designed, and the three are integrated into the beyond-visual-range air combat environment, the design of the state transfer function is completed, and the deep reinforcement learning framework for beyond-visual-range air combat is realized; S4: Using intelligent agents to simulate fighters, and using the deep reinforcement learning framework to conduct repeated simulated confrontations to optimize and adjust strategies and improve survivability and combat effectiveness in air combat; Through continuous iterative training of the reinforcement learning model, the intelligent agent can autonomously learn effective maneuvering strategies to deal with enemy aircraft and missile threats from different directions.

2. A method for generating a missile evasion and enemy-locking maneuver strategy according to claim 1, characterized in that: In step S1, the expressions of the kinematic model and the dynamic model are: Among them: (x, y, z) represents the three-dimensional coordinates of the aircraft, v represents the flight speed of the aircraft, is the flight path angle of the aircraft; n represents the normal overload of the aircraft, which is used to control the change in the direction of the aircraft's flight speed. y ,n z are the components of the normal overload in the y and z directions, g is the gravitational acceleration, C D0 is the drag coefficient, ρ is the atmospheric density, and S is the wing area of ​​our aircraft. In the dynamic control process of the aircraft, the overload and roll angle are used as the control variables of the aircraft. When considering the dynamic control of the missile, the normal overload in the y and z axes are used as the control variables.

3. The method for generating a missile evasion and enemy-locking maneuver strategy according to claim 1, characterized in that: In step S2, the beyond visual range air combat environment includes an aircraft motion module, a missile motion module, a relative motion module, and a missile guidance and control module; wherein: The input information of the aircraft motion module includes: combat altitude, initial speed, maneuver decision control quantity; the output information includes: aircraft position vector, aircraft velocity vector; The input information of the missile motion module includes: combat altitude, initial velocity, missile track angle acceleration control signal; its output information includes: missile position vector, missile velocity vector, missile track inclination, missile track deviation angle, missile velocity vector; The input information of the relative motion module includes: aircraft position vector, aircraft velocity vector, missile position vector, missile velocity vector; its output information includes: relative position vector, relative velocity vector; The input information package of the missile guidance and control module includes: relative position vector, relative velocity vector, missile track inclination, missile track deviation, and missile velocity vector; its output information includes: missile track angular acceleration control signal.

4. The method for generating a missile evasion and enemy-locking maneuver strategy according to claim 1, characterized in that: In step S2, the beyond-visual-range air combat environment includes a reset function for automatically restoring the system state to a preset initialization scene before each air combat simulation; the beyond-visual-range air combat environment also includes a termination condition for the air combat.

5. The method for generating a missile evasion and enemy-locking maneuver strategy according to claim 1, characterized in that: In step S3, the state space parameters of the state space include: the relative distance r between the enemy missile and the aircraft, the approach rate between the enemy missile and the aircraft Aircraft lead angle λ t , the missile's entry angle λ m 、The aircraft's flight altitude z t 、The aircraft's flight speed v t , the flight height of the enemy missile z m , enemy missile flight speed v m , the horizontal azimuth of the enemy aircraft relative to the aircraft The vertical azimuth of the enemy aircraft relative to the aircraft 6. A method for generating a missile evasion and enemy-locking maneuver strategy according to claim 5, characterized in that: In step S3, the action space uses the normal overload and the roll angle as control variables, and discretizes each control variable to obtain a discretized action space.

7. A method for generating a missile evasion and enemy-locking maneuver strategy according to claim 5, characterized in that: In step S3, the reward function includes a sparse reward and a process reward; Sparse rewards are obtained based on the results of the air combat. When the result of the air combat is a successful turn, the sparse reward r t is 100; when the result of the air combat is aircraft stall, aircraft crash or engine failure, reaching the maximum engagement time, or the aircraft is hit, the sparse reward r t = -100; when the result of the air combat is that the missile fails, the sparse reward r t is 0; Process Rewards s The expression is: Among them: c1, c2, c3, c4, c5, c6, c7, c8, c9 are respectively the distance reward weight, the closing rate reward weight, the missile speed reward weight, the missile altitude reward weight, the lead angle reward weight, the missile entry angle reward weight, the energy reward weight, the avoidance time reward weight, the radar detection target horizontal azimuth reward weight and the radar detection target vertical azimuth reward weight; t is the avoidance time of our aircraft; v′ m and h′ m They are respectively the missile speed and altitude at the previous moment; The final reward function is: R=γ (tmax-t) r t +r s Where: γ is the discount factor of the process reward; t is the training time.

8. The method for generating a missile evasion and enemy-locking maneuver strategy according to claim 1, characterized in that: In step S4, the intelligent agent selects a feedforward neural network with a total of 4 layers; among them, the state space of the beyond-visual-range air combat environment is taken as input in the form of a 10-dimensional state vector, and the intelligent agent's maneuver decision is taken as output, corresponding to the action space.

9. The method for generating a missile evasion and enemy-locking maneuver strategy according to claim 7, characterized in that: In step S4, the ∈-greedy strategy is used to select actions in the action space, the deep reinforcement learning framework is used to update the network, and the memory library is updated, thereby realizing the network parameter update of the deep reinforcement learning framework.

Images