A method of aircraft penetration based on deep reinforcement learning

Abstract

The present application belongs to the technical field of aircraft penetration strategy design. The disclosed embodiment is about an aircraft penetration method based on deep reinforcement learning. The disclosed embodiment introduces a regression network to enhance the structure of the deep neural network; according to the requirements of miss amount and energy optimization, a reward function in the form of terminal and process combination, strong reward and weak incentive guidance is proposed, and a strategy of energy-saving factor is added to the reward function to achieve quantitative regulation of maneuvering energy consumption; random noise is introduced in observation, and environmental uncertainty is incorporated into the training process; in addition, a strategy combining hierarchical training with course learning is proposed to accelerate the convergence speed of the algorithm and reduce the difficulty of training; a line of sight angular velocity correction factor is introduced to enhance Gaochao's sensitivity to line of sight angular velocity and increase Gaochao's penetration success rate.

Description

Technical Field

[0001] The disclosed embodiments relate to the technical field of aircraft penetration strategy design, and in particular to an aircraft penetration method based on deep reinforcement learning. Background Art

[0002] With the development of anti-near-space aircraft technology, especially the progress of high-speed aircraft cooperative interception technology, when a high-speed aircraft encounters a cooperative reverse-orbit interception by two interceptors, its survival space is greatly compressed and its high-speed advantage is also weakened. Therefore, combining intelligent methods to study the penetration game scenario between a high-speed aircraft and two cooperative reverse-orbit interceptors, and designing corresponding maneuvering strategies to achieve successful penetration are key scientific issues to be solved.

[0003] On the problem of high-speed aircraft penetration game, the research using traditional methods at home and abroad mainly focuses on the penetration guidance law based on unilateral optimal theory and the penetration guidance law based on differential game. With the rise of intelligent technologies such as machine learning and artificial intelligence, new solutions are provided for solving the penetration game problem of high-speed aircraft. As a branch of machine learning, reinforcement learning has developed rapidly in recent years. In order to use the intelligent method of deep reinforcement learning to solve the penetration problem of two interceptors coordinating reverse track interception, it is necessary to model it as a Markov decision process (MDP) and build an attack and defense multi-agent confrontation environment. The intelligent agent interacts with the environment through state, action and reward. In the process of interacting with the environment, the intelligent agent can take specific actions and obtain rewards by observing the state of the offensive and defensive parties in the environment, and then iteratively update its own strategy according to the reward, so that the intelligent agent can obtain more rewards at the next moment. It is an unsupervised heuristic algorithm that does not require the establishment of an accurate model. In the penetration game problem, commonly used deep reinforcement learning algorithms include DDPG (Deep Deterministic Policy Gradient) algorithm, TD3 (twin delayed deep deterministic gradient), etc., but the existing algorithms often have problems such as high training difficulty, weak generalization performance, and difficulty in defining reward functions. Summary of the invention

[0004] In order to avoid the shortcomings of the prior art, the present application provides an aircraft penetration method based on deep reinforcement learning to solve the problems existing in the prior art, such as high training difficulty, weak generalization performance, and difficulty in defining reward functions.

[0005] According to an embodiment of the present disclosure, a method for aircraft penetration based on deep reinforcement learning is provided, the method comprising:

[0006] Based on the multi-body penetration game scenario with two interceptors, a mathematical model of attack and defense confrontation between the aircraft and the interceptor is established in the lateral plane;

[0007] Based on the attack-defense confrontation mathematical model, the ITD3 network model, the reward function of the ITD3 network model and the state observation of the ITD3 network model are designed; the ITD3 network model includes a regression network, an Actor network and a Critic network, the regression network includes a discriminator and a filter, and random noise is introduced into the state observation of the ITD3 network model;

[0008] Using the training data, the ITD3 network model is trained in a hierarchical course to obtain the target ITD3 network model;

[0009] Introducing a sight line angular rate correction factor into the target ITD3 network model, and testing the target ITD3 network model after the sight line angular rate correction factor is introduced using test data to obtain a tested target ITD3 network model;

[0010] The tested target ITD3 network model is used to process the situation information of the aircraft and the two interceptors to obtain the penetration strategy.

[0011] Furthermore, the attack-defense confrontation mathematical model includes:

[0012] The three-degree-of-freedom dynamic model of the aircraft center of mass and the interceptor center of mass, the kinematic model of the aircraft center of mass and the interceptor center of mass, the autopilot model of the interceptor and the autopilot model of the aircraft.

[0013] Furthermore, the steps of establishing the three-degree-of-freedom dynamic model of the aircraft mass center and the interceptor mass center and the kinematic model of the aircraft mass center and the interceptor mass center include:

[0014] For the multi-body attack and defense game scenario in the horizontal plane, the relative motion equation between the aircraft and the interceptor is established; among them, the initial line of sight from the aircraft to the interceptor is the X-axis, the Y-axis is perpendicular to the X-axis in the horizontal plane, and the Z-axis is perpendicular to the X-axis in the fixed height plane. The relative motion equation between the aircraft and the interceptor is:

[0015]

[0016] Among them, E is the aircraft, P i (i=1,2) is the interceptor, is the relative distance between the aircraft and the interceptor, is the relative closing speed of the aircraft and the interceptor, is the relative acceleration of the vehicle and the interceptor, is the line of sight azimuth of the aircraft and interceptor, is the line-of-sight angular velocity of the aircraft and interceptor, is the line-of-sight angular acceleration of the aircraft and interceptor, V E is the speed of the aircraft, ψ VE is the trajectory angle of the aircraft, is the speed of the interceptor, is the interceptor's ballistic deviation angle, u is the actual overload of the aircraft, v i is the actual overload of the interceptor;

[0017] According to the relative motion equations of the aircraft and the interceptor, a three-degree-of-freedom dynamic model of the aircraft mass center and the interceptor mass center and a kinematic model of the aircraft mass center and the interceptor mass center are established;

[0018] The expression of the three-degree-of-freedom dynamic model of the center of mass of the aircraft and the center of mass of the interceptor is:

[0019]

[0020] The kinematic model of the center of mass of the aircraft and the center of mass of the interceptor is expressed as:

[0021]

[0022] Where j = E, P i (i=1,2), g is the acceleration due to gravity, V j is the speed of the aircraft or interceptor, θ j is the ballistic inclination angle of the aircraft or interceptor, n xj is the overload of the aircraft or interceptor in the X-axis direction, n yj is the overload of the aircraft or interceptor in the Y-axis direction, n zj is the overload of the aircraft or interceptor in the Z-axis direction, ψ Vj is the velocity trajectory angle of the aircraft or interceptor, is the acceleration of the aircraft or interceptor, is the angular velocity of the trajectory inclination of the aircraft or interceptor, is the angular velocity of the trajectory deviation of the aircraft or interceptor, is the component of the velocity vector of the aircraft or interceptor on the X-axis, is the component of the velocity vector of the aircraft or interceptor on the Y axis, is the component of the velocity vector of the aircraft or interceptor on the Z axis.

[0023] Furthermore, the steps of establishing the autopilot model of the interceptor and the autopilot model of the aircraft include:

[0024] The interceptor adopts a two-dimensional corrected proportional guidance law, so the lateral overload instruction expression of the interceptor is:

[0025]

[0026] Among them, N i (i=1,2) is the proportionality coefficient;

[0027] Assuming the autopilot model of the aircraft and the autopilot model of the interceptor are first-order inertial links, the expression of the autopilot model of the interceptor is:

[0028]

[0029] Among them, v i (s) is v i is the actual overload of the interceptor, v c,i (s) is v c,i It is the lateral overload instruction of the interceptor. is the time constant of the interceptor, s is the dynamic characteristic;

[0030] The expression of the aircraft's autopilot model is:

[0031]

[0032] Among them, u is the actual overload of the aircraft, u c is the overload instruction of the aircraft, τ E is the time constant of the aircraft;

[0033] The control constraints of the aircraft are:

[0034] u(t)≤u max

[0035] Among them, u max The maximum available overload of the aircraft.

[0036] Furthermore, in the multi-body penetration game scenario, if the relative distances between the aircraft and the two interceptors are greater than the preset first critical miss amount and the second critical miss amount, the aircraft penetration is successful, that is:

[0037]

[0038] in, is the relative distance between the aircraft and the first interceptor, is the intersection moment between the aircraft and the first interceptor, δ1 is the first critical miss distance, is the relative distance between the aircraft and the first interceptor, is the intersection moment between the aircraft and the second interceptor, and δ2 is the second critical miss distance.

[0039] Furthermore, the multi-body penetration game scenario is divided into and Three scenarios.

[0040] Furthermore, the regression network includes:

[0041] Discriminator and filter; the discriminator is a CNN structure with 16-dimensional input and 1-dimensional output, including an input layer, a convolution layer, a pooling layer, a fully connected layer and an output layer connected in sequence; the filter includes an input layer, 4 hidden layers and an output layer, the 4 hidden layers are the first hidden layer of 8 neurons, the second hidden layer of 16 neurons, the third hidden layer of 16 neurons and the fourth hidden layer of 8 neurons, and the activation functions of the 4 hidden layers are all REUL.

[0042] Furthermore, the reward function of the ITD3 network model includes:

[0043] The first reward function of the first interceptor and the second reward function of the second interceptor; where,

[0044] The first reward function of the first interceptor is:

[0045]

[0046]

[0047] Among them, R d is the first sub-reward function related to the miss distance, R n is the second sub-reward function related to energy consumption, is the third sub-reward function related to the line of sight angular rate, is the fourth sub-reward function related to the trajectory deviation angle, k d1 is the first weight coefficient, k d2 is the second weight coefficient, k d3 is the third weight coefficient, k n is the fourth weight coefficient, is the fifth weight coefficient, k ψ is the sixth weight coefficient, ψ VE0 is the initial ballistic deflection angle of the aircraft, is the relative distance between the aircraft and the first interceptor, is the intersection moment between the aircraft and the first interceptor, u is the actual overload of the aircraft, is the line-of-sight angular velocity between the aircraft and the first interceptor, ψ VE is the trajectory deviation angle of the aircraft, δ1 is the first critical miss distance;

[0048] The second reward function of the second interceptor is:

[0049] R2=r 21+r 22 +r 23 +r 24

[0050]

[0051] Among them, R2 is the second reward function, r 21 is the first process reward, r 22 is the second process reward, r 23 is the third process reward, r 24 is the terminal reward, c1 is the eighth weight coefficient, c2 is the ninth weight coefficient, c3 is the tenth weight coefficient, is the line-of-sight angular velocity of the aircraft and the second interceptor, is the relative distance between the aircraft and the second interceptor, is the initial relative distance between the aircraft and the second interceptor, t is the time, E c is the energy saving factor;

[0052] The state observation of the ITD3 network model is:

[0053]

[0054] in, is the relative distance between the aircraft and the first interceptor, is the initial relative distance between the aircraft and the first interceptor, is the relative distance between the aircraft and the second interceptor, is the relative sight angle between the aircraft and the first interceptor, is the relative line of sight azimuth between the aircraft and the second interceptor, is the relative line-of-sight azimuth rate between the aircraft and the first interceptor, is the relative line-of-sight azimuth rate between the aircraft and the second interceptor, is the relative ballistic angle of the aircraft, is the relative ballistic angle of the first interceptor, The relative ballistic deviation angle of the second interceptor game;

[0055] Random noise is added to the state observation of the target ITD3 network model:

[0056]

[0057] In the formula, the mean of random noise is 1.0, N() is Gaussian distribution, and the standard deviation is ε obs n-dimensional independent Gaussian random variables, ε obs is an array and n is the number of observations.

[0058] Further, based on the idea of ​​course learning, the ITD3 network model is trained using the training data to obtain the target ITD3 network model, the sight line angular rate correction factor is introduced into the target ITD3 network model, and the target ITD3 network model after the sight line angular rate correction factor is introduced is tested using the test data to obtain the target ITD3 network model after the test, including:

[0059] Based on the idea of ​​curriculum learning, the ITD3 network model is pre-trained using a hierarchical training strategy according to the situation information of the aircraft and the first interceptor in the training data to obtain the initial ITD3 network model;

[0060] After the first interceptor successfully penetrates, based on the curriculum learning idea, the initial ITD3 network model is trained again using the situation information of the aircraft and the second interceptor in the training data to obtain the target ITD3 network model;

[0061] Introducing the line-of-sight angular rate correction factor into the target ITD3 network model;

[0062] The target ITD3 network model after introducing the line of sight angular rate correction factor is tested using test data to verify the effectiveness of the target ITD3 network model.

[0063] Furthermore, the expression of the line of sight angular rate correction factor is:

[0064]

[0065] Wherein, k1 is the first correction coefficient, and k2 is the second correction coefficient.

[0066] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0067] In the embodiments of the present disclosure, through the above-mentioned aircraft penetration method based on deep reinforcement learning, firstly, a regression network is introduced to enhance the structure of the deep neural network; secondly, according to the requirements of miss amount and energy optimization, a reward function in the form of terminal and process combination, strong reward and weak incentive guidance is proposed, and a strategy of energy-saving factor is added to the reward function to achieve quantitative regulation of maneuvering energy consumption; random noise is introduced in the observation, and the uncertainty of the environment is incorporated into the training process; in addition, a strategy of combining hierarchical training with curriculum learning is proposed to accelerate the convergence speed of the algorithm and reduce the difficulty of training; finally, a line of sight angular rate correction factor is introduced to enhance the sensitivity of the aircraft to the change of line of sight angular rate and the accuracy of penetration timing judgment, so that the algorithm can be corrected while generating decisions online, further improving the effectiveness and generalization ability of the proposed algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0069] Figure 1 A step diagram showing an aircraft penetration method based on deep reinforcement learning in an exemplary embodiment of the present disclosure is shown;

[0070] Figure 2 A schematic diagram of a classic two-dimensional multi-body penetration game scenario in an exemplary embodiment of the present disclosure is shown;

[0071] Figure 3 The multi-body penetration game scenario in the exemplary embodiment of the present disclosure is shown as EP1 λ EP2 >0 when the schematic diagram;

[0072] Figure 4 The multi-body penetration game scenario in the exemplary embodiment of the present disclosure is shown as EP1 λ EP2 Schematic diagram when <0;

[0073] Figure 5 The multi-body penetration game scenario in the exemplary embodiment of the present disclosure is shown as EP1 λ EP2 = 0 when the schematic diagram;

[0074] Figure 6 A schematic diagram showing the structure of a deep neural network of a traditional TD3 algorithm in an exemplary embodiment of the present disclosure;

[0075] Figure 7 A schematic diagram showing the structure of a deep neural network of the ITD3 algorithm proposed in the present application in an exemplary embodiment of the present disclosure;

[0076] Figure 8 A schematic diagram showing that two interceptors do not form a coordinated interception in an exemplary embodiment of the present disclosure;

[0077] Fig. 9 A schematic diagram showing a hierarchical training strategy in an exemplary embodiment of the present disclosure;

[0078] Fig.10 The change of the reward value of the ITD3 algorithm during the training process in the exemplary embodiment of the present disclosure is shown;

[0079] Fig.11 The comparison results of the average reward values ​​of the ITD3 algorithm and the TD3 algorithm in the exemplary embodiment of the present disclosure are shown;

[0080] Fig.12 A comparison diagram of the game trajectories of the ITD3 algorithm without introducing a correction factor and introducing a correction factor under an extreme initial situation in an exemplary embodiment of the present disclosure is shown;

[0081] Fig.13 Shown in Figure 3 Simulation trajectory diagram of the classic optimal control algorithm in the scenario shown;

[0082] Fig.14 Shown in Figure 3 The simulation trajectory diagram of the ITD3 algorithm proposed in this application in the scenario shown;

[0083] Fig.15 Shown in Figure 3 The simulation relative distance change diagram of the ITD3 algorithm proposed in this application under the scenario shown;

[0084] Fig.16 Shown in Figure 3 The simulation overload variation diagram of the ITD3 algorithm proposed in this application in the scenario shown;

[0085] Fig.17 Shown in Figure 4 Simulation trajectory diagram of the classic optimal control algorithm in the scenario shown;

[0086] Fig.18 Shown in Figure 4 The simulation trajectory diagram of the ITD3 algorithm proposed in this application in the scenario shown;

[0087] Fig.19 Shown in Figure 4 The simulation relative distance change diagram of the ITD3 algorithm proposed in this application under the scenario shown;

[0088] Fig. 20 Shown in Figure 4 The simulation overload variation diagram of the ITD3 algorithm proposed in this application in the scenario shown;

[0089] Fig.21 Shown in Figure 5 Simulation trajectory diagram of the classic optimal control algorithm in the scenario shown;

[0090] Fig. 22 Shown in Figure 5 The simulation trajectory diagram of the ITD3 algorithm proposed in this application in the scenario shown;

[0091] Fig.23 Shown in Figure 5 The simulation relative distance change diagram of the ITD3 algorithm proposed in this application under the scenario shown;

[0092] Fig.24 Shown in Figure 5 The simulation overload variation diagram of the ITD3 algorithm proposed in this application in the scenario shown;

[0093] Fig.25 The different energy saving factors E in the exemplary embodiment of the present disclosure are shown. c High-speed aircraft overload under

[0094] Fig.26 The energy saving factor E in the exemplary embodiment of the present disclosure is shown as follows: c = 0.0, energy consumption of high-speed aircraft;

[0095] Fig. 27 The energy saving factor E in the exemplary embodiment of the present disclosure is shown as follows: c =0.1, energy consumption of high-speed aircraft;

[0096] Fig.28 The energy saving factor E in the exemplary embodiment of the present disclosure is shown as follows: c =0.2, energy consumption of high-speed aircraft;

[0097] Fig.29 The energy saving factor E in the exemplary embodiment of the present disclosure is shown as follows: c =0.3, energy consumption of high-speed aircraft;

[0098] Fig.30 The energy saving factor E in the exemplary embodiment of the present disclosure is shown as follows: c =0.4, the energy consumption of high-speed aircraft;

[0099] Fig.31 The energy saving factor E in the exemplary embodiment of the present disclosure is shown as follows: c =0.5, energy consumption of high-speed aircraft;

[0100] Fig.32 The energy saving factor E in the exemplary embodiment of the present disclosure is shown as follows: c =0.6, energy consumption of high-speed aircraft;

[0101] Fig.33 The energy saving factor E in the exemplary embodiment of the present disclosure is shown as follows: c =0.7, the energy consumption of high-speed aircraft. DETAILED DESCRIPTION

[0102] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0103] In addition, the accompanying drawings are only schematic illustrations of the embodiments of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0104] This example embodiment provides an aircraft penetration method based on deep reinforcement learning. Figure 1 As shown in , the aircraft penetration method based on deep reinforcement learning may include: steps S101 to S104.

[0105] Step S101: Based on a multi-body penetration game scenario with two interceptors, a mathematical model of attack and defense confrontation between the aircraft and the interceptor is established in a lateral plane;

[0106] Step S102: Based on the attack-defense confrontation mathematical model, design an ITD3 network model, a reward function of the ITD3 network model, and a state observation of the ITD3 network model; wherein the ITD3 network model includes a regression network, an Actor network (i.e., an actor network), and a Critic network (i.e., a critic network), the regression network includes a discriminator and a filter, and random noise is introduced into the state observation of the ITD3 network model;

[0107] Step S103: Performing hierarchical course training on the ITD3 network model using the training data and the test data to obtain a target ITD3 network model;

[0108] Step S104: introducing the sight line angular rate correction factor into the target ITD3 network model, and testing the target ITD3 network model after the sight line angular rate correction factor is introduced using the test data to obtain the tested target ITD3 network model;

[0109] Step S105: Use the tested target ITD3 network model to process the situation information of the aircraft and the two interceptors to obtain a penetration strategy.

[0110] Through the above-mentioned aircraft penetration method based on deep reinforcement learning, firstly, the regression network is introduced to enhance the structure of the deep neural network; secondly, according to the needs of miss distance and energy optimization, a reward function in the form of terminal and process combination, strong reward and weak incentive guidance is proposed, and the strategy of energy-saving factor is added to the reward function to realize the quantitative regulation of maneuver energy consumption; random noise is introduced in the observation, and the uncertainty of the environment is incorporated into the training process; in addition, a strategy combining hierarchical training and curriculum learning is proposed to accelerate the convergence speed of the algorithm and reduce the difficulty of training; finally, the line of sight angular rate correction factor is introduced to enhance the sensitivity of the aircraft to the change of line of sight angular rate and the accuracy of penetration timing judgment, so that the algorithm can make corrections while generating decisions online, further improving the effectiveness and generalization ability of the proposed algorithm.

[0111] Next, we will refer to Figures 1 to 33 Each step of the above-mentioned aircraft penetration method based on deep reinforcement learning in this example implementation is described in more detail.

[0112] In step S101, a mathematical model of attack and defense confrontation between a high-speed aircraft (ie, aircraft) and two coordinated and approximately reverse-track interceptor situation interceptors is established in a lateral plane.

[0113] Specifically, the direction of the maneuvering flight of the high-speed aircraft in the cruise phase can be divided into longitudinal and lateral maneuvers, and the two maneuvers can exist separately or simultaneously. The present application gives priority to completing the penetration through lateral maneuvers, because: the drastic changes in attitude angle and attitude angular rate will affect the working state of the scramjet engine. Compared with longitudinal maneuvers, lateral maneuver penetration can be carried out when the aircraft is at a constant altitude and speed, avoiding the influence of the attitude angle and engine thrust due to changes in speed and altitude during the penetration process.

[0114] The approximate reverse-track interception situation refers to a specific angle range. When the initial speed pointing deviation between the high-speed aircraft and the interceptor is greater than this angle range, the high-speed aircraft does not need to perform a maneuver penetration and can complete the penetration using its own speed; when the initial speed pointing deviation between the two parties is less than this angle range, a maneuver penetration is required.

[0115] More specifically, for a multi-body penetration game scenario with two interceptors, the classic two-dimensional attack and defense confrontation scenario is as follows: Figure 2 As shown. Where E represents high-speed aircraft, P i (i=1,2) two interceptors, in Figure 2 In EP i The initial sight direction is taken as the X-axis, and the Z-axis is perpendicular to the X-axis in the fixed height plane. The definitions of related symbols in the figure are shown in Table 1.

[0116] Table 1 Parameter names of the aircraft and two interceptors in multi-body game scenario

[0117]

[0118] In the table: i = 1, 2j = E, P i .

[0119] Combination Figure 2 In the multi-body attack and defense game scenario in the horizontal plane shown, the relative motion equations of the high-speed aircraft and the two interceptors are as follows:

[0120]

[0121] By analyzing the forces on the aircraft during flight, considering its overall dynamic and kinematic characteristics, and performing coordinate transformation and simplification, a three-degree-of-freedom dynamic model and kinematic model of the center of mass of the high-speed aircraft and the center of mass of the interceptor are established, as shown below:

[0122]

[0123]

[0124] Where, j = E, P i (i=1,2), g is the acceleration due to gravity, V j is the speed of the aircraft or interceptor, θ j is the ballistic inclination angle of the aircraft or interceptor, n xj is the overload of the aircraft or interceptor in the X-axis direction, n yj is the overload of the aircraft or interceptor in the Y-axis direction, n zj is the overload of the aircraft or interceptor in the Z-axis direction, ψ Vj is the velocity trajectory angle of the aircraft or interceptor, is the acceleration of the aircraft or interceptor, is the angular velocity of the trajectory inclination of the aircraft or interceptor, is the angular velocity of the trajectory deviation of the aircraft or interceptor, is the component of the velocity vector of the aircraft or interceptor on the X-axis, is the component of the velocity vector of the aircraft or interceptor on the Y axis, is the component of the velocity vector of the aircraft or interceptor on the Z axis.

[0125] In order to improve the tracking capability of the interceptor, this application assumes that the interceptor adopts a two-dimensional modified proportional guidance law, and adds overload compensation on the basis of traditional proportional guidance. The lateral overload instruction expression is as follows:

[0126]

[0127] Among them, N i (i=1,2) is the proportionality coefficient, It is the closing speed of the high-speed aircraft and the interceptor.

[0128] In order to make the model more realistic, this application takes into account the characteristics of the autopilots of both parties during the penetration process. Assuming that the autopilot models of both parties are first-order inertial links, the expressions of the actual overload and overload command of the interceptor are as follows:

[0129]

[0130] Among them, v i (s) is v i is the actual overload of the interceptor, v c,i (s) is v c,i It is the lateral overload instruction of the interceptor. is the time constant of the interceptor, and s is the complex variable in the Laplace transform, representing the dynamic characteristics of the system.

[0131] The autopilot model expression for a high-speed aircraft is as follows:

[0132]

[0133] Among them, u c is the overload command of the aircraft, u is the actual overload response (i.e. the actual overload of the aircraft), τ E is the time constant of the aircraft.

[0134] Taking into account the characteristics of high-speed aircraft, the control constraints it is subject to are:

[0135] u(t)≤u max

[0136] Among them, u max The maximum available overload of the aircraft.

[0137] In the multi-body penetration game scenario, if the minimum distance between the high-speed aircraft and the two interceptors is greater than the minimum critical miss distance, the high-speed aircraft is considered to have successfully penetrated, that is:

[0138]

[0139] In the formula, is the relative distance between the aircraft and the first interceptor, is the intersection moment between the aircraft and the first interceptor, δ1 is the first critical miss distance, is the relative distance between the aircraft and the first interceptor, is the intersection moment between the aircraft and the second interceptor, and δ2 is the second critical miss distance.

[0140] In addition, this application Figure 2 Based on the cooperative interception of multiple interceptors in the same direction, the attack and defense game scenario is further expanded by changing the initial parameters. According to the change of the initial line of sight azimuth, it is divided into and Three scenarios, such as Figures 3 to 5 As shown; among them, Figure 3 For the multi-body penetration game scenario Schematic diagram of the time; Figure 4 For the multi-body penetration game scenario Schematic diagram of the time; Figure 5 For the multi-body penetration game scenario Schematic diagram of the time. In the figure, u ⊥ is the component of the actual overload of the aircraft in the Z direction, Z1 is the lateral relative distance between the aircraft and the first interceptor missile, is the component of the actual overload of the first interceptor in the Z direction, is the component of the actual overload of the second interceptor in the Z direction.

[0141] In step S102, to address the problem of weak generalization of the existing TD3 algorithm, a regression network is used to improve the original algorithm network to obtain an ITD3 (Improve Twin Delayed Deep Deterministic policy gradient algorithm) network model.

[0142] Specifically, during the TD3 training process, the existence of random factors makes the initial situation of the game have a certain degree of randomness, but the trained aircraft still has the situation of penetration failure. Therefore, in order to improve the generalization ability of the algorithm, this application improves the network structure of the TD3 algorithm by introducing a regression network into the original deep neural network. The network structure of the TD3 algorithm is a prior art and will not be described in detail here.

[0143] The traditional TD3 algorithm's deep neural network includes an Actor network and a Critic network, such as Figure 6 As shown in the figure, the number of layers, types and specific number of neurons of the neural network are marked in the figure. The improved network structure using the regression network is shown in the attached figure. Figure 7 As shown in the figure, based on the original Actor network and Critic network, a regression network including a discriminator and a filter is added.

[0144] Discriminator: The input is the situation information of the three parties in the game (15 dimensions, including relative distance, position, sight angle, sight angular velocity, trajectory deflection, etc.) and the overload instruction (1 dimension) of the high-speed aircraft under this situation. The discriminator determines whether the overload instruction can successfully penetrate the defense in this scenario. If successful, the overload instruction (1 dimension) is input into the filter. The discriminator is a CNN structure with a 16-dimensional input dimension and a 1-dimensional output dimension, including an input layer, a convolution layer, a pooling layer, a fully connected layer, and an output layer.

[0145] filter: A fully connected neural network used to fit the successful penetration overload command. The input and output are both 1-dimensional, including an input layer, 4 hidden layers (the number of neurons is 8, 16, 16, 8, and the activation function is REUL), and an output layer.

[0146] During the training of the agent, the regression network inputs the situation information of the three parties in the game (including the initial situation and maneuver instructions, etc.) as training samples into the regression network. Those samples that are successfully avoided are filtered by the discriminator as Then The input is fed into the filter to train the filter's neural network. The trained filter outputs appropriate maneuver commands based on different situations. Instead of being output by the original Actor network.

[0147] The discriminator is based on Determine the penetration samples, set the successful penetration samples to 1 and input the relevant information into the filter, and set the failed penetration samples to 0 and exclude them. The filter consists of a multi-layer neural network, which uses gradient descent to train the model and continuously adjusts the weights and thresholds between networks through back propagation of errors.

[0148] Therefore, the regression network first filters samples from the discriminator, and then fits the selected samples according to the filter. The overload output by the regression network is used as the overall output of the TD3 algorithm, which effectively improves the versatility of the intelligent penetration strategy.

[0149] In addition, in order to address the difficulty in defining the reward function, a reward function in the form of a combination of “terminal” and “process”, a “strong reward” and “weak incentive” guidance is designed, and an energy-saving factor strategy is introduced.

[0150] The reward function can train and guide the agent to generate appropriate maneuvering instructions in different scenarios. The quality of the training results and the convergence speed of the algorithm are directly related to it. The appropriate reward function is the key to using deep reinforcement learning algorithms to solve the penetration problem. During the training process of the agent, the terminal reward directly determines the success or failure of the training results, while the process reward will affect the agent's choice of appropriate maneuvering strategies in different states through interaction with the environment, mainly playing a role in guiding the agent to successfully penetrate. In addition, the weight values ​​of each sub-reward function will also significantly affect the training effect. The use of "strong reward" and "weak incentive" guidance can make the agent more sensitive to related parameters and optimize its own parameters more targeted.

[0151] Miss distance and energy consumption are the two most important indicators in the high-speed aircraft penetration game, and are also the focus of this application. By comparing the size of the terminal miss distance and the critical miss distance, the success of the high-speed aircraft penetration can be directly reflected, and the energy consumption during the game will affect the subsequent flight and target strike of the high-speed aircraft. In addition, in the entire penetration game of the high-speed aircraft, the miss distance and energy consumption are two contradictory indicators. The increase in the miss distance often requires a greater maneuvering overload and consumes more energy, while the reduction in energy consumption will inevitably lead to a reduction in the miss distance.

[0152] In the scenario of this application, energy consumption is represented in the form of the total overload of high-speed aircraft maneuvers. The sub-reward functions related to the miss amount and energy consumption are strong rewards, and the sub-reward function related to the line of sight angle is a weak incentive. The main function of this sub-reward function is to guide the agent to increase the line of sight azimuth to increase the probability of successful penetration, and the line of sight angle cannot determine the success of the penetration, so its rewards and penalties are relatively small. In addition, the sub-reward function related to the miss amount is a terminal reward, and the sub-reward functions related to the line of sight angle, energy consumption, distance and time during the game are process rewards. In order to quantitatively control the maneuvering energy consumption, an energy-saving factor is also added to the energy consumption-related sub-reward function. When the energy-saving factor is large, the agent is relatively sensitive to energy consumption and tries to reduce energy consumption as much as possible on the premise of successful penetration. When the energy-saving factor is small, the weight of the energy consumption-related sub-reward function is small, and there is no need to consider energy consumption too much during the penetration process.

[0153] Faced with the scenario of two interceptors cooperating in reverse-trajectory interception, reward functions are designed for the two interceptors separately.

[0154] The reward function of the first interceptor is as follows:

[0155]

[0156]

[0157] Among them, R d is the first sub-reward function related to the miss distance, Rn is the second sub-reward function related to energy consumption, is the third sub-reward function related to the line of sight angular rate, is the fourth sub-reward function related to the trajectory deviation angle, k d1 is the first weight coefficient, k d2 is the second weight coefficient, k d3 is the third weight coefficient, k n is the fourth weight coefficient, is the fifth weight coefficient, k ψ is the sixth weight coefficient, ψ VE0 is the initial ballistic deflection angle of the aircraft.

[0158] In the sub-reward function, in addition to the miss amount related reward function R d In addition to the terminal reward, R n , The main considerations for penetrating the first interceptor missile are the miss distance and energy consumption, so R d With R n For strong rewards, and It is a weak incentive. The closer the miss distance is to δ1, the smaller the penalty is; when the miss distance is greater than δ1, a larger reward should be given, and on this basis, the larger the miss distance, the greater the reward; when the line of sight of the attacker and the defender increases, the lateral distance increases, and a weak reward should be given, otherwise a weak penalty should be given. This sub-reward function The main function is to guide the agent to increase the sight angle to increase the probability of successful penetration. The sight angle cannot determine the success of the penetration, so its rewards and penalties are relatively small. In addition, in order to deviate from the original trajectory as little as possible, the deviation of the trajectory of the high-speed aircraft should be as small as possible from the initial moment. The smaller the punishment.

[0159] The second reward function of the second interceptor is as follows:

[0160] R2=r 21 +r 22 +r 23 +r 24

[0161]

[0162] Among them, R2 is the second reward function, r 21 is the first process reward, r 22 is the second process reward, r 23 is the first terminal reward, r 24is the second terminal reward, c1 is the eighth weight coefficient, c2 is the ninth weight coefficient, c3 is the tenth weight coefficient, is the line-of-sight angular velocity of the aircraft and the second interceptor, is the relative distance between the aircraft and the second interceptor, is the initial relative distance between the aircraft and the second interceptor, t is the time, E c is the energy saving factor.

[0163] r 21 , r 22 , r 23 are process rewards, r 24 is the terminal reward, and the sub-reward function r in the second interceptor 21 , r 22 The energy saving factor E is introduced in c Used to set the training tendency, whether to choose to penetrate with a larger miss amount or with a lower energy consumption. Considering the characteristics of terminal miss amount and energy consumption, the sum of each weight factor is set to 1. In addition, the energy saving factor strategy is only meaningful under the premise that the high-speed aircraft successfully penetrates. 22 , r 24 Relative r 21 , r 23 For "strong rewards".

[0164] The state observations of the ITD3 network model are as follows:

[0165]

[0166] in, is the relative distance between the aircraft and the first interceptor, is the initial relative distance between the aircraft and the first interceptor, is the relative distance between the aircraft and the second interceptor, is the relative sight angle between the aircraft and the first interceptor, is the relative line of sight azimuth between the aircraft and the second interceptor, is the relative line-of-sight azimuth rate between the aircraft and the first interceptor, is the relative line-of-sight azimuth rate between the aircraft and the second interceptor, is the relative ballistic angle of the aircraft, is the relative ballistic angle of the first interceptor, The relative ballistic deviation angle of the second interceptor.

[0167] In addition, random noise is added to the observations.

[0168]

[0169] In the formula, the mean of random noise is 1.0, N() is Gaussian distribution, and the standard deviation is ε obs n-dimensional independent Gaussian random variables, ε obs is an array and n is the number of observations.

[0170] In step S103, in order to solve the problems of difficult training and slow convergence of deep reinforcement learning, a strategy combining hierarchical training and curriculum learning is designed by analyzing the characteristics of multi-agent games.

[0171] Specifically, in the scenario of two interceptors intercepting, if the distance between the two interceptors is too large or too small, the effect of coordinated interception cannot be achieved. When the distance is too large, it can be regarded as a high-speed aircraft facing two separate approximately reverse-track interceptions. When the distance is too small, the two missiles can be regarded as one for penetration game. In the above two situations, the existing penetration algorithm can be used to successfully penetrate, such as Figure 8 As shown. Among them, Figure 8 In the figure, A is a schematic diagram when the distance between two interceptors to form a coordinated interception is too small; B is a schematic diagram when the distance between two interceptors to form a coordinated interception is too large.

[0172] This application is a scenario where a high-speed aircraft faces a coordinated reverse-track interception by two interceptors. When penetrating two interceptors with appropriate spacing, the overall penetration strategy of the two missiles must be considered at the same time, that is, when penetrating the first interceptor, the penetration strategy of the second missile must be considered at the same time, otherwise the probability of failure of the high-speed aircraft penetration is extremely high. However, if the intelligent agent is trained at the same time for the penetration strategy of the two interceptors, the training difficulty is too great and the convergence is slow. Considering the time sequence of the encounter between the high-speed aircraft and the interceptor, it can be seen that the premise for the high-speed aircraft to successfully penetrate the second missile is to successfully avoid the first interceptor. Therefore, the intelligent penetration strategy can be trained for the first interceptor first, and then trained for the second interceptor on this basis. According to the characteristics of the problem itself, the complex and highly dynamic multi-body penetration game problem is transformed into multiple one-on-one penetration problems, which effectively reduces the blindness of the algorithm in the early stage of exploration.

[0173] More specifically, first, the agent is trained on the penetration problem of the first interceptor using a hierarchical training strategy.

[0174] In the observation items and reward functions of the training agent, the weight of the correlation coefficient with the second interceptor is reduced, so that the agent mainly observes the state of the first interceptor and the high-speed aircraft, and generates appropriate maneuvering instructions through training, so that the high-speed aircraft can successfully penetrate the first interceptor. The parameters related to the initial situation of the multi-body game are shown in Table 2, and the parameters of the deep neural network are shown in Table 3:

[0175] Table 2 Multi-player game situation related parameter settings

[0176]

[0177] Table 3: Deep neural network and training parameters

[0178] Parameter name Parameter Value Parameter name Parameter Value Actor network learning rate 0.005 Sampling time 0.1 Critic network learning rate 0.005 Batch size 128 Discount Factor 0.9 Optimizer Adam Inertia Factor 0.99 Regression network learning rate 0.01 Soft update rate 0.001 Minimum training error 0.001 Experience pool size 4000 Minimum performance gradient <![CDATA[1*10 -6 ]]>

[0179] Secondly, based on the successful penetration of the first interceptor, the agent is trained on the penetration problem of the second interceptor.

[0180] Adjust the weights related to the second interceptor in the agent's observation items and reward functions, so that the agent is equally sensitive to the relative situation of the two interceptors and the high-speed aircraft. The agent is trained to find the optimal penetration strategy through the feedback of the observation items and reward functions, taking energy consumption into consideration while successfully penetrating the two interceptors. Due to the pre-training of penetrating the first interceptor, the agent can find a successful maneuver strategy for penetrating the two interceptors more quickly. The schematic diagram of the hierarchical training strategy is shown in the figure below. Fig. 9 shown.

[0181] In addition to tiered training, the idea of ​​curriculum learning was also introduced. During the two training sessions, the guidance law of the interceptor missile was adjusted from easy to difficult, and the tracking efficiency of the guidance law was gradually improved. In the first 50 rounds of training, the interceptor did not perform lateral maneuvers; in rounds 51 to 100, the interceptor performed sinusoidal maneuvers; in rounds 101 to 150, the interceptor used a proportional guidance with a proportional coefficient of 3 for tracking; in rounds 151 to 200, the interceptor used a proportional guidance with a proportional coefficient of 4 for tracking; from round 201 to the end of the training, the interceptor used a corrected proportional guidance for tracking.

[0182] The above-mentioned hierarchical course learning training method reduces the difficulty of pilgrimage and improves the convergence speed of the network.

[0183] In step S104, in order to solve the problem that hypersonic aircraft (i.e., high-speed ultrasonic aircraft, i.e., aircraft) is not as sensitive to line-of-sight angular velocity as interceptors, which may cause penetration failure in some cases, a line-of-sight angular velocity correction factor is introduced based on the trained ITD3 network model to improve the sensitivity of hypersonic aircraft to line-of-sight velocity. The formula is as follows:

[0184]

[0185] Specifically, the trained ITD3 network model will generate penetration maneuver instructions based on the initial situation of the game, and on this basis, introduce correction items that are strongly related to the line of sight angular rate to enhance the sensitivity of the hypersonic to the line of sight rate. When penetrating the first interceptor, the weight of the line of sight angular rate correction factor with the second interceptor is extremely low, and the focus is on the penetration of the first interceptor; when the penetration of the first interceptor is successful, the line of sight angular rate correction factor with the first interceptor is set to 0. In addition, k1 is the first correction coefficient and k2 is the second correction coefficient, which are also related to the game situation and game time.

[0186] Finally, the intelligent agent trained in the previous step is used to simulate typical scenarios and compare them with traditional maneuver strategies to verify the effectiveness of the proposed algorithm.

[0187] The comparison of the training process of the ITD3 (Improve Twin Delayed Deep Deterministic policy gradient algorithm) algorithm proposed in this application and the traditional TD3 algorithm is shown in the figure below. Fig.10 and Fig.11 As shown in Figure 2, the ITD3 algorithm can converge to the optimal solution in less than 500 rounds. Fig.10 As shown, the TD3 algorithm Fig.11 The convergence trend began to show after 2000 rounds. The comparison shows that the hierarchical training strategy proposed in the algorithm training process is effective and can well solve the problems of excessive randomness and high difficulty in agent training. It improves the stability and convergence of the algorithm in the high-speed aircraft multi-body penetration game problem.

[0188] The intelligent agent obtained through training can successfully penetrate in most game situations, but in some extreme situations, it will fail to penetrate. After analysis, it is found that this is caused by the different sensitivity of hypersonics and interceptors to changes in line of sight angular rate. Therefore, a line of sight angular rate correction factor is introduced on the basis of the maneuver instructions generated by the intelligent agent to improve the sensitivity of hypersonics to line of sight angular rate and the accuracy of its penetration timing judgment. The algorithm can make corrections while making online decisions, further improving the effectiveness and generalization ability of the proposed algorithm. Under extreme conditions, the trajectory simulation of the ITD3 intelligent strategy without the introduction of the correction factor and the ITD3 intelligent strategy with the introduction of the correction factor is shown in the figure below. Fig.12 shown.

[0189] Using the trained agent, after introducing the line of sight angular rate correction factor, Figures 3 to 5 The simulation is performed under the three typical scenarios shown in the figure. The simulation results are as follows Figures 13 to 24 In the figure, Gaochao is the aircraft, Interceptor 1 is the first interceptor, and Interceptor 2 is the second interceptor.

[0190] Figures 13 to 16 It shows the situation that the two interceptors and the high-speed aircraft are in the same straight line at the initial moment. At this time, the two interceptors and the high-speed aircraft form a strict reverse track interception situation, that is, the line of sight azimuth and ballistic deviation angle of the two interceptors and the high-speed aircraft are both 0. Fig.13 As shown in the figure, when the classic penetration strategy faces the first interceptor, the miss distance at the terminal moment is 2.2135m, which is less than the critical miss distance, and the penetration fails. However, the two miss distances of the maneuverable penetration using the proposed strategy are both greater than the critical miss distance, as shown in Fig.14 and Fig.15 As shown in Figure 2, the strategy can successfully avoid the second interceptor under strict reverse trajectory. When facing the second interceptor, the proposed penetration strategy changes at 2.8s and is no longer a full overload maneuver, which causes the guidance rate of the second interceptor to change accordingly, as shown in Figure 2. Fig.16 shown.

[0191] Figures 17 to 20 The following figure shows the scenario of two interceptors coordinating to intercept from different directions. As shown in Figure 16, the traditional maneuver strategy can successfully evade the first interceptor, but it is very easy to be intercepted by the second interceptor. Fig.18 In the above figure, the maneuver command generated by the proposed intelligent penetration strategy can successfully avoid the coordinated interception of the two interceptors, and the miss amount is greater than the critical miss amount. Fig.19 In addition, unlike the classic maneuver strategy, the intelligent penetration strategy does not generate a downward overload command in the XZ two-dimensional plane when facing the first interceptor above the X axis. Instead, it drills through the two interceptors through the change of the overload command based on the initial upward maneuver, such as Fig. 20 It is different from the maneuvering instructions expected by humans, which means that the algorithm can generate maneuvering instructions that cannot be achieved by human subjective design through self-exploration and learning.

[0192] Figure 21 to Figure 24 The scene of two interceptors coordinating to intercept is shown. The high-speed aircraft using the classic maneuver strategy is intercepted by the second interceptor, and the miss distance is 3.5702m, which is less than the critical miss distance. Fig.21 When both interceptors intercept the high-speed aircraft on one side, the intelligent penetration strategy generates a maneuver command in the opposite direction, such as Fig. 22 In addition, when both interceptors are on one side, it is easier to avoid than the above two typical initial situations. Fig.21 To use the traditional strategy for the penetration trajectory, Fig. 22 For the penetration trajectory using the strategy proposed in this application, Fig.23 is the relative distance between the high-speed aircraft and the two interceptors using the strategy proposed in this application, Fig.24 Overload variation for a high-speed aircraft and two interceptors using the strategy proposed in this application.

[0193] Through simulation and comparative analysis under three typical attack and defense situations, the effectiveness of the intelligent penetration strategy in solving the multi-body penetration game problem of high-speed aircraft is verified. Compared with the classical maneuver strategy, this strategy has better performance under difficult initial conditions.

[0194] Using the target ITD3 network model that introduces the line of sight angular rate correction factor Figure 3 The biasing and Monte Carlo simulations are carried out in typical scenarios to verify the generalization and energy consumption reduction of the proposed algorithm.

[0195] Using the trained agent, both interceptors form a strict reverse-orbit interception situation with the high-speed aircraft ( Figure 3 The initial sight azimuth and relative initial distance of the interceptor are deviated in the typical scenario shown in Figure 1. The Monte Carlo simulation results show that:

[0196] The penetration strategy generated by the traditional TD3 without regression network still has the situation of penetration failure. The penetration strategy generated by adding regression network can achieve successful penetration in all harsh situations, and the miss range is linearly distributed. For the two cooperative interceptors, the proposed strategy can achieve effective penetration and has high generalization; when the line of sight azimuth is the same, the miss range of the high-speed aircraft with the first interceptor is greater than that with the second interceptor.

[0197] To further verify the energy saving of the algorithm, in a typical scenario where both interceptors form a strict reverse-track interception situation with the high-speed aircraft, only the energy saving factor E is adjusted. c The simulation results are as follows Figures 25 to 33 shown.

[0198] exist Fig.25 In the above example, the overload of the high-speed aircraft changes with the adjustment of the energy-saving factor. c = 0, the high-speed aircraft selects full overload maneuver; when E c When it increases in the range of [0,0.7], the energy consumption is reduced while ensuring the successful penetration of high-speed aircraft. c The specific values ​​of high-speed aircraft energy consumption under the value are shown in Table 4.

[0199] Table 4 Different energy saving factors E c Energy consumption of high-speed aircraft

[0200] Energy saving factor value Energy consumption of high-speed aircraft 0.0 85017.34 0.1 84121.90 0.2 83461.65 0.3 82796.48 0.4 81026.42 0.5 79831.94 0.6 78049.78 0.7 76938.68

[0201] Figure 26 to Figure 33 As shown in the figure, the actual energy consumption of the intelligent penetration strategy generated by ITD3 training is less than that of the traditional penetration strategy, and the energy saving factor E can be adjusted. cThe energy consumption during the penetration process is adjusted by the value of, which proves the quantitative energy-saving ability of the intelligent penetration strategy. Fig.26 Energy saving factor E c = 0.0, energy consumption of high-speed aircraft; Fig. 27 Energy saving factor E c =0.1, energy consumption of high-speed aircraft; Fig.28 Energy saving factor E c =0.2, energy consumption of high-speed aircraft; Fig.29 Energy saving factor E c =0.3, energy consumption of high-speed aircraft; Fig.30 Energy saving factor E c =0.4, the energy consumption of high-speed aircraft; Fig.31 Energy saving factor E c =0.5, energy consumption of high-speed aircraft; Fig.32 Energy saving factor E c =0.6, energy consumption of high-speed aircraft; Fig.33 Energy saving factor E c =0.7, the energy consumption of high-speed aircraft.

[0202] Through the above-mentioned aircraft penetration method based on deep reinforcement learning, firstly, the regression network is introduced to enhance the structure of the deep neural network; secondly, according to the needs of miss distance and energy optimization, a reward function in the form of terminal and process combination, strong reward and weak incentive guidance is proposed, and the strategy of energy-saving factor is added to the reward function to realize the quantitative regulation of maneuver energy consumption; in addition, a hierarchical training strategy is proposed to accelerate the convergence speed of the algorithm and reduce the training difficulty; finally, the line of sight angular rate correction factor is introduced to enhance the sensitivity of the hypersonic to the change of line of sight angular rate and the accuracy of penetration timing judgment, so that the algorithm can make corrections while generating decisions online, further improving the effectiveness and generalization ability of the proposed algorithm.

[0203] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0204] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0205] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. An aircraft penetration method based on deep reinforcement learning, characterized in that: The method includes: Based on the multi-body penetration game scenario with two interceptors, a mathematical model of attack and defense confrontation between the aircraft and the interceptor is established in the lateral plane; Based on the attack-defense confrontation mathematical model, the ITD3 network model, the reward function of the ITD3 network model and the state observation of the ITD3 network model are designed; the ITD3 network model includes a regression network, an Actor network and a Critic network, the regression network includes a discriminator and a filter, and random noise is introduced into the state observation of the ITD3 network model; Using the training data, the ITD3 network model is trained in a hierarchical course to obtain the target ITD3 network model; Introducing a sight line angular rate correction factor into the target ITD3 network model, and testing the target ITD3 network model after the sight line angular rate correction factor is introduced using test data to obtain a tested target ITD3 network model; The situation information of the aircraft and two interceptors is processed using the tested target ITD3 network model to obtain the penetration strategy; The steps of performing hierarchical course training on the ITD3 network model using the training data to obtain a target ITD3 network model, introducing a sight line angular rate correction factor into the target ITD3 network model, and testing the target ITD3 network model after the sight line angular rate correction factor is introduced using the test data to obtain the target ITD3 network model after the test, include: Based on the idea of ​​curriculum learning, the ITD3 network model is pre-trained using a hierarchical training strategy according to the situation information of the aircraft and the first interceptor in the training data to obtain the initial ITD3 network model; After the first interceptor successfully penetrates, based on the curriculum learning idea, the initial ITD3 network model is trained again using the situation information of the aircraft and the second interceptor in the training data to obtain the target ITD3 network model; Introducing the line-of-sight angular rate correction factor into the target ITD3 network model; The target ITD3 network model after introducing the line of sight angular rate correction factor is tested using test data to verify the validity of the target ITD3 network model; The reward function of the ITD3 network model includes: The first reward function of the first interceptor and the second reward function of the second interceptor; where, The first reward function of the first interceptor is: R1=R d +R n +R λ +R ψV Among them, R d is the first sub-reward function related to the miss distance, R n is the second sub-reward function related to energy consumption, is the third sub-reward function related to the line of sight angular rate, is the fourth sub-reward function related to the trajectory deviation angle, k d1 is the first weight coefficient, k d2 is the second weight coefficient, k d3 is the third weight coefficient, k n is the fourth weight coefficient, is the fifth weight coefficient, k ψ is the sixth weight coefficient, ψ VE0 is the initial ballistic deflection angle of the aircraft, is the relative distance between the aircraft and the first interceptor, is the intersection moment between the aircraft and the first interceptor, u is the actual overload of the aircraft, is the line-of-sight angular velocity between the aircraft and the first interceptor, ψ VE is the trajectory deviation angle of the aircraft, δ1 is the first critical miss distance; The second reward function of the second interceptor is: R2＝r 21 +r 22 +r 23 +r 24 Among them, R2 is the second reward function, r 21 is the first process reward, r 22 is the second process reward, r 23 is the third process reward, r 24 is the terminal reward, c1 is the eighth weight coefficient, c2 is the ninth weight coefficient, c3 is the tenth weight coefficient, is the line-of-sight angular velocity of the aircraft and the second interceptor, is the relative distance between the aircraft and the second interceptor, is the intersection moment between the spacecraft and the second interceptor, is the initial relative distance between the aircraft and the second interceptor, t is the time, E c is the energy saving factor, u max is the maximum available overload of the aircraft; The state observation of the ITD3 network model is: in, is the relative distance between the aircraft and the first interceptor, is the initial relative distance between the aircraft and the first interceptor, is the relative distance between the aircraft and the second interceptor, is the relative sight angle between the aircraft and the first interceptor, is the relative line of sight azimuth between the aircraft and the second interceptor, is the relative line-of-sight azimuth rate between the aircraft and the first interceptor, is the relative line-of-sight azimuth rate between the aircraft and the second interceptor, is the relative ballistic angle of the aircraft, is the relative ballistic angle of the first interceptor, The relative ballistic deviation angle of the second interceptor game; Random noise is added to the state observation of the target ITD3 network model: In the formula, the mean of random noise is 1.0, N() is Gaussian distribution, and the standard deviation is ε obs n-dimensional independent Gaussian random variables, ε obs is an array and n is the number of observations.

2. The aircraft penetration method based on deep reinforcement learning according to claim 1, characterized in that: The attack and defense mathematical model includes: The three-degree-of-freedom dynamic model of the aircraft center of mass and the interceptor center of mass, the kinematic model of the aircraft center of mass and the interceptor center of mass, the autopilot model of the interceptor and the autopilot model of the aircraft.

3. The aircraft penetration method based on deep reinforcement learning according to claim 2, characterized in that: The steps of establishing the three-degree-of-freedom dynamic model of the aircraft center of mass and the interceptor center of mass and the kinematic model of the aircraft center of mass and the interceptor center of mass include: For the multi-body attack and defense game scenario in the horizontal plane, the relative motion equation between the aircraft and the interceptor is established; among them, the initial line of sight from the aircraft to the interceptor is the X-axis, the Y-axis is perpendicular to the X-axis in the horizontal plane, and the Z-axis is perpendicular to the X-axis in the fixed height plane. The relative motion equation between the aircraft and the interceptor is: Among them, E is the aircraft, P i (i=1,2) is the interceptor, is the relative distance between the aircraft and the interceptor, is the relative closing speed of the aircraft and the interceptor, is the relative acceleration of the vehicle and the interceptor, is the line of sight azimuth of the aircraft and interceptor, is the line-of-sight angular velocity of the aircraft and interceptor, is the line-of-sight angular acceleration of the aircraft and interceptor, V E is the speed of the aircraft, ψ VE is the trajectory angle of the aircraft, is the speed of the interceptor, is the interceptor's ballistic deviation angle, u is the actual overload of the aircraft, v i is the actual overload of the interceptor; According to the relative motion equations of the aircraft and the interceptor, a three-degree-of-freedom dynamic model of the aircraft mass center and the interceptor mass center and a kinematic model of the aircraft mass center and the interceptor mass center are established; The expression of the three-degree-of-freedom dynamic model of the center of mass of the aircraft and the center of mass of the interceptor is: The kinematic model of the center of mass of the aircraft and the center of mass of the interceptor is expressed as: Where j = E, P i (i=1,2), g is the acceleration due to gravity, V j is the speed of the aircraft or interceptor, θ j is the ballistic inclination angle of the aircraft or interceptor, n xj is the overload of the aircraft or interceptor in the X-axis direction, n yj is the overload of the aircraft or interceptor in the Y-axis direction, n zj is the overload of the aircraft or interceptor in the Z-axis direction, ψ Vj is the velocity trajectory angle of the aircraft or interceptor, is the acceleration of the aircraft or interceptor, is the angular velocity of the ballistic inclination of the aircraft or interceptor, is the angular velocity of the trajectory deviation of the aircraft or interceptor, is the component of the velocity vector of the aircraft or interceptor on the X-axis, is the component of the velocity vector of the aircraft or interceptor on the Y axis, is the component of the velocity vector of the aircraft or interceptor on the Z axis.

4. The aircraft penetration method based on deep reinforcement learning according to claim 3 is characterized in that: The steps to build the interceptor autopilot model and the aircraft autopilot model include: The interceptor adopts a two-dimensional corrected proportional guidance law, so the lateral overload instruction expression of the interceptor is: Among them, N i (i=1,2) is the proportionality coefficient; Let the autopilot model of the aircraft and the autopilot model of the interceptor be first-order inertial links, then the expression of the autopilot model of the interceptor is: Among them, v i (s) is v i is the actual overload of the interceptor, v c,i (s) is v c,i It is the lateral overload instruction of the interceptor. is the time constant of the interceptor, s is the dynamic characteristic; The expression of the aircraft's autopilot model is: Among them, u is the actual overload of the aircraft, u c is the overload instruction of the aircraft, τ E is the time constant of the aircraft; The control constraints of the aircraft are: |u(t)|≤u max Among them, u max The maximum available overload of the aircraft.

5. The aircraft penetration method based on deep reinforcement learning according to claim 4 is characterized in that: In the multi-body penetration game scenario, if the relative distance between the aircraft and the two interceptors is greater than the preset first critical miss amount and second critical miss amount, the aircraft penetration is successful, that is: in, is the relative distance between the aircraft and the first interceptor, t f1 is the intersection moment between the aircraft and the first interceptor, δ1 is the first critical miss distance, is the relative distance between the aircraft and the first interceptor, is the intersection moment between the aircraft and the second interceptor, and δ2 is the second critical miss distance.

6. The aircraft penetration method based on deep reinforcement learning according to claim 5, characterized in that: The multi-body penetration game scenario is divided into the following types according to the change of the initial line of sight azimuth: and Three scenarios.

7. The aircraft penetration method based on deep reinforcement learning according to claim 6, characterized in that: The regression network consists of: Discriminator and filter; the discriminator is a CNN structure with 16-dimensional input and 1-dimensional output, including an input layer, a convolution layer, a pooling layer, a fully connected layer and an output layer connected in sequence; the filter includes an input layer, 4 hidden layers and an output layer, the 4 hidden layers are the first hidden layer of 8 neurons, the second hidden layer of 16 neurons, the third hidden layer of 16 neurons and the fourth hidden layer of 8 neurons, and the activation functions of the 4 hidden layers are all REUL.

8. The aircraft penetration method based on deep reinforcement learning according to claim 7, characterized in that: The expression of the line of sight angular rate correction factor is: Wherein, k1 is the first correction coefficient, and k2 is the second correction coefficient.

Images