Aircraft continuous maneuvering method based on DDPG algorithm, electronic equipment and storage medium

Abstract

The invention discloses an aircraft continuous maneuvering method based on a DDPG algorithm, electronic equipment and a storage medium, and belongs to the technical field of aircraft control. And maneuvering strategy calculation is carried out in real time to realize rapid moving of a target. The method comprises the following steps: establishing a kinematic model of the hypersonic aircraft according to the characteristics of the hypersonic aircraft; designing a training environment and mode for the hypersonic flight vehicle to execute continuous maneuvering actions; training an aircraft continuous maneuvering strategy based on a DDPG method; and performing simulation verification on the continuous maneuvering strategy training result of the aircraft. According to the method, firstly, a three-degree-of-freedom motion equation of the hypersonic aircraft is established, a flight environment with threats is designed based on the model, the continuous maneuvering strategy of the aircraft is trained by using a DDPG algorithm, and finally, the effectiveness and rapidity of the maneuvering strategy are verified through numerical simulation and Monte Carlo targeting.

Description

Technical Field

[0001] The present invention belongs to the technical field of aircraft control, and in particular relates to an aircraft continuous maneuvering method based on a DDPG algorithm, an electronic device and a storage medium. Background Art

[0002] Hypersonic aircraft have the characteristics of fast time-varying and high speed. Traditional maneuvering and avoidance algorithms have two main problems: the coding is complex and requires high storage space for onboard computers; the algorithm has poor real-time performance and cannot perform real-time maneuvering strategy solutions for fast-moving targets.

[0003] Reinforcement learning belongs to machine learning and is a hot topic in machine learning. Reinforcement learning is a method that mainly relies on the interaction between the intelligent agent and the environment, gains experience through the interaction process, and then learns through experience. In the process of studying the game maneuvering method of hypersonic aircraft, it was found that for the flight environment of hypersonic aircraft, deep reinforcement learning algorithms need to face extremely high learning difficulties. Due to the fast time-varying characteristics of hypersonic aircraft, as the position and speed of the game aircraft and itself are constantly updated, a large number of data points will be generated. The neural network may not be able to take successful data points during sampling training, so the training process needs to be optimized. Summary of the invention

[0004] The problem to be solved by the present invention is to realize real-time maneuvering strategy solution of a fast moving target, and propose an aircraft continuous maneuvering method, electronic equipment and storage medium based on the DDPG algorithm.

[0005] To achieve the above object, the present invention is implemented through the following technical solutions:

[0006] A method for continuous maneuvering of an aircraft based on a DDPG algorithm comprises the following steps:

[0007] S1. Establish a kinematic model of a hypersonic aircraft based on its characteristics;

[0008] S2. Design the training environment and methods for hypersonic aircraft to perform continuous maneuvers;

[0009] S3. Train the continuous maneuvering strategy of the aircraft based on the DDPG method;

[0010] S4. Perform simulation verification on the aircraft continuous maneuver strategy training results obtained in step S3.

[0011] Furthermore, the specific implementation method of step S1 includes the following steps:

[0012] S1.1. Set up common coordinate systems and related definitions involved in the hypersonic vehicle avoidance process;

[0013] S1.2. Establish the external environmental factors during the flight of the aircraft, including geophysical, atmospheric and gravity factors;

[0014] S1.3. Establish a hypersonic vehicle motion model, the expression is:

[0015]

[0016]

[0017] in, is the dimensionless rate of change of the distance from the center of the Earth with respect to time, is the dimensionless distance from the center of the Earth, that is is the rate of change of the current aircraft's longitude over time, is the rate of change of the current aircraft's latitude over time, θ and φ are the current aircraft's longitude and latitude, respectively. is the rate of change of dimensionless velocity with respect to time, is the dimensionless velocity, g 0 is the gravitational acceleration at sea level, and are the change rates of the flight path pitch angle and heading angle with respect to time, ψ and γ are the flight path pitch angle and heading angle, is the angular velocity of the Earth's rotation, which is dimensionless using the rotational radius vector, σ is the tilt angle, and are lift acceleration and drag acceleration respectively, and dynamic pressure is used for dimensionless processing.

[0018] Furthermore, the specific implementation method of step S2 includes the following steps:

[0019] S2.1. Set the training environment to a sequential continuous maneuver training environment, convert the continuous maneuver into multiple maneuvers, and then perform regression of the landing point deviation after performing multiple maneuvers;

[0020] S2.2. Set the training process to transfer training combined with progressive training:

[0021] Transfer training involves first conducting a round of game and landing point regression training. When the game results and landing point deviation of a round meet the requirements, the next round of training is conducted to achieve the effect of completing the final goal step by step.

[0022] Progressive training is to complete the game confrontation training, and then use a larger landing point deviation as the first training standard for the landing point regression problem under multiple waves. As the training process progresses, the training deviation value gradually decreases to the set threshold;

[0023] S2.3. Set the training environment parameters as follows: the dimension of the data collected by the hypersonic aircraft, i.e., the state variable, is 27 dimensions, including the velocity components of the aircraft in three directions, the detection information obtained at five time points, each detection information contains the components of the relative position of the threat in three directions and the rate of change of the relative distance, the distance between the aircraft and the target landing point in three directions, and the input regularization parameters include relative position and speed.

[0024] Furthermore, the specific implementation method of step S3 includes the following steps:

[0025] S3.1. Construct a continuous maneuver strategy model for the aircraft based on the deep deterministic policy gradient algorithm DDPG, including the strategy network π(a|s;θ), the target network of the strategy network, and the value network Q π (s,a|w) and the target network of its value network, where s is the state, a is the action, w is the value network weight, and θ is the action network weight;

[0026] The state value function V(s;w,θ) of DDPG is obtained by network fitting, and its expression is:

[0027]

[0028] Setting up Stochastic Policy Gradients for DDPG The expression is:

[0029]

[0030] The expression for setting the deterministic policy gradient g(θ) of DDPG is:

[0031]

[0032] Set the DDPG loss function L(θ Q ) is:

[0033]

[0034] The core goal of the DDPG algorithm is to minimize its loss function, that is, formula (1.10), where the policy gradient calculation method used when updating the Actor network parameters is as follows:

[0035]

[0036] Among them, N is the minimum batch size. The strategy adopted by the DDPG algorithm when updating the target network is a delayed momentum update method, that is, the target network parameters are copied from the corresponding network, and the momentum term τ is added to obtain the expression:

[0037]

[0038] After a lot of offline training, we get the deep deterministic policy gradient algorithm DDPG which contains the problem model information.

[0039] S3.2. Based on step S3.1, a model of the aircraft continuous maneuvering strategy is constructed. When the situation information at the initial moment is input into the deep deterministic policy gradient algorithm DDPG constructed in step S3.1, the maneuvering action variables output by the intelligent agent are obtained for the aircraft continuous maneuvering strategy;

[0040] The expression for setting the state space of the aircraft is:

[0041]

[0042] Among them, V xw0 ,V yw0 ,V zw0 are the three-direction components of the initial velocity of the aircraft in the reference coordinate system, Δx m (t N ), Δy m (t N ), Δz m (t N ) is the mth game aircraft at time t N The relative position of the three components in the reference coordinate system is: is the mth game aircraft at time t N The rate of change of distance;

[0043] Set the expression of the aircraft's action space to a = [t m ,σ,θ], where t m is the maneuvering time, σ, θ are the inclination and declination of the maneuvering direction in the reference coordinate system;

[0044] The input layer of the Actor network is set as the state variable. In the scenario where one aircraft plays a game against two game aircraft, the number of input layer nodes is 27, including the three components of the aircraft's speed, the three components of the relative positions of the two game aircraft at three consecutive time points, and the distance change rate. The number of output layer nodes is the dimension of the action space. In the design of the above action space, the number of output layer nodes of the Actor network is 3, because the dimension of the action variable is 3;

[0045] Set the number of input layer nodes of the Critic network to the total dimension of the environment state variable + action space, the number of input layer nodes to 30, and the number of output layer nodes to 1;

[0046] In order to achieve the goal of the aircraft reaching the target point while avoiding defense, the reward function is set as parameters related to the avoidance result and the target arrival. Combined with the agent training environment, the expression of the reward function is:

[0047]

[0048] Among them, zem represents the miss distance, and error represents the actual flight target point deviation;

[0049] S3.3. Based on the training environment obtained in step S2, a reward value is generated according to the reward function set in step S3.2. The aircraft agent stores the state space data, action space data, and reward data generated by the game process into the experience pool. The DDPG algorithm randomly extracts the data stored in the experience pool for offline training to obtain a trained aircraft continuous maneuvering strategy model.

[0050] Furthermore, step S4 applies the aircraft continuous maneuvering strategy model trained in step S3 to the online game process, generates an online game strategy, and verifies the effectiveness of the game strategy through numerical simulation and Monte Carlo shooting.

[0051] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a method for continuous maneuvering of an aircraft based on a DDPG algorithm when executing the computer program.

[0052] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for continuous maneuvering of an aircraft based on a DDPG algorithm.

[0053] Beneficial effects of the present invention:

[0054] The present invention discloses a method for continuous maneuvering of an aircraft based on the DDPG algorithm. First, a three-degree-of-freedom motion equation of a hypersonic aircraft is established. A threatening flight environment is designed based on the model. The DDPG algorithm is used to train the continuous maneuvering strategy of the aircraft. Finally, the effectiveness and rapidity of the maneuvering strategy are verified through numerical simulation and Monte Carlo shooting. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of a method for continuous maneuvering of an aircraft based on a DDPG algorithm according to the present invention;

[0056] Figure 2 A side view of a heat-insulating outer cover for protecting the temperature of an elevator traction machine from low temperatures according to the present invention;

[0057] Figure 3 is the off-target amount factor reward curve of the present invention;

[0058] Figure 4 The landing point deviation reward curve of the present invention;

[0059] Figure 5 This is a flowchart of the overall training and simulation verification process of the present invention;

[0060] Figure 6 The target deviation of the present invention is less than 1000km success rate curve;

[0061] Figure 7 The target deviation of the present invention is less than 800km success rate curve;

[0062] Figure 8 The target deviation of the present invention is less than 600km success rate curve;

[0063] Fig. 9 The target deviation of the present invention is less than 400km success rate curve;

[0064] Fig.10 The target deviation of the present invention is less than 200km success rate curve;

[0065] Fig.11 The target deviation of the present invention is less than 100km success rate curve;

[0066] Fig.12 is the Actor network loss curve of the present invention;

[0067] Fig.13 is the Critic network loss curve of the present invention;

[0068] Fig.14 is the final training process curve of the intelligent agent of the present invention;

[0069] Fig.15 The maneuvering time output curve of the present invention

[0070] Fig.16 is the action output curve of the present invention;

[0071] Fig.17 is the off-target amount monitoring curve of the present invention;

[0072] Fig.18 The exploratory noise output curve of the present invention;

[0073] Fig.19 is the average loss curve of the action network of the present invention;

[0074] Fig. 20 It is the average loss curve of the evaluation network of the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0076] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0077] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 -Attached Fig. 20 The detailed instructions are as follows: Specific implementation method one:

[0079] A method for continuous maneuvering of an aircraft based on a DDPG algorithm comprises the following steps:

[0080] S1. Establish a kinematic model of a hypersonic aircraft based on its characteristics;

[0081] Furthermore, the specific implementation method of step S1 includes the following steps:

[0082] S1.1. Set up common coordinate systems and related definitions involved in the hypersonic vehicle avoidance process;

[0083] S1.2. Establish the external environmental factors during the flight of the aircraft, including geophysical, atmospheric and gravity factors;

[0084] S1.3. Establish a hypersonic vehicle motion model, the expression is:

[0085]

[0086]

[0087] in, is the dimensionless rate of change of the distance from the center of the Earth with respect to time, is the dimensionless distance from the center of the Earth, that is is the rate of change of the current aircraft's longitude over time, is the rate of change of the current aircraft's latitude over time, θ and φ are the current aircraft's longitude and latitude, respectively. is the rate of change of dimensionless velocity with respect to time, is the dimensionless velocity, g 0 is the gravitational acceleration at sea level, and are the change rates of the flight path pitch angle and heading angle with respect to time, ψ and γ are the flight path pitch angle and heading angle, is the angular velocity of the Earth's rotation, which is dimensionless using the rotational radius vector, σ is the tilt angle, and are lift acceleration and drag acceleration respectively, and dynamic pressure is used for dimensionless processing.

[0088] S2. Design the training environment and methods for hypersonic aircraft to perform continuous maneuvers;

[0089] Furthermore, the specific implementation method of step S2 includes the following steps:

[0090] S2.1. Set the training environment to a sequential continuous maneuver training environment, convert the continuous maneuver into multiple maneuvers, and then perform regression of the landing point deviation after performing multiple maneuvers;

[0091] S2.2. Set the training process to transfer training combined with progressive training:

[0092] Transfer training involves first conducting a round of game and landing point regression training. When the game results and landing point deviation of a round meet the requirements, the next round of training is conducted to achieve the effect of completing the final goal step by step.

[0093] Progressive training is to complete the game confrontation training. For the problem of landing point regression under multiple waves, a larger landing point deviation is used as the first training standard. As the training process progresses, the training deviation value gradually decreases to the set threshold, from 1000km training to within 100km.

[0094] S2.3. Set the training environment parameters as follows: the dimension of the data collected by the hypersonic aircraft, i.e., the state variable, is 27 dimensions, including the velocity components of the aircraft in three directions, the detection information obtained at five time points, each detection information contains the components of the relative position of the threat in three directions and the rate of change of the relative distance, the distance between the aircraft and the target landing point in three directions, and the input regularization parameters include relative position and speed.

[0095] Further, in the traditional training environment, the hypersonic aircraft performs continuous maneuvers, that is, after the speed changes, completes 2 waves of games, and finally makes the distance between the landing point and the target point less than a certain value. After simulation verification, the use of continuous maneuvers for training will cause the data in the training pool to contain more failure information, and does not contain information on the improvement of the success rate, which will lead to the inability to improve the reward value of the algorithm, and the inability to continue deep reinforcement learning. Therefore, according to the existing situation, the training is carried out in the form of transfer learning and progressive learning. The training environment adopted in this embodiment is a sequential continuous maneuvering training environment, which converts continuous maneuvers into multiple maneuvers, and then regresses the landing point deviation after multiple maneuvers. Such training process processing can greatly accelerate the training process and obtain better results. The entire training process is split. For the problem that it is difficult to successfully play the game under multiple waves, a wave of game and landing point regression training is first carried out. When the game results and landing point deviation of a wave meet the requirements, the next wave of training is carried out, so as to achieve the effect of completing the final goal step by step, which is transfer training. After completing the game confrontation training, for the landing point regression problem under multiple waves, a larger landing point deviation is used as the first training standard. As the training process progresses, the training deviation value gradually decreases, which is progressive training.

[0096] S3. Train the continuous maneuvering strategy of the aircraft based on the DDPG method;

[0097] Furthermore, the specific implementation method of step S3 includes the following steps:

[0098] S3.1. Construct a continuous maneuver strategy model for the aircraft based on the deep deterministic policy gradient algorithm DDPG, including the strategy network π(a|s;θ), the target network of the strategy network, and the value network Q π (s,a|w) and the target network of its value network, where s is the state, a is the action, w is the value network weight, and θ is the action network weight;

[0099] The state value function V(s;w,θ) of DDPG is obtained by network fitting, and its expression is:

[0100]

[0101] Setting up Stochastic Policy Gradients for DDPG The expression is:

[0102]

[0103] The expression for setting the deterministic policy gradient g(θ) of DDPG is:

[0104]

[0105] Set the DDPG loss function L(θ Q ) is:

[0106]

[0107] The core goal of the DDPG algorithm is to minimize its loss function, that is, formula (1.10), where the policy gradient calculation method used when updating the Actor network parameters is as follows:

[0108]

[0109] Among them, N is the minimum batch size. The strategy adopted by the DDPG algorithm when updating the target network is a delayed momentum update method, that is, the target network parameters are copied from the corresponding network, and the momentum term τ is added to obtain the expression:

[0110]

[0111] After a lot of offline training, we get the deep deterministic policy gradient algorithm DDPG which contains the problem model information.

[0112] S3.2. Based on step S3.1, a model of the aircraft continuous maneuvering strategy is constructed. When the situation information at the initial moment is input into the deep deterministic policy gradient algorithm DDPG constructed in step S3.1, the maneuvering action variables output by the intelligent agent are obtained for the aircraft continuous maneuvering strategy;

[0113] The expression for setting the state space of the aircraft is:

[0114]

[0115] Among them, V xw0 ,V yw0 ,V zw0 are the three-direction components of the initial velocity of the aircraft in the reference coordinate system, Δx m (t N ), Δy m (t N ), Δz m (t N ) is the mth game aircraft at time t N The relative position of the three components in the reference coordinate system is: is the mth game aircraft at time t N The rate of change of distance;

[0116] Set the expression of the aircraft's action space to a = [t m ,σ,θ], where t mis the maneuvering time, σ, θ are the inclination and declination of the maneuvering direction in the reference coordinate system;

[0117] The input layer of the Actor network is set as the state variable. In the scenario where one aircraft plays a game against two game aircraft, the number of input layer nodes is 27, including the three components of the aircraft's speed, the three components of the relative positions of the two game aircraft at three consecutive time points, and the distance change rate. The number of output layer nodes is the dimension of the action space. In the design of the above action space, the number of output layer nodes of the Actor network is 3, because the dimension of the action variable is 3;

[0118] Set the number of input layer nodes of the Critic network to the total dimension of the environment state variable + action space, the number of input layer nodes to 30, and the number of output layer nodes to 1;

[0119] In order to achieve the goal of the aircraft reaching the target point while avoiding defense, the reward function is set as parameters related to the avoidance result and the target arrival. Combined with the agent training environment, the expression of the reward function is:

[0120]

[0121] Among them, zem represents the miss distance, and error represents the actual flight target point deviation;

[0122] S3.3. Based on the training environment obtained in step S2, a reward value is generated according to the reward function set in step S3.2. The aircraft agent stores the state space data, action space data, and reward data generated by the game process into the experience pool. The DDPG algorithm randomly extracts the data stored in the experience pool for offline training to obtain a trained aircraft continuous maneuvering strategy model.

[0123] Furthermore, let the off-target threshold be zem 0 = 10m, then the function graph of the reward function related to the terminal miss distance is as follows Figure 3 and Figure 4 shown.

[0124] S4. Perform simulation verification on the aircraft continuous maneuver strategy training results obtained in step S3.

[0125] Furthermore, step S4 applies the aircraft continuous maneuvering strategy model trained in step S3 to the online game process, generates an online game strategy, and verifies the effectiveness of the game strategy through numerical simulation and Monte Carlo shooting.

[0126] Furthermore, the algorithm hyperparameter values ​​are shown in Table 1:

[0127] Table 1 Algorithm hyperparameter values

[0128]

[0129]

[0130] The setting of the agent training simulation environment is shown in Table 2:

[0131] Table 2 Training environment settings

[0132]

[0133] Figures 6 to 11 The reward curve obtained by the agent after progressive landing point deviation training in one wave is shown. It can be seen that in each wave of training, the aircraft has achieved the goal after training. Fig.12 , Fig.13 The loss function curves of the Actor network and the Critic network during the training process show that the loss function gradually converges. After actual testing, the target completion rate of the agent reached 84.31%. After further training, the final results are as follows: Figures 14 to 20 As shown: Fig.14 The success rate (percentage) of the aircraft in completing the task in the final training phase is shown, and it can be seen that the aircraft finally completed the set goal; Fig.15 The output curve of the aircraft's judgment on the start time of maneuver is shown; Fig.16 The motion output curve of the aircraft is shown; Fig.17 The miss curve of the gaming aircraft is shown; Fig.18 It shows the change curve of the algorithm's exploration noise as the training process progresses; Fig.19 Shows the average loss curve of the Actor network; Fig. 20 The average loss curve of the Critic network is shown; finally, through Monte Carlo shooting, the hypersonic aircraft using the DDPG algorithm achieved a performance success rate of 90.43% in game confrontation after two waves of training. This shows that the DDPG algorithm can effectively complete the required tasks by transferring training and gradually optimizing the goals.

[0134] Furthermore, through Monte Carlo shooting, the hypersonic aircraft using the DDPG algorithm achieved a performance success rate of 90.43% in game confrontation after two waves of training. This shows that the DDPG algorithm can effectively complete the required tasks by transferring training and gradually optimizing the goals. Specific implementation method 2:

[0136] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a method for continuous maneuvering of an aircraft based on a DDPG algorithm described in a specific implementation method 1 are implemented.

[0137] The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit, etc. Moreover, the processor is used to implement the steps of the above-mentioned method of continuous maneuvering of an aircraft based on the DDPG algorithm when executing the computer program stored in the memory.

[0138] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0139] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Specific implementation method three:

[0141] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for continuous maneuvering of an aircraft based on a DDPG algorithm as described in a first specific implementation.

[0142] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by the processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. The computer-readable storage medium stores a computer program. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned method of continuous maneuvering of an aircraft based on the DDPG algorithm can be implemented.

[0143] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0144] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0145] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for continuous maneuvering of an aircraft based on the DDPG algorithm, characterized in that: The steps include: S1. Establish a kinematic model of a hypersonic aircraft based on its characteristics; S2. Design the training environment and methods for hypersonic aircraft to perform continuous maneuvers; S3. Train the continuous maneuvering strategy of the aircraft based on the DDPG method; S4. Perform simulation verification on the aircraft continuous maneuver strategy training results obtained in step S3.

2. The method for continuous maneuvering of an aircraft based on the DDPG algorithm according to claim 1, characterized in that: The specific implementation method of step S1 includes the following steps: S1.

1. Set up common coordinate systems and related definitions involved in the hypersonic vehicle avoidance process; S1.

2. Establish the external environmental factors during the flight of the aircraft, including geophysical, atmospheric and gravity factors; S1.

3. Establish a hypersonic vehicle motion model, the expression is: in, is the dimensionless rate of change of the distance from the center of the Earth with respect to time, is the dimensionless distance from the center of the Earth, that is is the rate of change of the current aircraft's longitude over time, is the rate of change of the current aircraft's latitude over time, θ and φ are the current aircraft's longitude and latitude, respectively. is the rate of change of dimensionless velocity with respect to time, is the dimensionless velocity, g0 is the gravitational acceleration at sea level, and are the change rates of the flight path pitch angle and heading angle with respect to time, ψ and γ are the flight path pitch angle and heading angle, is the angular velocity of the Earth's rotation, which is dimensionless using the rotational radius vector, σ is the tilt angle, and are lift acceleration and drag acceleration respectively, and dynamic pressure is used for dimensionless processing.

3. The method for continuous maneuvering of an aircraft based on the DDPG algorithm according to claim 2, characterized in that: The specific implementation method of step S2 includes the following steps: S2.

1. Set the training environment to a sequential continuous maneuver training environment, convert the continuous maneuver into multiple maneuvers, and then perform regression of the landing point deviation after performing multiple maneuvers; S2.

2. Set the training process to transfer training combined with progressive training: Transfer training involves first conducting a round of game and landing point regression training. When the game results and landing point deviation of a round meet the requirements, the next round of training is conducted to achieve the effect of completing the final goal step by step. Progressive training is to complete the game confrontation training, and then use a larger landing point deviation as the first training standard for the landing point regression problem under multiple waves. As the training process progresses, the training deviation value gradually decreases to the set threshold; S2.

3. Set the training environment parameters as follows: the dimension of the data collected by the hypersonic aircraft, i.e., the state variable, is 27 dimensions, including the velocity components of the aircraft in three directions, the detection information obtained at five time points, each detection information contains the components of the relative position of the threat in three directions and the rate of change of the relative distance, the distance between the aircraft and the target landing point in three directions, and the input regularization parameters include relative position and speed.

4. The method for continuous maneuvering of an aircraft based on the DDPG algorithm according to claim 3, characterized in that: The specific implementation method of step S3 includes the following steps: S3.

1. Construct a continuous maneuver strategy model for the aircraft based on the deep deterministic policy gradient algorithm DDPG, including the strategy network π(a|s;θ), the target network of the strategy network, and the value network Q π (s,a|w) and the target network of its value network, where s is the state, a is the action, w is the value network weight, and θ is the action network weight; The state value function V(s;w,θ) of DDPG is obtained by network fitting, and its expression is: Setting up Stochastic Policy Gradients for DDPG The expression is: The expression for setting the deterministic policy gradient g(θ) of DDPG is: Set the DDPG loss function L(θ Q ) is: The core goal of the DDPG algorithm is to minimize its loss function, that is, formula (1.10), where the policy gradient calculation method used when updating the Actor network parameters is as follows: Among them, N is the minimum batch size. The strategy adopted by the DDPG algorithm when updating the target network is a delayed momentum update method, that is, the target network parameters are copied from the corresponding network, and the momentum term τ is added to obtain the expression: After a lot of offline training, we get the deep deterministic policy gradient algorithm DDPG which contains the problem model information. S3.

2. Based on step S3.1, a model of the aircraft continuous maneuvering strategy is constructed. When the situation information at the initial moment is input into the deep deterministic policy gradient algorithm DDPG constructed in step S3.1, the maneuvering action variables output by the intelligent agent are obtained for the aircraft continuous maneuvering strategy; The expression for setting the state space of the aircraft is: Among them, V xw0 ,V yw0 ,V zw0 are the three-direction components of the initial velocity of the aircraft in the reference coordinate system, Δx m (t N ), Δy m (t N ), Δz m (t N ) is the mth game aircraft at time t N The relative position of the three components in the reference coordinate system is: is the mth game aircraft at time t N The rate of change of distance; Set the expression of the aircraft's action space to a = [t m ,σ,θ], where t m is the maneuvering time, σ, θ are the inclination and declination of the maneuvering direction in the reference coordinate system; The input layer of the Actor network is set as the state variable. In the scenario where one aircraft plays a game against two game aircraft, the number of input layer nodes is 27, including the three components of the aircraft's speed, the three components of the relative positions of the two game aircraft at three consecutive time points, and the distance change rate. The number of output layer nodes is the dimension of the action space. In the design of the above action space, the number of output layer nodes of the Actor network is 3, because the dimension of the action variable is 3; Set the number of input layer nodes of the Critic network to the total dimension of the environment state variable + action space, the number of input layer nodes to 30, and the number of output layer nodes to 1; In order to achieve the goal of the aircraft reaching the target point while avoiding defense, the reward function is set as parameters related to the avoidance result and the target arrival. Combined with the agent training environment, the expression of the reward function is: Among them, zem represents the miss distance, and error represents the actual flight target point deviation; S3.

3. Based on the training environment obtained in step S2, a reward value is generated according to the reward function set in step S3.

2. The aircraft agent stores the state space data, action space data, and reward data generated by the game process into the experience pool. The DDPG algorithm randomly extracts the data stored in the experience pool for offline training to obtain a trained aircraft continuous maneuvering strategy model.

5. The method for continuous maneuvering of an aircraft based on the DDPG algorithm according to claim 4, characterized in that: Step S4 applies the aircraft continuous maneuvering strategy model trained in step S3 to the online game process, generates an online game strategy, and verifies the effectiveness of the game strategy through numerical simulation and Monte Carlo shooting.

6. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a method for continuous maneuvering of an aircraft based on a DDPG algorithm as described in any one of claims 1 to 5 when executing the computer program.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for continuous maneuvering of an aircraft based on the DDPG algorithm described in any one of claims 1 to 5 is implemented.

Images