Aircraft intelligent control method based on dynamic teaching deep reinforcement learning

Abstract

The invention provides an aircraft intelligent control method based on dynamic teaching deep reinforcement learning, and the method comprises the steps: constructing an aircraft feedback control architecture: taking a deep neural network as an aircraft intelligent controller, taking a command signal and a real-time state as neural network input, and taking an execution mechanism control signal as neural network output; an aircraft simulation environment containing hidden parameter random pulling deviation is constructed; constructing an expert controller according to the hidden parameters; constructing a neural network model of the intelligent controller: adopting an Actor-Critic network structure when the intelligent controller is trained; constructing a mixed experience pool, wherein data maintained by the mixed experience pool comprises an experience data set, a priority data set and an environment similarity data set; and training the deep neural network by adopting a determined strategy gradient in combination with a dynamic expert teaching deep reinforcement learning algorithm to obtain a control strategy for realizing a flight index to the maximum extent. According to the invention, the control quality of the aircraft control method for high uncertainty of hidden kinetic parameters can be improved.

Description

Technical Field

[0001] This invention belongs to the field of aircraft guidance and control technology, and particularly relates to an intelligent control method for aircraft based on dynamic teaching deep reinforcement learning. Background Technology

[0002] High-speed flight is influenced by many unknown factors. Currently, ground-based wind tunnels are insufficient to accurately simulate high-Mach flight environments. Models built using theoretical analysis, ground tests, and limited flight test data cannot yet describe these uncertainties. Furthermore, boundary layer transition and aeroelasticity models for high-speed flight contain many factors that cannot currently be modeled and are typically treated as disturbances. These disturbances, along with random disturbances, further increase the uncertainty in aircraft modeling.

[0003] Existing traditional control theories, such as gain-scheduled PID control and extended observers in active disturbance rejection control, can improve the adaptability of control systems to complex characteristics to some extent. However, they have limitations when dealing with the large uncertainties in implicit parameters and complex models of high-speed aircraft. For example, for the same flight characteristic point, the dynamic characteristics can vary over a wide range, and if the characteristic parameter representing this variation is implicit and unobservable, it will be impossible to effectively schedule the control gain. When the controlled object model is highly complex, it will affect the accuracy of state estimation and identification, greatly weakening the observation compensation effect. These problems can reduce the quality of the controller and even lead to aircraft instability. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0005] This invention provides an intelligent control method for aircraft based on dynamic teaching deep reinforcement learning, the method comprising:

[0006] Constructing an aircraft feedback control architecture: using a deep neural network as the aircraft's intelligent controller, with command signals and real-time status as neural network inputs, and actuator control signals as neural network outputs;

[0007] Construct an aircraft simulation environment with implicit parameters and random bias: The aircraft simulation environment is a longitudinal dynamics model of the aircraft;

[0008] Construct an expert controller based on implicit parameters;

[0009] Constructing a neural network model for the intelligent controller: The intelligent controller employs an Actor-Critic network structure during training;

[0010] Construct a hybrid experience pool, which maintains data including: experience datasets, priority datasets, and environment similarity datasets;

[0011] A deep neural network is trained using a deterministic policy gradient combined with a dynamic expert teaching deep reinforcement learning algorithm to obtain a control strategy that maximizes flight performance.

[0012] Furthermore, in the constructed aircraft feedback control architecture, the input is set as... , These represent the error between the program signal and the angle signal, the integral of the angular velocity signal and the error signal, respectively. The controller action is as follows: .

[0013] Furthermore, within the constructed aircraft simulation environment, a longitudinal channel dynamic model of the unpowered glider is established:

[0014]

[0015] Among them, subscript Indicates the geocentric coordinate system, subscript Indicates the navigation coordinate system, subscript Indicates the projectile coordinate system, subscript Represents a velocity coordinate system; These represent the angle of attack and the sideslip angle, respectively. They represent Tie The trajectory inclination angle, trajectory deviation angle, and tilt angle are determined according to the anti-Euler rotation sequence. They represent System relative to The rotational angular velocity is in Projection onto the x, y, and z axes; They represent System relative to The rotational angular velocity is in Projections onto the x and z axes; Indicates the lift coefficient at zero angle of attack. This represents the partial derivative of the lift coefficient with respect to the angle of attack. This represents the partial derivative of the lift coefficient with respect to the rudder deflection angle. Indicates the pitch moment coefficient at zero angle of attack. This represents the partial derivative of the pitch moment coefficient with respect to the angle of attack. This represents the partial derivative of the pitch moment coefficient with respect to the pitch angular velocity. This represents the partial derivative of the pitching moment coefficient with respect to the rudder deflection angle. For flight pressure, For equivalent area, Equivalent length; Indicates quality, Indicates speed, Indicates lift. Represents gravity. Let x, y, and z represent the moments of inertia along the x, y, and z axes, respectively. Represents the product of inertia. Indicates control torque. They represent System relative to The rotational angular velocity is in Projections onto the x, y, and z axes Indicates the lift coefficient. Indicates the pitching moment coefficient. Represents the rudder deflection angle; the model input is... and The model output is , and Selection of hidden dynamic parameters Latent parameters are randomly biased. The range of values ​​is , It represents the pitch angular velocity.

[0016] Furthermore, within the constructed aircraft simulation environment, a servo motor model is established:

[0017]

[0018] in, This represents a servo motor model with a working range of... The random deviation range is , Indicates the aircraft's status.

[0019] Furthermore, in the constructed aircraft simulation environment, the reward function As shown in the following formula:

[0020]

[0021] in, Normalization refers to controlling the corresponding variable within a certain fixed range of values. The relevant item is a tracking reward; This represents angular velocity, and the related term is the stability bonus. The relevant item represents the energy reward for the implementing agency. Related terms indicate smoothed rewards; To encourage high precision, this means that the normalized angle error is less than a preset value. and normalized angular velocity less than preset value A "significant reward" will be given at that time; parameters to The scale coefficients for tracking rewards, stability rewards, energy rewards, and smoothing rewards respectively determine the importance percentage of the corresponding rewards in the total rewards; to These are the sensitivity coefficients for tracking rewards, stability rewards, energy rewards, and smoothing rewards, respectively, which determine the degree of precision the agent needs to control its state or actions to obtain the corresponding reward.

[0022] Furthermore, the expert controller employs a PID controller, and the PID controller parameter design rule uses the pole placement method, which is applied in each round of agent training based on implicit dynamic parameters. The pull-off condition determines the second-order model of the controlled object. After correction, the system pole position is selected so that the principal pole of the servo model is 1 / 6 of the real axis.

[0023] Furthermore, the intelligent controller neural network model adopts an Actor-Critic network structure. Both the Actor and Critic networks use a 4-layer fully connected structure with 128 nodes in each intermediate layer. The Actor activation layer uses the tanh function, and the output is between [-1, 1]. The output of each layer of the Actor and Critic networks is normalized, and the network weights are initialized using an orthogonal initialization method.

[0024] Furthermore, the dataset maintained by the hybrid experience pool includes experience datasets. Priority dataset and environment similarity dataset The elements in the three datasets E, P, and Sr correspond one-to-one; the elements in E are represented as... , For the first Empirical data from training rounds, including , , , , and These represent, in order, the aircraft state, controller action, reward size, next state, whether it is an expert controller action, and the degree of random bias in the implicit parameters; the elements in P are represented as... , For the first Data priority in training rounds; data in Sr is represented as , For the first Environmental similarity factor during training rounds.

[0025] Furthermore, a deterministic policy gradient combined with a dynamic expert-led deep reinforcement learning algorithm is used to train the deep neural network, specifically including:

[0026] (1) Pre-generate expert data before formal training begins; during training, set the first 80% of training rounds to use the expert controller with a 20% probability and the agent to use the intelligent agent with an 80% probability; and use the intelligent agent to control the last 20% of training rounds.

[0027] (2) Before the start of the current training round, the implicit parameters in the aircraft simulation environment are randomly biased, and an expert controller is dynamically constructed according to the bias; the environment similarity dataset in the hybrid experience pool is updated. :

[0028]

[0029] In the above formula, ,when hour, ; This indicates the degree of skewness caused by implicit parameters in historical data. This indicates the degree of bias in the latent parameters during the current training round;

[0030] (3) During training, add noise to the actions output by the agent (Actor):

[0031]

[0032] In the above formula, and These are two constants, used to control the noise level. The upper and lower limits; It is a random number; Indicates the current round number. Indicates the maximum number of rounds; Indicates the Actor's output. Indicates the result after adding noise. ;

[0033] Initial expert data has a higher priority than initial actor data, and the priority of initial expert data gradually decreases during training.

[0034]

[0035] In the above formula, It is a constant. This indicates the priority of expert experience for the current training step. Indicates the initial priority of expert experience. Indicates the attenuation rate. Indicates the number of training steps;

[0036] After each simulation step in each training round, the dataset is prioritized. Similarity dataset with environment The results of multiplying the corresponding data are used to apply a priority playback mechanism to the empirical dataset. Sampling is performed, with 128 sets of data collected each time, and a deep neural network is trained based on the sampling results;

[0037] (4) After training, the trained agent Actor is embedded into the constructed aircraft feedback control architecture and used as the aircraft intelligent controller for simulation testing. During the test, the implicit dynamic coefficients are randomly biased.

[0038] Furthermore, in step (3), the training method adopts expert Q-value supervision, by adjusting the parameters. To control the degree of expert guidance:

[0039]

[0040] In the above formula, The overall loss function of the Critic network represents the degree of error in the Critic network. These are weighting coefficients used to balance the losses of the two parts; Indicates timing difference error; This indicates sampling from the mixed experience pool D; This indicates that the current Critic network is in state [state name missing]. Next, execute the expert's actions The rating given; This represents the target Critic network.

[0041] This invention provides an intelligent control method for aircraft based on dynamic teaching deep reinforcement learning. This method uses a deep neural network as the intelligent controller for the aircraft, constructing an aircraft simulation environment with randomly biased latent parameters, an expert controller, and a hybrid experience pool. It trains the deep neural network using a deterministic policy gradient combined with a dynamic expert teaching deep reinforcement learning algorithm to obtain the control strategy. This invention leverages the powerful fitting ability of neural networks to fit the relationships between state parameters and latent parameters, and between latent parameters and the adaptive controller, improving the dynamic performance and robustness of the system under conditions of high uncertainty in latent parameters. Compared with existing technologies, this invention improves the control quality of existing aircraft control methods when facing high uncertainty in latent dynamic parameters. Attached Figure Description

[0042] The accompanying drawings, which form part of this specification, are provided to further illustrate embodiments of the invention and, together with the textual description, explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0043] Figure 1 A block diagram of a dynamic teaching deep reinforcement learning method for training intelligent control of aircraft is shown.

[0044] Figures 2 to 8 The diagram shows a comparison between the agent control performance and the expert-taught controller performance under various implicit parameters (rudder effect parameters) and pull conditions; among them Figures 2 to 8 These correspond to the nominal value, 11% deviation, 27% deviation, 50% deviation, -16% deviation, -29% deviation, and -50% deviation, respectively. Detailed Implementation

[0045] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0047] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0048] like Figure 1 As shown in the figure, a specific embodiment of the present invention provides an intelligent control method for aircraft based on dynamic teaching deep reinforcement learning, the method comprising:

[0049] Constructing an aircraft feedback control architecture: using a deep neural network as the aircraft's intelligent controller, with command signals and real-time status as neural network inputs, and actuator control signals as neural network outputs;

[0050] Construct an aircraft simulation environment with implicit parameters and random bias: The aircraft simulation environment is a longitudinal dynamics model of the aircraft;

[0051] Construct an expert controller based on implicit parameters;

[0052] Constructing a neural network model for the intelligent controller: The intelligent controller employs an Actor-Critic network structure during training;

[0053] Construct a hybrid experience pool D, which includes: experience dataset E, priority dataset P, and environment similarity dataset Sr;

[0054] A deep neural network is trained using a deterministic policy gradient combined with a dynamic expert teaching deep reinforcement learning algorithm to obtain a control strategy that maximizes flight performance.

[0055] This configuration provides an intelligent control method for aircraft based on dynamic teaching deep reinforcement learning. This method uses a deep neural network as the intelligent controller, constructing an aircraft simulation environment with randomly biased latent parameters, an expert controller, and a hybrid experience pool. The deep neural network is trained using a deterministic policy gradient combined with a dynamic expert teaching deep reinforcement learning algorithm to obtain the control policy. This method leverages the powerful fitting ability of neural networks to fit the relationships between state parameters and latent parameters, and between latent parameters and the adaptive controller, improving the dynamic performance and robustness of the system under conditions of high uncertainty in latent parameters.

[0056] Deep reinforcement learning technology is suitable for solving complex and difficult-to-model application scenarios. Research on applying this technology to improve the autonomy and intelligence of flight control in the field of aircraft dynamics and control has attracted much attention. Applying this technology to improve the control quality of high-speed aircraft is a new technical approach.

[0057] In this invention, a feedback control architecture for an aircraft is first constructed: a deep neural network is used as the intelligent controller of the aircraft, command signals and real-time status are used as the inputs of the neural network, and control signals of the actuators are used as the outputs of the neural network.

[0058] As a specific embodiment of the present invention, a feedback control architecture is adopted, directly using a deep neural network as the controller, and setting its input as... , These represent the error between the program signal and the angle signal, the integral of the angular velocity signal and the error signal, respectively. The controller action is as follows: Training data is generated by the interaction between the control network and the aircraft simulation environment. The control performance of the network is gradually improved according to the designed reward function, and finally a controller that meets the accuracy requirements is learned.

[0059] Furthermore, after constructing the aircraft feedback control architecture, an aircraft simulation environment containing implicit parameters with random bias is constructed: the aircraft simulation environment is the longitudinal dynamics model of the aircraft.

[0060] As a specific embodiment of the present invention, a six-degree-of-freedom dynamic model of the aircraft and a servo motor model are established.

[0061] This embodiment establishes a longitudinal channel dynamics model for an unpowered glider, as shown below:

[0062]

[0063] Among them, subscript Indicates the geocentric coordinate system, subscript Indicates the navigation coordinate system, subscript Indicates the projectile coordinate system, subscript Represents a velocity coordinate system; These represent the angle of attack and the sideslip angle, respectively. They represent Tie The trajectory inclination angle, trajectory deviation angle, and tilt angle are determined according to the anti-Euler rotation sequence. They represent System relative to The rotational angular velocity is in Projection onto the x, y, and z axes; They represent System relative to The rotational angular velocity is in Projections onto the x and z axes; Indicates the lift coefficient at zero angle of attack. This represents the partial derivative of the lift coefficient with respect to the angle of attack. This represents the partial derivative of the lift coefficient with respect to the rudder deflection angle. Indicates the pitch moment coefficient at zero angle of attack. This represents the partial derivative of the pitch moment coefficient with respect to the angle of attack. This represents the partial derivative of the pitch moment coefficient with respect to the pitch angular velocity. This represents the partial derivative of the pitching moment coefficient with respect to the rudder deflection angle. For flight pressure, For equivalent area, Equivalent length; Indicates quality, Indicates speed, Indicates lift. Represents gravity. Let x, y, and z represent the moments of inertia of rotation along the x, y, and z axes, respectively. Represents the product of inertia. Indicates control torque. They represent System relative to The rotational angular velocity is in Projections onto the x, y, and z axes Indicates the lift coefficient. Indicates the pitching moment coefficient. Represents the rudder deflection angle; the model input is... and The model output is , and Selection of hidden dynamic parameters Latent parameters are randomly biased. The range of values ​​is , It represents the pitch angular velocity.

[0064] The servo model is a second-order model, and its working range is... The random deviation range is :

[0065]

[0066] in, This represents a servo motor model. Indicates the aircraft's status.

[0067] reward function As shown in the following formula:

[0068]

[0069] in, Normalization refers to controlling the corresponding variable within a certain fixed range of values. The relevant item is a tracking reward; This represents angular velocity, and the related term is the stability bonus. The relevant item represents the energy reward for the implementing agency. Related terms indicate smoothed rewards; To encourage high precision, this means that the normalized angle error is less than a preset value. and normalized angular velocity less than preset value A "significant reward" will be given at that time; parameters to The scale coefficients for tracking rewards, stability rewards, energy rewards, and smoothing rewards respectively determine the importance percentage of the corresponding rewards in the total rewards; to These are the sensitivity coefficients for tracking rewards, stability rewards, energy rewards, and smoothing rewards, respectively, which determine the degree of precision the agent needs to control its state or actions to obtain the corresponding reward.

[0070] Furthermore, after constructing the aircraft simulation environment, an expert controller is built based on the implicit parameters.

[0071] The expert controller can be a PID controller. As a specific embodiment of the invention, the PID controller parameter design rule adopts the pole placement method. In each round of agent training, according to… The second-order model of the controlled object is determined by the pull-off condition. After correction, the position of the system pole is selected so that the principal pole of the servo model is 1 / 6 on the real axis, and then the three parameters of the PID controller are calculated.

[0072] Furthermore, after constructing the expert controller, a neural network model for the intelligent controller is constructed.

[0073] As a specific embodiment of the present invention, the intelligent controller neural network model adopts an Actor-Critic network structure. Both the Actor and Critic networks use a 4-layer, fully connected structure. Each intermediate layer has 128 nodes. The Actor activation layer uses the tanh function to ensure that the output is between [-1, 1]. The output of each layer of the Actor and Critic networks needs to be normalized, and the network weights are initialized using an orthogonal initialization method.

[0074] Furthermore, after constructing the neural network model of the intelligent controller, a hybrid experience pool is constructed. The dataset maintained by the hybrid experience pool includes: experience dataset E, priority dataset P, and environment similarity dataset Sr.

[0075] As a specific embodiment of the present invention, the data set maintained by the hybrid experience pool includes an experience dataset. Priority dataset and environment similarity dataset The elements in these three datasets correspond one-to-one. The elements in E are represented as... , For the first Empirical data from training rounds, including , , , , and , The elements in P represent, in order, the aircraft state, controller action, reward size, next state, whether it is an expert controller action, and the degree of random bias in the implicit parameters. The elements in P are represented as follows: , For the first Data priority in training rounds. Data in Sr is represented as... , For the first Environmental similarity factor during training rounds.

[0076] In this embodiment, the main functions of the hybrid experience pool include: initialization, adding data, data sampling, updating the environment similarity dataset, and updating the priority dataset.

[0077] Experience dataset The data includes three categories: pre-generated data from the dynamic expert controller before training, dynamic expert control result data during training, and agent (Actor) control result data during training. Each piece of empirical data in E... Each corresponds to a priority. The initial priority of expert data is set to 5, and the initial priority of non-expert data is set to 1. After sampling and training are completed, the weights corresponding to the sampled data are updated to reflect the Td error during the Critic network training process. The environment similarity dataset Sr is updated before each round based on the bias in that round; the update method will be explained in the next step. Initial values ​​are assigned to the data. Each set of empirical data Corresponding environmental similarity factor Stored in Sr.

[0078] Furthermore, after constructing a hybrid priority playback experience data pool, a deterministic policy gradient combined with a dynamic expert teaching deep reinforcement learning algorithm is used to train the deep neural network to obtain a control strategy that maximizes flight performance.

[0079] As a specific embodiment of the present invention, this step specifically includes:

[0080] (1) Pre-generate expert data before formal training begins. The pre-generated expert data can be set to 20% of the maximum number of training rounds. In the training process, this invention can be set to use an expert controller with a 20% probability and an agent controller with an 80% probability in the first 80% of training rounds. In the last 20% of training rounds, the agent controller is used for control.

[0081] (2) Randomly bias the implicit parameters in the aircraft simulation environment before the start of the current training round. Simultaneously, an expert controller is dynamically constructed based on the aforementioned steps to address biased scenarios. The environmental similarity dataset in the hybrid experience pool is updated. Its size depends on the degree of skewness of implicit parameters in historical data. Degree of bias compared to this round The degree of closeness, when and The closer the similarity, the higher the similarity factor and the higher the priority. The environmental similarity factor update method is as follows:

[0082]

[0083] In the above formula, ,when hour, It can be designed according to the actual situation of the implicit parameters.

[0084] (3) During the training process, noise is added to the output actions of the agent (Actor). This noise gradually decreases as the number of training rounds increases, as shown in the following formula:

[0085]

[0086] In the above formula, and For two constants ( ), respectively used to control noise level The upper and lower limits; It is a random number; Indicates the current round number. Indicates the maximum number of rounds; Indicates the Actor's output. Indicates the result after adding noise. .

[0087] The initial expert data has a higher priority than the initial agent data, but the priority of the initial expert data gradually decreases as the training process progresses, as shown in the following formula.

[0088]

[0089] In the above formula, It is a constant. This indicates the priority of expert experience for the current training step. Indicates the initial priority of expert experience. Indicates the attenuation rate. This represents the number of training steps. Expert priority decays exponentially with the number of training steps, but expert data may still be sampled in the later stages of training.

[0090] After each simulation step in each training round, the dataset is prioritized. Similarity dataset with environment The results of multiplying the corresponding data are used to apply a priority playback mechanism to the empirical dataset. Sampling is performed, with 128 data sets collected each time. A deep neural network is trained based on the sampling results. The training method employs expert Q-value supervision, where parameters are adjusted... To control the degree of expert guidance:

[0091]

[0092] In the above formula, The overall loss function of the Critic network represents the degree of error in the Critic network. These are weighting coefficients used to balance the losses of the two parts; Indicates timing difference error; This indicates sampling from the mixed experience pool D; This indicates that the current Critic network is in state [state name missing]. Next, execute the expert's actions The rating given for this behavior; This represents the target Critic network.

[0093] (4) After training, the trained agent Actor is embedded into the aircraft feedback control architecture in the initial step and used as the aircraft intelligent controller for simulation testing. During the test, the implicit dynamic coefficients are randomly biased.

[0094] This invention constructs a training environment based on random bias conditions of the aircraft's latent parameters. It utilizes the DDPG framework combined with an expert-taught controller dynamically designed according to random bias conditions and pre-defined design rules. Leveraging the powerful fitting capabilities of neural networks, it fits the relationships between state parameters and latent parameters, and between latent parameters and the adaptive controller, thereby improving the system's dynamic performance and robustness under conditions of significant latent parameter uncertainty. The advantages of this invention compared to existing technologies are:

[0095] (1) This invention designs an aircraft attitude controller based on a deep reinforcement learning algorithm. It utilizes the powerful fitting ability of deep neural networks to provide a new solution for scenarios where the large uncertainty of latent parameters makes it difficult to implement the gain scheduling method and the complex model observation compensation effect is poor.

[0096] (2) Compared with other teaching deep reinforcement learning algorithms, this invention dynamically designs an expert teaching controller based on random bias and preset design rules during the neural network training process, and controls the proportion of expert data in the experience data, which improves the quality of expert data, guides the training process, and ensures the autonomous exploration ability of the agent.

[0097] (3) This invention is based on deep reinforcement learning, but classical control technology is used in the teaching assistance process. While accelerating the convergence of the neural network, it improves the consistency of the dynamic performance of the neural network under the condition of implicit parameter bias, and realizes the cross-integration and complementary advantages of classical control theory and deep reinforcement learning technology.

[0098] To gain a further understanding of the present invention, the intelligent control method for aircraft based on dynamic teaching deep reinforcement learning of the present invention will be described in detail below with reference to specific embodiments.

[0099] This embodiment provides an intelligent control method for aircraft based on dynamic teaching deep reinforcement learning, such as... Figure 1 As shown, a simulation environment for an aircraft with latent parameters and random yaw is constructed, and deep reinforcement learning algorithms are applied to the aircraft's attitude control. The DDPG algorithm is combined with an expert controller dynamically designed based on the random yaw situation and preset rules for teaching-assisted training. This provides a novel control scheme for scenarios where the large uncertainty of latent parameters makes gain scheduling methods difficult to implement and complex model observation compensation effects are poor. Figure 1The diagram illustrates a dynamic teaching deep reinforcement learning method for training intelligent control of aircraft. This method is based on the Actor-Critic paradigm and mainly includes an Actor network (also known as a policy network), a Critic network (also known as a value network), an expert PID controller, an aircraft simulation model, and a hybrid experience pool.

[0100] The intelligent control method for aircraft based on dynamic teaching deep reinforcement learning in this embodiment specifically includes the following steps.

[0101] Step 1: Construct the aircraft feedback control architecture: Use a deep neural network as the aircraft's intelligent controller, with command signals and real-time status as neural network inputs, and actuator control signals as neural network outputs.

[0102] A feedback control architecture is adopted, in which a deep neural network is directly used as the controller. The command signal and real-time state are used as the input of the neural network, and the control signal of the actuator is used as the output of the neural network. Training data is generated by the interaction between the control network and the aircraft simulation environment, and the control performance of the network is gradually improved according to the designed reward function. Finally, a controller that meets the accuracy requirements is learned.

[0103] Step 2: Construct an aircraft simulation environment with hidden parameters for random bias: The aircraft simulation environment is the longitudinal dynamics model of the aircraft.

[0104] The aircraft simulation environment is a longitudinal dynamics model of the aircraft, with basic implicit dynamic parameters selected. The random deviation range is The reward function is shown in the following formula:

[0105]

[0106] The parameter values ​​are shown in the table below:

[0107] Table 1. List of parameters for the reward function

[0108]

[0109] Step 3: Construct an expert controller based on the implicit parameters.

[0110] The expert controller employs a PID controller, and the PID controller parameter design rule uses the pole placement method. In each round of agent training, based on... The second-order model of the controlled object is determined by the pull-off condition. After correction, the position of the system pole is selected so that the principal pole of the servo model is 1 / 6 on the real axis, and then the three parameters of the PID controller are calculated.

[0111] Step 4: Construct the intelligent controller neural network model: The intelligent controller neural network model adopts a dual Actor-Critic network structure.

[0112] The neural network model employs a dual Actor-Critic network structure. Both the Actor and Critic networks use a 4-layer, fully connected structure. Each intermediate layer has 128 nodes. The Actor activation layer uses the tanh function to ensure the output is between [-1, 1]. The output of each layer in both the Actor and Critic networks needs to be normalized, and the network weights are initialized using an orthogonal initialization method.

[0113] Step 5: Construct a hybrid experience pool. The dataset maintained by the hybrid experience pool includes: experience dataset, priority dataset, and environment similarity dataset.

[0114] The hybrid experience pool maintains three datasets: the experience dataset and the experience dataset. Priority dataset and environment similarity dataset Implemented using the Python programming language, its main functions include: initialization, adding data, data sampling, updating the environment similarity dataset, and updating the priority dataset.

[0115] Step six: Train the deep neural network using a deterministic policy gradient combined with a dynamic expert teaching deep reinforcement learning algorithm to obtain a control strategy that maximizes flight performance.

[0116] (1) Initialize the parameters of the evaluation network and policy network orthogonally;

[0117] (2) Set hyperparameters;

[0118] Table 2 Hyperparameter List

[0119]

[0120] (3) Before formal training, generate 20% of the expert data in the rounds and put it into the mixed experience pool;

[0121] (4) for episode=1,Max_episode do;

[0122] (5) Randomly adjust the implicit parameters, record the degree of adjustment, initialize the simulation model, and obtain the initial observation state. Design an expert controller based on the degree of adjustment and expected performance;

[0123] (6) Update the environment similarity dataset based on the degree of bias of the implicit parameters in this round;

[0124] (7) for t=1,T do;

[0125] (8) Control the aircraft model using an expert controller with a 20% probability, and in Data in the set is labeled as expert data and assigned an initial priority value, which gradually decays during training; the aircraft model is controlled by an agent (Actor) with 80% probability, and noise is added. Mark the data in the set as non-expert data and assign it an initial priority value;

[0126] (9) According to Sets and The result of multiplying corresponding elements of a set A set of data is obtained by sampling the dataset, the loss function of the evaluation network is calculated, and the evaluation network, policy network, and target network are updated.

[0127] (10) According to renew Data priority within a set;

[0128] (11) end for, corresponding to step (7);

[0129] (12) end for, corresponding to step (4).

[0130] In summary, this invention provides an intelligent control method for aircraft based on dynamic teaching deep reinforcement learning. This method obtains the nonlinear mapping between state parameters and latent parameters, and between latent parameters and adaptive controllers through deep neural networks, thereby enhancing the robustness of the system.

[0131] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for intelligent control of aircraft based on dynamic teaching deep reinforcement learning, characterized in that, The intelligent control method for aircraft based on dynamic teaching deep reinforcement learning includes: Constructing an aircraft feedback control architecture: using a deep neural network as the aircraft's intelligent controller, with command signals and real-time status as neural network inputs, and actuator control signals as neural network outputs; Construct an aircraft simulation environment with implicit parameters and random bias: The aircraft simulation environment is a longitudinal dynamics model of the aircraft; Construct an expert controller based on implicit parameters; Constructing a neural network model for the intelligent controller: The intelligent controller employs an Actor-Critic network structure during training; Construct a hybrid experience pool, which includes: an experience dataset, a priority dataset, and an environment similarity dataset; A deep neural network is trained using a deterministic policy gradient combined with a dynamic expert teaching deep reinforcement learning algorithm to obtain a control strategy that maximizes flight performance.

2. The intelligent control method for aircraft based on dynamic teaching deep reinforcement learning according to claim 1, characterized in that, In the constructed aircraft feedback control architecture, the input is set as... , These represent the error between the program signal and the angle signal, the integral of the angular velocity signal and the error signal, respectively. The controller action is as follows: .

3. The intelligent control method for aircraft based on dynamic teaching deep reinforcement learning according to claim 2, characterized in that, In the constructed aircraft simulation environment, a longitudinal channel dynamic model of the unpowered glider is established: , Among them, subscript Indicates the geocentric coordinate system, subscript Indicates the navigation coordinate system, subscript Indicates the projectile coordinate system, subscript Represents a velocity coordinate system; These represent the angle of attack and the sideslip angle, respectively. They represent Tie The trajectory inclination angle, trajectory deviation angle, and tilt angle are determined according to the anti-Euler rotation sequence. They represent System relative to The rotational angular velocity is in Projection onto the x, y, and z axes; They represent System relative to The rotational angular velocity is in Projections onto the x and z axes; Indicates the lift coefficient at zero angle of attack. This represents the partial derivative of the lift coefficient with respect to the angle of attack. This represents the partial derivative of the lift coefficient with respect to the rudder deflection angle. Indicates the pitch moment coefficient at zero angle of attack. This represents the partial derivative of the pitch moment coefficient with respect to the angle of attack. This represents the partial derivative of the pitch moment coefficient with respect to the pitch angular velocity. This represents the partial derivative of the pitching moment coefficient with respect to the rudder deflection angle. For flight pressure, For equivalent area, Equivalent length; Indicates quality, Indicates speed, Indicates lift. Represents gravity. Let x, y, and z represent the moments of inertia along the x, y, and z axes, respectively. Represents the product of inertia. Indicates control torque. They represent System relative to The rotational angular velocity is in Projections onto the x, y, and z axes Indicates the lift coefficient. Indicates the pitch moment coefficient. Represents the rudder deflection angle; the model input is... and The model output is , and Selection of hidden dynamic parameters Latent parameters are randomly biased. The range of values ​​is , It represents the pitch angular velocity.

4. The intelligent control method for aircraft based on dynamic teaching deep reinforcement learning according to claim 3, characterized in that, In the constructed aircraft simulation environment, a servo motor model is created: ,in, This represents a servo motor model with a working range of... The random deviation range is , Indicates the aircraft's status.

5. The intelligent control method for aircraft based on dynamic teaching deep reinforcement learning according to claim 4, characterized in that, In the constructed aircraft simulation environment, the reward function As shown in the following formula: ， in, Normalization refers to controlling the corresponding variable within a certain fixed range of values. The relevant item is a tracking reward; This represents angular velocity, and the related term is the stability bonus. The relevant item represents the energy reward for the implementing agency. Related terms indicate smoothed rewards; To encourage high precision, this means that the normalized angle error is less than a preset value. and normalized angular velocity less than preset value A "significant reward" will be given at that time; parameters to The scale coefficients for tracking rewards, stability rewards, energy rewards, and smoothing rewards respectively determine the importance percentage of the corresponding rewards in the total rewards; to These are the sensitivity coefficients for tracking rewards, stability rewards, energy rewards, and smoothing rewards, respectively, which determine the degree of precision the agent needs to control its state or actions to obtain the corresponding reward.

6. The intelligent control method for aircraft based on dynamic teaching deep reinforcement learning according to claim 5, characterized in that, The expert controller employs a PID controller, and the PID controller parameter design rule uses the pole placement method. During each round of agent training, the implicit dynamic parameters are used... The pull-off condition determines the second-order model of the controlled object. After correction, the system pole position is selected so that the principal pole of the servo model is 1 / 6 of the real axis.

7. The intelligent control method for aircraft based on dynamic teaching deep reinforcement learning according to claim 1, characterized in that, The intelligent controller neural network model adopts an Actor-Critic network structure. Both the Actor and Critic networks are 4-layer networks with a fully connected structure and 128 nodes in each intermediate layer. The Actor activation layer uses the tanh function, and the output is between [-1, 1]. The output of each layer of the Actor and Critic networks is normalized, and the network weights are initialized using an orthogonal initialization method.

8. The intelligent control method for aircraft based on dynamic teaching deep reinforcement learning according to claim 6, characterized in that, The dataset maintained by the hybrid experience pool includes experience datasets. Priority dataset and environment similarity dataset The elements in the three datasets E, P, and Sr correspond one-to-one; the elements in E are represented as... , For the first Empirical data from training rounds, including , , , , and , respectively representing the aircraft state, controller action, reward size, next state, whether it is an expert controller action, and the degree of random bias of implicit parameters; the elements in P are represented as , For the first Data priority in training rounds; data in Sr is represented as , For the first Environmental similarity factor during training rounds.

9. The intelligent control method for aircraft based on dynamic teaching deep reinforcement learning according to claim 8, characterized in that, The deep neural network is trained using a deterministic policy gradient combined with a dynamic expert teaching deep reinforcement learning algorithm, specifically including: (1) Pre-generate expert data before formal training begins; during training, set the first 80% of training rounds to use the expert controller with a 20% probability and the agent to use the intelligent agent with an 80% probability; and use the intelligent agent to control the last 20% of training rounds. (2) Before the start of the current training round, the implicit parameters in the aircraft simulation environment are randomly biased, and an expert controller is dynamically constructed according to the bias; the environment similarity dataset in the hybrid experience pool is updated. : ， In the above formula, ,when hour, ; This indicates the degree of skewness caused by implicit parameters in historical data. This indicates the degree of bias in the latent parameters during the current training round; (3) During training, add noise to the actions output by the agent (Actor): ， In the above formula, and These are two constants, used to control the noise level. The upper and lower limits; It is a random number; Indicates the current round number. Indicates the maximum number of rounds; Indicates the Actor's output. Indicates the result after adding noise. ; Initial expert data has a higher priority than initial actor data, and the priority of initial expert data gradually decreases during training. ， In the above formula, It is a constant. This indicates the priority of expert experience for the current training step. Indicates the initial priority of expert experience. Indicates the attenuation rate. Indicates the number of training steps; After each simulation step in each training round, the dataset is prioritized. Similarity dataset with environment The results of multiplying the corresponding data are used to apply a priority playback mechanism to the empirical dataset. Sampling is performed, with 128 sets of data collected each time, and a deep neural network is trained based on the sampling results; (4) After training, the trained agent Actor is embedded into the constructed aircraft feedback control architecture and used as the aircraft intelligent controller for simulation testing. During the test, the implicit dynamic coefficients are randomly biased.

10. The intelligent control method for aircraft based on dynamic teaching deep reinforcement learning according to claim 9, characterized in that, In step (3), the training method adopts expert Q-value supervision, which is achieved by adjusting the parameters. To control the degree of expert guidance: ， In the above formula, The overall loss function of the Critic network represents the degree of error in the Critic network. These are weighting coefficients used to balance the losses of the two parts; Indicates timing difference error; This indicates sampling from the mixed experience pool D; This indicates that the current Critic network is in state [state name missing]. Next, execute the expert's actions The rating given; This represents the target Critic network.

Images