Method and device for generating aircraft attitude control model, equipment and medium

Abstract

The embodiment of the invention provides an aircraft attitude control model generation method and device, equipment and a medium, and is applied to the technical field of aircraft control. The method comprises the following steps: acquiring a flight task and a flight environment; according to a pre-constructed simulation model, carrying out analogue simulation on the flight task and the flight environment, and outputting simulated flight data; training a pre-constructed reinforcement learning model by taking a flight strategy as an output and the simulation flight data as an input; and when a preset stop condition is met, stopping training, and taking the trained reinforcement learning model as an aircraft attitude control model. In this way, the training period and training cost of the controller can be reduced, and the aircraft attitude control model with a better control effect can be trained.

Description

Technical Field

[0001] The present disclosure relates to the field of aircraft control technology, and in particular to a method, device, equipment and medium for generating an aircraft attitude control model. Background Art

[0002] In recent years, aircraft have been widely promoted and applied in civil fields such as agricultural plant protection, power inspection, urban security, forest fire prevention, environmental monitoring, low-altitude aerial photography and aerial search and rescue, and in military fields such as battlefield environmental reconnaissance, transportation support, patrol surveillance, etc. Aircraft have the advantages of fast maneuverability, wide coverage, the ability to observe ground conditions from a "God's perspective", and rapid response to commands. In the future, they will become more popular and have richer application scenarios.

[0003] Attitude control is the core of aircraft flight control, and its quality directly affects the effect of the entire flight control. Conventional attitude control uses a cascade PID control structure, which outputs the control law to the servo by controlling the errors of the aircraft angle loop and angular velocity loop. However, there are interferences caused by unmodeled dynamics and uncertainties during the flight process, such as gusts, parameter perturbations and other continuous, time-varying disturbances. Traditional methods are difficult to resist the influence of interference, have low adaptability, and the debugging of control parameters is relatively complicated, requiring continuous trial and error, resulting in reduced stability margin of aircraft control and even the risk of loss of control.

[0004] Model predictive control is a classic control method for aircraft control. However, this method requires detailed modeling of the aircraft and the environment. When there are errors in the model, the control accuracy will decrease.

[0005] Reinforcement learning uses an end-to-end trial-and-error training method and an established reward and punishment mechanism to guide the controlled object to select the optimal action to apply to the environment, maximize the cumulative reward value, and find the optimal strategy. During the training process, factors such as uncertain interference and noise are introduced. Based on the empirical knowledge of the training data, the strategy learning mechanism is explored and utilized, which has strong adaptability and autonomous learning ability for complex disturbance environments. Using reinforcement learning methods to achieve aircraft control is a new control method in recent years. This method does not require the construction of a dynamic model of the aircraft and the environment. Instead, it allows the aircraft to "trial and error" in the environment, thereby continuously training the controller based on the reward, and ultimately achieving a better control effect. However, this method has a long training cycle and a high training cost. Summary of the invention

[0006] The present invention provides a method, device, equipment and medium for generating an aircraft attitude control model.

[0007] According to a first aspect of the present disclosure, a method for generating an aircraft attitude control model is provided. The method comprises:

[0008] Obtain flight mission and flight environment;

[0009] According to the pre-built simulation model, the flight mission and the flight environment are simulated and the simulated flight data is output;

[0010] The pre-built reinforcement learning model is trained with the flight strategy as output and the simulated flight data as input;

[0011] When the preset stop condition is met, the training is stopped and the trained reinforcement learning model is used as the aircraft attitude control model.

[0012] According to the above aspects and any possible implementation, an implementation is further provided, wherein the simulation model is represented by the following formula:

[0013]

[0014] x=[α,ω z1 ] T , u=[δ z1 , P] T ,

[0015]

[0016] Where α is the angle of attack, ω z1 is the pitch angular velocity, δ z1 To control the rudder deflection angle, P is the prediction step number, q is the velocity head, S is the characteristic cross-sectional area, L is the characteristic length of the fuselage, m is the control step number, is the aerodynamic coefficient at angle of attack, is the aerodynamic moment coefficient, is the control rudder aerodynamic coefficient, v is the flight speed, J z1 is the rotational inertia of the body.

[0017] According to the above aspects and any possible implementation, an implementation is further provided, wherein the reinforcement learning model is constructed based on an actor-critic network;

[0018] Among them, the actor network is composed of pre-built strategy models.

[0019] According to the above aspects and any possible implementation, there is further provided an implementation, wherein the reinforcement learning model is configured with a generalized advantage function;

[0020] The generalized advantage function is shown in the following formula:

[0021]

[0022] Among them, V(s t) is the state value estimated by the neural network, γ, λ are the hyperparameters for adjusting the bias and variance, l is the discount number, r t+l is the reward at time t+l, s t is the state at time t.

[0023] According to the above aspects and any possible implementation, an implementation is further provided, wherein the update gradient of the reinforcement learning model is performed according to the following formula:

[0024]

[0025] Among them, θ′ is the gradient after update; θ is the gradient before update; H is solved by using the conjugate gradient algorithm x = H -1 g is the Hessian matrix of the KL divergence obtained; α is the original update step size; j∈{1, 2, ..., K} is a hyperparameter. In the actual gradient update, the value of j is the minimum value obtained by iteration that can make θ′ satisfy the KL divergence constraint.

[0026] According to the above aspects and any possible implementation, an implementation is further provided, wherein the policy model is constructed by the following steps:

[0027] The strategy model is constructed according to the simulation model and a preset second algorithm.

[0028] According to the above aspects and any possible implementation, there is further provided an implementation, wherein the strategy model is pre-built with a reward function;

[0029] The reward function is shown in the following formula:

[0030]

[0031] Among them, (σ 0 , σ 1 , y) is the preference triple, D is the data set, y is the label, σ 0 ={s k , a k ,…,s k+H , a k+H}0 and σ 1 ={s k , a k ,…,s k+H , a k+H}1 is the trajectory segment, y(0) is σ 0 The corresponding rolling time domain optimization label, y(1) is σ 1 The corresponding rolling horizon optimization label, σ i >σ jindicates that trajectory segment i is better than trajectory segment j, P Ψ The reward function based on the Bradley-Terry model is a predictor model of preference, which is expressed as follows:

[0032]

[0033] Status Next action Reward.

[0034] According to a second aspect of the present disclosure, a device for generating an aircraft attitude control model is provided. The device comprises:

[0035] Data acquisition module, used to obtain flight missions and flight environment;

[0036] A data simulation module, used to simulate the flight mission and the flight environment according to a pre-built simulation model, and output simulated flight data;

[0037] A model training module, used to train a pre-built reinforcement learning model with the flight strategy as output and the simulated flight data as input;

[0038] The model generation module is used to stop training when the preset stop condition is met, and use the trained reinforcement learning model as the aircraft attitude control model.

[0039] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0040] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0041] According to the method, device, equipment and medium for generating an aircraft attitude control model provided by the embodiments of the present disclosure, by collecting data on the manipulation of various tasks in the simulation system and the autonomous interaction of the intelligent agent with the simulation environment, strategy learning is completed using a reinforcement learning decision-making intelligent algorithm, thereby reducing the training cycle and training cost of the controller, and the trained aircraft attitude control model has a better control effect.

[0042] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0044] Figure 1 A flow chart of a method for generating an aircraft attitude control model according to an embodiment of the present disclosure is shown;

[0045] Figure 2 A block diagram of a device for generating an aircraft attitude control model according to an embodiment of the present disclosure is shown;

[0046] Figure 3 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0048] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0049] In this disclosure, based on the proximal strategy optimization method, a highly scalable reinforcement learning framework suitable for multiple tasks is proposed. By manipulating various tasks in the simulation system and collecting data on the autonomous interaction between the intelligent agent and the simulation environment, strategy learning is completed using the reinforcement learning decision-making intelligent algorithm.

[0050] Figure 1 A flow chart of a method 100 for generating an aircraft attitude control model according to an embodiment of the present disclosure is shown. The method 100 includes:

[0051] Step 110, obtaining the flight mission and flight environment.

[0052] In some embodiments, the flight mission and flight environment are obtained. For example, the flight environment may include mountains, sea, wind (specific wind direction, wind level, etc.), rain, etc. The flight mission is to fly from a to b (covering the flight environment set above).

[0053] Step 120, simulating the flight mission and the flight environment according to the pre-built simulation model, and outputting simulated flight data.

[0054] In some embodiments, the simulation model in the controller performs simulation according to the provided flight mission and flight environment, that is, plans the flight path and flight attitude, etc. by itself, and then generates a large amount of simulated flight data (that is, flight path and flight attitude).

[0055] Specifically, the construction process of the simulation model is as follows:

[0056] First, the simulation model is laid out with the angle of attack α and sideslip angle β of the aircraft during flight, which are used to represent the relationship between the aircraft body coordinate system and the velocity coordinate system. Angle of attack α: The angle that the velocity axis rotates around the Oz2 axis in the positive direction. It represents the angle between the body velocity vector and the Ox1z1 plane. When the body is facing upward (when the Oy1 axis is upward), α is positive when the velocity vector is below the Ox1z1 plane, and negative otherwise. Sideslip angle β: After the velocity axis rotates around the Oz2 axis in the positive direction by α, the Oy'2 axis coincides with the Oy1 axis, and the angle rotated around the Oy1 axis in the positive direction. It represents the angle between the projection of the body velocity vector on the Ox1z1 plane and the body longitudinal axis Ox1. When the body is facing upward, looking toward the nose, β is positive when the velocity vector is on the right side of the Ox1 axis, and negative otherwise.

[0057] The transformation relationship between two coordinates It can be expressed as a matrix:

[0058]

[0059] The two angles can be related by the component v of the velocity vector in the body coordinate system x1 , v y1 , v z1 To express it, that is, formula 2:

[0060]

[0061] Where v is the flight speed.

[0062] According to the momentum theorem, the body rotation equation around the center is expressed as formula 3:

[0063]

[0064] In the formula, H is the sum of the momentum vector of the body, M is the various torques acting on the body, t is time, and ω is the angular velocity. Here, the three components of the momentum in the body coordinate system (Ox1y1z1) are defined as Formula 4:

[0065]

[0066] In order to simplify the modeling part, the aircraft body studied in this application is axisymmetric, so it is assumed that J x1y1 , J y1z1 , J x1z1 is zero, only the moment of inertia J of the body is considered x1 , J y1 and J z1 .

[0067] Substituting Formula 4 into Formula 3, we can obtain the differential equation of the angular velocity of the body rotating around the center of mass, as shown in Formula 5:

[0068]

[0069] Among them, M x1 、M y1 、M z1 They represent the axial, lateral and normal moments of the fuselage respectively. Since the gravity is located at the center of mass, there is no moment effect. The thrust effect coincides with the longitudinal axis without considering the eccentric interference, so no moment effect is generated. The moment effect mainly comes from the aerodynamic moment, which is divided into rolling moment, yaw moment and pitch moment on Ox1y1z1, which can be expressed as formula 6 respectively:

[0070]

[0071] Where q is the velocity head, S is the characteristic cross-sectional area, L is the characteristic length of the fuselage, and the aerodynamic moment coefficient m is x1 , m y1 , m z1 After linearization, it is expressed as formula 7:

[0072]

[0073] In the formula, is the aerodynamic moment coefficient, is the perturbation part of the aerodynamic moment coefficient, is the high-order nonlinear part of the aerodynamic moment coefficient, δ x1 ,δ y1 ,δ z1 They control the rudder components on the x1, y1 and z1 axes respectively.

[0074] According to the conversion relationship between the ground coordinate system and the body coordinate system, the kinematic equation for the body's rotation around the center of mass is Formula 8:

[0075]

[0076] Formula 2 has given the relationship between the angle of attack, sideslip angle and speed. By taking the derivative of both ends of Formula 2, the attitude equation for the change of the angle of attack and sideslip angle can be obtained as shown in Formula 9:

[0077]

[0078] Among them, G a is the component of gravity on the a-axis, is the aerodynamic parameter of b along the a-axis, is the uncertainty of the aerodynamic parameters of b along the a-axis.

[0079] Formula 5 gives the equation for the change of attitude angular velocity, where the roll angular velocity ω x1 is a small amount, and the corresponding roll controller can be designed to strictly control it, so ω x1 ≈0, and because the aircraft is basically symmetrical, J y1 and J z1 The difference is not big, so Formula 5 can be decoupled. Substituting Formula 7 and Formula 6 into the decoupled Formula 5, the control equation of the attitude angular velocity can be obtained as shown in Formula 10:

[0080]

[0081] The attitude control equations of the longitudinal and lateral directions of the aircraft have been obtained above. In the design of the attitude control system of the aircraft, without considering the deviation caused by the asymmetric factors of the aircraft, the channels in the two directions can be decomposed. For the simplicity of the control system design, the longitudinal channel is analyzed independently in the present invention, and only the movement of the aircraft in the longitudinal plane (plane Ox1y1) is considered, that is, the changing motion parameters are the flight speed V, the angle of attack α, and the pitch angle Ballistic inclination angle θ, pitch angular velocity ω z1 , the parameters of lateral motion are all small and take the value as zero.

[0082] After the longitudinal channel is separated, the attack angle α and the pitch angular velocity ω are taken into account in formula 9 and formula 10 respectively. z1 The control equation of , ignoring the high-order nonlinear terms of aerodynamic parameters, are all 0, and the perturbation part of the aerodynamic parameters All of them are considered as interference to the control system. At the same time, since the aircraft is flying at a very high speed, the influence of gravity on the attitude angle variable is small. Therefore, the gravity part in the equation is ignored. After simplification, the attitude model of the longitudinal channel of the projectile can be expressed as formula 11:

[0083]

[0084] Where, d α and d ω Represents aerodynamic parameter perturbations and some unknown disturbances. The controlled variables of the longitudinal channel are the angle of attack α and the pitch angular velocity ω z1 .

[0085] Furthermore, the attitude nonlinear model of the longitudinal channel described by Formula 11 and its aerodynamic parameters are appropriately simplified; an improved function replacement method is used to study the LPV model description of the longitudinal nonlinear dynamics with the influence of elastic coupling factors.

[0086] The necessary simplification of the nonlinear longitudinal model, that is, the linear simplification of formula 11, can obtain formula 12:

[0087]

[0088] Choosing α and ω z1 As the state of the system, the above formula is transformed into formula 13:

[0089]

[0090] Thus, the polyhedral description of the aircraft, i.e. the simulation model, is obtained, as shown in Formula 14:

[0091]

[0092] Where x = [α, ω z1 ] T , u=[δ z1 , P] T .

[0093] Step 130 , training a pre-built reinforcement learning model with the flight strategy as output and the simulated flight data as input.

[0094] In some embodiments, the reinforcement learning model is constructed based on an actor-critic network, wherein the actor network is composed of a pre-constructed policy model. The policy model is constructed by the following steps: constructing the policy model according to the simulation model and a preset second algorithm.

[0095] Specifically, based on the above simulation model, the construction process of the strategy model is as follows:

[0096] In formula 14, the matrices A(p(t)), B(p(t)) belong to the polytope Ω d , the subscript d represents the result obtained after discretization. That is, Ω d ={[A 1d , B1d ], ..., [A Ld , B Ld ]}, L = 4. Within the range of altitude and velocity, A(p(t)), B(p(t)) can be represented by the weighted representation of the four discretized vertices, and the weights are still the angle of attack α and the pitch angular velocity ω. z1 The function of 0≤p j (k)≤1.

[0097] Assume the number of prediction steps is p, and record the reference trajectory as formula 15:

[0098]

[0099] The number of control steps is m, and the incremental sequence of the control quantity is formula 16:

[0100]

[0101] The system output can be obtained according to the LPV model of the system, as shown in Formula 17:

[0102]

[0103] Then the control objective function of the model predictive control is formula 18:

[0104] J(x(k), ΔU(k))=||Γ y (Y p，c (k+1|k)-R(k+1))|| 2 +||Γ u ΔU(k)|| 2 ,

[0105] Among them, Γ y =diag{Γ y，1 , Γ y，2 , ..., Γ y，p} is the weight matrix of tracking error, Γ u =diag{Γ u，1 , Γ u，2 , ..., Γ u，p} is the weighted matrix of control cost.

[0106] Then the model predictive control problem can be described as the following optimization problem formula 19 and formula 20:

[0107]

[0108] For MPC control of LPV systems, the method based on linear matrix inequality (LMI) is the most widely used. In this method, the stability of the system is ensured by solving LMI at the grid points, and the state prediction is performed by solving the quadratic programming (QP) optimization problem. Although the QP problem can be solved online through effective mathematical methods, it is unrealistic to solve LMI online, especially for LPV systems with grid division, which need to solve the LMI group.

[0109] If formula 19 is converted into the following QP problem, namely formula 21 and formula 22:

[0110]

[0111] stCz≥b,

[0112] The interior point method can be used to solve the problem. This application uses the following method to complete the transformation:

[0113] 1) Formula 19 is transformed into Formula 21:

[0114] Define the zero input control error variable as Equation 23:

[0115] E p (k+1|k)=R(k+1)-A(ρ)Δx(k),

[0116] For the sake of simplicity, formula 23 is denoted as E. After transformation, formula 19 can be written as follows, that is, formula 24:

[0117]

[0118] Since the last term in Equation 24 It is independent of ΔU(k), so it can be removed from the optimization objective function. So Formula 24 can be written as Formula 25:

[0119]

[0120] in,

[0121] It can be seen that Formula 25 and Formula 21 have the same form.

[0122] 2) Formula 20 is transformed into Formula 22:

[0123] Using the incremental method to represent the control input, we have formula 26:

[0124] u(k+1)=u(k-1)+Δu(k)+Δu(k+1),

[0125] It can be seen that formula 27:

[0126]

[0127] For i=0, 1, 2, ..., m-1, we have formula 28:

[0128]

[0129] Writing formula 28 in matrix form gives formula 29:

[0130]

[0131] In formula 29,

[0132] Then Formula 29 has the same form as Formula 22, that is, it can be solved by QP.

[0133] In some embodiments, in order to complete the task of aircraft attitude stabilization control in complex environments, this application introduces a model predictive control algorithm to accelerate the convergence speed of attitude control of deep reinforcement learning in complex scenarios and improve control accuracy, and guide the aircraft to autonomously learn a reward model that meets the complex environment settings based on model predictive control traction, avoiding the problems of inaccurate modeling and low versatility caused by manually designed reward functions. This application follows the framework of learning reward functions from preferences, in which the reward model is trained to be consistent with the model predictive control strategy. In this framework, the trajectory segment σ is a sequence of observations and actions {s k , a k ,…,s k+H , a k+H}.

[0134] The preferences are expressed as triples (σ 0 , σ 1 , y) in the form of data set D. By following the Bradley-Terry model, the preference predictor is modeled using the reward function as follows:

[0135]

[0136] where σ i >σ j denotes that trajectory segment i is preferred over trajectory segment j. This can be intuitively interpreted as assuming that the probability that the MPC policy prefers a trajectory segment depends exponentially on the sum of the trajectory segments of the underlying reward function. Although not a binary classifier, the reward model can be learned for binary classification using the labels y provided by the MPC. Specifically, the reward function, modeled as a neural network with parameter ψ, is updated by minimizing the following loss:

[0137]

[0138] The reward function is learned to guide the aircraft to perform attitude stabilization control. To achieve this goal, the reward function of the aircraft attitude control is learned by establishing a probability preference model of the model predictive control strategy for the sampled trajectory segments. The probability preference mentioned here is positively correlated with the reward index of the segment, that is, the segment with a high reward will be given a higher probability of selection. In the task setting of a complex environment, due to the limited amount of information collected, it is necessary to sample the sample trajectory selectively. At this time, the trajectory segment with the highest expected information value (MEIV) should be selected to express the preference, because this can maximize the utility level of the existing information and provide richer and more effective information support for the subsequent learning of the aircraft attitude control strategy.

[0139] In actual calculations, it is difficult to accurately estimate MEIV, mainly because it is necessary to perform a full traversal expectation calculation of all possible trajectories generated under all new strategies in a probability-weighted manner. In order to reduce the computational complexity, three trajectory segment selectors with sampling schemes are proposed:

[0140] (a) Uniform sampling, i.e., randomly sampling all trajectories with uniform probability;

[0141] (b) Set-based sampling: This scheme first samples multiple trajectory segments of length k, then uses the reward prediction model trained in the corresponding set to score the trajectory segments, and finally selects the trajectory segment with the largest prediction result variance in the set;

[0142] (c) Entropy-based sampling: This method attempts to improve the sample information quality by reducing the uncertainty of samples near boundary points. Specifically, it first samples multiple pairs of trajectory segments, then calculates the joint entropy of each pair of segments, and finally selects the one that maximizes the entropy H(P ψ ) of the trajectory segments.

[0143] In order to improve the accuracy and stability of the reward model's prediction of the reward value for actions performed in complex environments, a reward predictor based on ensemble learning is proposed. Specifically, an ensemble of a set of predictors is fitted, each of which is trained on a preference data triple sampled from the same trajectory pool. By normalizing each of these predictors independently and then averaging the results, a more accurate reward estimate is obtained:

[0144]

[0145] Where N is the number of reward predictors. For example, 1 / e of the data is retained as the validation set for each predictor. L2 regularization is used, and the regularization coefficient is adjusted to keep the validation loss between 1.1 and 1.5 times the training loss. For each reward predictor, a three-layer neural network is used, each with 256 hidden units and a leakyReLU activation function. The tanh function is used to constrain the output between multiple reward predictors. Each reward predictor is optimized using the ADAM learning rule with an initial learning rate of 0.0003.

[0146] In some embodiments, with the reward function, any reinforcement learning algorithm can be used to update the policy π and the action value function Q. Some methods use online reinforcement learning algorithms to learn policy models, such as the TRPO algorithm and the A2C algorithm. However, in complex environments, the reward function may be non-stationary. In addition, it is difficult for many scenarios to collect a large amount of data in real time to meet the needs of online reinforcement learning algorithms, and their poor sampling efficiency will also lead to poor feedback efficiency of the entire reinforcement learning method based on model predictive control preferences. Therefore, the present application improves the offline reinforcement learning algorithm to adapt to the policy learning scenario under the traction of model predictive control in complex environments. The SAC algorithm is used as the basic offline policy reinforcement learning algorithm, which provides sample efficient learning by reusing past experience stored in the playback buffer. SAC is an offline policy actor-critic method based on the maximum entropy reinforcement learning framework, which encourages the agent to explore complex environments and enhance robustness to noise by maximizing the weighted objectives of reward and policy entropy. In order to update the parameters, SAC alternates between soft policy evaluation and soft policy improvement. At the soft policy evaluation step, the soft action value function Q, which is modeled as a neural network with parameters θ, is updated by minimizing the following soft Bellman residual:

[0147]

[0148] in,

[0149] Here τ t =(s t , a t ,s t+1 , r t ) is the state transfer data, B is the replay buffer, is the delay parameter and α is the temperature parameter. In the soft policy improvement step, the policy π is updated by minimizing the following objective:

[0150]

[0151] SAC has better sample efficiency than online policy reinforcement learning algorithms by reusing its past experience. However, traditional offline policy reinforcement learning algorithms have the problem of unstable training during the training process. Specifically, this type of method updates the strategy by memorizing the previously collected environmental feedback experience, and the reward values ​​​​annotated by these memorized data at the beginning are obtained using the previous old version of the reward model, which deviates from the understanding of the reward in the current state. In order to solve this problem, this application will recalculate and re-label all historical trajectory data observed by the agent so far according to the latest model after each round of reward function update. This simple technique can significantly stabilize the training process of offline reinforcement learning and greatly improve the performance of the final strategy.

[0152] This application uses a reward model based on model predictive control preference training to train a policy that can produce higher quality outputs based on model predictive control. To prevent the policy behavior patterns generated during reinforcement learning from deviating too much from the model predictive control expectations and thus deviating from the expected reward value, a KL divergence regularization constraint is added to the reward. This constraint penalty has a parameter The KL deviation between the reinforcement learning strategy and the pre-trained model is used to suppress the excessive deviation of the strategy behavior from the previously learned model predictive control strategy. The complete reward function R is expressed as follows:

[0153]

[0154] here represents a reinforcement learning policy with parameters φ, π pre The behavior distribution within the pre-trained model. This design plays two main roles: first, it plays the role of entropy addition, encouraging the strategy to continue to maintain the diversity of behavior distribution and avoid the collapse of the strategy model into a single mode; it can significantly suppress the deviation between the final strategy and the rolling time domain optimization preference, ensuring that the strategy generator's behavior pattern still meets the subjective expectations of model predictive control while pursuing the maximization of environmental rewards.

[0155] In some embodiments, the policy model obtained through the above process is used as the actor model in the subsequent actor-critic method for reinforcement learning training.

[0156] In some embodiments, when selecting a reinforcement learning algorithm, considering that the task involves a high-dimensional continuous action space, in order to ensure the convergence and stability of the algorithm, the present invention selects a proximal policy optimization algorithm to build a reinforcement learning training framework. Specifically, the proximal policy optimization algorithm uses a policy gradient optimization method based on the actor-critic method to maximize the expectation of cumulative returns. Among them, the proximal policy optimization algorithm adopts an actor-critic method. This method combines policy-based and value-based reinforcement learning methods, and solves the problem of high variance by introducing a policy evaluation mechanism. The policy model is an actor network that outputs actions; the value network is a critic network that is used to evaluate the quality of the actions selected by the actor network, and generates a time difference (TD-error) signal while guiding the update of the actor-critic network. The input of the actor network is the state, and the output action. For continuous actions, the output layer of the neural network can use tanh or sigmod, and for discrete actions, the softmax layer is used as the output layer to achieve the effect of probability output. The critic network inputs the state and action, and the output is the Q value. It should be noted that the actions here are the flight strategies (i.e., the next flight attitude, next flight path, next flight speed, etc. of the aircraft).

[0157] In some embodiments, the reinforcement learning model is configured with a generalized advantage function; the advantage estimated by the generalized advantage function can be effectively reduced in the variance of the gradient estimation when applied in the policy gradient method, thereby reducing the samples required for training.

[0158] The generalized advantage function is shown in the following formula:

[0159]

[0160] Among them, V(s t ) is the state value estimated by the neural network, γ, λ are the hyperparameters for adjusting the bias and variance, l is the discount number, r t+l is the reward at time t+l, s t is the state at time t.

[0161] In some embodiments, during the reinforcement learning training, when performing policy gradient updates, similar to the trust region policy optimization algorithm, the proximal policy optimization considers the effective update of the policy, that is, how to use the current data to take the greatest possible improvement measures on the policy without accidentally updating to a position that causes performance crashes. The loss function of the trust region policy optimization method is:

[0162]

[0163] in, is the importance sampling coefficient, which is used to use the behavior strategy The collected data estimates the target policy π θ The advantage function of , in order to achieve the maximum step size under effective update, the constraint function defined by KL divergence is given as follows:

[0164]

[0165] In this way, the gradient solution of trust region strategy optimization is transformed into an optimization problem:

[0166]

[0167]

[0168] By making an approximate estimate of the KL divergence and using the Lagrange dual optimization theory to find an analytical solution, the step size of each update can be determined. The specific gradient update formula is as follows:

[0169]

[0170] Among them, θ′ is the gradient after update; θ is the gradient before update; H is solved by using the conjugate gradient algorithm x = H -1 g is the Hessian matrix of the KL divergence obtained; α is the original update step size; j∈{1, 2, ..., K} is a hyperparameter. In the actual gradient update, the value of j is the minimum value obtained by iteration that can make θ′ satisfy the KL divergence constraint.

[0171] Step 140, when the preset stop condition is met, stop the training, and use the trained reinforcement learning model as the aircraft attitude control model.

[0172] In some embodiments, for example, when the number of training rounds reaches a preset number and / or the model accuracy reaches a preset threshold, the training of the reinforcement learning model is stopped to obtain an aircraft attitude control model to achieve a better effect of controlling the flight of the aircraft.

[0173] In addition, since reinforcement learning is performed on the basis of the manipulation of various tasks in the simulation system and the collection of data on the autonomous interaction between the intelligent agent and the simulation environment, there is no need to build a flight dynamics model, which reduces the training cycle and training cost of the controller.

[0174] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0175] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0176] Figure 2 FIG. 2 shows a block diagram of a device 200 for generating an aircraft attitude control model according to an embodiment of the present disclosure. Figure 2 As shown, the device 200 includes:

[0177] A data acquisition module 210 is used to acquire flight missions and flight environments;

[0178] The data simulation module 220 is used to simulate the flight mission and the flight environment according to a pre-built simulation model, and output simulated flight data;

[0179] A model training module 230 is used to train a pre-built reinforcement learning model using the flight strategy as output and the simulated flight data as input;

[0180] The model generation module 240 is used to stop the training when the preset stop condition is met, and use the trained reinforcement learning model as the aircraft attitude control model.

[0181] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0182] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0183] Figure 3A schematic block diagram of an electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0184] The electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a ROM 302 or a computer program loaded from a storage unit 308 into a RAM 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0185] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0186] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as a method for generating an aircraft attitude control model. For example, in some embodiments, the method for generating an aircraft attitude control model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method for generating the aircraft attitude control model described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to execute the method for generating the aircraft attitude control model in any other appropriate manner (for example, by means of firmware).

[0187] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0188] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0189] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0190] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0191] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0192] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0193] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0194] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating an aircraft attitude control model, characterized in that: include: Obtain flight mission and flight environment; According to the pre-built simulation model, the flight mission and the flight environment are simulated and the simulated flight data is output; The pre-built reinforcement learning model is trained with the flight strategy as output and the simulated flight data as input; When the preset stop condition is met, the training is stopped and the trained reinforcement learning model is used as the aircraft attitude control model.

2. The method according to claim 1, characterized in that The simulation model is expressed by the following formula: x=[a,ω z1 ] T ,u=[δ z1 ,P] T , Where α is the angle of attack, ω z1 is the pitch angular velocity, δ z1 To control the rudder deflection angle, P is the prediction step number, q is the velocity head, S is the characteristic cross-sectional area, L is the characteristic length of the fuselage, m is the control step number, is the aerodynamic coefficient at angle of attack, is the aerodynamic moment coefficient, is the control rudder aerodynamic coefficient, v is the flight speed, J z1 is the rotational inertia of the body.

3. The method according to claim 2, characterized in that The reinforcement learning model is constructed based on an actor-critic network; Among them, the actor network is composed of pre-built strategy models.

4. The method according to claim 3, characterized in that The reinforcement learning model is configured with a generalized advantage function; The generalized advantage function is shown in the following formula: Among them, V(s t ) is the state value estimated by the neural network, γ, λ are the hyperparameters for adjusting the bias and variance, l is the discount number, r t+l is the reward at time t+l, s t is the state at time t.

5. The method according to claim 3, characterized in that: The update gradient of the reinforcement learning model is performed according to the following formula: Among them, θ′ is the gradient after update; θ is the gradient before update; H is solved by using the conjugate gradient algorithm x = H -1 g is the Hessian matrix of the KL divergence obtained; α is the original update step size; j∈{1, 2, ..., K} is a hyperparameter. In the actual gradient update, the value of j is the minimum value obtained by iteration that can make θ′ satisfy the KL divergence constraint.

6. The method according to claim 3, characterized in that The policy model is constructed by the following steps: The strategy model is constructed according to the simulation model and a preset second algorithm.

7. The method according to claim 6, characterized in that The policy model is pre-built with a reward function; The reward function is shown in the following formula: Among them, (σ 0 , σ 1 , y) is the preference triple, D is the data set, y is the label, σ 0 ={s k , a k ,…,s k+H , a k+H }0 and σ 1 ={s k , a k ,…,s k+H , a k+H }1 is the trajectory segment, y(0) is σ 0 The corresponding rolling time domain optimization label, y(1) is σ 1 The corresponding rolling horizon optimization label, σ i >σ j indicates that trajectory segment i is better than trajectory segment j, P Ψ The reward function based on the Bradley-Terry model is a predictor model of preference, which is expressed as follows: Status Next action Reward.

8. A device for generating an aircraft attitude control model, characterized in that: include: Data acquisition module, used to obtain flight missions and flight environment; A data simulation module, used to simulate the flight mission and the flight environment according to a pre-built simulation model, and output simulated flight data; A model training module, used to train a pre-built reinforcement learning model with the flight strategy as output and the simulated flight data as input; The model generation module is used to stop training when the preset stop condition is met, and use the trained reinforcement learning model as the aircraft attitude control model.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images