Method and device for generating an aircraft attitude control decision model

Abstract

Embodiments of the present disclosure provide a method and device for generating an aircraft attitude control decision model, and are applied to the technical field of aircraft control. The method comprises: obtaining historical running trajectory data of a target aircraft; obtaining expert decisions and network decisions; comparing the expert decisions and the network decisions to obtain a first loss value; updating parameters of a decision network according to the first loss value; obtaining simulated running trajectory data; comparing the simulated running trajectory data and actual running trajectory data to obtain a second loss value; the actual running trajectory data is historical running trajectory data at the same time point as the simulated running trajectory data; updating the parameters of the decision network according to the second loss value; when the parameters of the decision network meet a preset condition, training of the decision network is completed; and the trained decision network is used as an aircraft attitude control decision model. In this way, the training period and training cost of the controller can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of aircraft control technology, and in particular to a method and device for generating an aircraft attitude control decision model. Background Art

[0002] In recent years, aircraft have been widely promoted and applied in civilian fields such as agricultural plant protection, power inspection, urban security, forest fire prevention, environmental monitoring, low-altitude aerial photography, and aerial search and rescue, as well as in military fields such as battlefield reconnaissance, transportation support, and patrol surveillance. Due to their advantages such as high maneuverability, wide coverage, the ability to observe ground conditions from a global perspective, and rapid response to commands, aircraft are expected to become even more prevalent and have a wider range of applications in the future.

[0003] Attitude control is the core of aircraft flight control, and its performance directly impacts the overall flight control effectiveness. Conventional attitude control utilizes a cascaded PID control structure, which controls the errors in the aircraft's angle and angular velocity loops and outputs the control law to the servo. However, during flight, interference from unmodeled dynamics and uncertainties, such as gusts of wind, parameter perturbations, and other persistent, time-varying disturbances, exists. Traditional methods are difficult to resist and lack adaptability. Furthermore, debugging control parameters is complex and requires continuous trial and error, resulting in reduced stability margins for aircraft control and even the risk of loss of control.

[0004] Model predictive control is a classic control method for aircraft control. However, this method requires detailed modeling of the aircraft and the environment. When there are errors in the model, the control accuracy will decrease.

[0005] Reinforcement learning uses an end-to-end trial-and-error training method, based on an established reward-and-penalty mechanism, to guide the controlled object to select the optimal action to apply in the environment, maximizing the cumulative reward and finding the optimal strategy. During training, factors such as uncertain interference and noise are introduced. Based on the empirical knowledge of the training data, the policy learning mechanism explores and utilizes this knowledge, resulting in strong adaptability and autonomous learning capabilities in complex and disturbed environments. Using reinforcement learning to control aircraft is a new control method that has emerged in recent years. This method eliminates the need to construct dynamic models of the aircraft and the environment. Instead, it uses a "trial and error" process in the environment to continuously train the controller based on rewards, ultimately achieving better control results. However, this method has long training cycles and high training costs. Summary of the Invention

[0006] The present disclosure provides a method and device for generating an aircraft attitude control decision model.

[0007] According to a first aspect of the present disclosure, a method for generating an aircraft attitude control decision model is provided. The method comprises:

[0008] Obtain historical trajectory data of the target aircraft;

[0009] Obtaining expert decisions and network decisions; the expert decisions are obtained by expert training a pre-built decision network based on the historical operation trajectory data; the network decisions are obtained by using the decision network to make decisions based on the historical operation trajectory data;

[0010] Comparing and calculating the expert decision and the network decision to obtain a first loss value;

[0011] Updating parameters of the decision network according to the first loss value;

[0012] Acquiring simulated operation trajectory data; the simulated operation trajectory data is obtained by performing reinforcement learning on the decision network after updating parameters based on the historical operation trajectory data and the preset flight scenario;

[0013] Comparing and calculating the simulated running trajectory data and the actual running trajectory data to obtain a second loss value; the actual running trajectory data is the historical running trajectory data at the same time point as the simulated running trajectory data;

[0014] Updating the parameters of the decision network according to the second loss value;

[0015] When the parameters of the decision network meet the preset conditions, completing the training of the decision network;

[0016] The trained decision network is used as the aircraft attitude control decision model.

[0017] According to the above aspects and any possible implementation, there is further provided an implementation, wherein the historical trajectory data includes angle of attack data and sideslip angle data of the aircraft flight;

[0018] The decision network is constructed through the following steps:

[0019] Constructing a polyhedral description model of the aircraft according to the angle of attack data and the sideslip angle data and a preset first algorithm;

[0020] Constructing an attitude control strategy model for the aircraft based on the polyhedral description model and a preset second algorithm;

[0021] The posture control strategy model is used as a decision network.

[0022] According to the above aspects and any possible implementation, an implementation is further provided, wherein the polytope description model is as shown in the following formula:

[0023]

[0024] x=[α,ω z1 ] T ,u=[δ z1 , P] T ,

[0025]

[0026]

[0027] Where α is the angle of attack, ω z1 is the pitch angular velocity, δ z1 To control the rudder deflection angle, P is the prediction step number, q is the velocity head, S is the characteristic cross-sectional area, L is the characteristic length of the body, m is the control step number, is the aerodynamic coefficient at angle of attack, is the aerodynamic moment coefficient, is the aerodynamic coefficient of the control rudder, v is the flight speed, J z1 is the rotational inertia of the body.

[0028] According to the above aspects and any possible implementation, there is further provided an implementation, wherein the decision network is pre-built with a reward function;

[0029] The reward function is shown in the following formula:

[0030]

[0031] Among them, (σ 0 ,σ 1 ,y) is the preference triple, D is the data set, y is the label, σ 0 ={s k , a k ,…,s k+H , a k+H}0 and σ 1 ={s k , a k ,…,s k+H , a k+H}1 is the trajectory segment, y(0) is σ 0 The corresponding rolling time domain optimization label, y(1) is σ 1 The corresponding rolling horizon optimization label, σ i >σ j indicates that trajectory segment i is better than trajectory segment j, P Ψ The reward function is a predictor model of preference established according to the Bradley-Terry model, and its expression is as follows:

[0032]

[0033] Status Next action rewards.

[0034] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the method further includes:

[0035] Calculating the prediction factor obtained by the reward function to obtain a reward estimate;

[0036] The formula for calculating the predicted factor obtained by the reward function to obtain the reward estimate is as follows:

[0037]

[0038] in, is the reward estimate, N is the number of predictors, For the first i The reward of a trajectory, s is the state on the trajectory, and a is the action on the trajectory.

[0039] According to the above aspects and any possible implementation, there is further provided an implementation, wherein the trajectory segments in the preference triples are represented by selecting the trajectory segment with the highest MEIV value;

[0040] And the sampling methods of trajectory segments include: uniform sampling, set-based sampling and entropy-based sampling.

[0041] According to a second aspect of the present disclosure, a method for generating an attitude control decision for an aircraft is provided. The method comprises:

[0042] Obtain flight trajectory data of the aircraft to be controlled;

[0043] Inputting the flight trajectory data into an aircraft attitude control decision model trained according to the method of the first aspect of the present disclosure, and outputting a flight control strategy;

[0044] Based on the output flight control strategy, flight control is performed on the aircraft to be controlled.

[0045] According to a third aspect of the present disclosure, a device for generating an aircraft attitude control decision model is provided. The device comprises:

[0046] A data acquisition module is used to obtain the historical trajectory data of the target aircraft;

[0047] A decision acquisition module is used to obtain expert decisions and network decisions; the expert decisions are obtained by expert training a pre-built decision network based on the historical operation trajectory data; the network decisions are obtained by using the decision network to make decisions based on the historical operation trajectory data;

[0048] a calculation module, configured to compare and calculate the expert decision and the network decision to obtain a first loss value;

[0049] A parameter updating module, configured to update the parameters of the decision network according to the first loss value;

[0050] The data acquisition module is further used to acquire simulated operation trajectory data; the simulated operation trajectory data is obtained by performing reinforcement learning on the decision network after updating parameters based on the historical operation trajectory data and the preset flight scenario;

[0051] The calculation module is further configured to compare and calculate the simulated running trajectory data and the actual running trajectory data to obtain a second loss value; the actual running trajectory data is historical running trajectory data at the same time point as the simulated running trajectory data;

[0052] The parameter updating module is further used to update the parameters of the decision network according to the second loss value;

[0053] A model generation module, configured to complete the training of the decision network when the parameters of the decision network meet preset conditions;

[0054] The model generation module is also used to use the trained decision network as an aircraft attitude control decision model.

[0055] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor implements the method according to the first aspect of the present disclosure when executing the program.

[0056] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0057] The disclosed embodiments provide a method and apparatus for generating an aircraft attitude control decision model. First, aircraft dynamics modeling is performed, and a model predictive control method is used to generate an aircraft attitude control sample set, providing training samples for behavioral cloning. Second, a behavioral cloning reinforcement learning method is used to complete offline learning of the aircraft attitude controller. This reduces the controller training cycle and training cost. Furthermore, during the dynamics model construction step, high modeling accuracy is not required, as the subsequent reinforcement learning method does not require the dynamics model.

[0058] It should be understood that the contents described in the Summary of the Invention section are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0060] Figure 1 A flow chart showing a method for generating an aircraft attitude control decision model according to an embodiment of the present disclosure is shown;

[0061] Figure 2 A diagram showing the relationship between the velocity coordinate system and the body coordinate system of an aircraft according to an embodiment of the present disclosure is shown;

[0062] Figure 3 A flow chart of a method for generating attitude control decisions for an aircraft according to an embodiment of the present disclosure is shown;

[0063] Figure 4 A block diagram of a device for generating an aircraft attitude control decision model according to an embodiment of the present disclosure is shown;

[0064] Figure 5 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0065] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0066] In this document, the term "and / or" simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " generally indicates that the associated objects are in an "or" relationship.

[0067] This application focuses on the problems of low data utilization efficiency and difficulty in effective data collection in supervised reinforcement learning driven by model predictive control, and proposes a new reinforcement learning training framework that combines the behavior cloning method and the reward reconstruction method to improve the training effect of the neural network. The behavior cloning method uses supervised learning to allow the neural network to directly learn the pre-constructed model predictive control decision set to quickly improve the decision-making ability of the neural network. The advantage of this method is that it can make full use of existing data and improve data utilization efficiency. In addition, the reward reconstruction method improves the training effect of reinforcement learning through a more sophisticated reward value function design. This method can better measure the quality of each action, so that the neural network can learn the correct decision-making strategy more accurately. By combining these two methods, a more efficient and accurate reinforcement learning training framework can be obtained, which provides new ideas and methods for solving aircraft attitude control problems.

[0068] This application incorporates the behavioral cloning concept from imitation learning into the deep reinforcement learning framework to design an aircraft attitude control strategy. This strategy organically combines deep reinforcement learning algorithms with behavioral cloning, enabling the aircraft to imitate a pre-constructed model predictive control decision set while autonomously exploring. Specifically, the training method of this application involves training a deep reinforcement learning model, enabling the aircraft to autonomously explore and generate a series of control instructions based on its current state and environmental information to achieve attitude control. Simultaneously, leveraging the concept of behavioral cloning, the pre-constructed model predictive control decision set is incorporated into the model training process, enabling the aircraft to continuously imitate and learn from these excellent control decisions during autonomous exploration. Combining behavioral cloning with deep reinforcement learning enables the aircraft to simultaneously explore and imitate, thereby improving the model's generalization and the strategy's applicability. Furthermore, this strategy is efficient and interpretable. By leveraging the concept of behavioral cloning, data can be more efficiently utilized during training, and the model's decision logic can be more clearly explained. This makes the strategy of this application easier to understand and accept, as well as easier to debug and optimize. In summary, this application designs an effective aircraft attitude control strategy by integrating the behavioral cloning concept from imitation learning into a deep reinforcement learning framework. This strategy can effectively improve the utilization of model predictive control data and enhance the effectiveness of reinforcement learning algorithms. It also provides valuable reference for future research.

[0069] Behavioral cloning methods are remarkably efficient in deep reinforcement learning training and can achieve expert-level decision-making capabilities. However, despite their numerous advantages, behavioral cloning methods still have certain limitations. When a large amount of expert experience is collected, its coverage may be relatively limited, failing to fully encompass all possible states of a real system. Therefore, even if a deep reinforcement learning algorithm can acquire good decision-making capabilities through expert training, it will only excel at path planning tasks encompassed by the expert experience. When encountering system states not previously encountered in the expert experience, decision-making results remain subject to randomness, resulting in either good or poor decisions. This randomness can cause changes in the parameters of the neural network and may even cause the deep reinforcement learning algorithm to lose its previously learned decision-making capabilities.

[0070] In response to this phenomenon, this application has conducted research on collective data aggregation algorithms under a probabilistic space framework for behavioral cloning security and designed an alternating learning framework. Throughout the training process, the neural network uses expert training and reinforcement learning alternately to train the neural network under a probabilistic framework. This training method is similar to the need for humans to frequently review what their teacher has taught in the learning process to consolidate the knowledge they have learned. In this way, the neural network can not only learn decision-making methods in unknown states, but also always retain its acquired expert decision-making capabilities. This training method is innovative and provides an effective solution to the limitations of behavioral cloning methods.

[0071] Figure 1 A flow chart of a method 100 for generating an aircraft attitude control decision model according to an embodiment of the present disclosure is shown. The method 100 includes:

[0072] Step 110: Acquire historical trajectory data of the target aircraft.

[0073] In some embodiments, historical trajectory data of the target aircraft is obtained as data support for model training.

[0074] Step 120: Obtain expert decision and network decision.

[0075] The expert decision is obtained by expert training a pre-built decision network based on the historical operation trajectory data; the network decision is obtained by using the decision network to make a decision on the historical operation trajectory data.

[0076] In some embodiments, the historical trajectory data includes the angle of attack data and sideslip angle data of the aircraft; the decision network is constructed by the following steps: constructing a polyhedral description model of the aircraft based on the angle of attack data and the sideslip angle data and a preset first algorithm; constructing an attitude control strategy model of the aircraft based on the polyhedral description model and a preset second algorithm; and using the attitude control strategy model as the decision network. Figure 2 The relationship between the velocity coordinate system and the body coordinate system of the aircraft is shown in the figure. The relationship between the aircraft body coordinate system and the velocity coordinate system can be expressed by the angle of attack α and the sideslip angle β. The angle of attack α is the angle through which the velocity axis rotates in the positive direction around the Oz2 axis. It represents the angle between the body velocity vector and the Ox1z1 plane. When the aircraft is facing upward (Oy1 axis is facing upward), α is positive when the velocity vector is below the Ox1z1 plane, and negative otherwise. The sideslip angle β is the angle through which the velocity axis rotates in the positive direction around the Oz2 axis, so that the Oy′2 axis coincides with the Oy1 axis, and rotates in the positive direction around the Oy1 axis. It represents the angle between the projection of the body velocity vector on the Ox1z1 plane and the longitudinal axis Ox1 of the aircraft. When the aircraft is facing upward, looking toward the nose, β is positive when the velocity vector is to the right of the Ox1 axis, and negative otherwise.

[0077] The transformation relationship between two coordinates It can be expressed as a matrix:

[0078]

[0079] The relationship between the two angles can be expressed by the component v of the velocity vector in the body coordinate system. x1 , v y1 , v z1 To express it, that is, Formula 2:

[0080]

[0081] Where v is the flight speed.

[0082] According to the angular momentum theorem, the body's rotation equation around the center is expressed as Formula 3:

[0083]

[0084] Where H is the sum of the body's momentum vector, M is the various torques acting on the body, t is time, and ω is the angular velocity. Here, the three components of the momentum in the body coordinate system (Ox1y1z1) are defined as Formula 4:

[0085]

[0086] Specifically, the pre-construction of the decision network is described in detail through the following process.

[0087] In order to simplify the modeling part, the aircraft body studied in this application is axisymmetric, so it is assumed that J x1y1 、J y1z1 、J x1z1 is zero, only the moment of inertia J of the body is considered x1 、J y1 and J z1 .

[0088] Substituting Formula 4 into Formula 3 yields the differential equation for the angular velocity of the body rotating around its center of mass, as shown in Formula 5:

[0089]

[0090] Among them, M x1 、M y1 、M z1 They represent the axial, lateral and normal moments of the aircraft respectively. Since gravity is located at the center of mass, there is no moment effect. The thrust effect coincides with the longitudinal axis without considering eccentric interference, so no moment effect is generated. The moment effect mainly comes from the aerodynamic moment, which is divided into rolling moment, yaw moment and pitching moment on Ox1y1z1, which can be expressed as formula 6 respectively:

[0091]

[0092] Where q is the velocity head, S is the characteristic cross-sectional area, L is the characteristic length of the fuselage, and the aerodynamic moment coefficient m is x1 , m y1 , m z1 After linearization, it is expressed as Formula 7:

[0093]

[0094] Where, is the aerodynamic moment coefficient, is the perturbation part of the aerodynamic moment coefficient, is the high-order nonlinear part of the aerodynamic moment coefficient, δ x1 , δ y1 , δ z1 They are the components of the rudder on the x1, y1 and z1 axes respectively.

[0095] According to the conversion relationship between the ground coordinate system and the body coordinate system, the kinematic equation for the body's rotation around the center of mass is Formula 8:

[0096]

[0097] in, is the pitch angle, ψ is the yaw angle, and γ is the roll angle.

[0098] Formula 2 has given the relationship between the angle of attack, sideslip angle and speed. Taking the derivative of both ends of Formula 2, the attitude equation for the change of angle of attack and sideslip angle can be obtained as shown in Formula 9:

[0099]

[0100] Among them, G a is the component of gravity on the a-axis, is the aerodynamic parameter of b along axis a, is the uncertainty of the aerodynamic parameters of b along the a-axis.

[0101] Formula 5 gives the equation for the change of attitude angular velocity, where the roll angular velocity ω x1 is a small amount, and the corresponding roll controller can be designed to strictly control it, so ω x1 ≈0, and since the aircraft is basically symmetrical, J y1 and J z1 The difference is not big, so Formula 5 can be decoupled. Substituting Formula 7 and Formula 6 into the decoupled Formula 5, the control equation of the attitude angular velocity can be obtained as shown in Formula 10:

[0102]

[0103] The attitude control equations for the longitudinal and lateral directions of the aircraft have been obtained above. In the design of the aircraft's attitude control system, without considering the deviation caused by the asymmetric factors of the aircraft, the channels in the two directions can be decomposed. For the simplicity of the control system design, the longitudinal channel is analyzed independently in the present invention, and only the motion of the aircraft in the longitudinal plane (plane Ox1y1) is considered, that is, the changing motion parameters are the flight speed v, the angle of attack α, and the pitch angle. Ballistic inclination angle θ, pitch angular velocity ω z1 , the parameters of lateral motion are all small and take the value as zero.

[0104] After separating the longitudinal channel, formulas 9 and 10 take the angle of attack α and pitch angular velocity ω respectively. z1 The control equation of , ignoring the high-order nonlinear terms of aerodynamic parameters, are all 0, and the perturbation part of the aerodynamic parameters All of them are considered as interference to the control system. At the same time, since the aircraft is flying at a very high speed, the influence of gravity on the attitude angle variable is small. Therefore, the gravity part in the equation is ignored. After simplification, the attitude model of the longitudinal channel of the projectile can be expressed as formula 11:

[0105]

[0106] Where, dα and d ω Represents aerodynamic parameter perturbations and some unknown disturbances. The controlled variables of the longitudinal channel are the angle of attack α and the pitch angular velocity ω z1 .

[0107] Furthermore, the attitude nonlinear model of the longitudinal channel described by Formula 11 and its aerodynamic parameters are appropriately simplified; an improved function replacement method is used to study the LPV model description of the longitudinal nonlinear dynamics with the influence of elastic coupling factors.

[0108] The necessary simplification of the nonlinear longitudinal model is performed, that is, the linear simplification of Formula 11 can be obtained to obtain Formula 12:

[0109]

[0110] Choosing α and ω z1 As the state of the system, the above formula is transformed into formula 13:

[0111]

[0112] This results in a polyhedral description of the aircraft, as shown in Formula 14:

[0113]

[0114] Where x = [α, ω z1 ] T ,u=[δ z1 , P] T .

[0115] In formula 14, the matrices A(p(t)) and B(p(t)) belong to the polytope Ω d , the subscript d represents the result obtained after discretization. That is, Ω d ={[A 1d , B 1d ], ..., [A Ld , B Ld ]}, L = 4. Within the set range of altitude and speed, A(p(t)) and B(p(t)) can be represented by the weighted representation of these four discretized vertices, and the weights are still the angle of attack α and the pitch angular velocity ω. z1 The function of

[0116] Let the number of prediction steps be p, and the reference trajectory be Formula 15:

[0117]

[0118] The number of control steps is m, and the incremental sequence of the control quantity is formula 16:

[0119]

[0120] The system output can be obtained according to the system's LPV model, expressed as Formula 17:

[0121]

[0122] Then the control objective function of the model predictive control is formula 18:

[0123] J(x(k), ΔU(k))=||Γ y (Y p，c (k+1|k)-R(k+1))|| 2 +||Γ u ΔU(k)|| 2 ,

[0124] Among them, Γ y =diag{Γ y，1 , Γ y，2 ,...,Γ y，p} is the weighted matrix of tracking error, Γ u =diag{Γ u，1 , Γ u，2 ,...,Γ u，p} is the weighted matrix of control cost.

[0125] The model predictive control problem can be described as the following optimization problem:

[0126]

[0127] For MPC control of LPV systems, the most widely used method is based on linear matrix inequality (LMI). In this method, system stability is ensured by solving the LMI at grid points, and state prediction is performed by solving a quadratic programming (QP) optimization problem. Although QP problems can be solved online using effective mathematical methods, solving LMI online is unrealistic, especially for gridded LPV systems, which require solving a group of LMIs.

[0128] If formula 19 is converted into the following QP problem, namely formula 21 and formula 22:

[0129]

[0130] stCz≥b,

[0131] The interior point method can be used to solve the problem. This application uses the following method to complete the conversion:

[0132] 1) Convert Formula 19 to Formula 21:

[0133] Define the zero input control error variable as Equation 23:

[0134] E p (k+1|k)=R(k+1)-A(ρ)Δx(k),

[0135] For the sake of simplicity, Formula 23 is denoted as E. After transformation, Formula 19 can be written as follows, that is, Formula 24:

[0136]

[0137] Since the last term in Equation 24 It is independent of ΔU(k), so it can be removed from the optimization objective function. Therefore, Equation 24 can be written as Equation 25:

[0138]

[0139] in,

[0140] It can be seen that Formula 25 and Formula 21 have the same form.

[0141] 2) Formula 20 is converted to Formula 22:

[0142] Using incremental method to express the control input, we have formula 26:

[0143] u(k+1)=u(k-1)+Δu(k)+Δu(k+1),

[0144] It can be seen from formula 27:

[0145]

[0146] For i=0, 1, 2, ..., m-1, we have formula 28:

[0147]

[0148] Writing formula 28 into matrix form gives formula 29:

[0149]

[0150] In formula 29,

[0151] Then Formula 29 has the same form as Formula 22, that is, it can be solved by QP.

[0152] In some embodiments, in order to complete the task of aircraft attitude stabilization control in complex environments, this application introduces a model predictive control algorithm to accelerate the convergence speed of attitude control of deep reinforcement learning in complex scenarios and improve the control accuracy. The aircraft is guided to autonomously learn a reward model that conforms to the complex environment settings based on model predictive control traction, avoiding the problems of inaccurate modeling and low versatility caused by manually designed reward functions. This application follows the framework of learning reward functions from preferences, in which the reward model is trained to be consistent with the model predictive control strategy. In this framework, the trajectory segment σ is a sequence of observations and actions {s k , a k ,…,s k+H , a k+H}.

[0153] The preference is expressed as a triple (σ 0 ,σ 1 , y) in the form of data set D. By following the Bradley-Terry model, the preference predictor is modeled using the reward function as follows:

[0154]

[0155] where σ i >σ j =\begin{aligned}\begin{aligned} y denotes that trajectory segment i is preferred over trajectory segment j. This can be intuitively interpreted as assuming that the probability that the MPC policy prefers a trajectory segment depends exponentially on the sum of the trajectory segments of the underlying reward function. Although not a binary classifier, the reward model can be learned for binary classification using the labels y provided by the MPC. Specifically, the reward function, modeled as a neural network with parameter ψ, is updated by minimizing the following loss:

[0156]

[0157] Learning a reward function to guide the vehicle's attitude stabilization control. To achieve this goal, the reward function for vehicle attitude control is learned by establishing a probabilistic preference model for sampled trajectory segments within the model predictive control strategy. This probabilistic preference is positively correlated with the reward index of the segment, meaning that segments with high rewards are given a higher probability of selection. In complex environment mission settings, due to limited information collection, selective sampling of sample trajectories is necessary. In this case, the trajectory segments with the highest maximum expected information value (MEIV) should be selected to express preference, as this maximizes the utility of the existing information and provides richer and more effective information support for subsequent learning of the vehicle's attitude control strategy.

[0158] In actual calculations, it is difficult to accurately estimate the MEIV, mainly because it is necessary to perform a full traversal expectation calculation of all possible trajectories generated under all new strategies in a probability-weighted manner. To reduce the computational complexity, three trajectory segment selectors with sampling schemes are proposed:

[0159] (a) Uniform sampling, i.e., randomly sampling all trajectories with uniform probability;

[0160] (b) Set-based sampling: This scheme first samples multiple trajectory segments of length k, then uses the reward prediction model trained in the corresponding set to score the trajectory segments, and finally selects the trajectory segment with the largest prediction variance in the set;

[0161] (c) Entropy-based sampling: This method attempts to improve the quality of sample information by reducing the uncertainty of samples near boundary points. Specifically, it first samples multiple pairs of trajectory segments, then calculates the joint entropy of each pair of segments, and finally selects the one that maximizes the entropy H(P ψ ) of the trajectory segments.

[0162] To improve the accuracy and stability of the reward model's predictions of reward values ​​for actions performed in complex environments, a reward predictor based on ensemble learning is proposed. Specifically, an ensemble of predictors is fitted, each trained on preference data triplets sampled from the same trajectory pool. By normalizing each of these predictors independently and then averaging the results, a more accurate reward estimate is obtained:

[0163]

[0164] Where N is the number of reward predictors. For example, 1 / e of the data is retained as the validation set for each predictor. L2 regularization is used, and the regularization coefficient is adjusted to keep the validation loss between 1.1 and 1.5 times the training loss. For each reward predictor, a three-layer neural network with 256 hidden units and a leakyReLU activation function is used. The outputs of multiple reward predictors are constrained using a tanh function. Each reward predictor is optimized using the ADAM learning rule with an initial learning rate of 0.0003.

[0165] In some embodiments, with the reward function, any reinforcement learning algorithm can be used to update the policy π and action-value function Q. Some methods use online reinforcement learning algorithms to learn policy models, such as the TRPO algorithm and the A2C algorithm. However, in complex environments, the reward function may be non-stationary. In addition, many scenarios find it difficult to collect large amounts of data in real time to meet the needs of online reinforcement learning algorithms, and their poor sampling efficiency also leads to poor feedback efficiency of the entire reinforcement learning method based on model predictive control preferences. Therefore, the present application improves the offline reinforcement learning algorithm to adapt to policy learning scenarios under model predictive control in complex environments. The SAC algorithm is used as the basic offline policy reinforcement learning algorithm, which provides sample-efficient learning by reusing past experience stored in the replay buffer. SAC is an offline policy actor-critic method based on the maximum entropy reinforcement learning framework, which encourages the agent to explore complex environments and enhance robustness to noise by maximizing the weighted objectives of reward and policy entropy. In order to update the parameters, SAC alternates between soft policy evaluation and soft policy improvement. In the soft policy evaluation step, the soft action-value function Q is updated by minimizing the following soft Bellman residual, which is modeled as a neural network with parameters θ:

[0166]

[0167] in,

[0168] Here τ t =(s t , a t , s t+1 , r t ) is the state transfer data, B is the replay buffer, is the delay parameter and α is the temperature parameter. In the soft policy improvement step, the policy π is updated by minimizing the following objective:

[0169]

[0170] SAC has better sample efficiency than online policy reinforcement learning algorithms by reusing its past experience. However, traditional offline policy reinforcement learning algorithms have the problem of unstable training during the training process. Specifically, this type of method updates the strategy by memorizing the previously collected environmental feedback experience, and the reward values ​​initially marked with these memorized data are obtained using the previous old version of the reward model, which deviates from the understanding of the reward in the current state. To solve this problem, this application will recalculate and re-label all historical trajectory data observed by the agent so far according to the latest model after each round of reward function update. This simple technique can significantly stabilize the training process of offline reinforcement learning and greatly improve the performance of the final strategy.

[0171] This application uses a reward model based on model predictive control preference training to train a policy that can produce higher quality outputs based on model predictive control. To prevent the policy behavior patterns generated during reinforcement learning from deviating excessively from the model predictive control expectations and thus deviating from the expected reward value, a KL divergence regularization constraint is added to the reward. This constraint penalty has a parameter The KL deviation between the reinforcement learning strategy and the pre-trained model is used to suppress the excessive deviation of the strategy behavior from the previously learned model predictive control strategy. The complete reward function R is expressed as follows:

[0172]

[0173] here represents a reinforcement learning policy with parameters φ, π pre This design serves two main purposes: first, it acts as an entropy generator, encouraging the policy to maintain a diverse behavioral distribution and preventing the policy model from collapsing into a single mode; and second, it significantly suppresses the deviation between the final policy and the rolling horizon optimization preference, ensuring that the policy generator's behavior remains consistent with the subjective expectations of model predictive control while maximizing environmental rewards.

[0174] In some embodiments, when training begins, a decision network Policy Network is first randomly generated, and all its parameters are also randomly generated. Expert training is performed on the decision network Policy Network, and experience is sampled from the expert experience pool and the experience is standardized (for the convenience of comparison with the network decision, such as corresponding scoring, etc.) to obtain the expert decision; the decision network Policy Network is used to make a decision to obtain a network decision. Note that the decision here refers to the decision to guide how the aircraft should fly, such as the flight angle, speed, etc.

[0175] Step 130 : Compare and calculate the expert decision and the network decision to obtain a first loss value.

[0176] In some embodiments, the expert decision and the network decision are compared to calculate a loss value.

[0177] Step 140: Update the parameters of the decision network according to the first loss value.

[0178] In some embodiments, the loss value calculated in step 130 is used to update the parameters of the decision network PolicyNetwork.

[0179] Step 150: Acquire simulation running trajectory data.

[0180] The simulated operation trajectory data is obtained by performing reinforcement learning on the decision network after updating parameters based on the historical operation trajectory data and the preset flight scenario.

[0181] In some embodiments, a new round of training begins, with the flight scenario initialized and a series of randomly generated picking tasks. Reinforcement learning training is then performed on the policy network: the flight scenario interacts with the Policy Gradient algorithm, and data generated throughout the training round, i.e., simulated trajectory data, is collected.

[0182] Step 160 : Compare and calculate the simulated running trajectory data and the actual running trajectory data to obtain a second loss value.

[0183] The actual running trajectory data is historical running trajectory data at the same time point as the simulated running trajectory data.

[0184] In some embodiments, the loss value is calculated by using the recorded actual running trajectory data and the simulated running trajectory data obtained in step 150 .

[0185] Step 170: Update the parameters of the decision network according to the second loss value.

[0186] In some embodiments, the loss value calculated in step 160 is used to update the parameters of the decision network PolicyNetwork, and this round of training ends.

[0187] Step 180: When the parameters of the decision network meet the preset conditions, the training of the decision network is completed.

[0188] In some embodiments, it is determined whether a training termination condition is met, for example, whether the parameters of the decision network meet a preset accuracy. If the termination condition is met, the training ends; otherwise, the process proceeds to step 120 to enter a new round of training.

[0189] Step 190: Use the trained decision network as the aircraft attitude control decision model.

[0190] In some embodiments, the trained decision network is used as the aircraft attitude control decision model. The above training process is essentially a game between expert training and reinforcement learning. When the learning rate used in the expert training process is larger, the expert training has a greater impact on the neural network; when the learning rate in reinforcement learning is larger, the reinforcement learning has a greater impact on the neural network. However, this game is controlled by the "training termination condition". For example, the termination condition of training can also be that the neural network can control the aircraft to complete all picking tasks without error in 100 consecutive training rounds. Therefore, whether there is more expert training or more reinforcement learning, the neural network must reach parameter convergence and be able to make correct decisions continuously.

[0191] Figure 3 A flow chart of a method 300 for generating an attitude control decision for an aircraft according to an embodiment of the present disclosure is shown. The method 300 includes:

[0192] Step 310, obtaining flight trajectory data of the aircraft to be controlled;

[0193] Step 320 , inputting the flight trajectory data into the aircraft attitude control decision model trained according to the method 100 of the present disclosure, and outputting a flight control strategy;

[0194] Step 330: Perform flight control on the aircraft to be flight controlled based on the output flight control strategy.

[0195] In some embodiments, by collecting flight trajectory data of the aircraft to be controlled and inputting it into a pre-trained aircraft attitude control decision model, a flight control strategy can be output to control the flight of the aircraft.

[0196] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0197] The above is an introduction to the method embodiment. The following is a further explanation of the solution disclosed in the present disclosure through an apparatus embodiment.

[0198] Figure 4FIG. 4 shows a block diagram of a device 400 for generating an aircraft attitude control decision model according to an embodiment of the present disclosure. Figure 4 As shown, the apparatus 400 includes:

[0199] The data acquisition module 410 is used to obtain the historical trajectory data of the target aircraft;

[0200] Decision acquisition module 420, for acquiring expert decisions and network decisions; the expert decisions are obtained by expert training a pre-built decision network based on the historical operation trajectory data; the network decisions are obtained by using the decision network to make decisions based on the historical operation trajectory data;

[0201] A calculation module 430 is configured to compare and calculate the expert decision and the network decision to obtain a first loss value;

[0202] A parameter updating module 440, configured to update the parameters of the decision network according to the first loss value;

[0203] The data acquisition module 410 is further configured to acquire simulated trajectory data; the simulated trajectory data is obtained by performing reinforcement learning on the decision network after updating parameters based on the historical trajectory data and the preset flight scenario;

[0204] The calculation module 430 is further configured to compare and calculate the simulated running trajectory data and the actual running trajectory data to obtain a second loss value; the actual running trajectory data is the historical running trajectory data at the same time point as the simulated running trajectory data;

[0205] The parameter updating module 440 is further configured to update the parameters of the decision network according to the second loss value;

[0206] A model generation module 450 is configured to complete the training of the decision network when the parameters of the decision network meet preset conditions;

[0207] The model generation module 450 is further configured to use the trained decision network as an aircraft attitude control decision model.

[0208] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0209] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device and a readable storage medium.

[0210] Figure 5A schematic block diagram of an electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0211] The electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a ROM 502 or a computer program loaded from a storage unit 508 into a RAM 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An I / O interface 505 is also connected to the bus 504.

[0212] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0213] The computing unit 501 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the method for generating an aircraft attitude control decision model. For example, in some embodiments, the method for generating an aircraft attitude control decision model can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method for generating an aircraft attitude control decision model described above can be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the method for generating the aircraft attitude control decision model in any other appropriate manner (for example, by means of firmware).

[0214] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0215] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0216] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0217] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0218] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0219] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0220] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0221] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for generating an aircraft attitude control decision model, characterized in that: include: Obtain historical trajectory data of the target aircraft; Obtaining expert decisions and network decisions; the expert decisions are obtained by expert training a pre-built decision network based on the historical operation trajectory data; the network decisions are obtained by using the decision network to make decisions based on the historical operation trajectory data; Comparing and calculating the expert decision and the network decision to obtain a first loss value; Updating parameters of the decision network according to the first loss value; Acquiring simulated operation trajectory data; the simulated operation trajectory data is obtained by performing reinforcement learning on the decision network after updating parameters based on the historical operation trajectory data and the preset flight scenario; Comparing and calculating the simulated running trajectory data and the actual running trajectory data to obtain a second loss value; the actual running trajectory data is the historical running trajectory data at the same time point as the simulated running trajectory data; Updating the parameters of the decision network according to the second loss value; When the parameters of the decision network meet the preset conditions, completing the training of the decision network; The trained decision network is used as the aircraft attitude control decision model.

2. The method according to claim 1, characterized in that The historical trajectory data includes the angle of attack data and sideslip angle data of the aircraft; The decision network is constructed through the following steps: Constructing a polyhedral description model of the aircraft according to the angle of attack data and the sideslip angle data and a preset first algorithm; Constructing an attitude control strategy model for the aircraft based on the polyhedral description model and a preset second algorithm; The posture control strategy model is used as a decision network.

3. The method according to claim 2, characterized in that The polytope description model is shown in the following formula: ， ， ， ， ， in, is the angle of attack, is the pitch angular velocity, To control the rudder deflection angle, P is the prediction step number, q is the velocity head, S is the characteristic cross-sectional area, L is the characteristic length of the body, m is the control step number, is the aerodynamic coefficient at angle of attack, 、 、 is the aerodynamic moment coefficient, is the aerodynamic coefficient of the control rudder, v is the flight speed, is the rotational inertia of the body.

4. The method according to claim 2, characterized in that The decision network is pre-built with a reward function; The reward function is shown in the following formula: , in, is the preference triple, D is the data set, y is the label, and is the trajectory segment, y(0) is The corresponding rolling time domain optimization label, y(1) is The corresponding rolling time domain optimization label, indicates that trajectory segment i is better than trajectory segment j, The reward function is a predictor model of preference established according to the Bradley-Terry model, and its expression is as follows: , Status Next action rewards.

5. The method according to claim 4, characterized in that The method further comprises: Calculating the prediction factor obtained by the reward function to obtain a reward estimate; The formula for calculating the predicted factor obtained by the reward function to obtain the reward estimate is as follows: ， in, is the reward estimate, N is the number of predictors, For the The reward of a trajectory, s is the state on the trajectory, and a is the action on the trajectory.

6. The method according to claim 4, characterized in that The trajectory segments in the preference triplet are represented by selecting the trajectory segment with the highest MEIV value; And the sampling methods of trajectory segments include: uniform sampling, set-based sampling and entropy-based sampling.

7. A method for generating attitude control decisions for an aircraft, characterized in that: include: Obtain flight trajectory data of the aircraft to be controlled; Inputting the flight trajectory data into an aircraft attitude control decision model trained according to any one of the methods of claims 1-6, and outputting a flight control strategy; Based on the output flight control strategy, flight control is performed on the aircraft to be controlled.

8. A device for generating an aircraft attitude control decision model, characterized in that: include: A data acquisition module is used to obtain the historical trajectory data of the target aircraft; A decision acquisition module is used to obtain expert decisions and network decisions; the expert decisions are obtained by expert training a pre-built decision network based on the historical operation trajectory data; the network decisions are obtained by using the decision network to make decisions based on the historical operation trajectory data; a calculation module, configured to compare and calculate the expert decision and the network decision to obtain a first loss value; A parameter updating module, configured to update the parameters of the decision network according to the first loss value; The data acquisition module is further used to acquire simulated operation trajectory data; the simulated operation trajectory data is obtained by performing reinforcement learning on the decision network after updating parameters based on the historical operation trajectory data and the preset flight scenario; The calculation module is further configured to compare and calculate the simulated running trajectory data and the actual running trajectory data to obtain a second loss value; the actual running trajectory data is historical running trajectory data at the same time point as the simulated running trajectory data; The parameter updating module is further used to update the parameters of the decision network according to the second loss value; A model generation module, configured to complete the training of the decision network when the parameters of the decision network meet preset conditions; The model generation module is also used to use the trained decision network as an aircraft attitude control decision model.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

Images