A longitudinal control method for fixed-wing UAV based on output feedback Q-learning

Abstract

The present invention provides a method for longitudinal control of a fixed-wing UAV based on output feedback Q learning, comprising: establishing a longitudinal dynamics model of a fixed-wing UAV, and reconstructing state information using input and output data; secondly, constructing a Q function of output feedback, and based on the Bellman equation, identifying the kernel matrix of the Q function using input and output data; finally, based on the learned kernel matrix, designing the optimal output feedback control gain and control input, and verifying the learned control law on a nonlinear model. The present invention utilizes Q learning technology to realize adaptive learning of output feedback control of a fixed-wing UAV based on the input and output data of the system. Compared with the traditional PID control method, the present invention not only has better adaptive capabilities to model uncertainty and operating point changes, but also has a simpler controller parameter design process than PID parameter setting, and the control effect is also better than a fixed-parameter PID controller.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle flight control, and in particular relates to a fixed-wing unmanned aerial vehicle longitudinal control method based on output feedback Q learning. Background Art

[0002] At present, most UAV flight control systems use the control strategy of the classic PID control theory.

[0003] The algorithm has a simple control structure and principle, and is easy to implement in engineering. However, its disadvantages are poor robustness, weak control system resistance to external interference, poor adaptability to systems with complex models and model uncertainty, and a complex process for adjusting system parameters with unknown models.

[0004] As the aerodynamic structure of drones is becoming more and more complex, and the flight missions of drones are becoming more and more diverse, drone control systems need to have a certain degree of intelligence and autonomy, so as to solve the problem of complex and unknown models of drones themselves, and at the same time, when the external environment changes, the drone control system can adaptively learn the optimal control law. Therefore, there is an urgent need for a model-free control algorithm that can solve the unknown model and environment, and design control laws in a data-driven environment to keep the drone's optimal control performance. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a fixed-wing UAV longitudinal control method based on output feedback Q learning in view of the shortcomings of the prior art. The method uses the input and output data of the UAV system to adaptively learn the optimal longitudinal control law, improve the adaptability of the UAV control system to the environment, and realize that the design of the control law is not limited to the known system dynamics model.

[0006] The method of the present invention comprises the following steps:

[0007] Step 1: Perform kinematic analysis and dynamic modeling on the UAV, and obtain the state space model of the longitudinal motion of the UAV after linearization;

[0008] Step 2: Under the condition that the system is observable, reconstruct the state information using the system input and output data;

[0009] Step 3: Use the reconstructed state information to construct the output feedback Q function, and derive the output feedback Q function to obtain the optimal control law matrix of the output feedback;

[0010] Step 4: Parameterize the output feedback Q function, collect system input and output data, and solve the optimal control matrix online through iteration;

[0011] Step 5: The elements in the optimal control matrix are combined into the optimal control law, and the optimal control input is calculated.

[0012] In step 1, the state space model of the longitudinal motion of the UAV includes four state variables, namely, velocity V, angle of attack α, pitch rate q and pitch angle θ, and the control input is the elevator deflection angle δ e , the nonlinear differential equation is:

[0013]

[0014]

[0015]

[0016]

[0017] in is the time derivative of velocity V, is the time derivative of the angle of attack α, is the time derivative of the angle of attack q, is the derivative of the angle of attack θ with respect to time; T t is the thrust, m is the mass of the drone, and g is the acceleration due to gravity (9.8 m / s in SI units). 2 ), ρ is the air density, I y is the moment of inertia around the y-axis in the body coordinate system, C L , C D and C m are the lift coefficient, drag coefficient and pitching moment coefficient, respectively. is the mean aerodynamic chord and S is the wing area.

[0018] In step 1, after the dynamic modeling, the nonlinear differential equation is linearized by using the trim() and linmod() functions in Matlab to obtain the expression of the state space model of the longitudinal motion of the UAV:

[0019] x k+1 =Ax k +Bu k

[0020] y k =Cx k ,

[0021] in represents a real number set, n is the dimension of the state variable, n1 is the dimension of the control input, and p is the dimension of the system output; the state variable x k is the discretized UAV speed V, angle of attack α, pitch rate q, and pitch angle θ at the current moment; x k+1u is the UAV speed V, angle of attack α, pitch rate q, and pitch angle θ at the next moment after discretization; k is the elevator deflection angle δ e ; A is the system matrix, B is the control matrix, and C is the observation (output) matrix; y k Output values ​​for the UAV system.

[0022] Step 2 includes: reconstructing the state information by the following formula:

[0023]

[0024] Among them, M u Reconstruct the matrix for the control input, M y is the system output reconstruction matrix, is the control input vector composed of control inputs at different times, is the output vector consisting of the system output at different times, and

[0025]

[0026] in represents the control input at time kN, represents the system output at time kN, the superscript T is the matrix transpose; U N is the controllability matrix of the system, V N is the observability matrix of the system, D N is the system coupling matrix, N is the number of steps used in the past, and the value of N should be greater than or equal to the system dimension n; the symbols are expressed as follows:

[0027]

[0028] U N =[B AB…A N-1 B],V N =[(CA N-1 ) T …(CA) T C T ] T ,

[0029]

[0030] In step 3, construct the following output feedback Q function Q(x k ,u k ):

[0031]

[0032] Where Q1 and R are weight matrices, γ is the discount factor and 0<γ≤1, u kis the control input, x k is the state variable;

[0033] P is the solution of the Riccati equation, z k is a vector composed of control input and system output in the past N moments, and the output feedback Q function Q(x k ,u k ) is further written as:

[0034]

[0035] Where H is the Q function kernel matrix, which is expressed as follows:

[0036]

[0037] in for The performance function matrix, for and The coupling performance function matrix, for and The coupling performance function matrix, for and u k The coupling performance function matrix, for u k and The coupling performance function matrix, for The performance function matrix, for with u k The coupling performance function matrix, for u k and The coupling performance function matrix, H uu for u k The performance function matrix is ​​expressed as follows:

[0038]

[0039]

[0040] H uu =R+γB T PB.

[0041] In step 4, the input and output data of the system in the time range [kN, k-1] are collected, and the collected u k-1 ,u k-2 ,...,u k-N ,yk-1 ,y k-2 ,...,y k-N The optimal control matrix H is obtained using a policy iteration algorithm based on output feedback Q function learning.

[0042] In step 4, the policy iteration algorithm based on output feedback Q function learning specifically includes:

[0043] Step 4-1, initialize H 0 , H 0 is the initialization value of H, initialize j = 0, select a stable initial control strategy u0, and initialize the weight matrix Q y and R, the initial discount factor γ;

[0044] Step 4-2, strategy evaluation: Collect the input and output data of the system, collect a data set L of sufficient step length to solve the output feedback Q function Bellman equation, where L satisfies L≥(n1N+pN+n1)×(n1N+pN+n1+1) / 2:

[0045]

[0046] Among them, H j is the H matrix of the jth iteration, Q y is the weight matrix;

[0047] Step 4-3, strategy update: Use the elements in the solved kernel matrix H to update the control law:

[0048]

[0049] is the control input at the j+1th iteration of the kth step;

[0050] Step 4-4, termination condition: when the convergence condition || H is reached j -H j-1 The iteration is stopped when ||≤ε, and H is the optimal control matrix at this time; ε is a pre-designed convergence condition constant, and the value of ε is generally a relatively small positive number, for example, ε is 0.01. If the condition for stopping the iteration is not met, j=j+1 is updated and the iteration continues.

[0051] In step 5, the elements in the optimal control matrix are combined into the optimal control law. The optimal control law is multiplied by the input and output to obtain the optimal control input. The optimal control matrix and the optimal control law are expressed as:

[0052]

[0053] H * Indicates that the convergence condition ||H is satisfied j -Hj-1 The H matrix when ||≤ε is the optimal control matrix.

[0054] Furthermore, the present invention also provides a fixed-wing UAV longitudinal control device based on output feedback Q learning, comprising:

[0055] The model building module is used to perform kinematic analysis and dynamic modeling on the UAV, and obtain the state space model of the longitudinal motion of the UAV after linearization;

[0056] A state information reconstruction module is used to reconstruct state information using system input and output data under system observable conditions;

[0057] An optimal control law matrix calculation module is used to construct an output feedback Q function using the reconstructed state information, and to obtain an optimal control law matrix of the output feedback by taking the derivative of the output feedback Q function;

[0058] The optimal control matrix calculation module is used to parameterize the output feedback Q function, collect system input and output data, and iterate online to solve the optimal control matrix;

[0059] The optimal control input calculation module is used to combine the elements in the optimal control matrix into an optimal control law and calculate the optimal control input.

[0060] Furthermore, the present invention also provides a storage medium storing a computer program or instruction, which, when executed, implements the fixed-wing UAV longitudinal control method based on output feedback Q learning.

[0061] The present invention has the following beneficial effects:

[0062] 1. The present invention is a method for longitudinal control of fixed-wing UAV based on output feedback Q learning, which overcomes the problem that the traditional control method requires a UAV dynamics model when designing the control law for the UAV. The flight control law design of this method only needs the input and output data of the system. In this data-driven environment, the Q learning strategy iteration algorithm is used to adaptively learn the control law online without the need for a system model. If the UAV model is unknown and only a coupled state information (observation) can be obtained, the Q learning algorithm based on output feedback can be used.

[0063] 2. Since this method completely utilizes data to learn the flight control law, the model unknown problem is solved, and the model uncertainty problem and modeling complexity problem in the design of traditional control methods can also be solved.

[0064] 3. Compared with the classic PID control algorithm, the present invention not only has the advantage of model-free control, but also has a much better control effect on the sample drone than PID control. In addition, the verification of the control law learned by the algorithm on the nonlinear system also proves the effectiveness and reliability of the present invention.

[0065] 4. The control law design method of the present invention is obtained through online iterative learning. This method is essentially to solve quadratic problems and is suitable for complex systems that do not have sufficient computing power or storage capacity, such as micro air vehicles. Since the Q learning method uses a simple quadratic cost function, the strategy parameters of the kernel matrix converge very quickly, which makes the Q learning method a candidate solution for the application of online adaptive control methods in actual engineering projects. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0067] Figure 1 It is a flow chart for implementing the method of the present invention.

[0068] Figure 2 It is a schematic diagram of the angle of attack change under the output feedback Q learning algorithm.

[0069] Figure 3 It is a schematic diagram of the pitch angle rate change under the output feedback Q learning algorithm.

[0070] Figure 4 It is a schematic diagram of the pitch angle change under the output feedback Q learning algorithm.

[0071] Figure 5 It is a schematic diagram of the change of elevator deflection angle under the output feedback Q learning algorithm.

[0072] Figure 6 It is a schematic diagram of the convergence of each element of the H matrix under the Q learning algorithm.

[0073] Figure 7 is the online learning H matrix and the theoretical H * Schematic diagram of norm change.

[0074] Figure 8 This is a schematic diagram of the angle of attack change when comparing Q learning and PID control methods.

[0075] Fig. 9 This is a schematic diagram of the pitch angle rate change under the comparison of Q learning and PID control methods.

[0076] Fig.10 It is a schematic diagram of pitch angle change under the comparison of Q learning and PID control methods.

[0077] Fig.11 It is a schematic diagram of the angle of attack change (verification) of the nonlinear system under the optimal control law.

[0078] Fig.12 It is a schematic diagram of the pitch angle rate change (verification) of the nonlinear system under the optimal control law.

[0079] Fig.13 It is a schematic diagram of the pitch angle change (verification) of the nonlinear system under the optimal control law.

[0080] Fig.14 It is a schematic diagram of the change of elevator deflection angle of nonlinear system and linear system under the optimal control law. DETAILED DESCRIPTION

[0081] like Figure 1 As shown, the present invention provides a fixed-wing UAV longitudinal control method based on output feedback Q learning, comprising:

[0082] Step 1: Perform kinematic analysis and dynamic modeling on the UAV, and obtain the state space model of the longitudinal motion of the UAV after linearization;

[0083] The longitudinal model of the UAV includes four state variables, namely velocity V, angle of attack α, pitch rate q, pitch angle θ, and the control input is the elevator δ e . The nonlinear differential equation is:

[0084]

[0085]

[0086]

[0087]

[0088] Where T t is the thrust, m is the mass of the drone, ρ is the air density, I y is the moment of inertia about the y-axis, C L , C D and C m are the lift coefficient, drag coefficient and pitching moment coefficient, respectively. is the mean aerodynamic chord and S is the wing area.

[0089] In this example, the speed V is kept constant at 200 m / s. In this case, the state space of the system is transformed into a third-order model. After linearizing and discretizing the model, the state space expression is obtained as follows:

[0090]

[0091] where x k =[α q θ] T ,u k =δ e .

[0092] The system is set to be unable to measure the angle of attack, and can only obtain information on the pitch angle and pitch rate. Therefore, the system output is designed as a sensor fusion mode:

[0093] y k =[0 57.3 57.3]x k

[0094] Step 2: Since the output feedback control law design is studied, the state information is reconstructed using the system input and output data under the condition that the system is observable:

[0095]

[0096] in

[0097]

[0098] U N =[B AB A 2 B],V N =[(CA 2 ) T (CA) T C T ] T ,

[0099]

[0100] Step 3: Define an output feedback Q function:

[0101]

[0102] P is the solution of the Riccati equation, The Q function is further written as:

[0103]

[0104] in:

[0105]

[0106]

[0107] H uu =R+γB T PB;

[0108] Here R = 5, Qy =1, γ = 0.5, the solutions of the Riccati equation and Q1 are:

[0109] Q1=C T Q y C

[0110] Substituting into the formula, we can get the theoretical value of H as:

[0111]

[0112] The corresponding control law matrix is:

[0113]

[0114] Step 4: Collect the input and output data of the system in the time range [kN, k-1]. Here, the longitudinal model is of order 3, so N = 3. Since the corresponding H is 7×7, L ≥ (7×8) / 2, L can be 30, and H 0 =0.01*I, I is the unit matrix. k-1 ,u k-2 ,u k-3 ,y k-1 ,y k-2 ,y k-3 Use the least squares method to solve the Q function Bellman equation. When the convergence condition ||H is met j -H j-1 ||≤ε, stop the iteration, and H is the optimal control matrix.

[0115] Step 5: Some elements in the optimal control matrix H form the optimal control law. The optimal control law is multiplied by the input and output to obtain the optimal control input. Finally, the optimal control matrix and the optimal control law are expressed as:

[0116]

[0117] The steps of the policy iteration algorithm based on output feedback Q learning are as follows:

[0118] 1) Initialize H 0 , select a stable initial control strategy u0 and initialize the weight matrix Q y and R, initializing the discount factor γ

[0119] 2) Strategy evaluation: Collect the input and output data of the system, and collect a data set with sufficient step length to solve the Q function Bellman equation:

[0120]

[0121] 3) Strategy update: Use the solved kernel matrix H elements to update the control law:

[0122]

[0123] 4) Termination condition: When the convergence condition || H is reached j -H j-1 The iteration is stopped when ||≤ε, where ε is a pre-designed convergence condition constant, otherwise the iteration continues with j=j+1.

[0124] The kernel matrix H obtained by online learning of the above algorithm is:

[0125]

[0126]

[0127] H uu =49.7140;

[0128] The corresponding control law matrix is:

[0129]

[0130] The controller of the present invention overcomes the problem that traditional control methods require a UAV dynamics model when designing UAV control laws. The flight control law design of this method only requires the input and output data of the system. In this data-driven environment, the Q learning strategy iterative algorithm is used to adaptively learn the control law online without the need for a system model. Figure 1 Flow chart for realizing the method of the present invention; Figure 2 , Figure 3 , Figure 4 , Figure 5 This is a schematic diagram of the changes in the angle of attack, pitch rate, pitch angle, and elevator deflection angle under the output feedback Q-learning algorithm. Each state can quickly reach a stable state; Figure 6 and Figure 7 The convergence of each element of H matrix in the learning process and the relationship between H matrix and H in theory * The norm of , as can be seen from the figure, the optimal control matrix can be learned in the second iteration; Figure 8 , Fig. 9 , Fig.10 This is a control response comparison between the method of the present invention and the classic PID method. It can be seen from the figure that the method has a faster adjustment speed and smaller overshoot on the sample UAV model, and can even achieve smooth adjustment without overshoot, which has obvious control performance advantages; Fig.11 , Fig.12 , Fig.13 , Fig.14The effect of applying the learned optimal control law to a nonlinear system is compared with that of a linear system. As can be seen from the figure, in addition to a smaller steady-state error, the optimal control law learned under linear conditions can complete the regulation of the original nonlinear system, and the regulation effect is not much different, which proves the effectiveness of the method of the present invention.

[0131] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of a method for longitudinal control of a fixed-wing unmanned aerial vehicle based on output feedback Q learning provided by the present invention and some or all of the steps in each embodiment can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0132] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of a computer program, i.e., a software product, which can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0133] The present invention provides a method for longitudinal control of a fixed-wing UAV based on output feedback Q learning. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A method for longitudinal control of a fixed-wing UAV based on output feedback Q learning, characterized in that: The following steps are involved: Step 1: Perform kinematic analysis and dynamic modeling on the UAV, and obtain the state space model of the longitudinal motion of the UAV after linearization; Step 2: Under the observable condition of the UAV system, reconstruct the state information using the system input and output data; Step 3: Use the reconstructed state information to construct the output feedback Q function, and derive the output feedback Q function to obtain the optimal control law matrix of the output feedback; Step 4: Parameterize the output feedback Q function, collect system input and output data, and solve the optimal control matrix online through iteration; Step 5: The elements in the optimal control matrix are combined into an optimal control law, and the optimal control input is calculated; In step 1, the state space model of the longitudinal motion of the UAV includes four state variables, namely, velocity V, angle of attack α, pitch rate q and pitch angle θ, and the control input is the elevator deflection angle δ e , the nonlinear differential equation is: in is the time derivative of velocity V, is the time derivative of the angle of attack α, is the time derivative of the angle of attack q, is the derivative of the angle of attack θ with respect to time; T t is the thrust, m is the mass of the drone, g is the acceleration due to gravity, ρ is the air density, and I y is the moment of inertia around the y-axis in the body coordinate system, C L , C D and C m are the lift coefficient, drag coefficient and pitching moment coefficient, respectively. is the mean aerodynamic chord, S is the wing area; In step 1, after the dynamic modeling, the nonlinear differential equation is linearized by using the trim() and linmod() functions in Matlab to obtain the expression of the state space model of the longitudinal motion of the UAV: x k+1 =Ax k +Bu k y k =Cx k , in n is the dimension of state variables, n1 is the dimension of control input, and p is the dimension of system output; the state variable x k is the discretized UAV speed V, angle of attack α, pitch rate q, and pitch angle θ at the current moment; x k+1 u is the UAV speed V, angle of attack α, pitch rate q, and pitch angle θ at the next moment after discretization; k is the elevator deflection angle δ e ; A is the system matrix, B is the control matrix, C is the observation matrix; y k Output values ​​for the UAV system; Step 2 includes: reconstructing the state information by the following formula: Among them, M u Reconstruct the matrix for the control input, M y is the system output reconstruction matrix, is the control input vector composed of control inputs at different times, is the output vector consisting of the system output at different times, and in represents the control input at time kN, represents the system output at time kN, the superscript T is the matrix transpose; U N is the controllability matrix of the system, V N is the observability matrix of the system, D N is the system coupling matrix, N is the number of steps used in the past, and the value of N should be greater than or equal to the system dimension n; the symbols are expressed as follows: U N =[B AB … A N-1 B],V N =[(CA N-1 ) T … (CA) T C T ] T , In step 3, construct the following output feedback Q function Q(x k ,u k ): Where Q1 and R are weight matrices, γ is the discount factor and 0<γ≤1, u k is the control input, x k is the state variable; P is the solution of the Riccati equation, z k is a vector composed of control input and system output in the past N moments, and the output feedback Q function Q(x k ,u k ) is further written as: Where H is the Q function kernel matrix, which is expressed as follows: in for The performance function matrix, for and The coupling performance function matrix, for and The coupling performance function matrix, for and u k The coupling performance function matrix, for u k and The coupling performance function matrix, for The performance function matrix, for with u k The coupling performance function matrix, for u k and The coupling performance function matrix, H uu for u k The performance function matrix is ​​expressed as follows: In step 4, the input and output data of the system in the time range [kN, k-1] are collected, and the collected u k-1 ,u k-2 ,...,u k-N ,y k-1 ,y k-2 ,...,y k-N Use the policy iteration algorithm based on output feedback Q function learning to obtain the optimal control matrix H; In step 4, the policy iteration algorithm based on output feedback Q function learning specifically includes: Step 4-1, initialize H 0 , H 0 is the initialization value of H, initialize j = 0, select a stable initial control strategy u0, and initialize the weight matrix Q y and R, the initial discount factor γ; Step 4-2, strategy evaluation: Collect the input and output data of the system, collect a data set L of sufficient step length to solve the output feedback Q function Bellman equation, where L satisfies L≥(n1N+pN+n1)×(n1N+pN+n1+1) / 2: Among them, H j is the H matrix of the jth iteration, Q y is the weight matrix; Step 4-3, strategy update: Use the elements in the solved kernel matrix H to update the control law: is the control input at the j+1th iteration of the kth step; Step 4-4, termination condition: when the convergence condition || H is reached j -H j-1 When ||≤ε, the iteration stops. At this time, H is the optimal control matrix. ε is a pre-designed convergence condition constant. If the condition for stopping the iteration is not met, j=j+1 is updated and the iteration continues. In step 5, the elements in the optimal control matrix are combined into the optimal control law. The optimal control law is multiplied by the input and output to obtain the optimal control input. The optimal control matrix and the optimal control law are expressed as: H * Indicates that the convergence condition ||H is satisfied j -H j-1 The H matrix when ||≤ε is the optimal control matrix.

Images