Nonlinear dynamic inverse control method based on reinforcement learning agent

Abstract

The invention discloses a nonlinear dynamic inverse control method based on a reinforcement learning agent. The nonlinear dynamic inverse control method comprises the following steps: establishing an attitude dynamic model of an aircraft; the attitude dynamics model is converted into an outer control loop and an inner control loop, the outer control loop controls the roll angle, the pitch angle and the yaw angle, and the inner control loop controls the roll angle speed, the pitch angle speed and the yaw angle speed; an outer loop control law is obtained based on the outer control loop, and control parameters of the outer loop control law are updated in real time based on reinforcement learning; an error is introduced on the basis of the inner control loop, a disturbance observer is constructed to estimate the error, and an inner loop control law is obtained based on the estimated error and the inner control loop; and performing flight control by combining the outer loop control law and the inner loop control law. The method is applied to the field of aircrafts, a reinforcement learning method is combined with a traditional nonlinear dynamic inverse method, controller parameters are dynamically optimized, and the robustness of a control system under the condition that model errors exist is improved.

Description

Technical Field

[0001] The present invention relates to the field of aircraft control technology, and in particular to a nonlinear dynamic inverse control method based on reinforcement learning intelligent agent. Background Art

[0002] Nonlinear Dynamic Inversion (NDI) is a control method with fast response speed, strong decoupling ability and good adaptability, and is widely used in aircraft control systems. This method constructs a feedback function by "inversion" to eliminate the nonlinear characteristics of the system and the coupling between channels, thereby achieving tracking control of the desired dynamics of the system. However, this method is heavily dependent on the accuracy of system modeling. Therefore, in the presence of model errors, how to improve the robustness of the control system has always been a research problem.

[0003] Some studies have used the model-independent incremental dynamic inversion method (Incremental NDI, INDI) to improve the robustness of the control system. However, this method requires accurate angular acceleration data. The actual aircraft is not equipped with such sensors to measure angular acceleration. Usually, the angular acceleration is approximately calculated using angular velocity differential or neural network estimation methods. The approximation process is prone to produce large errors, which affects the system control performance.

[0004] There are also studies that improve the robustness of dynamic inverse control methods by designing disturbance observers. Common disturbance observers include neural network-based disturbance observers and nonlinear disturbance observers. The former uses the approximation ability of neural networks for nonlinear functions to estimate the model inverse error and compensate it to the control input. This method has the characteristics of fast response speed and good control performance. The disadvantage is that the neural network needs to be trained offline, and the process of neural network error estimation has no clear physical meaning and cannot prove its stability. The latter constructs nonlinear functions to track the errors caused by various disturbances in the control system. This method has the characteristics of simple structure and good stability. The disadvantage is that it needs to obtain the derivative data of the aircraft output.

[0005] For nonlinear dynamic inverse controller, its control performance is greatly affected by control parameters, which are generally obtained through empirical debugging or parameter optimization. The prior art designs control parameters through empirical debugging or parameter optimization. Since the obtained control parameters are constant values, the control system is usually unable to adapt to tracking control under different motion states. Summary of the invention

[0006] In view of the deficiencies in the above-mentioned prior art, the present invention provides a nonlinear dynamic inverse control method based on reinforcement learning agent, which combines the reinforcement learning method with the traditional nonlinear dynamic inverse method, utilizes the powerful environmental interaction and learning ability of the reinforcement learning agent, and dynamically optimizes the controller parameters to improve the robustness of the control system.

[0007] To achieve the above object, the present invention provides a nonlinear dynamic inverse control method based on reinforcement learning agent, comprising the following steps:

[0008] Establish the attitude dynamics model of the aircraft;

[0009] Converting the attitude dynamics model into an outer control loop and an inner control loop, wherein the outer control loop is used to control the roll angle, pitch angle and yaw angle of the aircraft, and the inner control loop is used to control the roll angular velocity, pitch angular velocity and yaw angular velocity of the aircraft;

[0010] Obtaining an outer loop control law based on the outer control loop, and updating control parameters of the outer loop control law in real time based on reinforcement learning;

[0011] Introducing an error based on the inner control loop, constructing a disturbance observer to estimate the error, and obtaining an inner loop control law based on the estimated error and the inner control loop;

[0012] The outer loop control law and the inner loop control law are combined to perform flight control on the aircraft.

[0013] In one embodiment, in step 1, the posture dynamics model is:

[0014]

[0015] Among them, φ, θ, ψ represent the roll angle, pitch angle and yaw angle of the aircraft respectively, p, q, r represent the roll angular velocity, pitch angular velocity and yaw angular velocity of the aircraft respectively, Γ i (i=1,2,...8) are the coefficients of the moment equation, and L, M, and N represent the rolling moment, pitching moment, and yaw moment, respectively.

[0016] In one embodiment, the outer control loop is:

[0017]

[0018] Where, Ω = [φθψ] T represents the attitude angle, ω=[pqr] T represents the angular velocity, f1=0 represents the attitude angle change rate in the trim state, Represents the conversion matrix of attitude angle change rate and angular velocity, T is the transpose of the matrix;

[0019] The outer loop control law is:

[0020]

[0021] Among them, Ω d represents the desired attitude angle, ω d represents the angular velocity required to track the desired attitude angle, and K1 represents the designed parameter matrix, which is:

[0022]

[0023] Among them, K φ , K θ , K ψ Represents the control parameters of the outer loop control law.

[0024] In one embodiment, the control parameters of the outer loop control law are updated in real time based on reinforcement learning as follows:

[0025] A TD3-based reinforcement learning agent is used to observe the attitude control error of the aircraft, so as to continuously adjust the control parameters of the outer loop control law. The neural network structure in the TD3 reinforcement learning algorithm consists of a Critic network and an Actor network:

[0026] In the Critic network, the motion state Ω of the aircraft is d -Ω and the control parameters of the outer loop control law are respectively input as state variables and action variables, respectively pass through two fully connected layers containing 64 neurons, the two fully connected layers are connected through a combination layer, and sequentially pass through two fully connected layers containing 32 and 16 neurons to output the current state-action evaluation value;

[0027] In the Actor network, the motion state Ω of the aircraft is d -Ω is used as input, and then passes through the fully connected layers containing 64 and 32 neurons in turn, and finally outputs the action signal, i.e., the control parameter of the outer loop control law, through the fully connected layer containing three neurons.

[0028] In one embodiment, during the training of a reinforcement learning agent based on the TD3 reinforcement learning algorithm, the reward function r(t) is:

[0029]

[0030] in, represents the weight coefficient, Ω(t), Ω d (t) represent the actual attitude and target attitude of the aircraft at time t respectively.

[0031] In one embodiment, the inner control loop is:

[0032]

[0033] Where ω = [pqr] T represents the angular velocity, δ u =[δ a δ e δ r ] T represents the rudder deflection angle required to track the desired angular velocity, δ a ,δ e ,δ r They represent the aileron deflection angle, elevator deflection angle and rudder deflection angle of the aircraft respectively, f2 represents the angular velocity state transfer function, g2 represents the control input function from rudder deflection angle to angular acceleration, and T is the transpose of the matrix;

[0034]

[0035]

[0036] in, represents the aerodynamic pressure on the aircraft during flight, ρ represents the air density, V a represents airspeed, S, b, c represent wing area, wingspan and average aerodynamic chord length respectively, α and β represent angle of attack and sideslip angle respectively, C l0 , C m0 , C n0 Represents the basic coefficient of aerodynamic moment of each axis, C lβ , C mα , C nβ They represent the aerodynamic moment stability derivative, C lp , C lr , C mq , C np , C nr They represent the aerodynamic derivatives of the torque of each axis with respect to the angular velocity, C lδa , C lδr , C mδe , C nδa , C nδr They represent the control derivatives of the torque of each axis relative to the control surface, I y Represents the moment of inertia of the aircraft relative to the y-axis;

[0037] Introducing an error on the basis of the inner control loop, we get:

[0038]

[0039] Among them, D frepresents the composite error, Δf2 and Δg2 represent the model error, and d f represents interference error;

[0040] Finally, the inner loop control law is obtained as:

[0041]

[0042] Among them, ω d represents the angular velocity required to track the desired attitude angle, Denotes the composite error D obtained by the disturbance observer f The estimated value of , K1 represents the designed parameter matrix, which is:

[0043]

[0044] Among them, K p , K q , K r Represents the control parameters of the inner loop control law.

[0045] In one embodiment, the interference observer is specifically:

[0046]

[0047] Among them, z f represents the error estimation function of the disturbance observer, Φ f (ω)=[L f1 p L f2 q L f3 r] T represents the error estimate associated with the angular velocity, represents the parameter matrix, L f1 , L f2 , L f3 Represents design parameters.

[0048] Compared with the prior art, the present invention has the following beneficial technical effects:

[0049] 1. The present invention combines the reinforcement learning method with the traditional nonlinear dynamic inversion method, introduces the reinforcement learning agent based on the TD3 algorithm as the parameter regulator in the outer control loop, and adjusts the control parameters of the outer loop control law in real time according to different flight states of the aircraft, thereby effectively improving the robustness of the control system;

[0050] 2. The present invention uses a nonlinear disturbance observer in dynamic inverse control, and solves the dependence of traditional dynamic inverse methods on accurate models by estimating model errors and compensating controller outputs. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0052] Figure 1 Flow chart of a nonlinear dynamic inverse control method based on a reinforcement learning agent in an embodiment of the present invention;

[0053] Figure 2 This is a block diagram of the aircraft attitude control in an embodiment of the present invention;

[0054] Figure 3 is a structural block diagram of a controller in an embodiment of the present invention;

[0055] Figure 4 This is a diagram of the TD3 neural network structure in an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of the training results of the control parameter adjuster in an embodiment of the present invention;

[0057] Figure 6 Schematic diagram of the attitude angle control of the aircraft in the embodiment of the present invention, wherein: (a) is a schematic diagram of the roll angle, (b) is a schematic diagram of the pitch angle, and (c) is a schematic diagram of the yaw angle;

[0058] Figure 7 Schematic diagram of the change of the angular velocity of the aircraft attitude in the embodiment of the present invention, wherein: (a) is a schematic diagram of the roll angular velocity, (b) is a schematic diagram of the pitch angular velocity, and (c) is a schematic diagram of the yaw angular velocity;

[0059] Figure 8 Schematic diagrams of deflection of various control surfaces of an aircraft in an embodiment of the present invention, wherein: (a) is a schematic diagram of aileron deflection, (b) is a schematic diagram of elevator deflection, and (c) is a schematic diagram of rudder deflection;

[0060] Fig. 9 Schematic diagrams of error tracking in an embodiment of the present invention, wherein: (a) is a schematic diagram of error tracking in the rolling direction, (b) is a schematic diagram of error tracking in the pitching direction, and (c) is a schematic diagram of error tracking in the yaw direction;

[0061] Fig.10 Schematic diagram of the change of control parameters of the inner control loop in an embodiment of the present invention.

[0062] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0065] like Figure 1 The figure shows a nonlinear dynamic inverse control method based on reinforcement learning agent disclosed in this embodiment, which mainly includes the following steps:

[0066] Establish the attitude dynamics model of the aircraft;

[0067] Converting the attitude dynamics model into an outer control loop and an inner control loop, wherein the outer control loop is used to control the roll angle, pitch angle and yaw angle of the aircraft, and the inner control loop is used to control the roll angular velocity, pitch angular velocity and yaw angular velocity of the aircraft;

[0068] An outer loop control law is obtained based on the outer control loop, and control parameters of the outer loop control law are updated in real time based on reinforcement learning;

[0069] An error is introduced on the basis of the inner control loop, and a disturbance observer is constructed to estimate the error. The inner loop control law is obtained based on the estimated error and the inner control loop.

[0070] The outer loop control law and the inner loop control law are combined to control the flight of the aircraft.

[0071] In the specific implementation process, the attitude dynamics model of the fixed-wing aircraft can be expressed as:

[0072]

[0073] Among them, φ, θ, ψ represent the roll angle, pitch angle and yaw angle of the aircraft respectively, p, q, r represent the roll angular velocity, pitch angular velocity and yaw angular velocity of the aircraft respectively, Γ i (i=1,2,...8) are the coefficients of the moment equation, L, M, N represent the rolling moment, pitching moment, and yaw moment, respectively;

[0074] The torque equation coefficient can be expressed as follows:

[0075]

[0076] Among them, I x ,I y ,I z ,I xz are the moments of inertia of the aircraft relative to each coordinate axis, and Γ represents an intermediate parameter;

[0077] The rolling moment, pitching moment and yaw moment can be expressed as follows:

[0078]

[0079] Where ρ represents the air density, V a represents airspeed, S, b, c represent wing area, wingspan and average aerodynamic chord length respectively, α and β represent angle of attack and sideslip angle respectively, δ a ,δ e ,δ r Represent the aileron, elevator and rudder angles respectively, C l0 , C m0 , C n0 Represents the basic coefficient of aerodynamic moment of each axis, C lβ , C mα , C nβ They represent the aerodynamic moment stability derivative, C lp , C lr , C mq , C np , C nr They represent the aerodynamic derivatives of the torque of each axis with respect to the angular velocity, C lδa , C lδr , C mδe , C nδa , C nδr They represent the control derivatives of the torque of each axis relative to the control surface.

[0080] The nonlinear dynamic inverse algorithm constructs the feedback function by inversion. For the state equation, we have:

[0081]

[0082] Therefore, the following control input can be constructed:

[0083]

[0084] Where u represents the control input, x represents the state, f(x) represents the state transfer function, G(x) represents the control input function, y represents the observable output of the system, and h(x) represents the observation function. Indicates a desired state;

[0085] However, the prerequisite for using dynamic inversion is that the state quantity and control quantity of the controlled object have the same dimension and the system is an affine nonlinear system. Therefore, this embodiment divides the attitude dynamics model of the fixed-wing aircraft into an outer control loop and an inner control loop to design corresponding nonlinear dynamic inverse controllers, that is, Figure 2 As shown, Figure 2 In, φ d ,θ d , d They represent the desired roll angle, pitch angle and yaw angle of the aircraft, respectively, and K φ , K θ , K ψ represents the control parameter of the outer loop control law, K p , K q , K r represents the control parameter of the inner loop control law, p d ,q d 、r d They represent the desired roll, pitch and yaw angular velocity of the aircraft, δ a ,δ e ,δ r They represent the aileron deflection angle, elevator deflection angle and rudder deflection angle of the aircraft respectively.

[0086] In the specific implementation process, the outer control loop rewrites equation (1) into an affine nonlinear form as follows:

[0087]

[0088] Where, Ω=[φθψ] T represents the attitude angle, ω=[pqr] T represents the angular velocity, f1=0 represents the attitude angle change rate in the trim state, Represents the conversion matrix of attitude angle change rate and angular velocity, T is the transpose of the matrix;

[0089] Therefore, the outer loop control law is:

[0090]

[0091] Among them, Ω d represents the desired attitude angle, ω d represents the angular velocity required to track the desired attitude angle, and K1 represents the designed parameter matrix, which is

[0092] In the specific implementation process, the inner control loop rewrites equation (2) into an affine nonlinear form as follows:

[0093]

[0094] Among them, δu =[δ a δ e δ r ] T represents the rudder deflection angle required to track the desired angular velocity, f2 represents the angular velocity state transfer function, g2 represents the control input function from rudder deflection angle to angular acceleration, Indicates the aerodynamic pressure that the aircraft is subjected to during flight;

[0095] According to the inner control loop of formula (9), the inner loop control law when no error is introduced can be obtained:

[0096]

[0097] Among them, K2 represents the designed parameter matrix, which is

[0098] Due to the complex nonlinearity and external interference of the attitude dynamics system, its mathematical model usually has errors. Therefore, this embodiment considers the model errors Δf2, Δg2 and the interference error d in the inner control loop of equation (9). f , which can be expressed as:

[0099]

[0100] Among them, D f represents the composite error;

[0101] Therefore, the inner loop control law is finally obtained as:

[0102]

[0103] in, Denotes the composite error D obtained by the disturbance observer f The estimated value of .

[0104] In this embodiment, the interference observer is specifically:

[0105]

[0106] Φ f (ω)=[L f1 p L f2 q L f3 r] T (16)

[0107]

[0108] Among them, z f represents the error estimation function of the disturbance observer, Φ f (ω) represents the error estimate associated with the angular velocity, l f(ω) represents the parameter matrix, L f1 , L f2 , L f3 >0 indicates the parameters to be designed;

[0109] Disturbance observer estimation error e f It can be expressed as:

[0110]

[0111] Assuming the composite error D f Slowly varying and bounded, that is According to formula (14) and formula (15), it can be deduced that:

[0112]

[0113] Because l f (ω) is a positive definite matrix, so the estimation error of the disturbance observer can converge to 0, that is, the estimation of the model error and external disturbance is realized.

[0114] Formula (16) has proved the stability of the closed-loop system after adding the disturbance observer. However, the control performance of the system is affected by the control parameters K1 and K2 of the outer control loop and the inner control loop. Different motion states of the aircraft have different requirements for control parameters. The control parameters in the traditional DO-NDI control method are fixed and cannot adapt to the attitude control of the aircraft in different motion states. This embodiment introduces a reinforcement learning agent based on TD3. The agent continuously adjusts the control parameter K1 of the outer control loop by observing the attitude control error of the aircraft. The controller structure is as follows: Figure 3 As shown, Figure 3 In the equation, e1 represents the attitude angle control error, e2 represents the angular velocity control error, and δ cmd Indicates the rudder angle command.

[0115] The neural network structure in the TD3 reinforcement learning algorithm in this embodiment consists of a Critic network and an Actor network. Figure 4 :

[0116] In the Critic network, the motion state of the aircraft (i.e., the attitude tracking error Ω d -Ω) and the control parameter K1 of the outer loop control law are used as the state variable and action variable input respectively, and pass through two fully connected layers containing 64 neurons respectively. The two fully connected layers are connected by a combination layer, and then pass through two fully connected layers containing 32 and 16 neurons in turn to output the current state-action evaluation value. The activation function is the Relu function;

[0117] In the Actor network, the motion state of the aircraft (i.e., the attitude tracking error Ω d-Ω) as input, and then passes through the fully connected layer containing 64 and 32 neurons in turn, and finally outputs the action signal, that is, the control parameter K1 of the outer loop control law, through the fully connected layer containing three neurons. The activation function of the fully connected layer is the Tanh function.

[0118] When designing parameter regulators, it is usually necessary to train and validate the TD3 agent. Figure 2 The closed-loop control system in is used as the training environment, the control parameter K1 is randomly generated, and the control parameter is adjusted according to the reward value of the environment feedback. After multiple training trials and errors and learning, the policy function finally converges to the optimal, and the agent can output the optimal control parameters to improve the attitude control performance of the aircraft system. The verification process refers to using the trained TD3 agent in the closed-loop system to adjust the slow loop control parameter K1 in real time to verify the feasibility and robustness of the control method under different flight conditions.

[0119] In this embodiment, the number of TD3 agent training is set to 800 times. The data for each training is derived from the flight simulation data of the aircraft under the same working conditions. The flight simulation time is 10s. The initial attitude angle and the expected attitude angle are set to Ω0 = [2deg 3deg 2deg] and Ω d =[-8deg 10deg 5deg], and the rest of the training parameter settings are shown in Table 1.

[0120] Table 1

[0121]

[0122] In this embodiment, the process of training the reinforcement learning agent based on the TD3 reinforcement learning algorithm is as follows:

[0123]

[0124] in, represents the weight coefficient, Ω(t), Ω d (t) represent the actual attitude and target attitude of the aircraft at time t respectively.

[0125] The rewards of each time step are accumulated to get the reward value of this training. When the reward value is greater than -1 or the number of training times reaches the maximum, the training stops. The reward value and the average reward value of each training are as follows: Figure 5 As shown, from Figure 5It can be seen that in the first 90 training sessions, the agent continuously explored the action output and optimized the control strategy based on the feedback reward value, and finally made the reward value converge to the maximum value at the 120th training session. The agent trained in the last training session was used as the control parameter regulator of the closed-loop system, and the flight simulation of the aircraft system was performed under different initial motion states to verify the feasibility and generalization ability of the parameter regulator.

[0126] Since the action output range of the TD3 agent is [1,15], the control parameter of the inner control loop is set to K φ , K θ , K ψ ∈[1, 15], therefore, using the TD3 agent as a parameter regulator of the closed-loop control system will not affect the stability of the proof of formula (19).

[0127] The nonlinear dynamic inverse control method based on the reinforcement learning agent in this embodiment is further explained below with reference to specific examples.

[0128] The main technical parameters of the conventional wing-shaped fixed-wing aircraft object disclosed in the paper “RW Beard, Small Unmanned Aircraft, Princeton University Press, 2012” include shape parameters, aerodynamic parameters, etc., as shown in Table 2 and Table 3 respectively.

[0129] Table 2

[0130]

[0131] Table 3

[0132]

[0133] During the simulation, the initial motion of the aircraft is the trimmed state of level flight at a flight speed of V0 = 30 m / s. Its initial attitude angle and angular velocity are Ω0 = [01.90] (deg) and ω0 =

[000] (deg / s), respectively. The target attitude angle is set to Ω d =[-56.9-5](deg). When there is a deviation in the model parameters, three control methods, namely NDI, DO-NDI and TD3-NDI (i.e., the method of this embodiment), are used to control the attitude of the aircraft.

[0134] When the shape parameter deviation e s = 2%, aerodynamic parameter deviation e a =50%, the flight simulation of the aircraft is carried out under three control methods: NDI, DO-NDI and TD3-NDI. The control parameter settings during the simulation are shown in Table 4.

[0135] Table 4

[0136]

[0137] For the control parameters of the external control loop, both NDI and DO-NDI are set to [K φ K θ K ψ ]=

[878] , and in the TD3-NDI control process, this parameter is adaptively adjusted by the reinforcement learning agent; for the control parameters of the external control loop, all three control methods are set to [K p K q K r ]=[302530]; For the control parameters in the disturbance observer, they are set to [L f1 L f2 L f3 ]=[101010], the flight simulation results of the aircraft under the three control methods are as follows Figures 6 to 10 shown.

[0138] During the flight simulation, the aircraft attitude angle control is as follows: Figure 6 As shown in the figure, it can be seen that when the model parameters are pulled off, the traditional NDI method cannot make the aircraft attitude converge to the expected value, and the steady-state errors of the roll angle, pitch angle and yaw angle are 0.08deg, 0.2deg and 0.7deg respectively. During the DO-NDI and TD3-NDI control processes, the attitude angles of the three axes of the aircraft can converge to the expected values. During the DO-NDI control process, the overshoots of the roll angle, pitch angle and yaw angle are 14.9%, 7.5% and 9.5% respectively, and the convergence times are 2.1s, 2.2s and 3.0s respectively. Under the TD3-NDI control method, the overshoots of the three-axis attitude angles are smaller, which are 0, 3.2% and 6.0% respectively, and the convergence times are 2.0s, 2.3s and 3.3s respectively.

[0139] The change of the aircraft's attitude angular velocity is as follows: Figure 7 As shown in the figure, it can be seen that in the three control methods, the three-axis attitude angular velocity of the aircraft finally converges to 0deg / s. In the DO-NDI control process, the roll angular velocity, pitch angular velocity and yaw angular velocity oscillate in the initial movement process, and the maximum angular velocities are -41.5deg / s, 24.4deg / s and -33.3deg / s, respectively, and the convergence times are 2.1s, 2.2s and 3.0s, respectively; in the TD3-NDI control process, the three-axis attitude angular velocity oscillation phenomenon is not obvious, and the maximum angular velocities are -22.5deg / s, 14.4deg / s and -17.8deg / s, respectively, and the convergence times are 2.0s, 2.3s and 3.3s, respectively.

[0140] The deflection of each control surface of the aircraft is as follows Figure 8 As shown in the figure, it can be seen that in the three control methods, the deflection angles of the three-axis rudder surfaces of the aircraft all converge to a balanced state. In the DO-NDI control process, the aileron, elevator and rudder oscillate during the 0-1s movement process, and the rudder deflection angle ranges are (-24.6deg, 0), (-22.8deg, 0) and (0, 16.8deg), respectively. The larger deflection range and higher oscillation frequency pose a challenge to the deflection performance of the servo; in the TD3-NDI control process, the three-axis rudder deflection oscillation frequency is lower and the angle range is smaller, which are (-15.6deg, 0), (-11.8deg, 0) and (0, 12.2deg), respectively. The rudder deflection under this control method is easier to achieve in actual flight.

[0141] The error tracking of each controller caused by model deviation is as follows: Fig. 9 As shown in the figure, it can be seen that due to the lack of interference observer, the traditional NDI method fails to eliminate the interference caused by model deviation, resulting in Figure 6 The attitude angle cannot converge to the expected value. The DO-NDI and TD3-NDI control methods with the disturbance observer can eliminate the disturbance caused by the model deviation and realize the corresponding rudder deviation compensation. In the DO-NDI method, the slow loop control parameters are fixed and unchanged, and obvious oscillation occurs during the disturbance estimation process. In the TD3-NDI control process, the slow loop control parameters are adaptively adjusted by the TD3 agent, and their changes are shown in Figure 2. Fig.10 As shown, the parameter adjustment process can effectively reduce the oscillation amplitude and oscillation frequency of error tracking.

[0142] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A nonlinear dynamic inverse control method based on reinforcement learning agent, characterized in that: The steps include: Establish the attitude dynamics model of the aircraft; Converting the attitude dynamics model into an outer control loop and an inner control loop, wherein the outer control loop is used to control the roll angle, pitch angle and yaw angle of the aircraft, and the inner control loop is used to control the roll angular velocity, pitch angular velocity and yaw angular velocity of the aircraft; Obtaining an outer loop control law based on the outer control loop, and updating control parameters of the outer loop control law in real time based on reinforcement learning; Introducing an error based on the inner control loop, constructing a disturbance observer to estimate the error, and obtaining an inner loop control law based on the estimated error and the inner control loop; The outer loop control law and the inner loop control law are combined to perform flight control on the aircraft.

2. The nonlinear dynamic inverse control method based on reinforcement learning agent according to claim 1, characterized in that: In step 1, the posture dynamics model is: Among them, φ, θ, ψ represent the roll angle, pitch angle and yaw angle of the aircraft respectively, p, q, r represent the roll angular velocity, pitch angular velocity and yaw angular velocity of the aircraft respectively, Γ i (i=1,2,...8) are the coefficients of the moment equation, and L, M, and N represent the rolling moment, pitching moment, and yaw moment, respectively.

3. The nonlinear dynamic inverse control method based on reinforcement learning agent according to claim 2, characterized in that: The external control loop is: Where, Ω = [φθψ] T represents the attitude angle, ω=[pqr] T represents the angular velocity, f1=0 represents the attitude angle change rate in the trim state, Represents the conversion matrix of attitude angle change rate and angular velocity, T is the transpose of the matrix; The outer loop control law is: Among them, Ω d represents the desired attitude angle, ω d represents the angular velocity required to track the desired attitude angle, and K1 represents the designed parameter matrix, which is: Among them, K φ , K θ , K ψ Represents the control parameters of the outer loop control law.

4. The nonlinear dynamic inverse control method based on reinforcement learning agent according to claim 3, characterized in that: The control parameters of the outer loop control law updated in real time based on reinforcement learning are specifically: A TD3-based reinforcement learning agent is used to observe the attitude control error of the aircraft, so as to continuously adjust the control parameters of the outer loop control law. The neural network structure in the TD3 reinforcement learning algorithm consists of a Critic network and an Actor network: In the Critic network, the motion state Ω of the aircraft is d -Ω and the control parameters of the outer loop control law are respectively input as state variables and action variables, respectively pass through two fully connected layers containing 64 neurons, the two fully connected layers are connected through a combination layer, and sequentially pass through two fully connected layers containing 32 and 16 neurons to output the current state-action evaluation value; In the Actor network, the motion state Ω of the aircraft is d -Ω is used as input, and then passes through the fully connected layers containing 64 and 32 neurons in turn, and finally outputs the action signal, i.e., the control parameter of the outer loop control law, through the fully connected layer containing three neurons.

5. The nonlinear dynamic inverse control method based on reinforcement learning agent according to claim 4, characterized in that: In the process of training a reinforcement learning agent based on the TD3 reinforcement learning algorithm, the reward function r(t) is: in, represents the weight coefficient, Ω(t), Ω d (t) represent the actual attitude and target attitude of the aircraft at time t respectively.

6. The nonlinear dynamic inverse control method based on reinforcement learning agent according to any one of claims 2 to 5, characterized in that: The inner control loop is: Where ω = [pqr] T represents the angular velocity, δ u =[δ a δ e δ r ] T represents the rudder deflection angle required to track the desired angular velocity, δ a ,δ e ,δ r They represent the aileron deflection angle, elevator deflection angle and rudder deflection angle of the aircraft respectively, f2 represents the angular velocity state transfer function, g2 represents the control input function from rudder deflection angle to angular acceleration, and T is the transpose of the matrix; in, represents the aerodynamic pressure on the aircraft during flight, ρ represents the air density, V a represents airspeed, S, b, c represent wing area, wingspan and average aerodynamic chord length respectively, α and β represent angle of attack and sideslip angle respectively, C l0 , C m0 , C n0 Represents the basic coefficient of aerodynamic moment of each axis, C lβ , C mα , C nβ They represent the aerodynamic moment stability derivative, C lp , C lr , C mq , C np , C nr They represent the aerodynamic derivatives of the torque of each axis with respect to the angular velocity, C lδa , C lδr , C mδe , C nδa , C nδr They represent the control derivatives of the torque of each axis relative to the control surface, I y Represents the moment of inertia of the aircraft relative to the y-axis; Introducing an error on the basis of the inner control loop, we get: Among them, D f represents the composite error, Δf2 and Δg2 represent the model error, and d f represents interference error; Finally, the inner loop control law is obtained as: Among them, ω d represents the angular velocity required to track the desired attitude angle, Denotes the composite error D obtained by the disturbance observer f The estimated value of , K1 represents the designed parameter matrix, which is: Among them, K p , K q , K r Represents the control parameters of the inner loop control law.

7. The nonlinear dynamic inverse control method based on reinforcement learning agent according to claim 6, characterized in that: The disturbance observer is specifically: Among them, z f represents the error estimation function of the disturbance observer, Φ f (ω)=[L f1 p L f2 q f3 r] T represents the error estimate related to the angular velocity, represents the parameter matrix, L f1 , L f2 , L f3 Represents design parameters.

Images