An actor-critic-based adaptive control method for small fixed-wing unmanned aerial vehicles
By combining the Actor-Critic framework with neural networks, an adaptive control method for UAVs was constructed, which solved the anti-interference problem of small fixed-wing UAVs in complex environments and achieved rapid and stable flight control and improved anti-interference capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2026-04-07
AI Technical Summary
Traditional control methods are difficult to effectively address the anti-interference capabilities of small fixed-wing UAVs in complex environments, especially in maintaining stable flight under external disturbances.
By employing the Actor-Critic framework combined with neural networks, an Actor network, a Critic network, and a Disturbance network are constructed. The weights are updated through the backpropagation algorithm to achieve adaptive control of the UAV and enhance its anti-interference capability.
In complex environments, drones can fly quickly and stably, improving their anti-interference capabilities and reducing jitter and oscillation caused by external disturbances, making them suitable for aircraft with limited computing power.
Smart Images
Figure CN116610133B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of unmanned aerial vehicle flight control, and particularly relates to a small fixed-wing unmanned aerial vehicle adaptive control method based on Actor-Critic. BACKGROUND
[0002] With the continuous development of reinforcement learning algorithm and computing power, an agent can complete complex decision-making tasks such as flight control, task planning and maneuvering decision-making through reinforcement learning. For the optimal control problem of a nonlinear system, the core idea mainly revolves around solving the Hamilton-Jacobi-Bellman (HJB) equation, and solving this equation is a very challenging problem. In recent years, many researchers have made a lot of research in this regard by using the function approximation characteristics of neural networks. Adaptive dynamic programming (ADP) is an effective optimal control framework, and its unique Actor-Critic design can control many complex MIMO (Multiple-input Multiple-output) systems. For the MIMO complex model of a 6-DoF (6-DoF) unmanned aerial vehicle, modeling and parameter design are required, which is theoretically complex and computationally intensive. Therefore, the flight control system not only needs to have the ability to handle the complex dynamics model of the unmanned aerial vehicle, but also needs to have high adaptability and anti-interference to the environment, which is difficult to apply to traditional control methods. SUMMARY
[0003] The purpose of the present application is to provide a small fixed-wing unmanned aerial vehicle adaptive control method based on Actor-Critic, which can fly more quickly and stably in a complex environment and improve the anti-interference ability of the unmanned aerial vehicle.
[0004] Technical scheme: The small fixed-wing unmanned aerial vehicle adaptive control method comprises the following steps:
[0005] S1, constructing a performance function according to the longitudinal dynamics of the small fixed-wing unmanned aerial vehicle;
[0006] S2, constructing an Actor network and a Critic network, and using a neural network with a momentum coefficient to strengthen the stability of the unmanned aerial vehicle controller;
[0007] S3, information transmission and construction are performed on the Actor neural network and the Critic neural network, and a Disturbance neural network is constructed to add disturbance to the unmanned aerial vehicle to test the flight ability of the unmanned aerial vehicle under interference;
[0008] S4, the target control law and disturbance extreme value are calculated by Critic, and then the weight value of Actor neural network, Critic neural network and Disturbance neural network is updated by back propagation algorithm;
[0009] S5, step S4 is repeated, when the convergence condition is met, the iteration of Actor neural network, Critic neural network and Disturbance neural network is stopped; if the convergence condition is not met, the iteration is continued.
[0010] Further, in step S1, the nonlinear state space expression of the unmanned aerial vehicle is as follows:
[0011] x(k+1)=f(x(k))+g(x(k))u(k)+h(x(k))d(k)
[0012] Wherein, is the state of the unmanned aerial vehicle system, is the system control independent dynamics, is the system control dynamics, is the system control input, is the system state quantity, is the external disturbance input, is the disturbance input matrix; is the real number set;
[0013] The longitudinal nonlinear discrete equation of the unmanned aerial vehicle is as follows:
[0014]
[0015]
[0016] α(k+1)=hq(k)-(γ(k+1)-γ(k))+α(k)
[0017]
[0018] The state variables v(k), γ(k), α(k) and q(k) represent the speed, flight path angle, angle of attack and pitch angle velocity of the unmanned aerial vehicle at time k respectively; h represents the sampling time interval, T represents the engine thrust, D represents the engine drag, L represents the engine lift, M represents the engine pitching moment, I yy is the moment of inertia of the body shaft system around the y axis, g is the acceleration of gravity, and m is the mass of the unmanned aerial vehicle;
[0019] Then the expression of the performance function based on the longitudinal control of the small fixed-wing unmanned aerial vehicle is as follows:
[0020]
[0021] V(k) is the system's performance function at time k, and the performance matrix is... Both are positive definite; λ is the performance function factor, γ d This is the disturbance attenuation factor; Let be the utility function used to evaluate the current flight state of the UAV; thus, the Bellman optimal equation is:
[0022]
[0023] V * (k) represents the performance function under the optimal state at time k;
[0024] Then Hamilton's equation is:
[0025]
[0026] Differentiating both sides of the Hamiltonian equation with respect to the control law u(k) yields the optimal control law:
[0027]
[0028] Similarly, differentiating both sides of the Hamiltonian equation with respect to the perturbation d(k) yields the extreme perturbation:
[0029]
[0030] Furthermore, in step S2, the Actor employs a single-hidden-layer, two-layer neural network to approximate the target control law:
[0031]
[0032] N a The number of neurons in the hidden layer of the Actor neural network. Let be the weights from the j-th input layer node to the i-th hidden layer node in the Actor network. Let σ be the weight from the j-th hidden layer node to the i-th output layer node of the Actor. a Let n be the Actor activation function. a Let x(k) be the number of nodes in the Actor's input layer, and x(k) = [v(k)γ(k)α(k)q(k)] T Status of the unmanned aerial vehicle (UAV) system;
[0033] For Critic, a single-hidden-layer two-layer neural network is used to approximate the performance function, and the Critic output is:
[0034]
[0035] N c The number of neurons in the Critic hidden layer. This represents the weights of the Critic from the j-th input layer node to the i-th hidden layer node. Let σ be the weight from the i-th hidden layer node to the output layer in Critic. c z is the activation function for Critic. c (k) is the input to Critic, n c This represents the number of output layer nodes for Critic.
[0036] Furthermore, in step S3, the Disturbance network is constructed using a neural network with momentum gradients, and the output of Disturbance is:
[0037]
[0038] The activation function σ of the Disturbance network d It is the hyperbolic tangent function;
[0039] The weights between the input layer and the hidden layer are defined as follows:
[0040]
[0041] N d The number of neurons in the Disturbance hidden layer. Let the weights from the j-th node in the input layer to the i-th node in the hidden layer be defined as follows; the weights between the hidden layer and the output layer are defined as follows:
[0042]
[0043] Let be the weight of Disturbance from the i-th hidden layer node to the output layer.
[0044] Furthermore, in step S4, the Critic approximates the performance function, and simultaneously calculates the Bellman equation error and performs backpropagation through a neural network to update the weights.
[0045]
[0046]
[0047]
[0048]
[0049]
[0050]
[0051] η cFor the Critic learning rate, e c (k) represents the TD error of Critic at time k, E c (k) represents the Critic error index at time k, and λ is the performance function decay factor.
[0052] The Actor is used to approximate the target control law, making the performance function optimal.
[0053]
[0054]
[0055]
[0056]
[0057]
[0058]
[0059]
[0060] e a (k) represents the approximation error of the Actor network at time k, E a (k) represents the Actor network error index at time k. For the Actor, use the weights at time k and the output at time k-1, u * (x(k-1)) is the target control law at time k-1.
[0061] Furthermore, Disturbance introduces interference to the drone based on its current state to test its flight and stability capabilities under interference:
[0062]
[0063] N d The number of neurons in the Disturbance hidden layer. Let be the weights from the j-th node in the Disturbance input layer to the i-th node in the hidden layer. Let σ be the weight of the disturbance from the i-th hidden layer node to the output layer. d Let n be the activation function for Disturbance. d Enter the number of layer nodes for Disturbance.
[0064] Furthermore, in step S5, when the convergence condition is met... When the weight updates for Actor, Critic, and Disturbance are stopped, the iteration stops, and ε is a convergence condition constant; if the convergence condition is not met, the iteration continues.
[0065] Compared with the prior art, the significant advantages of this invention are as follows:
[0066] 1. This invention relates to an Actor-Critic small fixed-wing UAV anti-disturbance control algorithm. Compared with traditional control methods, this method integrates the nonlinear approximation capability of neural networks and the learning capability of reinforcement learning, enabling adaptive control of UAV stability in complex external environments. Furthermore, it incorporates a Disturbance neural network to train the UAV's flight capability under disturbances, thereby improving the system's anti-disturbance capability and enhancing its accuracy and speed.
[0067] 2. In the process of designing the control law, this invention continuously collects data on system input, output and disturbance input. It uses this data to design an anti-interference adaptive controller. When the external environment changes, the control law will also adaptively adjust according to the environmental changes. This method makes the UAV more intelligent.
[0068] 3. In constructing the neural network, the present invention adds momentum coefficients to the neural network, which can effectively reduce the jitter and oscillation caused by external disturbances and greatly optimize the system performance of the UAV when subjected to external interference.
[0069] 4. The anti-interference control of this invention is essentially to solve the optimal control problem of discrete systems. The neural network solution process is simple and fast, so it is suitable for aircraft with weak computing power and small storage space. Attached Figure Description
[0070] Figure 1 This diagram illustrates the control block diagram for a small fixed-wing UAV based on reinforcement learning.
[0071] Figure 2 This is a flowchart of the Actor-Critic reinforcement learning controller under perturbation.
[0072] Figure 3 This represents the Critic output results under different attenuation coefficients;
[0073] Figure 4 This represents the output of the drone utility function;
[0074] Figure 5(a) shows the elevator output of Actor;
[0075] Figure 5(b) shows the output result of Actor's throttle valve;
[0076] Figure 6 This indicates the output of Disturbance;
[0077] Figure 7(a) shows a comparison of the control effects of the approach angle;
[0078] Figure 7(b) shows a comparison of the control effects of the flight path angle;
[0079] Figure 7(c) shows a comparison of the speed control effects;
[0080] Figure 7(d) shows a comparison of the control effects of pitch angular velocity. Detailed Implementation
[0081] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0082] To address the shortcomings of traditional methods in terms of anti-interference capability, this invention combines the Actor-Critic framework to design an anti-disturbance adaptive control law. This method not only enables UAVs to fly faster and more stably in complex environments, but also greatly improves the anti-interference capability of UAVs.
[0083] In this embodiment, the small fixed-wing UAV has a mass m = 13.5 kg, a wingspan b = 0.28956 m, and a wing area S = 0.55 m². 2 Moment of inertia I yy = 1.135 kg·m 2 The mean aerodynamic chord length of the wing is c = 0.18994 m, and the engine constant is K. motor =80.
[0084] like Figure 1 As shown, the specific steps of the adaptive control method for small fixed-wing UAVs based on Actor-Critic of this invention are as follows:
[0085] Step 1: Construct a performance function based on the longitudinal dynamics of the small fixed-wing UAV under disturbance, and use the performance function to evaluate the current state of the UAV.
[0086] The longitudinal nonlinear discrete equation of the UAV is:
[0087]
[0088] Wherein, state variables v(k), γ(k), α(k), and q(k) represent the velocity, track angle, angle of attack, and pitch rate at time k, respectively; state variables v(k+1), γ(k+1), α(k+1), and q(k+1) represent the velocity, track angle, angle of attack, and pitch rate at time k+1, respectively; h = 0.01s is the sampling time interval, T is the engine thrust, D is the pull, L is the lift, M is the pitch moment, and I...yy = 1.135 kg·m 2 Let g be the moment of inertia of the machine body's axis system about the y-axis, and g = 9.8 m / s². 2 Let m be the acceleration due to gravity, and m = 13.5 kg be the mass of the drone.
[0089] The formulas for calculating thrust, pull, lift, and pitch moment are as follows:
[0090]
[0091]
[0092]
[0093]
[0094] S = 0.55m 2 For the wing area, ρ = 1.2682 kg / m² 3 S is the density of air. prop =0.2027m 2 Where C is the propeller area, c = 0.18994m, and C is the mean aerodynamic chord of the wing. prop =1.0 is the propeller chord length, K motor =80 is the engine constant, δ T For throttle valve input; C L C D C m These are the lift coefficient, thrust coefficient, and pitching moment coefficient, respectively. The formulas for calculating each coefficient are as follows:
[0095]
[0096]
[0097]
[0098] e = 0.9 is the Oswald efficiency factor, AR = 0.152 is the wing width-to-height ratio, and δ e For elevator input, The lift coefficient at zero angle of attack. The lift coefficient caused by the angle of attack. The lift coefficient caused by the elevator. The drag coefficient generated by frictional resistance, The drag coefficient caused by the elevator. The pitching moment coefficient caused by the angle of attack. The pitch moment coefficient caused by the elevator. This is the pitch moment coefficient during level flight.
[0099] The longitudinal dynamics of the UAV are expressed as follows:
[0100] x(k+1)=f(x(k))+g(x(k))u(k)+h(k)d(k) (2)
[0101] x(k) = [v(k)a(k)γ(k)q(k)] T For system state variables, For the dynamics of the control-independent terms of the system, For the dynamics of the control terms of the system, For the system's control input, For external disturbance input; The disturbance input matrix represents the different effects of external disturbances on the UAV's different states due to the UAV's dynamic characteristics. Given a set of real numbers, the perturbation input matrix h(k) is obtained through simulation experiments in this example. T =[2 1 1 0.5].
[0102] The initial conditions for the UAV are chosen as x0 = [v0, γ0, a0, q0]. T =[75.7399,-0.0615,-0.1788,-9.7399e -5 ] T .
[0103] Based on the design performance function of a small fixed-wing UAV:
[0104]
[0105] U(i) is the utility function at time i, measuring the performance index of the UAV at that time; performance matrix Both are positive definite. The performance matrix Q measures the cost of control input to the UAV system, and R measures the performance cost of the UAV state to the system. The performance function attenuation factor λ = 0.9, and the error attenuation coefficient... Expanding the utility function U(k) yields:
[0106]
[0107] Where Q = diag(0.1,1,1,4); u = [δ e ,δ T ] represents the control law input, δ e δ T These represent the elevator input and the throttle valve input, respectively. u(i) is the control law input at time i, and R = diag(1,2).
[0108] Formula (4) can be written in iterative form to obtain the Bellman equation:
[0109]
[0110] The Bellman optimal equation can be obtained from formula (5):
[0111]
[0112] V * (k) represents the performance function under the optimal state at time k.
[0113] The Hamiltonian equation is defined by the Bellman equation as follows:
[0114]
[0115] The optimal control law can be obtained by differentiating both sides of the Hamiltonian equation (7) with respect to the control law u(k):
[0116]
[0117] Similarly, by differentiating both sides of the Hamiltonian equation (7) with respect to the perturbation d(k), the extreme perturbation can be obtained:
[0118]
[0119] Step 2: Set the parameters and momentum learning parameters in the performance function, and construct the Actor and Critic.
[0120] like Figure 2 As shown, Actor and Critic use neural networks with momentum coefficients, and Actor uses a two-layer neural network with a single hidden layer.
[0121] The Actor output is:
[0122]
[0123] Actor activation function σ a (.) = tanh(.), Actor output is The weight matrix between the input layer and the hidden layer is defined as follows:
[0124]
[0125] n a N represents the number of neurons in the Actor input layer. a The number of neurons in the hidden layer. The weights from the j-th input layer node to the i-th hidden layer node are given.
[0126] Define the weights from the hidden layer to the output layer as follows:
[0127]
[0128] Let be the weights from the j-th hidden layer node to the i-th output layer node, and m be the number of neurons in the Actor output layer.
[0129] For Critic, a two-layer neural network with a single hidden layer is used to approximate the performance function. The Critic output is:
[0130]
[0131] Critic input is z c (k)=[v(k)a(k)γ(k)q(k)d(k)δ e (k)δ T (k)] T Critic activation function σ c (.) = ReLU(.); The weights from the input layer to the hidden layer are:
[0132]
[0133] n c N represents the number of neurons in the Critic input layer. c The number of neurons in the hidden layer, where This represents the weight from the j-th input layer node to the i-th hidden layer node. The weight from the hidden layer to the output layer is defined as:
[0134]
[0135] Let be the weight from the i-th hidden layer node to the output layer.
[0136] In this embodiment, Critic employs a single-hidden-layer, two-layer neural network with 7 input layer nodes, 10 hidden layer nodes, and 1 output layer node, and the activation function σ. c For the ReLU function, ζ is the momentum learning parameter of the Critic. c =0.1, learning rate η c =0.9, performance function decay parameter λ=0.7; Actor uses a single-hidden-layer two-layer neural network, with 4 input layer nodes, 10 hidden layer nodes, and 2 output layer nodes; activation function σ a The hyperbolic tangent function is selected, and the threshold for the UAV control input is set to |δ e |<30deg,|δ T | <100%. Actor momentum learning parameter ζa =0.1, learning rate η a =0.8; While calculating the performance function, Critic calculates the target control law according to the Bellman optimal equation and transmits it to Actor for weight update. At the same time, it calculates the TD error at the current moment based on the system state at the previous moment for Critic's weight update.
[0137] Step 3: Design a disturbance network to introduce disturbances to the drone based on the current state in order to test the drone's flight status under disturbance.
[0138] The Disturbance network is constructed using a neural network with momentum coefficients. The main purpose of the Disturbance network is to output extreme perturbations based on the current state of the UAV in order to train the UAV's flight and stability capabilities under disturbances.
[0139] The output of Disturbance is:
[0140]
[0141] Disturbance activation function σ d It is the hyperbolic tangent function.
[0142] The weights between the Disturbance input layer and the hidden layer are defined as follows:
[0143]
[0144] N d n is the number of neurons in the Disturbance hidden layer. d The number of neurons in the Disturbance input layer. The weights from the j-th node in the input layer to the i-th node in the hidden layer are given for Disturbance.
[0145] Define the weights between the hidden layer and the output layer of the Disturbance network as follows:
[0146]
[0147] Let be the weights from the i-th hidden layer node to the output layer in the Disturbance network.
[0148] In this embodiment, Disturbance uses a single-hidden-layer, two-layer neural network with 4 nodes in the input layer, 10 nodes in the hidden layer, and 1 node in the output layer; the activation function σ dThis is a ReLU function. Critic calculates extreme disturbances based on the system's current state and inputs them into the UAV system to train the UAV's flight stability under disturbances.
[0149] Step four: Calculate the target control law and perturbation extrema using Critic, and then update the neural network weights using the backpropagation algorithm.
[0150] In this embodiment, when the UAV is in flight, the current state of the UAV is first input into the Critic to calculate the performance function. Then, the UAV state stored from the previous moment is used to calculate the TD (Time Difference) error to update the Critic weights; the TD error e c The formula for calculating (k) is as follows:
[0151]
[0152]
[0153] Among them, E c (k) is the error index of Critic;
[0154] The Critic update method uses the gradient descent algorithm:
[0155]
[0156]
[0157] η c >0 represents the Critic learning rate.
[0158] We can obtain the following using the chain rule:
[0159]
[0160]
[0161] Simultaneously, the target control law is calculated based on the Bellman equation:
[0162]
[0163] The Actor is updated based on the target control law calculated by the Critic. Specifically, for the Actor, the Actor error is set as the error between the actual control law and the target control law.
[0164]
[0165] For the Actor, use the weights at time k and the output at time k-1, u *(x(k-1)) is the target control law at time k-1.
[0166] Define the Actor weight error signal as the square of the Actor network error:
[0167]
[0168] The Actor's weights are updated using the gradient descent algorithm:
[0169]
[0170]
[0171] η a Let be the Actor's learning rate.
[0172] The weight update between the input layer and the hidden layer can be obtained using the chain rule:
[0173]
[0174] The weights between the hidden layer and the output layer are updated as follows:
[0175]
[0176] Disturbance outputs extreme value perturbations based on the drone's state to train drone control under perturbation. That is, for Disturbance, the error of the Disturbance network is set as the square of the difference between the neural network output and the calculated target value:
[0177]
[0178]
[0179] The Disturbance network weight update method uses the gradient descent algorithm:
[0180]
[0181]
[0182] Where η d >0 represents the Disturbance learning rate.
[0183] By the chain rule, we can obtain:
[0184]
[0185]
[0186] Considering that small fixed-wing UAVs are highly sensitive to external disturbances during actual operation, if the controller fails to adjust and restore the UAV to a stable state in time when disturbed, stalling or jittering is highly likely. To enhance the control effect of the UAV and reduce the operational deviation caused by disturbances, we consider adding momentum gradients to the Actor-Critic controller for neural network construction. Applying this to the neural network yields its general form of the update iteration equation:
[0187]
[0188] η is the learning rate of the neural network, ζ is the momentum learning gradient coefficient, and Δw(k) is the change in weights at time k. Applying formula (37) to the neural network in the Actor-Critic controller yields:
[0189]
[0190] ζ c ζ a ζ d The momentum gradient coefficients of Critic, Actor, and Disturbance, respectively, η c η a η d The learning rates are for Critic, Actor, and Disturbance, respectively.
[0191] In this embodiment, the weights of the first node in the Critic hidden layer converge to the following data:
[0192]
[0193] The weights of the first node in the Critic output layer converge to the following data:
[0194]
[0195] The first node in the Actor's hidden layer converges to the following data:
[0196]
[0197] The first node of the Actor output layer converges to the following data:
[0198]
[0199] The first node of the Disturbance hidden layer converges to the following data:
[0200]
[0201] The first node of the Disturbance output layer converges to the following data:
[0202]
[0203] Step 5: Repeat step 4 until the neural network meets the convergence condition. The iteration stops when ε is reached, where ε is the convergence condition constant. If the convergence condition is not met, the iteration continues.
[0204] Figure 1 The general framework for reinforcement learning control of small fixed-wing UAVs mainly consists of Actors and Critics. Actors are used to calculate the control law, and Critics are used to calculate the performance function and the target control law. Figure 2 The flowchart of the Actor-Critic reinforcement learning control for a drone under disturbance is shown, briefly illustrating the learning process of the adaptive controller during drone operation. Figure 3 The error curve of Critic shows that the Critic network can quickly approximate the fast learning characteristics reflected in the value function. Figure 4 The curves show the variation of the UAV utility function U(k). Under the regulation of the reinforcement learning controller, the UAV utility function U(k) converges and reaches a stable state in a short time. Figures 5(a) and 5(b) show the elevator δ, respectively. e and throttle valve δ T The output shows that, thanks to the rapid learning of the neural network, the elevator and throttle valve can quickly adjust the drone's stability and converge. Figure 6 The output of Disturbance shows that, under the control of the controller, its error converges rapidly and remains stable within a short time. Figures 7(a)-(d) are comparisons of the control effects of different control methods on the UAV under disturbance. It can be concluded that, compared with PID control and nonlinear dynamic inverse control, the reinforcement learning controller can quickly and stably control the UAV to maintain stability and system performance under disturbance, demonstrating the anti-interference and stability of the momentum-based AC framework.
Claims
1. An adaptive control method for a small fixed-wing unmanned aerial vehicle based on Actor-Critic, characterized in that, Includes the following steps: S1, construct the performance function based on the longitudinal dynamics of the small fixed-wing UAV; S2, construct the Actor network and Critic network, and use a neural network with momentum coefficients to enhance the stability of the UAV controller; S3 constructs information transmission between the Actor neural network and the Critic neural network, and simultaneously constructs a Disturbance neural network to introduce disturbances into the UAV to test the UAV's flight capability under interference. S4, calculate the target control law and disturbance extrema through Critic, and then update the weights of Actor neural network, Critic neural network and Disturbance neural network through backpropagation algorithm; S5. Repeat step S4. When the convergence condition is met, the Actor neural network, Critic neural network, and Disturbance neural network stop iterating; if the convergence condition is not met, the iteration continues. In step S2, the Actor uses a single-hidden-layer, two-layer neural network to approximate the target control law: N a The number of neurons in the hidden layer of the Actor neural network. Let be the weights from the j-th input layer node to the i-th hidden layer node in the Actor network. Let σ be the weight from the j-th hidden layer node to the i-th output layer node of the Actor. a Let n be the Actor activation function. a Let x(k) be the number of nodes in the Actor's input layer, and x(k) = [v(k)γ(k)α(k)q(k)] T Let v(k), γ(k), α(k), and q(k) represent the UAV system state at time k, respectively, where v(k), γ(k), α(k), and q(k) represent the UAV's velocity, track angle, angle of attack, and pitch rate at time k. For Critic, a single-hidden-layer two-layer neural network is used to approximate the performance function, and the Critic output is: N c The number of neurons in the Critic hidden layer. This represents the weights of the Critic from the j-th input layer node to the i-th hidden layer node. Let σ be the weight from the i-th hidden layer node to the output layer in Critic. c z is the activation function for Critic. c (k) is the input to Critic, n c This represents the number of output layer nodes for Critic. In step S3, the Disturbance network is constructed using a neural network with momentum gradient, and the output of Disturbance is: The activation function σ of the Disturbance network d It is the hyperbolic tangent function; The weights between the input layer and the hidden layer are defined as follows: N d The number of neurons in the Disturbance hidden layer. Let the weights from the j-th node in the input layer to the i-th node in the hidden layer be defined as follows: The weights between the hidden layer and the output layer are defined as follows: Let the Disturbance be the weight from the i-th hidden layer node to the output layer; In step S4, the Critic approximates the performance function, and simultaneously calculates the Bellman equation error and performs backpropagation through the neural network to update the weights. η c For the Critic learning rate, e c (k) represents the TD error of Critic at time k, E c (k) represents the Critic error index at time k, and λ is the performance function decay factor. The Actor is used to approximate the target control law, making the performance function optimal. e a (k) represents the approximation error of the Actor network at time k, E a (k) is the Actor network error index at time k. For the Actor, use the weights at time k and the output at time k-1, u * (x(k-1)) is the target control law at time k-1.
2. The adaptive control method for a small fixed-wing UAV based on Actor-Critic as described in claim 1, characterized in that, In step S1, the nonlinear state-space expression of the UAV is as follows: x(k+1)=f(x(k))+g(x(k))u(k)+h(x(k))d(k) in, The status of the unmanned aerial vehicle system. For system control-independent dynamics, For system control dynamics, For system control input, For system state variables, For external disturbance input. The perturbation input matrix; It is the set of real numbers; The longitudinal nonlinear discrete equation of the UAV is: α(k+1)=hq(k)-(γ(k+1)-γ(k))+α(k) State variables v(k), γ(k), α(k), and q(k) represent the UAV's velocity, track angle, angle of attack, and pitch rate at time k, respectively; h represents the sampling time interval; T is the engine thrust; D is the engine pull; L is the engine lift; M is the engine pitch moment; and I... yy Let g be the moment of inertia of the airframe about the y-axis, g be the acceleration due to gravity, and m be the mass of the UAV. The expression for the performance function based on the longitudinal control of a small fixed-wing UAV is as follows: V(k) is the system's performance function at time k, and the performance matrix is... Both are positive definite; λ is the performance function factor, γ d This is the disturbance attenuation factor; This is a utility function used to evaluate the flight status of the drone at the current time; Therefore, the Bellman optimal equation is: V * (k) represents the performance function under the optimal state at time k; Then Hamilton's equation is: Differentiating both sides of the Hamiltonian equation with respect to the control law u(k) yields the optimal control law: Similarly, differentiating both sides of the Hamiltonian equation with respect to the perturbation d(k) yields the extreme perturbation:
3. The adaptive control method for a small fixed-wing UAV based on Actor-Critic as described in claim 1, characterized in that, Disturbance introduces interference to the drone based on its current state to test its flight and stability capabilities under interference. N d The number of neurons in the Disturbance hidden layer. Let be the weights from the j-th node in the Disturbance input layer to the i-th node in the hidden layer. Let σ be the weight of the disturbance from the i-th hidden layer node to the output layer. d Let n be the activation function for Disturbance. d Enter the number of layer nodes for Disturbance.
4. The adaptive control method for a small fixed-wing UAV based on Actor-Critic as described in claim 1, characterized in that, In step S5, when the convergence condition is met... When the weight updates of Actor, Critic, and Disturbance are stopped, ε is a convergence condition constant; if the convergence condition is not met, the iteration continues.