A reinforcement learning-based method for high angle-of-attack control of aircraft

Abstract

This invention discloses a reinforcement learning-based method for high angle-of-attack (AOA) control of aircraft, comprising: establishing a longitudinal aerodynamic model of the aircraft and designing a dual-delay deep deterministic policy gradient algorithm; training the algorithm with reinforcement learning for flight control tasks of four different target aircraft angles of attack to obtain four reinforcement learning control policies; determining the target aircraft angle of attack range that each control policy can achieve; generating sequence data with fixed time steps at all achievable target aircraft angles of attack using the policies, storing the data to form a training sample set; constructing a feedforward cascaded network model; training the model based on samples from the training sample set; and fixing the parameters to obtain a comprehensive control policy to achieve stable control of the aircraft at any high angle of attack. This invention combines multiple different reinforcement learning policies to obtain a comprehensive control policy, thereby enabling stable control of any target aircraft angle of attack within the aircraft's control effect range.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and flight control technology, specifically relating to a high angle-of-attack control method for aircraft based on reinforcement learning. Background Technology

[0002] When an aircraft flies at high angles of attack, its aerodynamics exhibit strong nonlinear and unsteady characteristics, making its deviation characteristics highly sensitive to changes in angle of attack. In fact, deviation characteristics may change multiple times within a small angle of attack range. Therefore, how to design a stability augmentation control law to ensure sufficient deviation damping has always been a difficult problem in the field of aircraft design.

[0003] Currently, aircraft still employ stability augmentation control systems for control. Commonly used aircraft stability augmentation system design methods in engineering include root locus method, pole placement method, linear quadratic optimal method, and model tracking method. Aircraft are highly complex nonlinear systems. Linear design methods are mainly suitable for small angles of attack and can effectively control aircraft under normal conditions, but they are not applicable to complex nonlinear regions. Furthermore, some design methods suitable for nonlinear regions rely on accurate aerodynamic modeling. Due to the nonlinear and unsteady characteristics of aircraft dynamics at large angles of attack, accurate model building is very difficult, thus also presenting limitations.

[0004] In recent years, reinforcement learning algorithms have been continuously developing. Intelligent agents acquire data by interacting with the environment and learn optimal strategies from it. This data-driven approach is suitable for solving complex nonlinear problems and possesses autonomy and adaptive characteristics, thus providing a new means for flight control design. However, current research on fixed-wing aircraft flight control laws based on reinforcement learning is mostly focused on specific tasks. To achieve variable tasks, it suffers from insufficient generalization ability. For example, it can only recover from complex deviation states to level flight at a fixed angle of attack, but it cannot achieve stable control of the aircraft at arbitrarily large angles of attack when the target aircraft's angle of attack changes. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the present invention aims to provide a reinforcement learning-based method for controlling aircraft at high angles of attack, thereby solving the problems that existing stability augmentation control methods are unsuitable for the high angle-of-attack region and rely too heavily on accurate aerodynamic models. The present invention combines multiple different reinforcement learning strategies to obtain a comprehensive control strategy, thereby enabling stable control of the angle of attack of any target aircraft within the aircraft's rudder effect range.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] The present invention provides a high angle-of-attack control method for aircraft based on reinforcement learning, comprising the following steps:

[0008] Step S1: Establish the longitudinal aerodynamic model of the aircraft and design a dual-delay depth deterministic policy gradient algorithm. Perform reinforcement learning training for flight control tasks with four different target aircraft angles of attack, resulting in four reinforcement learning control policies, denoted as follows:

[0009] Step S2: Determine the control strategies respectively The achievable target aircraft angle of attack range is determined by using a strategy to generate sequence data of fixed time steps at all achievable target aircraft angles of attack, which are then stored to form a training sample set.

[0010] Step S3: Construct a feedforward cascaded network model, train it based on samples in the training sample set, and solidify the parameters to obtain a comprehensive control strategy π. * This enables stable control of aircraft flight at any high angle of attack.

[0011] Further, step S1 specifically includes:

[0012] Step S11: Establish the longitudinal aerodynamic model of the aircraft, the expression of which is:

[0013]

[0014] In the formula, L is the lift force, L=qsC L q represents dynamic pressure, s represents the aircraft reference area, and C L D represents the lift coefficient, obtained through wind tunnel testing; D is the drag coefficient, D = qcC. D , C D The drag coefficient is represented by M, obtained through wind tunnel testing; M is the pitching moment, M = qscC. M c represents the mean aerodynamic chord length, C M The pitching moment coefficient is obtained through wind tunnel experiments; T is the thrust; m is the mass of the aircraft; g is the acceleration due to gravity; and I is the thrust coefficient. y Let y be the moment of inertia along the y-axis, V be the airspeed of the aircraft, α be the angle of attack of the aircraft, q be the pitch rate of the aircraft, θ be the pitch angle of the aircraft, and t be the time step.

[0015] Step S12: Design a dual-delay deep deterministic policy gradient algorithm, including state design, action design, and reward function design;

[0016] State value s t Represented as:

[0017] s t =[α t -α target ,θ t -θ target ,q t -q target ]T

[0018] In the formula, α t θ t q t α represents the current values ​​of the aircraft's angle of attack, pitch angle, and pitch rate, respectively. target θ target q target α represents the target values ​​of the aircraft's angle of attack, pitch angle, and pitch rate, respectively. target =θ target q target =0 rad / s; the superscript T is the transpose sign;

[0019] Action value a t Represented as:

[0020] a t =[dz t ,dT t ] T

[0021] In the formula, dz t The deflection angle of the aircraft elevator affects the magnitude of L; dT t The control data for the aircraft throttle affects the magnitude of the thrust T;

[0022] reward function r state Represented as:

[0023]

[0024] In the formula, k is the scaling factor, adjusting k changes the position of the intersection point of the function curve and the horizontal axis, while the vertical threshold remains unchanged; state is the current state. target The target aircraft state; state can be α, θ, or q;

[0025] The total reward r for each step t The total reward is equal to the weighted sum of the rewards of each component in the state value. The maximum total reward for one step is 1, and the minimum total reward is -1. The expression is as follows:

[0026] r t =w α r α +w θ r θ +w q r q

[0027] In the formula, w α w θ w q These represent the weights of angle of attack, pitch angle, and pitch rate, respectively; the larger the weight, the greater the importance. α、r θ 、r q These represent the bonus values ​​for angle of attack, pitch angle, and pitch angular velocity, respectively.

[0028] Step S13: In the longitudinal aerodynamic model of the aircraft, reinforcement learning training is performed with four target aircraft angles of attack to obtain four reinforcement learning strategies, denoted as follows:

[0029] Furthermore, the reinforcement learning training step in step S13 specifically includes:

[0030] Step S131: Initialize the longitudinal aerodynamic model of the aircraft: For the reinforcement learning task of the target aircraft angle of attack, the aircraft is set to an equilibrium state at a random angle of attack in each training process, and the state value s is calculated based on the value of the equilibrium state and the value of the target aircraft angle of attack. t In equilibrium, the angle of attack is equal to the pitch angle, and the pitch velocity is 0 rad / s.

[0031] Step S132: Initialize the network parameters of the reinforcement learning algorithm: The dual-delay deep deterministic policy gradient algorithm contains six neural networks, namely: Actor network π φ Target Actor Network π φ′ Critic Network Target Critic Network 1 Critic Network 2 Target Critic Network II During random initialization, the network parameters φ = φ′.

[0032] Step S133: Obtain the state value s t As input, the Actor network π is input. φ The action value a is calculated. t ;

[0033] Step S134: Set the action value a t Inputting this into the aircraft's longitudinal aerodynamic model yields the next long state s. t+1 And calculate the reward value r. t ; will (s t ,a t ,s t+1 ,r t This data is stored as a set of empirical data in the experience pool.

[0034] Step S135: Randomly select M sets of data from the experience pool and update the network parameters; where s = [s1, s2, ..., s M ] Ta = [a1, a2, ..., a M ] T ,s′=[s′1,s′2,...,s′ M ] T r = [r1, r2, ..., r M ] T s is the state sequence of the current step size, a is the action sequence of the current step size, s′ is the state sequence of the next step size, and r is the reward value sequence of the current step size;

[0035] Step S136: Repeat training until the maximum number of rounds is reached; save the policy corresponding to the maximum average reward in the set fixed number of rounds as the structure and parameters of the Actor network; the control policies for the angle of attack of the four target aircraft are respectively denoted as...

[0036] Further, step S135 specifically includes:

[0037] Step S1351: For Critic network one and Critic Network II Update parameters and

[0038]

[0039] In the formula, π φ′ (s′) represents the input of s′ to the target Actor network π. φ′ The calculated action value, ε is the noise value, and N is a normal distribution. Let c be the variance and c be the noise limit. The motion value after adding noise;

[0040]

[0041] In the formula, γ is the target value, and γ is the discount factor;

[0042]

[0043] Step S1352: After d steps, update the Actor network π. φ Parameter φ:

[0044]

[0045] In the formula, Let J denote the gradient, and J denote the objective function with respect to the parameter φ.

[0046] Step S1353: For the target Actor network πφ′ Target Critic Network Target Critic Network II The parameter φ′, Update:

[0047] φ′=τφ+(1-τ)φ′, In the formula, τ is the soft update rate.

[0048] Further, step S2 specifically includes:

[0049] Step S21: Determine the strategies respectively The achievable target aircraft angle of attack range; for the range in α 10 and α 20 All initial angles of attack, under the control of the strategy, can achieve the target aircraft angle of attack with an error ≤ ±1°, indicating that the strategy can control the target aircraft angle of attack; by changing the target aircraft angle of attack value and testing, the strategy is finally determined. The achievable target aircraft angle of attack range is α. 11 To α 12 ;Strategy The achievable target aircraft angle of attack range is α. 21 To α 22 ;Strategy The achievable target aircraft angle of attack range is α. 31 To α 32 ;Strategy The achievable target aircraft angle of attack range is α. 41 To α 42 ; where α 10 α is the minimum initial angle of attack. 20 α is the maximum value of the initial angle of attack. 11 α 21 α 31 α 41 respectively strategy It can achieve the minimum angle of attack of the target aircraft, α 12 α 22 α 32 α 42 respectively strategy It can achieve the maximum angle of attack of the target aircraft;

[0050] Step S22: Generate sequence data of fixed time steps for all achievable target aircraft angles of attack using a strategy, store them to form a training sample set; at α 10 and α 20 Record strategy at the initial angle of attack within the range Achieve α 11 To α 12 α21 To α 22 α 31 To α 32 α 41 To α 42 The sequence data for all targets within the range at a fixed time step is expressed as follows:

[0051]

[0052] In the formula, T′ represents the total time step of each sequence, and s0 is the initial state value. s t Let a be the input state value at time step t. t This is the output action value at the t-th time step;

[0053] Using the target aircraft's angle of attack as a feature of the training sample set, the final input and output features of the training sample set are:

[0054] x i =[α target ;s i ]

[0055] y i =a i

[0056] In the formula, α target s is the target angle of attack value. i Let a be the three-dimensional state value of the i-th sample. i Let x be the two-dimensional action value of the i-th sample. i Let y be the four-dimensional input feature of the i-th sample. i Let be the two-dimensional output feature of the i-th sample.

[0057] Furthermore, step S3 specifically includes:

[0058] Step S31: Construct the input layer, hidden layer, and output layer of the feedforward cascaded network model; the input layer has 4 neurons, and the hidden layer has K layers with the number of neurons n1, n2, n3, ..., n. K K and n are adjustable hyperparameters, and the number of neurons in the output layer is 2.

[0059] Step S32: Supervised training of the feedforward cascaded network model based on the training sample set;

[0060] Step S33: Solidify the parameters of the feedforward cascade network to obtain the modified feedforward cascade network model as the integrated control strategy π. * This enables stable control of aircraft flight at any high angle of attack.

[0061] Furthermore, step S32 specifically includes:

[0062] Step S321: Each neuron in the feedforward cascade network model processes the input with cross-layer connections through a weighted sum operation and a nonlinear activation function, and produces an output. The current layer is set to l in the feedforward cascade network. The neuron's computation formula is expressed as:

[0063]

[0064] In the formula, a (l) The activation output of layer l; a (j) f is the activation output of the j-th layer; (l) W is the activation function for the l-th layer, which can be the Sigmoid function, the hyperbolic tangent function, or the ReLU function; (l,j) b is the weight matrix between layer l and layer j; (l) This is the bias term for the l-th layer; This represents the sum of the outputs of all layers from the input layer j=0 to the (l-1)th layer;

[0065] Step S322: Calculate the loss value and update the parameters of the feedforward cascaded network using gradient descent. The expression is:

[0066]

[0067] In the formula, Loss is the loss function, y i Let y be the output feature of the i-th sample, which is used as the true value during training, y be the predicted value, and B be the number of samples;

[0068]

[0069] In the formula, η is the learning rate, which is used to control the step size of the update;

[0070] Step S323: Determine the termination condition for training the feedforward cascaded network. If the number of validations is greater than or equal to the maximum number of validations, or the number of iterations is greater than or equal to the maximum number of iterations, or the loss function value is less than or equal to a certain threshold, the training is complete.

[0071] The beneficial effects of this invention are:

[0072] This invention employs four reinforcement learning strategies to achieve stable control of target aircraft angles of attack within four different ranges. To address the issue of significant differences in control strategies for different target aircraft angles of attack, the four learned strategies are used to generate control data for all achievable target aircraft angles of attack, and a comprehensive control strategy is trained to achieve flight stability control of any target aircraft angle of attack over a wider range.

[0073] Compared to traditional control methods, this invention employs a reinforcement learning algorithm to design a high angle-of-attack control strategy for aircraft. This strategy is more applicable to high angle-of-attack regions and does not rely on a sufficiently accurate aerodynamic model, providing a reference for the design of high angle-of-attack control methods for future high-maneuverability fighter jets. Attached Figure Description

[0074] Figure 1 This is a flowchart of the method of the present invention;

[0075] Figure 2 A schematic diagram for reinforcement learning;

[0076] Figure 3 Angle of attack variation diagram to achieve a target aircraft angle of attack of 20° for different initial angles of attack;

[0077] Figure 4 This is a diagram of the feedforward cascaded network structure used in this invention;

[0078] Figure 5a The diagram shows the change in angle of attack from an initial angle of attack of 10° to 55°.

[0079] Figure 5b The diagram shows the pitch angle change from an initial angle of attack of 10° to 55°.

[0080] Figure 5c The diagram shows the change in elevator deflection angle as the elevator is pulled up from an initial angle of attack of 10° to 55°.

[0081] Figure 5d The diagram shows the change in pitch angular velocity from an initial angle of attack of 10° to 55°.

[0082] Figure 5e This is a graph showing the change in throttle control data as the angle of attack increases from an initial angle of attack of 10° to 55°. Detailed Implementation

[0083] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0084] Reference Figure 1 As shown, the present invention provides a high angle-of-attack control method for aircraft based on reinforcement learning, comprising the following steps:

[0085] Step S1: Establish a longitudinal aerodynamic model of the aircraft, design a dual-delay depth deterministic policy gradient algorithm, and perform reinforcement learning training for flight control tasks with four different target aircraft angles of attack, resulting in four reinforcement learning control policies, denoted as follows:

[0086] Step S11: Establish the longitudinal aerodynamic model of the aircraft, the expression of which is:

[0087]

[0088] In the formula, L is the lift force, L=qsC L q represents dynamic pressure, s represents the aircraft reference area, and C L D represents the lift coefficient, obtained through wind tunnel testing; D is the drag coefficient, D = qcC. D , C D The drag coefficient is represented by M, obtained through wind tunnel testing; M is the pitching moment, M = qscC. M c represents the mean aerodynamic chord length, C M The pitching moment coefficient is obtained through wind tunnel experiments; T is the thrust; m is the mass of the aircraft; g is the acceleration due to gravity; and I is the thrust coefficient. y Let y be the moment of inertia along the y-axis, V be the airspeed of the aircraft, α be the angle of attack of the aircraft, q be the pitch rate of the aircraft, θ be the pitch angle of the aircraft, and t be the time step.

[0089] Step S12: Design a dual-delay deep deterministic policy gradient algorithm, including state design, action design, and reward function design;

[0090] State value s t Represented as:

[0091] s t =[α t -α target ,θ t -θ target ,q t -q target ] T

[0092] In the formula, α t θ t q t α represents the current values ​​of the aircraft's angle of attack, pitch angle, and pitch rate, respectively. target θ target q target α represents the target values ​​of the aircraft's angle of attack, pitch angle, and pitch rate, respectively. target =θ target q target =0 rad / s; the superscript T is the transpose sign;

[0093] Action value a t Represented as:

[0094] a t =[dz t ,dT t ] T

[0095] In the formula, dz tThe deflection angle of the aircraft elevator affects the magnitude of L; dT t The control data for the aircraft throttle affects the magnitude of the thrust T;

[0096] reward function r state Represented as:

[0097]

[0098] In the formula, k is the scaling factor, adjusting k changes the position of the intersection point of the function curve and the horizontal axis, while the vertical threshold remains unchanged; state is the current state. target The target aircraft state; state can be α, θ, or q;

[0099] The total reward r for each step t The total reward is equal to the weighted sum of the rewards of each component in the state value. The maximum total reward for one step is 1, and the minimum total reward is -1. The expression is as follows:

[0100] r t =w α r α +w θ r θ +w q r q

[0101] In the formula, w α w θ w q These represent the weights of angle of attack, pitch angle, and pitch rate, respectively; the larger the weight, the greater the importance. α 、r θ 、r q These represent the bonus values ​​for angle of attack, pitch angle, and pitch angular velocity, respectively.

[0102] Step S13: In the longitudinal aerodynamic model of the aircraft, reinforcement learning training is performed with four target aircraft angles of attack to obtain four reinforcement learning strategies, denoted as follows:

[0103] Specifically, the reinforcement learning training step in step S13 includes:

[0104] Step S131: Initialize the longitudinal aerodynamic model of the aircraft: For the reinforcement learning task of the target aircraft angle of attack, the aircraft is set to an equilibrium state at a random angle of attack in each training process, and the state value s is calculated based on the value of the equilibrium state and the value of the target aircraft angle of attack. t In equilibrium, the angle of attack is equal to the pitch angle, and the pitch velocity is 0 rad / s.

[0105] Step S132: Initialize the network parameters of the reinforcement learning algorithm: The dual-delay deep deterministic policy gradient algorithm contains six neural networks, namely: Actor network π φ Target Actor Network π φ′ Critic Network Target Critic Network 1 Critic Network 2 Target Critic Network II During random initialization, the network parameters φ = φ′.

[0106]

[0107] Step S133: Obtain the state value s t As input, the Actor network π is input. φ The action value a is calculated. t ;

[0108] Step S134: Set the action value a t Inputting this into the aircraft's longitudinal aerodynamic model yields the next long state s. t+1 And calculate the reward value r. t , will (s t ,a t ,s t+1 ,r t As a set of experiential data, it is stored in the experience pool. The data collection process embodies the principles of reinforcement learning, specifically as follows: Figure 2 As shown;

[0109] Step S135: Randomly select M sets of data from the experience pool and update the network parameters; where s = [s1, s2, ..., s M ] T a = [a1, a2, ..., a M ] T ,s′=[s′1,s′2,...,s′ M ] T r = [r1, r2, ..., r M ] T s is the state sequence of the current step size, a is the action sequence of the current step size, s′ is the state sequence of the next step size, and r is the reward value sequence of the current step size;

[0110] Step S136: Repeat training until the maximum number of rounds is reached; save the policy corresponding to the maximum average reward in the set fixed number of rounds as the structure and parameters of the Actor network; the control policies for the angle of attack of the four target aircraft are respectively denoted as...

[0111] Specifically, step S135 includes:

[0112] Step S1351: For Critic network one and Critic Network II Update parameters and

[0113]

[0114] In the formula, π φ′ (s′) represents the input of s′ to the target Actor network π. φ′ The calculated action value, ε is the noise value, and N is a normal distribution. Let c be the variance and c be the noise limit. The motion value after adding noise;

[0115]

[0116] In the formula, The target value is γ, which is the discount factor, typically set to 0.99.

[0117]

[0118] Step S1352: After d steps, update the Actor network π. φ Parameter φ:

[0119]

[0120] In the formula, Let J denote the gradient, and J denote the objective function with respect to the parameter φ.

[0121] Step S1353: For the target Actor network π φ′ Target Critic Network Target Critic Network II The parameter φ′, Update:

[0122] φ′=τφ+(1-τ)φ′, In the formula, τ is the soft update rate, which is generally set to 0.005.

[0123] Step S2: Determine the control strategies respectively The achievable target aircraft angle of attack range is determined. A strategy is employed to generate sequence data at fixed time steps for all achievable target aircraft angles of attack, which are then stored to form a training sample set. Specifically, this includes:

[0124] Step S21: Determine the strategies respectively The achievable target aircraft angle of attack range; for the range in α 10 and α 20 All initial angles of attack, under the control of the strategy, can achieve the target aircraft angle of attack with an error ≤ ±1°, indicating that the strategy can control the target aircraft angle of attack. The control effect of the strategy when the target aircraft angle of attack is 20° is as follows: Figure 3 As shown; the target aircraft's angle of attack was changed and tested to ultimately determine the strategy. The achievable target aircraft angle of attack range is α. 11 To α 12 ;Strategy The achievable target aircraft angle of attack range is α. 21 To α 22 ;Strategy The achievable target aircraft angle of attack range is α. 31 To α 32 ;Strategy The achievable target aircraft angle of attack range is α. 41 To α 42 ; where α 10 α is the minimum initial angle of attack. 20 α is the maximum value of the initial angle of attack. 11 α 21 α 31 α 41 respectively strategy It can achieve the minimum angle of attack of the target aircraft, α 12 α 22 α 32 α 42 respectively strategy It can achieve the maximum angle of attack of the target aircraft;

[0125] Step S22: Generate sequence data of fixed time steps for all achievable target aircraft angles of attack using a strategy, store them to form a training sample set; at α 10 and α 20 Record strategy at the initial angle of attack within the range Achieve α 11 To α 12 α 21 To α 22 α 31 To α 32 α 41 To α 42 The sequence data for all targets within the range at a fixed time step is expressed as follows:

[0126] {(s0,a0),(s1,a1),...,(s T′ ,a T′ )}

[0127] In the formula, T′ represents the total time step of each sequence, and s0 is the initial state value. s t Let a be the input state value at time step t. t This is the output action value at the t-th time step;

[0128] Using the target aircraft's angle of attack as a feature of the training sample set, the final input and output features of the training sample set are:

[0129] x i =[α target ;s i ]

[0130] y i =a i

[0131] In the formula, α target s is the target angle of attack value. i Let a be the three-dimensional state value of the i-th sample. i Let x be the two-dimensional action value of the i-th sample. i Let y be the four-dimensional input feature of the i-th sample. i Let be the two-dimensional output feature of the i-th sample.

[0132] Step S3: Construct a feedforward cascaded network model, train it based on samples in the training sample set, and solidify the parameters to obtain a comprehensive control strategy π. * To achieve stable control of aircraft flight at any high angle of attack; specifically including:

[0133] Step S31: Construct the input layer, hidden layer, and output layer of the feedforward cascaded network model; the input layer has 4 neurons, and the hidden layer has K layers with the number of neurons n1, n2, n3, ..., n. K K and n are adjustable hyperparameters, the number of neurons in the output layer is 2, and the specific structure of the final feedforward cascaded network model is as follows: Figure 4 As shown;

[0134] Step S32: Supervised training of the feedforward cascaded network model based on the training sample set;

[0135] Step S33: Solidify the parameters of the feedforward cascade network to obtain the modified feedforward cascade network model as the integrated control strategy π. * This enables stable control of aircraft flight at any high angle of attack.

[0136] Specifically, step S32 includes:

[0137] Step S321: Each neuron in the feedforward cascade network model processes the input with cross-layer connections through a weighted sum operation and a nonlinear activation function, and produces an output. The current layer is set to l in the feedforward cascade network. The neuron's computation formula is expressed as:

[0138]

[0139] In the formula, a (l) The activation output of layer l; a (j) f is the activation output of the j-th layer; (l) W is the activation function for the l-th layer, which can be the Sigmoid function, the hyperbolic tangent function, or the ReLU function; (l,j) b is the weight matrix between layer l and layer j; (l) This is the bias term for the l-th layer; This represents the sum of the outputs of all layers from the input layer j=0 to the (l-1)th layer;

[0140] Step S322: Calculate the loss value and update the parameters of the feedforward cascaded network using gradient descent. The expression is:

[0141]

[0142] In the formula, Loss is the loss function, y i Let y be the output feature of the i-th sample, which is used as the true value during training, y be the predicted value, and B be the number of samples;

[0143]

[0144] In the formula, η is the learning rate, which is used to control the step size of the update;

[0145] Step S323: Determine the termination condition for training the feedforward cascaded network. If the number of validations is greater than or equal to the maximum number of validations, or the number of iterations is greater than or equal to the maximum number of iterations, or the loss function value is less than or equal to a certain threshold, the training is complete.

[0146] To verify the integrated control strategy π * The control performance, the changes of various variables from the initial angle of attack of 10° to 55° in stages, are as follows: Figures 5a-5e As shown in the figure; tests have shown that this strategy can respond quickly to different initial angles of attack, pulling the aircraft to the set target angle of attack within 10 seconds of applying control and maintaining stability, while maintaining consistent flight quality at each stage, demonstrating excellent control performance, and ultimately achieving stable control of the aircraft flight at any large angle of attack.

[0147] This invention has many specific applications. The above description is only a preferred embodiment of this invention. It should be noted that for those skilled in the art, several improvements can be made without departing from the principle of this invention, and these improvements should also be considered within the scope of protection of this invention.

Claims

1. A high angle-of-attack control method for aircraft based on reinforcement learning, characterized in that, Here are the steps: Step S1: Establish the longitudinal aerodynamic model of the aircraft and design a dual-delay depth deterministic policy gradient algorithm. Perform reinforcement learning training for flight control tasks with four different target aircraft angles of attack, resulting in four reinforcement learning control policies, denoted as follows: Step S2: Determine the control strategies respectively The achievable target aircraft angle of attack range is determined by using a strategy to generate sequence data of fixed time steps at all achievable target aircraft angles of attack, which are then stored to form a training sample set. Step S3: Construct a feedforward cascaded network model, train it based on samples in the training sample set, and solidify the parameters to obtain a comprehensive control strategy π. * To achieve stable control of aircraft flight at any high angle of attack; Step S3 specifically includes: Step S31: Construct the input layer, hidden layer, and output layer of the feedforward cascaded network model; the input layer has 4 neurons, and the hidden layer has K layers with the number of neurons n1, n2, n3, ..., n. K K and n are adjustable hyperparameters, and the number of neurons in the output layer is 2. Step S32: Supervised training of the feedforward cascaded network model based on the training sample set; Step S33: Solidify the parameters of the feedforward cascade network to obtain the modified feedforward cascade network model as the integrated control strategy π. * This enables stable control of aircraft flight at any high angle of attack.

2. The reinforcement learning-based high angle-of-attack control method for aircraft according to claim 1, characterized in that, Step S1 specifically includes: Step S11: Establish the longitudinal aerodynamic model of the aircraft, the expression of which is: In the formula, L is the lift force, L=qsC L q represents dynamic pressure, s represents the aircraft reference area, and C L D represents the lift coefficient; D is the drag coefficient, D = qcC D C D M represents the drag coefficient; M is the pitching moment M = qscC M c represents the mean aerodynamic chord length, C M The pitch moment coefficient is represented by T; thrust is m; mass of the aircraft is g; and acceleration due to gravity is I. y Let y be the moment of inertia along the y-axis, V be the airspeed of the aircraft, α be the angle of attack of the aircraft, q be the pitch rate of the aircraft, θ be the pitch angle of the aircraft, and t be the time step. Step S12: Design a dual-delay deep deterministic policy gradient algorithm, including state design, action design, and reward function design; State value s t Represented as: s t =[a t -a target ,i t -θ target ,q t -q target ] T In the formula, α t θ t q t α represents the current values ​​of the aircraft's angle of attack, pitch angle, and pitch rate, respectively. target θ target q target α represents the target values ​​of the aircraft's angle of attack, pitch angle, and pitch rate, respectively. target =θ target q target = 0 rad / s; the superscript T is the transpose symbol; Action value a t Expressed as: a t =[dz t ,dT t ] T In the formula, dz t The deflection angle of the aircraft elevator; dT t This refers to the control data for the aircraft throttle. reward function r state Expressed as: In the formula, k is the scaling factor, adjusting k changes the position of the intersection point of the function curve and the horizontal axis, while the vertical threshold remains unchanged; state is the current state. target The target aircraft state; state can be α, θ, or q; The total reward r for each step t The total reward is equal to the weighted sum of the rewards of each component in the state value. The maximum total reward for one step is 1, and the minimum total reward is -1. The expression is as follows: r t =w α r α +w θ r θ +w q r q In the formula, w α 、w θ 、w q These represent the weights of the angle of attack, pitch angle, and pitch rate, respectively; r α r θ r q These represent the bonus values ​​for angle of attack, pitch angle, and pitch angular velocity, respectively. Step S13: In the longitudinal aerodynamic model of the aircraft, reinforcement learning training is performed with four target aircraft angles of attack to obtain four reinforcement learning strategies, denoted as follows:

3. The reinforcement learning-based high angle-of-attack control method for aircraft according to claim 2, characterized in that, The reinforcement learning training steps in step S13 specifically include: Step S131: Initialize the longitudinal aerodynamic model of the aircraft: For the reinforcement learning task of the target aircraft angle of attack, the aircraft is set to an equilibrium state at a random angle of attack in each training process, and the state value s is calculated based on the value of the equilibrium state and the value of the target aircraft angle of attack. t In equilibrium, the angle of attack is equal to the pitch angle, and the pitch velocity is 0 rad / s. Step S132: Initialize the network parameters of the reinforcement learning algorithm: The dual-delay deep deterministic policy gradient algorithm contains six neural networks, namely: Actor network Target Actor Network Critic Network Target Critic Network 1 Critic Network 2 Target Critic Network II During random initialization, network parameters = , Step S133: Obtain the state value s t As input, the Actor network is input. The action value a is calculated. t ; Step S134: Set the action value a t Inputting this into the aircraft's longitudinal aerodynamic model yields the next long state s. t+1 And calculate the reward value r. t ; will (s t ,a t ,s t+1 ,r t This data is stored as a set of empirical data in the experience pool. Step S135: Randomly select M sets of data from the experience pool and update the network parameters; where s = [s1, s2, ..., s M ] T a = [a1, a2, ..., a M ] T ,s′=[s′1,s′2,...,s′ M ] T r = [r1, r2, ..., r M ] T s is the state sequence of the current step size, a is the action sequence of the current step size, s′ is the state sequence of the next step size, and r is the reward value sequence of the current step size; Step S136: Repeat training until the maximum number of rounds is reached; save the policy corresponding to the maximum average reward in the set fixed number of rounds as the structure and parameters of the Actor network; the control policies for the angle of attack of the four target aircraft are respectively denoted as...

4. The reinforcement learning-based high angle-of-attack control method for aircraft according to claim 3, characterized in that, Step S135 specifically includes: Step S1351: For Critic network one and Critic Network II Update parameters and Where, This indicates that s′ is input into the target Actor network. The calculated action value, ε is the noise value, and N is a normal distribution. Let c be the variance and c be the noise limit. The motion value after adding noise; Where, γ is the target value, and γ is the discount factor; Step S1352: After d steps, update the Actor network. parameters : In the formula, ▽ represents the gradient, and J represents the gradient with respect to the parameters. The objective function; Step S1353: For the target Actor network Target Critic Network Target Critic Network II parameters , Update: =t +(1-t) , In the formula, τ is the soft update rate.

5. The reinforcement learning-based high angle-of-attack control method for aircraft according to claim 1, characterized in that, Step S2 specifically includes: Step S21: Determine the strategies respectively The achievable target aircraft angle of attack range; for the range in α 10 and α 20 All initial angles of attack, under the control of the strategy, can achieve the target aircraft angle of attack with an error ≤ ±1°, indicating that the strategy can control the target aircraft angle of attack; by changing the target aircraft angle of attack value and testing, the strategy is finally determined. The achievable target aircraft angle of attack range is α. 11 To α 12 ;Strategy The achievable target aircraft angle of attack range is α. 21 To α 22 ;Strategy The achievable target aircraft angle of attack range is α. 31 To α 32 ;Strategy The achievable target aircraft angle of attack range is α. 41 To α 42 ; where α 10 α is the minimum initial angle of attack. 20 α is the maximum value of the initial angle of attack. 11 , α 21 , α 31 , α 41 respectively strategy It can achieve the minimum angle of attack of the target aircraft, α 12 , α 22 , α 32 , α 42 respectively strategy It can achieve the maximum angle of attack of the target aircraft; Step S22: Generate sequence data of fixed time steps for all achievable target aircraft angles of attack using a strategy, store them to form a training sample set; at α 10 and α 20 Record strategy at the initial angle of attack within the range Achieve α 11 To α 12 , α 21 To α 22 , α 31 To α 32 , α 41 To α 42 The sequence data for all targets within the range at a fixed time step is expressed as follows: {(s0,a0),(s1,a1),...,(s T′ ,a T′ )} In the formula, T′ represents the total time step of each sequence, and s0 is the initial state value. s t Let a be the input state value at time step t. t This is the output action value at the t-th time step; Using the target aircraft's angle of attack as a feature of the training sample set, the final input and output features of the training sample set are: x i =[a target ;s i ] and i =a i In the formula, α target s is the target angle of attack value. i Let a be the three-dimensional state value of the i-th sample. i Let x be the two-dimensional action value of the i-th sample. i Let y be the four-dimensional input feature of the i-th sample. i Let be the two-dimensional output feature of the i-th sample.

6. The reinforcement learning-based high angle-of-attack control method for aircraft according to claim 5, characterized in that, Step S32 specifically includes: Step S321: Each neuron in the feedforward cascade network model processes the input with cross-layer connections through a weighted sum operation and a nonlinear activation function, and produces an output. The current layer is set to l in the feedforward cascade network. The neuron's computation formula is expressed as: In the formula, a (l) The activation output of layer l; a (j) f is the activation output of the j-th layer; (l) W is the activation function for the l-th layer, which can be the Sigmoid function, the hyperbolic tangent function, or the ReLU function. (l,j) b is the weight matrix between layer l and layer j; (l) This is the bias term for the l-th layer; This represents the sum of the outputs of all layers from the input layer j=0 to the (l-1)th layer; Step S322: Calculate the loss value and update the parameters of the feedforward cascaded network using gradient descent. The expression is: In the formula, Loss is the loss function, y i Let y be the output feature of the i-th sample, which is used as the true value during training, y be the predicted value, and B be the number of samples; In the formula, η is the learning rate, which is used to control the step size of the update; Step S323: Determine the termination condition for training the feedforward cascaded network. If the number of validations is greater than or equal to the maximum number of validations, or the number of iterations is greater than or equal to the maximum number of iterations, or the loss function value is less than or equal to a certain threshold, the training is complete.

Images