Aircraft high attack angle control method based on reinforcement learning

Abstract

The invention discloses an aircraft high-attack-angle control method based on reinforcement learning, and the method comprises the steps: building an aircraft longitudinal aerodynamic model, designing a double-delay depth deterministic strategy gradient algorithm, carrying out the reinforcement learning training of flight control tasks of four different target aircraft attack angles, and obtaining four reinforcement learning control strategies; respectively determining a target aircraft attack angle range which can be realized by the control strategy, generating sequence data of a fixed time step length under all realizable target aircraft attack angles by adopting the strategy, and storing the sequence data to form a training sample set; and constructing a feed-forward cascade network model, training based on samples in the training sample set, and solidifying parameters to obtain a comprehensive control strategy so as to realize stable control of aircraft flight at any large angle of attack. According to the method, a plurality of different reinforcement learning strategies are combined to obtain a comprehensive control strategy, so that stable control of any target aircraft attack angle in an aircraft steering range can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and flight control, and in particular relates to a high angle of attack control method for an aircraft based on reinforcement learning. Background Art

[0002] When an aircraft flies in the high angle of attack area, the aerodynamic force exhibits strong nonlinear and unsteady characteristics, which makes the deviation characteristics very sensitive to the angle of attack changes, and may even change multiple times within a small angle of attack range. Therefore, how to design a stability augmentation control law to ensure sufficient deviation damping has always been a difficult problem in the field of aircraft design.

[0003] At present, aircraft are still controlled by stability augmentation control systems. The commonly used aircraft stability augmentation system design methods in engineering include: root locus method, pole configuration method, linear quadratic optimal method and model tracking method. Aircraft is a very complex nonlinear system. Linear design methods are mainly applicable to small angles of attack. They can be effectively controlled under normal circumstances, but are not applicable to complex nonlinear areas. Some design methods applicable to nonlinear areas need to rely on accurate aerodynamic modeling. Due to the nonlinear and unsteady characteristics of aircraft high angle of attack dynamics, it is very difficult to accurately establish the model, so there are also limitations.

[0004] In recent years, reinforcement learning algorithms have been continuously developed. Intelligent agents acquire data through interaction with the environment and learn the optimal strategy from it. This data-driven approach is suitable for solving complex nonlinear problems and has autonomous and adaptive characteristics, thus providing a new means for flight control design. However, the current design of fixed-wing aircraft flight control laws based on reinforcement learning is mostly only studied for specific tasks. There is a problem of insufficient generalization ability to achieve variable tasks. For example, it can only recover from complex deviation states to fixed angle of attack and level flight. When the target aircraft angle of attack is changed, stable control of the aircraft at any large angle of attack cannot be achieved. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method for controlling an aircraft at a high angle of attack based on reinforcement learning, so as to solve the problem that the stabilization control method in the prior art is not suitable for the large angle of attack area and is too dependent on the precise aerodynamic model; the present invention combines multiple different reinforcement learning strategies to obtain a comprehensive control strategy, thereby being able to achieve stable control of any target aircraft angle of attack within the aircraft rudder efficiency range.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] The invention provides a method for controlling an aircraft at a high angle of attack based on reinforcement learning, comprising the following steps:

[0008] Step S1: Establish the aircraft longitudinal aerodynamic model, design the double-delayed deep deterministic policy gradient algorithm, conduct reinforcement learning training for the flight control tasks of four different target aircraft angles of attack, and obtain four reinforcement learning control strategies, which are respectively denoted as

[0009] Step S2: Determine the control strategies respectively The target aircraft angle of attack range that can be achieved, using strategies to generate sequence data of fixed time steps under all achievable target aircraft angles of attack, and storing them to form a training sample set;

[0010] Step S3: Construct a feedforward cascade network model, train based on the samples in the training sample set, and solidify the parameters to obtain a comprehensive control strategy π * , in order to achieve stable control of aircraft flight at any large angle of attack.

[0011] Furthermore, the step S1 specifically includes:

[0012] Step S11: Establish the aircraft longitudinal aerodynamic model, the expression is:

[0013]

[0014] Where L is lift, L = qsC L , q represents dynamic pressure, s represents aircraft reference area, C L represents the lift coefficient, obtained through wind tunnel test; D is the drag, D = qcC D , C D represents the drag coefficient, obtained through wind tunnel tests; M is the pitching moment M = qscC M , c represents the average aerodynamic chord length, C M represents the pitch moment coefficient, obtained through wind tunnel experiments; T is thrust; m is the mass of the aircraft, g is the acceleration of gravity, I y is the moment of inertia about the y-axis, V is the aircraft airspeed, α is the aircraft angle of attack, q is the aircraft pitch angular velocity, θ is the aircraft pitch angle, and t is the time step.

[0015] Step S12: Design a double-delayed deep deterministic policy gradient algorithm, including state design, action design, and reward function design;

[0016] Status value t It is expressed as:

[0017] s t =[α t -α target ,θ t -θ target ,q t -q target ]T

[0018] In the formula, α t ,θ t ,q t Respectively represent the current values ​​of the aircraft's angle of attack, pitch angle, and pitch velocity, α target ,θ target ,q target Respectively represent the target values ​​of the aircraft's angle of attack, pitch angle, and pitch velocity, α target =θ target ,q target =0rad / s; the superscript T is the transposition symbol;

[0019] Action value a t It is expressed as:

[0020] a t =[dz t ,dT t ] T

[0021] In the formula, dz t is the deflection angle of the aircraft elevator, which affects the size of L; dT t It is the control data of the aircraft throttle, which affects the thrust T.

[0022] Reward function r state It is expressed as:

[0023]

[0024] In the formula, k is the scaling factor. Adjusting k changes the intersection of the function curve and the horizontal axis, while the vertical threshold remains unchanged. state is the current state. target is the state of the target aircraft; state can be α, θ or q;

[0025] The total reward for each step length r t It is equal to the weighted sum of the rewards of each component in the state value. The maximum value of the total reward for a step is 1 and the minimum value is -1. The expression is as follows:

[0026] r t =w α r α +w θ r θ +w q r q

[0027] In the formula, w α 、w θ 、w q Respectively represent the weights of angle of attack, pitch angle, and pitch angular velocity. The larger the weight, the higher the importance. α、r θ 、r q Represent the reward values ​​of angle of attack, pitch angle, and pitch angular velocity respectively;

[0028] Step S13: In the aircraft longitudinal aerodynamic model, reinforcement learning training is performed with four target aircraft angles of attack to obtain four reinforcement learning strategies, which are respectively recorded as

[0029] Furthermore, the reinforcement learning training step in step S13 specifically includes:

[0030] Step S131: Initialize the aircraft longitudinal aerodynamic model: For the reinforcement learning task of the target aircraft angle of attack, the aircraft is set to an equilibrium state at a random angle of attack in each training process, and the state value s is calculated based on the value of the equilibrium state and the value at the set target aircraft angle of attack. t , where the angle of attack value is equal to the pitch angle value in the equilibrium state, and the pitch angle velocity is 0 rad / s;

[0031] Step S132: Initialize the network parameters of the reinforcement learning algorithm: The double-delayed deep deterministic policy gradient algorithm contains six neural networks, namely: Actor network π φ 、Target Actor Network π φ′ 、Critic Network 1 Target Critic Network 1 Critic Network 2 Target Critic Network 2 When randomly initialized, the network parameter φ=φ′,

[0032] Step S133: The obtained state value s t As input, enter the Actor network π φ , calculate the action value a t ;

[0033] Step S134: Set the action value a t Input into the aircraft longitudinal aerodynamic model to obtain the next step length state s t+1 , and calculate the reward value r t ; will (s t ,a t ,s t+1 ,r t ) is stored in the experience pool as a set of experience data;

[0034] Step S135: randomly extract M groups of data from the experience pool and update the network parameters; where s = [s1, s2, ..., s M ] T,a=[a1,a2,...,a M ] T ,s′=[s′1,s′2,...,s′ M ] T ,r=[r1,r2,...,r M ] T , s is the state sequence of the current step, a is the action sequence of the current step, s′ is the state sequence of the next step, and r is the reward value sequence of the current step;

[0035] Step S136: Repeat the training until the maximum round is reached; save the strategy corresponding to the maximum average reward in the fixed round, which is the structure and parameters of the Actor network; the control strategies of the four target aircraft angles of attack are recorded as

[0036] Furthermore, the step S135 specifically includes:

[0037] Step S1351: For Critic network 1 and Critic Network 2 Update Parameters and

[0038]

[0039] In the formula, π φ′ (s′) represents the input of s′ to the target Actor network π φ′ The action value calculated later, ε is the noise value, N is the normal distribution, is the variance, c is the noise limit, is the action value after adding noise;

[0040]

[0041] In the formula, is the target value, γ is the discount factor;

[0042]

[0043] Step S1352: After d steps, update the Actor network π φ Parameter φ:

[0044]

[0045] In the formula, represents the gradient, J represents the objective function with respect to the parameter φ;

[0046] Step S1353: Target Actor network πφ′ , target critic network Target Critic Network 2 The parameter φ′, To update:

[0047] φ′=τφ+(1-τ)φ′, Where τ is the soft update rate.

[0048] Furthermore, the step S2 specifically includes:

[0049] Step S21: Determine strategies respectively The target aircraft angle of attack range that can be achieved; for the range α 10 and α 20 All initial angles of attack can achieve a target aircraft angle of attack with an error of ≤±1° under the control of the strategy, indicating that the strategy can achieve the control of the target aircraft angle of attack; change the target aircraft angle of attack value and test it, and finally determine the strategy The target aircraft angle of attack range that can be achieved is α 11 to α 12 ;Strategy The target aircraft angle of attack range that can be achieved is α 21 to α 22 ;Strategy The target aircraft angle of attack range that can be achieved is α 31 to α 32 ;Strategy The target aircraft angle of attack range that can be achieved is α 41 to α 42 ; Among them, α 10 is the minimum value of the initial angle of attack, α 20 is the maximum value of the initial angle of attack, α 11 , α 21 , α 31 , α 41 Strategy The minimum angle of attack of the target aircraft can be achieved, α 12 , α 22 , α 32 , α 42 Strategy Able to achieve the maximum angle of attack of the target aircraft;

[0050] Step S22: Generate all achievable target aircraft angle of attack sequence data with fixed time steps using a strategy, and store them to form a training sample set; 10 and α 20 At the initial angle of attack within the range, the recording strategy Implementing α 11 to α 12 , α21 to α 22 , α 31 to α 32 , α 41 to α 42 The sequence data of all target angles of attack within the range at a fixed time step is expressed as follows:

[0051]

[0052] In the formula, T′ represents the total time step of each sequence, s0 is the initial state value, s t is the input state value of the tth time step, a t is the output action value of the tth time step;

[0053] The target aircraft angle of attack is used as the feature of the training sample set. The final input and output features of the training sample set are:

[0054] x i =[α target ;s i ]

[0055] y i =a i

[0056] In the formula, α target is the target angle of attack, s i is the three-dimensional state value of the i-th sample, a i is the two-dimensional action value of the i-th sample, x i is the four-dimensional input feature of the i-th sample, y i is the two-dimensional output feature of the i-th sample.

[0057] Furthermore, the step S3 specifically includes:

[0058] Step S31: construct the input layer, hidden layer, and output layer of the feedforward cascade network model; the number of neurons in the input layer is 4, and the number of neurons in the hidden layer is K layers, and the number of neurons is n1, n2, n3, ..., n respectively. K , K and n are adjustable hyperparameters, and the number of neurons in the output layer is 2;

[0059] Step S32: performing supervised training on the feedforward cascade network model based on the training sample set;

[0060] Step S33: Solidify the feedforward cascade network parameters to obtain the modified feedforward cascade network model as the comprehensive control strategy π * , achieving stable control of aircraft flight at any large angle of attack.

[0061] Furthermore, the step S32 specifically includes:

[0062] Step S321: Each neuron in the feedforward cascade network model processes the input with cross-layer connection through a weighted sum operation and a nonlinear activation function, and generates an output. Assuming that the current layer is the lth layer of the feedforward cascade network, the calculation formula of the neuron is expressed as:

[0063]

[0064] In the formula, a (l) is the activation output of layer l; a (j) is the activation output of the jth layer; f (l) is the activation function of the lth layer, which is the Sigmoid function, the hyperbolic tangent function or the ReLU function; W (l,j) is the weight matrix between the lth layer and the jth layer; b (l) is the bias term of the lth layer; represents the output accumulation of all layers from input layer j=0 to layer l-1;

[0065] Step S322: Calculate the loss value and use the gradient descent method to update the parameters of the feedforward cascade network. The expression is:

[0066]

[0067] In the formula, Loss is the loss function, y i is the output feature of the i-th sample, which is the true value during training, y is the predicted value, and B is the number of samples;

[0068]

[0069] In the formula, η is the learning rate, which is used to control the step size of the update;

[0070] Step S323: Determine the termination condition of the feedforward cascade network training. If the number of verifications ≥ the maximum number of verifications or the number of iterations ≥ the maximum number of iterations or the loss function value ≤ a certain threshold, the training is completed.

[0071] Beneficial effects of the present invention:

[0072] The present invention adopts four reinforcement learning strategies to realize stable control of target aircraft angles of attack in four different ranges. Aiming at the problem that the control strategies for different target aircraft angles of attack vary greatly, the four learned strategies are used to generate control data for all achievable target aircraft angles of attack, and a comprehensive control strategy is obtained through training, thereby realizing flight stable control at any target aircraft angle of attack in a larger range.

[0073] Compared with traditional control methods, the aircraft high angle of attack control strategy designed by the present invention adopts reinforcement learning algorithm, which is more applicable to the area of ​​high angle of attack and does not rely on a sufficiently accurate aerodynamic model. It provides a certain reference for the design of control methods for high angle of attack of future highly maneuverable fighters. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flow chart of the method of the present invention;

[0075] Figure 2 This is a schematic diagram of the principle of reinforcement learning;

[0076] Figure 3 The angle of attack variation diagram for achieving the target aircraft angle of attack of 20° for different initial angles of attack;

[0077] Figure 4 A diagram showing the structure of a feedforward cascade network used in the present invention;

[0078] Figure 5a This is the angle of attack change diagram from the initial angle of attack of 10° to 55°;

[0079] Figure 5b This is the pitch angle change diagram from the initial angle of attack of 10° to 55°;

[0080] Figure 5c This is the change diagram of the elevator rudder deflection angle from the initial angle of attack of 10° to 55°;

[0081] Figure 5d This is the pitch angle velocity change diagram from the initial angle of attack of 10° to 55°;

[0082] Figure 5e This is a diagram showing the changes in throttle and rudder control data from an initial angle of attack of 10° to 55°. DETAILED DESCRIPTION

[0083] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0084] Reference Figure 1 As shown, a high angle of attack control method of an aircraft based on reinforcement learning of the present invention comprises the following steps:

[0085] Step S1: Establish the aircraft longitudinal aerodynamic model, design the double-delayed deep deterministic policy gradient algorithm, conduct reinforcement learning training for the flight control tasks of four different target aircraft angles of attack, and obtain four reinforcement learning control strategies, which are respectively denoted as

[0086] Step S11: Establish the aircraft longitudinal aerodynamic model, the expression is:

[0087]

[0088] Where L is lift, L = qsC L , q represents dynamic pressure, s represents aircraft reference area, C L represents the lift coefficient, obtained through wind tunnel test; D is the drag, D = qcC D , C D represents the drag coefficient, obtained through wind tunnel tests; M is the pitching moment M = qscC M , c represents the average aerodynamic chord length, C M represents the pitch moment coefficient, obtained through wind tunnel experiments; T is thrust; m is the mass of the aircraft, g is the acceleration of gravity, I y is the moment of inertia about the y-axis, V is the aircraft airspeed, α is the aircraft angle of attack, q is the aircraft pitch angular velocity, θ is the aircraft pitch angle, and t is the time step.

[0089] Step S12: Design a double-delayed deep deterministic policy gradient algorithm, including state design, action design, and reward function design;

[0090] Status value t It is expressed as:

[0091] s t =[α t -α target ,θ t -θ target ,q t -q target ] T

[0092] In the formula, α t ,θ t ,q t Respectively represent the current values ​​of the aircraft's angle of attack, pitch angle, and pitch velocity, α target ,θ target ,q target Respectively represent the target values ​​of the aircraft's angle of attack, pitch angle, and pitch velocity, α target =θ target ,q target =0rad / s; the superscript T is the transposition symbol;

[0093] Action value a t It is expressed as:

[0094] a t =[dz t ,dT t ] T

[0095] In the formula, dz tis the deflection angle of the aircraft elevator, which affects the size of L; dT t It is the control data of the aircraft throttle, which affects the thrust T.

[0096] Reward function r state It is expressed as:

[0097]

[0098] In the formula, k is the scaling factor. Adjusting k changes the intersection of the function curve and the horizontal axis, while the vertical threshold remains unchanged. state is the current state. target is the state of the target aircraft; state can be α, θ or q;

[0099] The total reward for each step length r t It is equal to the weighted sum of the rewards of each component in the state value. The maximum value of the total reward for a step is 1 and the minimum value is -1. The expression is as follows:

[0100] r t =w α r α +w θ r θ +w q r q

[0101] In the formula, w α 、w θ 、w q Respectively represent the weights of angle of attack, pitch angle, and pitch angular velocity. The larger the weight, the higher the importance. α 、r θ 、r q Represent the reward values ​​of angle of attack, pitch angle, and pitch angular velocity respectively;

[0102] Step S13: In the aircraft longitudinal aerodynamic model, reinforcement learning training is performed with four target aircraft angles of attack to obtain four reinforcement learning strategies, which are respectively recorded as

[0103] The reinforcement learning training step in step S13 specifically includes:

[0104] Step S131: Initialize the aircraft longitudinal aerodynamic model: For the reinforcement learning task of the target aircraft angle of attack, the aircraft is set to an equilibrium state at a random angle of attack in each training process, and the state value s is calculated based on the value of the equilibrium state and the value at the set target aircraft angle of attack. t , where the angle of attack value is equal to the pitch angle value in the equilibrium state, and the pitch angle velocity is 0 rad / s;

[0105] Step S132: Initialize the network parameters of the reinforcement learning algorithm: The double-delayed deep deterministic policy gradient algorithm contains six neural networks, namely: Actor network π φ 、Target Actor Network π φ′ 、Critic Network 1 Target Critic Network 1 Critic Network 2 Target Critic Network 2 When randomly initialized, the network parameter φ=φ′,

[0106]

[0107] Step S133: The obtained state value s t As input, enter the Actor network π φ , calculate the action value a t ;

[0108] Step S134: Set the action value a t Input into the aircraft longitudinal aerodynamic model to obtain the next step length state s t+1 , and calculate the reward value r t , will (s t ,a t ,s t+1 ,r t ) is stored in the experience pool as a set of experience data. The data collection process reflects the principle of reinforcement learning. Figure 2 As shown;

[0109] Step S135: randomly extract M groups of data from the experience pool and update the network parameters; where s = [s1, s2, ..., s M ] T ,a=[a1,a2,...,a M ] T ,s′=[s′1,s′2,...,s′ M ] T ,r=[r1,r2,...,r M ] T , s is the state sequence of the current step, a is the action sequence of the current step, s′ is the state sequence of the next step, and r is the reward value sequence of the current step;

[0110] Step S136: Repeat the training until the maximum round is reached; save the strategy corresponding to the maximum average reward in the fixed round, which is the structure and parameters of the Actor network; the control strategies of the four target aircraft angles of attack are recorded as

[0111] Specifically, the step S135 includes:

[0112] Step S1351: For Critic network 1 and Critic Network 2 Update Parameters and

[0113]

[0114] In the formula, π φ′ (s′) represents the input of s′ to the target Actor network π φ′ The action value calculated later, ε is the noise value, N is the normal distribution, is the variance, c is the noise limit, is the action value after adding noise;

[0115]

[0116] In the formula, is the target value, γ is the discount factor, which is generally set to 0.99;

[0117]

[0118] Step S1352: After d steps, update the Actor network π φ Parameter φ:

[0119]

[0120] In the formula, represents the gradient, J represents the objective function with respect to the parameter φ;

[0121] Step S1353: Target Actor network π φ′ , target critic network Target Critic Network 2 The parameter φ′, To update:

[0122] φ′=τφ+(1-τ)φ′, Where τ is the soft update rate, which is generally set to 0.005.

[0123] Step S2: Determine the control strategies respectively The target aircraft angle of attack range that can be achieved uses a strategy to generate sequence data of fixed time steps under all achievable target aircraft angles of attack, which are stored to form a training sample set; specifically, it includes:

[0124] Step S21: Determine strategies respectively The target aircraft angle of attack range that can be achieved; for the range α 10 and α 20 For all initial angles of attack, the target aircraft angle of attack with an error of ≤±1° can be achieved under the control of the strategy, indicating that the strategy can achieve the control of the target aircraft angle of attack. When the target aircraft angle of attack is 20°, the control effect of the strategy is as follows: Figure 3 As shown; change the target aircraft angle of attack and test, and finally determine the strategy The target aircraft angle of attack range that can be achieved is α 11 to α 12 ;Strategy The target aircraft angle of attack range that can be achieved is α 21 to α 22 ;Strategy The target aircraft angle of attack range that can be achieved is α 31 to α 32 ;Strategy The target aircraft angle of attack range that can be achieved is α 41 to α 42 ; Among them, α 10 is the minimum value of the initial angle of attack, α 20 is the maximum value of the initial angle of attack, α 11 , α 21 , α 31 , α 41 Strategy The minimum angle of attack of the target aircraft can be achieved, α 12 , α 22 , α 32 , α 42 Strategy Able to achieve the maximum angle of attack of the target aircraft;

[0125] Step S22: Generate all achievable target aircraft angle of attack sequence data with fixed time steps using a strategy, and store them to form a training sample set; 10 and α 20 At the initial angle of attack within the range, the recording strategy Implementing α 11 to α 12 , α 21 to α 22 , α 31 to α 32 , α 41 to α 42 The sequence data of all target angles of attack within the range at a fixed time step is expressed as follows:

[0126] {(s0,a0),(s1,a1),...,(s T′ ,a T′ )}

[0127] In the formula, T′ represents the total time step of each sequence, s0 is the initial state value, s t is the input state value of the tth time step, a t is the output action value of the tth time step;

[0128] The target aircraft angle of attack is used as the feature of the training sample set. The final input and output features of the training sample set are:

[0129] x i =[α target ;s i ]

[0130] y i =a i

[0131] In the formula, α target is the target angle of attack, s i is the three-dimensional state value of the i-th sample, a i is the two-dimensional action value of the i-th sample, x i is the four-dimensional input feature of the i-th sample, y i is the two-dimensional output feature of the i-th sample.

[0132] Step S3: Construct a feedforward cascade network model, train based on the samples in the training sample set, and solidify the parameters to obtain a comprehensive control strategy π * , in order to achieve stable control of the aircraft flight at any large angle of attack; specifically including:

[0133] Step S31: construct the input layer, hidden layer, and output layer of the feedforward cascade network model; the number of neurons in the input layer is 4, and the number of neurons in the hidden layer is K layers, and the number of neurons is n1, n2, n3, ..., n respectively. K , K and n are adjustable hyperparameters, the number of neurons in the output layer is 2, and the specific structure of the feedforward cascade network model finally adopted is as follows Figure 4 As shown;

[0134] Step S32: performing supervised training on the feedforward cascade network model based on the training sample set;

[0135] Step S33: Solidify the feedforward cascade network parameters to obtain the modified feedforward cascade network model as the comprehensive control strategy π * , achieving stable control of aircraft flight at any large angle of attack.

[0136] Wherein, the step S32 specifically includes:

[0137] Step S321: Each neuron in the feedforward cascade network model processes the input with cross-layer connection through a weighted sum operation and a nonlinear activation function, and generates an output. Assuming that the current layer is the lth layer of the feedforward cascade network, the calculation formula of the neuron is expressed as:

[0138]

[0139] In the formula, a (l) is the activation output of layer l; a (j) is the activation output of the jth layer; f (l) is the activation function of the lth layer, which is the Sigmoid function, the hyperbolic tangent function or the ReLU function; W (l,j) is the weight matrix between the lth layer and the jth layer; b (l) is the bias term of the lth layer; represents the output accumulation of all layers from input layer j=0 to layer l-1;

[0140] Step S322: Calculate the loss value and use the gradient descent method to update the parameters of the feedforward cascade network. The expression is:

[0141]

[0142] In the formula, Loss is the loss function, y i is the output feature of the i-th sample, which is the true value during training, y is the predicted value, and B is the number of samples;

[0143]

[0144] In the formula, η is the learning rate, which is used to control the step size of the update;

[0145] Step S323: Determine the termination condition of the feedforward cascade network training. If the number of verifications ≥ the maximum number of verifications or the number of iterations ≥ the maximum number of iterations or the loss function value ≤ a certain threshold, the training is completed.

[0146] To verify the comprehensive control strategy π * The control performance of the system is shown in the following figure. The variables change from the initial angle of attack of 10° to 55° in stages. Figure 5a-Figure 5e As shown; the test shows that for different initial angles of attack, the strategy can respond quickly, pull it up to the set target aircraft angle of attack and keep it stable within 10 seconds of applying control, and maintain consistent flight quality in all stages, with excellent control effect, and finally achieve stable control of aircraft flight at any large angle of attack.

[0147] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims

1. A high angle of attack control method for an aircraft based on reinforcement learning, characterized in that: Here are the steps: Step S1: Establish the aircraft longitudinal aerodynamic model, design the double-delayed deep deterministic policy gradient algorithm, conduct reinforcement learning training for the flight control tasks of four different target aircraft angles of attack, and obtain four reinforcement learning control strategies, which are respectively denoted as Step S2: Determine the control strategies respectively The target aircraft angle of attack range that can be achieved, using strategies to generate sequence data of fixed time steps under all achievable target aircraft angles of attack, and storing them to form a training sample set; Step S3: Construct a feedforward cascade network model, train based on the samples in the training sample set, and solidify the parameters to obtain a comprehensive control strategy π * , in order to achieve stable control of aircraft flight at any large angle of attack.

2. The aircraft high angle of attack control method based on reinforcement learning according to claim 1, characterized in that: The step S1 specifically includes: Step S11: Establish the aircraft longitudinal aerodynamic model, the expression is: Where L is lift, L = qsC L , q represents dynamic pressure, s represents aircraft reference area, C L represents the lift coefficient; D is the drag, D = qcC D , C D represents the drag coefficient; M is the pitching moment M = qscC M , c represents the average aerodynamic chord length, C M represents the pitch moment coefficient; T is the thrust; m is the mass of the aircraft, g is the acceleration of gravity, I y is the moment of inertia about the y-axis, V is the aircraft airspeed, α is the aircraft angle of attack, q is the aircraft pitch velocity, θ is the aircraft pitch angle, and t is the time step; Step S12: Design a double-delayed deep deterministic policy gradient algorithm, including state design, action design, and reward function design; Status value t It is expressed as: s t =[a t -a target ,i t -θ target ,q t -q target ] T In the formula, α t ,θ t ,q t Respectively represent the current values ​​of the aircraft's angle of attack, pitch angle, and pitch velocity, α target ,θ target ,q target Respectively represent the target values ​​of the aircraft's angle of attack, pitch angle, and pitch velocity, α target =θ target ,q target =0rad / s; the superscript T is the transposition symbol; Action value a t It is expressed as: a t =[dz t ,dT t ] T In the formula, dz t is the rudder deflection angle of the aircraft elevator; dT t It is the control data of aircraft throttle; Reward function r state It is expressed as: In the formula, k is the scaling factor. Adjusting k changes the intersection of the function curve and the horizontal axis, while the vertical threshold remains unchanged. state is the current state. target is the state of the target aircraft; state can be α, θ or q; The total reward for each step length r t It is equal to the weighted sum of the rewards of each component in the state value. The maximum value of the total reward for a step is 1 and the minimum value is -1. The expression is as follows: r t =w α r α +w θ r θ +w q r q In the formula, w α 、w θ 、w q Respectively represent the weights of angle of attack, pitch angle, and pitch angular velocity; r α 、r θ 、r q Represent the reward values ​​of angle of attack, pitch angle, and pitch angular velocity respectively; Step S13: In the aircraft longitudinal aerodynamic model, reinforcement learning training is performed with four target aircraft angles of attack to obtain four reinforcement learning strategies, which are respectively recorded as 3. The aircraft high angle of attack control method based on reinforcement learning according to claim 2, characterized in that: The reinforcement learning training step in step S13 specifically includes: Step S131: Initialize the aircraft longitudinal aerodynamic model: For the reinforcement learning task of the target aircraft angle of attack, the aircraft is set to an equilibrium state at a random angle of attack in each training process, and the state value s is calculated based on the value of the equilibrium state and the value at the set target aircraft angle of attack. t , where the angle of attack value is equal to the pitch angle value in the equilibrium state, and the pitch angle velocity is 0 rad / s; Step S132: Initialize the network parameters of the reinforcement learning algorithm: The double-delayed deep deterministic policy gradient algorithm contains six neural networks, namely: Actor network π φ 、Target Actor Network π φ′ 、Critic Network 1 Target Critic Network 1 Critic Network 2 Target Critic Network 2 When randomly initialized, the network parameter φ=φ′, Step S133: The obtained state value s t As input, enter the Actor network π φ , calculate the action value a t ; Step S134: Set the action value a t Input into the aircraft longitudinal aerodynamic model to obtain the next step length state s t+1 , and calculate the reward value r t ; will (s t ,a t ,s t+1 ,r t ) is stored in the experience pool as a set of experience data; Step S135: randomly extract M groups of data from the experience pool and update the network parameters; where s = [s1, s2, ..., s M ] T ,a=[a1,a2,...,a M ] T ,s′=[s′1,s′2,...,s′ M ] T ,r=[r1,r2,...,r M ] T , s is the state sequence of the current step, a is the action sequence of the current step, s′ is the state sequence of the next step, and r is the reward value sequence of the current step; Step S136: Repeat the training until the maximum round is reached; save the strategy corresponding to the maximum average reward in the fixed round, which is the structure and parameters of the Actor network; the control strategies of the four target aircraft angles of attack are recorded as 4. The aircraft high angle of attack control method based on reinforcement learning according to claim 3, characterized in that: The step S135 specifically includes: Step S1351: For Critic network 1 and Critic Network 2 Update Parameters and In the formula, π φ′ (s′) represents the input of s′ to the target Actor network π φ′ The action value calculated later, ε is the noise value, N is the normal distribution, is the variance, c is the noise limit, is the action value after adding noise; In the formula, is the target value, γ is the discount factor; Step S1352: After d steps, update the Actor network π φ Parameter φ: In the formula, represents the gradient, J represents the objective function with respect to the parameter φ; Step S1353: Target Actor network π φ ′, target Critic network Target Critic Network 2 The parameter φ′, To update: Where τ is the soft update rate.

5. The aircraft high angle of attack control method based on reinforcement learning according to claim 1, characterized in that: The step S2 specifically includes: Step S21: Determine strategies respectively The target aircraft angle of attack range that can be achieved; for the range α 10 and α 20 All initial angles of attack can achieve a target aircraft angle of attack with an error of ≤±1° under the control of the strategy, indicating that the strategy can achieve the control of the target aircraft angle of attack; change the target aircraft angle of attack value and test it, and finally determine the strategy The target aircraft angle of attack range that can be achieved is α 11 to α 12 ;Strategy The target aircraft angle of attack range that can be achieved is α 21 to α 22 ;Strategy The target aircraft angle of attack range that can be achieved is α 31 to α 32 ;Strategy The target aircraft angle of attack range that can be achieved is α 41 to α 42 ; Among them, α 10 is the minimum value of the initial angle of attack, α 20 is the maximum value of the initial angle of attack, α 11 , α 21 , α 31 , α 41 Strategy The minimum angle of attack of the target aircraft can be achieved, α 12 , α 22 , α 32 , α 42 Strategy Able to achieve the maximum angle of attack of the target aircraft; Step S22: Generate all achievable target aircraft angle of attack sequence data with fixed time steps using a strategy, and store them to form a training sample set; 10 and α 20 At the initial angle of attack within the range, the recording strategy Implementing α 11 to α 12 , α 21 to α 22 , α 31 to α 32 , α 41 to α 42 The sequence data of all target angles of attack within the range at a fixed time step is expressed as follows: {(s0,a0),(s1,a1),...,(s T′ ,a T′ )} In the formula, T′ represents the total time step of each sequence, s0 is the initial state value, k=1,2,3,4,t=0,1,...,T′,s t is the input state value of the tth time step, a t is the output action value of the tth time step; The target aircraft angle of attack is used as the feature of the training sample set. The final input and output features of the training sample set are: x i =[a target ;s i ] and i =a i In the formula, α target is the target angle of attack, s i is the three-dimensional state value of the i-th sample, a i is the two-dimensional action value of the i-th sample, x i is the four-dimensional input feature of the i-th sample, y i is the two-dimensional output feature of the i-th sample.

6. The aircraft high angle of attack control method based on reinforcement learning according to claim 1, characterized in that: The step S3 specifically includes: Step S31: construct the input layer, hidden layer, and output layer of the feedforward cascade network model; the number of neurons in the input layer is 4, and the number of neurons in the hidden layer is K layers, and the number of neurons is n1, n2, n3, ..., n respectively. K , K and n are adjustable hyperparameters, and the number of neurons in the output layer is 2; Step S32: performing supervised training on the feedforward cascade network model based on the training sample set; Step S33: Solidify the feedforward cascade network parameters to obtain the modified feedforward cascade network model as the comprehensive control strategy π * , achieving stable control of aircraft flight at any large angle of attack.

7. The aircraft high angle of attack control method based on reinforcement learning according to claim 6, characterized in that: The step S32 specifically includes: Step S321: Each neuron in the feedforward cascade network model processes the input with cross-layer connection through a weighted sum operation and a nonlinear activation function, and generates an output. Assuming that the current layer is the lth layer of the feedforward cascade network, the calculation formula of the neuron is expressed as: In the formula, a (l) is the activation output of layer l; a (j) is the activation output of the jth layer; f (l) is the activation function of the lth layer, which is the Sigmoid function, the hyperbolic tangent function or the ReLU function; W (l,j) is the weight matrix between the lth layer and the jth layer; b (l) is the bias term of the lth layer; represents the output accumulation of all layers from input layer j=0 to layer l-1; Step S322: Calculate the loss value and use the gradient descent method to update the parameters of the feedforward cascade network. The expression is: In the formula, Loss is the loss function, y i is the output feature of the i-th sample, which is the true value during training, y is the predicted value, and B is the number of samples; In the formula, η is the learning rate, which is used to control the step size of the update; Step S323: Determine the termination condition of the feedforward cascade network training. If the number of verifications ≥ the maximum number of verifications or the number of iterations ≥ the maximum number of iterations or the loss function value ≤ a certain threshold, the training is completed.

Images