Aircraft attitude control method based on interpretable reinforcement learning

Abstract

The invention belongs to the technical field of reinforcement learning and aircraft attitude control, and particularly relates to an aircraft attitude control method based on explainable reinforcement learning, which comprises the following steps: firstly, designing an aircraft attitude controller with strong robustness, introducing a multi-channel control network and combining a reinforcement learning algorithm to control the attitude of an aircraft; the controller can effectively cope with challenges brought by external interference and system nonlinearity, and the adaptability and robustness of an aircraft control system are remarkably improved. Secondly, the invention provides an interpretability method combined with visual analysis. The t-SNE algorithm is used for carrying out dimensionality reduction visualization on the input characteristics of the aircraft control system, the SHAP value is combined for carrying out attribution analysis on the input characteristics, key factors in the model decision making process can be revealed, and understanding how the aircraft control system makes decisions according to different input characteristics is helped. According to the invention, high adaptability and transparency of the aircraft control system in a complex environment are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of reinforcement learning and aircraft attitude control, and specifically relates to an aircraft attitude control method based on explainable reinforcement learning. Background Art

[0002] With the continuous development of aircraft technology, especially the increasing demand for online control in complex environments, how to ensure that aircraft can achieve stable performance in dynamically changing environments has become a major challenge in the aerospace field. Although traditional aircraft control methods can ensure the stability of aircraft to a certain extent, they often fail to provide sufficient adaptability and flexibility when facing online environmental changes, real-time task adjustments, and sudden interference. In addition, aircraft control algorithms often rely on a large amount of offline data for training, resulting in weak generalization capabilities of the algorithms when facing new environments or unseen situations. Therefore, improving the online learning ability and interpretability of aircraft control algorithms has become a key issue that needs to be urgently addressed in the field of aircraft intelligent control. Summary of the invention

[0003] The purpose of the present invention is to provide an aircraft attitude control method based on interpretable reinforcement learning, specifically involving the use of a training method based on a Soft Actor-Critic (SAC) reinforcement learning method to achieve flight stability control, and interpretable analysis based on a visualization method; in response to the online learning and interpretability requirements of aircraft control, a reinforcement learning algorithm is introduced, and the control strategy is optimized in combination with real-time data. At the same time, the interpretability of the model is improved through t-SNE visualization and SHAP value analysis, thereby achieving high adaptability and transparency of the aircraft control system in a complex environment.

[0004] The technical solution adopted by the present invention is as follows:

[0005] The aircraft attitude control method based on interpretable reinforcement learning includes the following steps:

[0006] Step 1: Establish a Markov decision model for aircraft attitude control and design an aircraft offline attitude controller;

[0007] Preferably, in step 1, considering the flight characteristics of the aircraft in a large airspace and a wide speed range, based on the requirements for stable control of the aircraft under the influence of complex external interference, a Markov model including a state set, an action set, a reward function, a state transition probability and a discount factor is established as follows;

[0008] The state set: The three-channel attitude error, attitude angle, and attitude angular velocity obtained by the aircraft using the sensor are taken as the state, and the state set S is defined as: S = {e1, Θ, ω}; where Θ = [α, β, σ] Tis the three-channel attitude angle, where α is the angle of attack, β is the sideslip angle, σ is the roll angle, and ω=[p,q,r] T They are the angular velocities of the three channels: roll, pitch and yaw, e1 = Θ - Θ d is the vehicle attitude tracking error;

[0009] The action set: Considering the stable tracking control requirements of the aircraft, based on the aircraft attitude control model, the action set includes the three-direction control rudder deviation of the aircraft, thereby controlling the network action set A c Defined as: in δ ψ ,δ γ They are the control rudder deflections in pitch, yaw and roll directions respectively;

[0010] The reward function establishes a reward and punishment mechanism based on the requirements of ensuring the steady-state performance and dynamic performance of the aircraft, including:

[0011] The error between the current pose and the desired pose: Among them, e1i represents the attitude tracking error in the three directions of roll, pitch and yaw;

[0012] Rudder Smoothing Bonus:

[0013] Where X is the input three-dimensional array with n samples and m dimensions, and R smooth The smaller the value, the smoother the data. Based on this constraint, the continuity of the rudder deviation change is guaranteed, which can better meet the actual engineering needs.

[0014] Dynamic performance: The overshoot and adjustment time during the operation of the aircraft are mainly considered as dynamic performance indicators, as shown in the following formula: In the formula, the overshoot in three directions is defined, α d ,β d ,σ d is the expected attitude in three directions, and the overshoot in three directions is averaged to get the total overshoot: The adjustment time is defined as the time it takes for a sequence to first enter the 2% error band and remain there, as shown in the following equation: Define the adjustment time in three directions, take the average of the adjustment time in three directions, and get the total adjustment time: The dynamic performance reward is defined as:

[0015] R D =-0.05·OS avg -0.1T set ; Combining the above three indicators, the single-step decision reward value R of the aircraft is:

[0016] R=R e +R smooth +R D ;

[0017] The state transition probability: For the aircraft attitude control problem, the aircraft attitude control model is used as the state transition function; that is, the probability of reaching the next state S' after executing action A under the environment state S;

[0018] Discount factor: The discount factor γ indicates the importance of future benefits relative to current benefits. When γ = 0, it is equivalent to only considering current benefits without considering future benefits. When γ = 1, future benefits and current benefits are considered equally important.

[0019] By defining the Markov model of aircraft control, the aircraft control process can be re-described as follows: for the state set s(t) consisting of the aircraft attitude angle and angular velocity obtained by the sensor at time t, according to the designed immediate benefit value R i (t), determine the optimal action A under the premise of maximum benefit c (t), and interact with the environment, and obtain the aircraft state s(t+1) at the next moment by the state transfer function, and repeat the process until the desired stable tracking control goal is achieved;

[0020] Preferably, in step 1, under the established aircraft control network framework, the network back propagation algorithm is studied, and aircraft control training is performed in the constructed simulation training environment to achieve stable tracking control of aircraft multi-channel attitude without decoupling; during the training process, the adopted SAC algorithm has two policy networks and four value networks, the policy network is represented by π, and the value network is described by φ; the value network (only runs in the training stage) uses the aircraft state and action information to obtain the evaluation value of the action taken on the current state;

[0021] Preferably, in step 1, the strategy is guided and updated during the training phase; a complete training process of the SAC algorithm includes a data collection process and a parameter training process:

[0022] Data collection process: The aircraft observes the environment to obtain the current state S, sends it to the policy network to output the corresponding operation A, and obtains the reward value R and the next state S′; after executing the above operations, each iteration round will store the data in the following form: (S, A, R, S′);

[0023] This process requires a large number of rounds to obtain enough experience vectors, all of which are stored in the experience pool. This stage does not involve the value evaluation network, but only includes the interaction between the policy network and the environment. In order to avoid the negative impact of policy relevance on the training effect, an online policy sampling and update mechanism is introduced. By gradually updating the strategy in the process of interaction between the policy network and the environment, and dynamically adjusting the policy network parameters, the data of each collection round is independent and diverse. This policy update method not only improves the coverage and representativeness of the data in the experience pool, but also enhances the robustness and generalization ability of model training;

[0024] Parameter training process: The core idea of ​​the SAC algorithm is to expand the traditional expected cumulative reward maximization objective into an entropy regularization objective by introducing a policy entropy term:

[0025]

[0026] In the formula, r(s t ,a t ) is the immediate reward, E π [] represents the expected value under strategy π, is the entropy of the strategy, which measures the randomness of the strategy, α is the entropy temperature, a hyperparameter that controls the importance of entropy, and E at-π [] represents the expected probability distribution of action a according to strategy π, which measures the probability of strategy π in state s t Through this objective function, SAC's strategy can not only effectively utilize the reward signal in the environment, but also maintain enough randomness to explore the possible optimal strategy in complex environments.

[0027] During the value network training process, the SAC algorithm minimizes the loss function L Q (θ i ) to update the network parameters:

[0028]

[0029] In the formula, represents the Q value estimate given by the current value network, θ i Represents the parameters of the current network, It represents sampling state, action, reward value, and next moment state from the experience replay pool. The target value y is generated by the target network and is defined as follows:

[0030]

[0031] Where γ is the discount factor, a t+1 ′ is the action sampled from the policy network, is the smaller Q value at the future moment;

[0032] The innovation of the SAC algorithm lies in the use of two value networks at the same time to reduce the over-estimation problem of Q value, and the introduction of entropy regularization to balance exploration and utilization, thereby improving the robustness of the strategy; in the training of the policy network, the SAC algorithm updates the policy network parameters by minimizing the following loss function:

[0033]

[0034] Where: D is the sampling space of the state, E[g] represents the mean square error function;

[0035] In addition, the entropy temperature parameter α is updated; the loss function for optimizing the temperature parameter α is defined as:

[0036]

[0037] In the formula, Indicates sampling the action distribution and calculating the expected value of the sampled data, -logπ φ (a t |s t ) indicates action a t The negative of the log probability in the policy distribution, which is used to measure the uncertainty of the policy, is the target entropy;

[0038] Since the update of the value network requires the current Q value and the Q′ value at the next moment, so as to make the training data relevant and improve the network convergence speed, the target strategy network and target value network are designed, and the parameters are regularly copied through the soft update method. The update method is:

[0039] θ′←τθ+(1-τ)θ′

[0040]

[0041] Where θ′ is the parameter of the target policy network, θ is the policy network parameter, are the parameters of the two target value networks, are the parameters of the two value networks respectively; τ is the soft update coefficient.

[0042] Step 2: Based on the trained Markov decision model, a visualization analysis is performed based on the t-SNE algorithm to achieve an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller;

[0043] Preferably, in step 2, a t-SNE visualization analysis of the network output is performed: when the model is trained, the output of the model is visualized, and the output of the last hidden layer of the evaluation network is extracted. Each data sample generates a corresponding high-dimensional feature vector at each time step, and these vectors represent the model's internal representation of the input data at that time step; since the dimensions of these vectors are usually high, it is very difficult to directly visualize these eigenvalues; therefore, we need to reduce the dimensions of these high-dimensional feature vectors to two-dimensional space for subsequent visualization analysis; for this purpose, t-SNE is used to map these high-dimensional features to two-dimensional space, maintaining the relative distance between similar data points while separating dissimilar data points; then the t-SNE algorithm is used to reduce the dimensions to two-dimensional space, complete the visualization analysis, and achieve an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller.

[0044] Step 3: Based on the SHAP value feature analysis, the influence of multiple hidden layers in the aircraft reinforcement learning controller on the network output is sorted out to achieve the interpretability analysis of the aircraft neural network model;

[0045] Preferably, in step 3, the process of calculating the SHAP value can be regarded as traversing all possible feature combinations to evaluate the marginal contribution of the features to the model output; in the calculation process, it is necessary to consider the changes in the model output when the features are added in different orders, so as to obtain the average marginal contribution of the features and finally obtain the SHAP value of the feature; the specific calculation formula is:

[0046]

[0047] In the formula, φ i is the SHAP value of feature i, S is the feature subset, N is the set of all features, |S| is the number of features in subset S, |N| is the number of features in the full feature set, |S|! (|N|-|S|-1)! is the product of the number of permutations of subset S and the number of permutations of the remaining features, which is used to calculate the combined weights of feature i at different positions, |N! is the number of permutations of the full feature set, which is used to normalize the weights, f(S∪{i}) represents the output value of the model when using subset S and feature i for prediction, and f(S) represents the model prediction value corresponding to the feature subset; by summing all possible feature combinations, it ensures that the contribution of the features is fully considered.

[0048] The technical effects achieved by the present invention are:

[0049] The present invention specifically involves the use of a training method based on the Soft Actor-Critic (SAC) reinforcement learning method to achieve flight stability control, and interpretable analysis based on a visualization method; in response to the online learning and interpretability requirements of aircraft control, a reinforcement learning algorithm is introduced, and the control strategy is optimized in combination with real-time data. At the same time, the interpretability of the model is improved through t-SNE visualization and SHAP value analysis, thereby achieving high adaptability and transparency of the aircraft control system in a complex environment.

[0050] In the present invention, considering the external interference and model uncertainty problems faced by aircraft in complex environments, the present invention aims to propose an interpretable aircraft intelligent robust controller design method to cope with the limitations of traditional control methods in practical applications. Specifically, firstly, a robust aircraft attitude controller is designed to address the attitude control problem of aircraft in complex environments. By introducing a multi-channel control network and combining a reinforcement learning algorithm, the controller can effectively cope with the challenges brought by external interference and system nonlinearity, and significantly improve the adaptability and robustness of the aircraft control system. Secondly, based on the t-SNE (t-Distributed Stochastic Neighbor Embedding) algorithm, the high-dimensional data of the aircraft control network is mapped to a two-dimensional space. By analyzing the distribution of different states or actions in the low-dimensional space, the change trend or potential structural characteristics of the strategy are determined; finally, based on the SHAP (SHAPley Additive Explanations) value, an effective method is provided to analyze and explain the prediction characteristics of the model, and the importance of the network characteristics of different layers of the aircraft reinforcement learning controller in the decision-making process is quantified, thereby realizing the interpretability of the model from a more intuitive perspective.

[0051] In the present invention, a robust control strategy based on reinforcement learning is designed, considering the external interference and model uncertainty problems of the aircraft in a complex operating environment. A reward function including the three-channel attitude tracking error of the aircraft is established, and the engineering needs of improving the dynamic performance of the operation process are considered. Dynamic performance indicators such as attitude control overshoot are further added to the reward function, thereby realizing steady-state tracking and dynamic performance optimization of the aircraft in a complex environment.

[0052] In the present invention, considering the uninterpretability of reinforcement learning algorithms in aircraft control, an interpretable analysis framework of an aircraft reinforcement learning controller based on the t-SNE algorithm is designed. By extracting the output of the last hidden layer of the evaluation network of the aircraft reinforcement learning controller, a visual representation of the changing trend or potential structural characteristics of the control strategy is achieved.

[0053] In the present invention, further, a method for interpreting aircraft reinforcement learning network relationships based on SHAP value analysis is designed to quantify the importance of different layer network features of the aircraft reinforcement learning controller in the decision-making and control process, intuitively show which features contribute the most to the model output, reveal the transparency and logic of the control decision, and thus improve the credibility of the aircraft control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a diagram of the architecture of the robust aircraft control algorithm of the present invention;

[0055] Figure 2 is the control network reward function curve in step 1 of the present invention;

[0056] Figure 3 is the completion rate of every 50 rounds in step 1 of the present invention;

[0057] Figure 4 is the number of steps per round in step 1 of the present invention;

[0058] Figure 5 is the posture error curve in step 1 of the present invention and an enlarged view of the corresponding area;

[0059] Figure 6 is the attitude error curve in step 1 of the present invention without considering the dynamic performance;

[0060] Figure 7 is the attitude angular velocity curve in step 1 of the present invention;

[0061] Figure 8 is the deflection angle of the rudder surface in step 1 of the present invention;

[0062] Fig. 9 It is a visualization diagram of the control network output based on t-SNE in step 2 of the present invention;

[0063] Fig.10 It is the step 2 of the present invention of mapping the Q value color to the posture error curve graph;

[0064] Fig.11 It is a feature analysis diagram based on SHAP value in step 3 of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is specifically described below in conjunction with embodiments. It should be understood that the following text is only used to describe one or several specific embodiments of the present invention, and does not strictly limit the scope of protection of the specific claims of the present invention.

[0066] The present invention proposes an interpretable design method for an aircraft intelligent robust controller, aiming to solve the external interference and model uncertainty problems faced by aircraft in complex environments and overcome the limitations of traditional control methods. By introducing a multi-channel control network and a reinforcement learning algorithm, a highly robust aircraft attitude controller is designed, which can effectively cope with external interference and nonlinear challenges, and significantly improve the adaptability and robustness of the system. At the same time, in response to the unexplainable problem of reinforcement learning algorithms in applications, methods combining t-SNE visualization and SHAP value feature attribution are proposed, which effectively reveal the decision-making process of the aircraft control model, thereby providing a more intuitive and interpretable decision-making basis for the control system.

[0067] like Figure 1-Figure 11 As shown, this embodiment discloses: an aircraft attitude control method based on interpretable reinforcement learning, comprising the following steps:

[0068] Step 1: Establish a Markov decision model for aircraft attitude control and design an aircraft offline attitude controller;

[0069] Step 2: Based on the trained Markov decision model, a visualization analysis is performed based on the t-SNE algorithm to achieve an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller;

[0070] Step 3: Based on the SHAP value feature analysis, the influence of multiple hidden layers in the aircraft reinforcement learning controller on the network output is sorted out to achieve the interpretability analysis of the aircraft neural network model.

[0071] Specific implementation of step 1:

[0072] Establish a Markov decision model for aircraft attitude control and design an aircraft offline attitude controller:

[0073] Considering the flight characteristics of the aircraft in a large airspace and wide speed range, and based on the requirements for stable control of the aircraft under the influence of complex external interference, the following Markov model including state set, action set, reward function, state transition probability and discount factor is established:

[0074] State Set:

[0075] The three-channel attitude error, attitude angle, attitude angular velocity, etc. obtained by the aircraft using the sensor are taken as the state s, and the state set S is defined as:

[0076] S={e1,Θ,ω} (1)

[0077] Where Θ = [α, β, σ] T is the three-channel attitude angle, where α is the angle of attack, β is the sideslip angle, σ is the roll angle, and ω=[p,q,r]T They are the angular velocities of the three channels: roll, pitch and yaw, e1 = Θ - Θ d is the vehicle attitude tracking error.

[0078] Action Set:

[0079] Considering the stable tracking control requirements of the aircraft, based on the aircraft attitude control model, the action set includes the three-direction control rudder deviation of the aircraft, so the control network action set A c Defined as:

[0080]

[0081] in, δ ψ ,δ γ They are the control rudder deflections in pitch, yaw and roll directions respectively.

[0082] Reward function:

[0083] In order to ensure the control performance of the aircraft under the influence of strong interference, the selection of the reward function is crucial. The present invention comprehensively considers two parts of requirements when designing the reward function: on the one hand, the stable tracking control requirements of the aircraft under the influence of unknown external interference of the aircraft are considered, and the three-channel attitude tracking error is used as a reference. The smaller the tracking error, the greater the reward, and the greater the tracking error, the greater the penalty, so as to ensure the steady-state control performance of the aircraft under the influence of interference; on the other hand, the dynamic characteristics such as overshoot during the operation of the aircraft are also crucial to the control performance of the aircraft, so that dynamic characteristics indicators (such as aircraft control overshoot, response time, etc.) are further added to the reward function setting. The better the dynamic performance indicators, the greater the reward, so as to ensure the dynamic performance changes during the operation of the aircraft, and then improve the control performance of the aircraft.

[0084] In summary, based on the above requirements to ensure the steady-state performance and dynamic performance of the aircraft, a reward and punishment mechanism is established, which mainly includes the following three parts:

[0085] The error between the current pose and the desired pose:

[0086]

[0087] Where e 1i Represents the attitude tracking error in the three directions of roll, pitch and yaw.

[0088] Rudder Smoothing Bonus:

[0089]

[0090] Where X is the input three-dimensional array with n samples and m dimensions, R smoothThe smaller the value, the smoother the data. Based on this constraint, the continuity of the rudder deviation change is guaranteed, which better meets the actual engineering needs.

[0091] Dynamic performance:

[0092] The overshoot and adjustment time during the operation of the aircraft are mainly considered as dynamic performance indicators, as shown in the following formula:

[0093]

[0094] In the formula, the overshoot in three directions is defined, α d ,β d ,σ d is the expected attitude in three directions, and the overshoot in three directions is averaged to get the total overshoot:

[0095]

[0096] The adjustment time is defined as the time it takes for a sequence to first enter the 2% error band and remain there, as shown in the following equation: Define the adjustment time in three directions, take the average of the adjustment time in three directions, and get the total adjustment time: The dynamic performance reward is defined as:

[0097] R D =-0.05·OS avg -0.1T set (7)

[0098] Combining the above three indicators, the single-step decision reward value R of the aircraft is:

[0099] R=R e +R smooth +R D (8)

[0100] State transition probability:

[0101] For the aircraft attitude control problem, the aircraft attitude control model is used as the state transfer function, that is, the probability of reaching the next state S' after executing action A under the environment state S.

[0102] Discount Factor:

[0103] The discount factor γ indicates the importance of future benefits relative to current benefits. When γ = 0, it is equivalent to only considering current benefits without considering future benefits. When γ = 1, future benefits and current benefits are considered equally important.

[0104] By defining the Markov model of aircraft control, the aircraft control process can be re-described as follows: for the state set s(t) consisting of the aircraft attitude angle and angular velocity obtained by the sensor at time t, according to the designed immediate benefit value R i (t), determine the optimal action A under the premise of maximum benefit c (t), and interact with the environment, and obtain the vehicle state s(t+1) at the next moment by the state transfer function, and repeat the process until the desired stable tracking control goal is achieved.

[0105] Considering that the Critic target network in the deep deterministic policy gradient (DDPG) reinforcement learning algorithm easily overestimates the state value when calculating the behavioral value of the target action due to the lack of interaction with the environment, it leads to poor policy performance. The Twin delayed deep deterministic policy gradient (TD3) algorithm reduces the estimation bias by adding a dual Q network, but the robustness of the policy is poor. To address this problem, the present invention designs a controller based on the reinforcement learning algorithm of Soft Actor-Critic (SAC). SAC is a deep reinforcement learning algorithm based on the maximum entropy reinforcement learning framework. By adding a policy entropy term to the objective function, the learning process not only pursues the maximization of the cumulative reward, but also encourages the policy output to have a higher randomness, thereby enhancing the exploratory and robust nature of the policy. The overall network architecture is as follows: Figure 1 Shown

[0106] Under the established aircraft control network framework, the network back propagation algorithm is studied, and aircraft control training is carried out in the constructed simulation training environment to achieve stable tracking control of aircraft multi-channel attitude without decoupling. During the training process, the SAC algorithm used has two policy networks and four value networks. The policy network is represented by π and the value network is described by φ. The value network (only runs in the training phase) uses the aircraft state and action information to obtain the evaluation value of the action taken for the current state, so it guides the update of the strategy during the training phase. A complete training process of SAC includes data collection process and parameter training process:

[0107] Data collection process

[0108] During data collection, the aircraft observes the environment to obtain the current state S, sends it to the policy network to output the corresponding operation A, and obtains the reward value R and the next state S'. After executing the above operations, each iteration round will store the data in the following form:

[0109] (S,A,R,S′)(9)

[0110] This process requires a large number of rounds to obtain enough experience vectors, all of which are stored in the experience pool. This stage does not involve the value evaluation network, but only includes the interaction between the policy network and the environment. In order to avoid the negative impact of policy relevance on the training effect, an online policy sampling and update mechanism is introduced. By gradually updating the policy in the process of interaction between the policy network and the environment, the policy network parameters are dynamically adjusted, so that the data of each acquisition round is independent and diverse. This policy update method not only improves the coverage and representativeness of the data in the experience pool, but also enhances the robustness and generalization ability of model training.

[0111] Parameter update process:

[0112] The core idea of ​​SAC is to expand the traditional expected cumulative reward maximization objective into an entropy regularization objective by introducing a policy entropy term:

[0113]

[0114] In the formula, r(s t ,a t ) is the immediate reward, E π [] represents the expected value under strategy π,

[0115] is the entropy of the strategy, which measures the randomness of the strategy, α is the entropy temperature, a hyperparameter that controls the importance of entropy, and E at-π [] represents the expected probability distribution of action a according to strategy π, which measures the probability of strategy π in state s t Through this objective function, SAC's strategy can not only effectively utilize the reward signal in the environment, but also maintain sufficient randomness to explore the possible optimal strategy in a complex environment.

[0116] During the value network training process, the SAC algorithm minimizes the loss function L Q (θ i ) to update the network parameters:

[0117]

[0118] In the formula, represents the Q value estimate given by the current value network, θ i Represents the parameters of the current network, It represents sampling state, action, reward value, and next moment state from the experience replay pool. The target value y is generated by the target network and is defined as follows:

[0119]

[0120] Where γ is the discount factor, a t+1 ′ is the action sampled from the policy network, is the smaller Q value at the future moment.

[0121] The innovation of the SAC algorithm is to use two value networks at the same time to reduce the overestimation problem of Q value, and introduce entropy regularization to balance exploration and utilization, thereby improving the robustness of the strategy. In the training of the policy network, the SAC algorithm updates the policy network parameters by minimizing the following loss function:

[0122]

[0123] Where: D is the sampling space of the state, and E[g] represents the mean square error function.

[0124] In addition, the entropy temperature parameter α is updated; the loss function for optimizing the temperature parameter α is defined as:

[0125]

[0126] In the formula, Indicates sampling the action distribution and calculating the expected value of the sampled data, -logπ φ (a t |s t ) indicates action a t The negative of the log probability in the policy distribution, used to measure the uncertainty of the policy, is the target entropy.

[0127] Since the update of the value network requires the current Q value and the Q′ value at the next moment, so as to make the training data relevant and improve the network convergence speed, the target strategy network and target value network are designed, and the parameters are regularly copied through the soft update method. The update method is:

[0128]

[0129] Where θ′ is the parameter of the target policy network, θ is the policy network parameter, are the parameters of the two target value networks, are the parameters of the two value networks respectively; τ is the soft update coefficient.

[0130] This embodiment performs simulation verification on step 1:

[0131] Parameter settings;

[0132] Aircraft physical parameter settings: dynamic pressure is q = 5189.62528, aircraft characteristic area S = 4.17, aircraft characteristic length L = 4.12, system set inertial parameter is I xx=321,I yy =4785,I zz =4625,I xz =0, the initial value of the attitude angle is Θ0 = [α0, β0, σ0] T =[0.17 0.05 0.34] T rad, the initial value of the attitude angular velocity is designed to be: ω0 = [p0, q0, r0] T =[0 0 0] T rad. Reference attitude angle Θ d ~Uniform(-0.5,0.5)rad represents Θ d is a random number uniformly distributed between -0.5 and 0.5. During training, the control network outputs the rudder deviation range in (-20°, 20°). The maximum time step number of each training round is set to 3000 steps, and the duration of a time step is 0.005s. The termination condition of each training round is: the attitude error of the aircraft in three directions does not exceed 1° or the number of training steps reaches the maximum time step. After multiple parameter adjustments, the training hyperparameter settings of the reinforcement learning algorithm are shown in Table 1.

[0133] Table 1 Hyperparameters based on the robust adversarial reinforcement learning algorithm

[0134]

[0135] Simulation Results Analysis

[0136] After continuous iterative training, the simulation results are as follows Figure 2-Figure 8 As shown. Among them, Figure 2 This is the reward value curve obtained for the aircraft control network. It can be seen from the figure that as the number of iterations increases, the reward value becomes larger and larger, indicating that the control network has converged and has good control performance. Figure 3 It represents the completion rate of every 50 rounds during the training process, that is, the proportion of rounds in which the steady-state error of each channel is no more than 1° in 50 rounds. It can be seen that the completion rate is the highest when the horizontal axis is 10, reaching 98%. The control network at this time is saved as the final model. Figure 4 This is the number of rounds required to achieve the required control accuracy during training. As the number of training times increases, the number of rounds gradually decreases. Under the optimal control network, only 1400 steps are needed to converge.

[0137] Figure 5 This is the attitude error curve of the control network after offline training. It can be seen from the figure that the attitude error tends to be stable within 5 seconds and meets the control accuracy requirements. Then, from 8 to 11 seconds, interference is added to the pitch, roll and yaw channels respectively to verify the robustness of the control network. The interference size is [50sin(2t), 50cos(t), -60sin(t)] TIt can be seen that the attitude error remains stable, and the final three-channel tracking error is (-0.0034, 0.0075, 0.0012) rad, which verifies the effectiveness of the strong robustness aircraft attitude control method proposed in this invention. Figure 6 Compared with the error curve without considering the dynamic performance, the overshoot and adjustment time are greatly improved. Figure 7 It means that the attitude angular velocity converges within 5 seconds. When the disturbance occurs in the 8th second, it always maintains a stable state, indicating that the robust control method proposed in the present invention can better handle the deviation disturbance. Figure 8 To control the rudder deflection curve of the aircraft, after interference occurs, the output of the control network will change accordingly to suppress the interference, thereby verifying the effectiveness of the network.

[0138] Specific implementation of step 2:

[0139] According to the trained model, visual analysis is performed based on the t-SNE algorithm to achieve an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller.

[0140] The interpretability of reinforcement learning models is one of the important factors that limit their application in the aerospace field. In order to improve the transparency and interpretability of reinforcement learning models, the present invention carries out visualization analysis of network models from two aspects: First, considering that t-SNE (t-Distributed Stochastic Neighbor Embedding) is an effective visualization tool, by mapping high-dimensional data to two-dimensional or three-dimensional space, the local structure between data points is retained, so that we can intuitively understand the decision-making process of the model. In reinforcement learning, t-SNE can be used to visualize high-dimensional data such as policy space, state space, and value function, and by analyzing the distribution of different states or actions in low-dimensional space, it helps us discover the changing trend or potential structural characteristics of the strategy; secondly, considering that SHAP (SHAPley Additive Explanations) values ​​provide an effective method to analyze and explain the predictive features of the model, especially for deep reinforcement learning networks. By using SHAP values, the importance of different features in the decision-making process can be quantified, thereby improving the interpretability of the model. In the training process of reinforcement learning, the agent usually makes action decisions based on state features, and SHAP values ​​can help us understand how each state feature affects the final decision result. By visualizing the SHAP value, we can intuitively understand which features contribute most to the model output, thereby identifying potential influencing factors and optimizing the model design. Therefore, the following will introduce the visualization of the aircraft reinforcement learning controller from these two aspects.

[0141] Network output t-SNE visualization analysis

[0142] t-SNE is a nonlinear technique for dimensionality reduction of high-dimensional data, mainly used to visualize complex high-dimensional data sets. It embeds high-dimensional data into two-dimensional or three-dimensional space so that similar points remain close in low-dimensional space, thereby helping people observe and analyze the distribution and structure of data in low-dimensional space. The core goal of t-SNE is to maintain the local structure of the data by minimizing the difference between the two distributions.

[0143] When the model is trained, visualize the output of the model and extract the output of the last hidden layer of the evaluation network. Each data sample will generate a corresponding high-dimensional feature vector at each time step. These vectors represent the model's internal representation of the input data at that time step. Since the dimensions of these vectors are usually high, it is very difficult to directly visualize these eigenvalues. Therefore, we need to reduce these high-dimensional feature vectors to two-dimensional space for subsequent visualization analysis. To this end, t-SNE is used to map these high-dimensional features to two-dimensional space, maintaining the relative distance between similar data points while separating dissimilar data points. Then use the t-SNE algorithm to reduce the dimension to two-dimensional space, such as Fig. 9 As shown, the left picture has no interference, and the right picture has interference. One round includes 3000 steps. Each point represents a high-dimensional data sample in one time step. The color of the point represents the Q value. Dark colors represent smaller Q values, and light colors represent larger Q values. Through color mapping, we can intuitively see which data points have larger or smaller output values. The clusters in the figure represent the similarity between samples, while the distance between samples reflects their similarity in high-dimensional space.

[0144] As can be observed from the figure, steps with high Q values ​​are clustered in the same area in the embedding space, indicating that the model has learned to represent states with similar Q values ​​in two-dimensional space. The path in the round is represented by the continuous transition of color, which shows that the aircraft goes through different states in one round. And there is a clear separation between the purple and yellow areas, indicating that the model makes a key decision, which is the moment when the attitude error meets the conditions. In addition, t-SNE visualization provides a deep understanding of the structure of the control strategy in high-dimensional space and can help determine the direction of strategy improvement. For example, in the right figure, the points with low Q values ​​are scattered in the high Q value (warm color) state, indicating that the system control error increases after the disturbance is strengthened, and returns to the high Q value level as the control network is adjusted. Fig.10 To map the color of the Q value to the attitude error curve, as the entire round time goes by, the attitude error continues to decrease, and the Q value increases accordingly. After the interference is added, the attitude error increases and the corresponding color becomes darker. This shows that after the interference is added, the color changes, and the interference affects the attitude control. The final color shows that stable control under interference is achieved.

[0145] Specific implementation of step 3:

[0146] Based on the SHAP value feature analysis, the important procedures of the influence of multiple hidden layers in the aircraft reinforcement learning controller on the network output (i.e., three-directional control rudder deviation) are sorted out, and the interpretability analysis of the aircraft neural network model is realized.

[0147] As machine learning models become increasingly complex, especially deep learning and ensemble learning models, the "black box" characteristics of the models lead to a lack of transparency in the decision-making process. Therefore, model interpretability methods have gradually become a research hotspot. The SHAP value is a feature importance measurement method based on game theory that can help reveal the decision-making mechanism of the model, thereby improving the interpretability of the model. The theoretical basis of the SHAP value comes from the Shapley value in game theory. The Shapley value was first proposed by economist Lloyd Shapley to solve the problem of how to reasonably distribute benefits in cooperative games. In game theory, the Shapley value is used to evaluate the contribution of each participant to the overall benefit. In machine learning, features are regarded as "cooperative participants", and the prediction result of the model is the "total benefit". The SHAP value is used to measure the independent contribution of each feature to the prediction result.

[0148] The process of calculating the SHAP value can be seen as traversing all possible feature combinations to evaluate the marginal contribution of the feature to the model output. In the calculation process, it is necessary to consider the changes in the model output when the features are added in different orders, so that the average marginal contribution of the feature can be obtained, and finally the SHAP value of the feature can be obtained. The specific calculation formula is:

[0149]

[0150] In the formula, S is the feature subset, N is the set of all features, and f(S) represents the model prediction value corresponding to the feature subset. By summing all possible feature combinations, we can ensure that the contribution of the features is fully considered.

[0151] In order to explain the contribution of each layer of the network to the model output, a visual analysis is performed based on the SHAP value. The model in this section has three continuous action outputs, namely three-directional control rudder deviation, and the input is the attitude angle and angular velocity of the aircraft, a total of 7 input features. For the explanation, all features are sorted according to their attribution to the action output, and then the two most important features are selected to explain the model prediction. First, the output of each layer of the network is extracted, and the SHAP value is calculated with the final model output. Then, the principal component analysis method is used to extract the most important SHAP value as the SHAP value of this layer of the network. Finally, the re-extracted SHAP values ​​of each layer are spliced ​​to obtain Fig.11 .

[0152] In this set of SHAP value graphs, each graph shows the impact of the output of each layer of the deep network (such as L1, L2, L3, L4, L5) on the final three-way rudder output of the model. The horizontal axis represents the SHAP value, reflecting the impact of each feature on the model output (positive or negative); the vertical axis lists the network name, sorted by the importance of the impact on the model (the higher the layer, the greater the impact on the output); the size of the eigenvalue is represented by color: red represents a larger eigenvalue, blue represents a smaller eigenvalue, and the distribution of points of different colors shows how changes in eigenvalues ​​affect the SHAP value.

[0153] from Fig.11 From the analysis we can get:

[0154] from Fig.11 From the perspective of the middle structure, L3 is the most important feature in all three rudder outputs, and its SHAP value is widely distributed, indicating that it has a significant and diverse impact on the model prediction; L2 and L1 also have obvious contributions to the model, but their influence is lower than that of L3; the influence of L4 and L5 layers is small and concentrated, indicating that their effect on the prediction results is limited.

[0155] From the density and variation of the distribution, L3 has a large number of data points in the entire SHAP value range, indicating that its influence covers most samples, indicating that L3's contribution to the output result is global. L3's distribution is not only wide, but also has a higher sample density within the SHAP value range, while L2 has a greater impact on extreme values, but the overall contribution is less, and its distribution length is caused by outliers in a few samples, indicating that most samples are significantly affected by L3, rather than dominated by a few samples.

[0156] From the color distribution, the red and blue (high and low eigenvalues) in L3 have strong performance in the positive and negative SHAP value ranges, indicating that it significantly affects the output at different values. Specifically, the high-value features are concentrated in the negative SHAP value area, indicating that its high-value features reduce the final rudder output, and the low-value features are concentrated in the positive SHAP value area, indicating that its low-value features increase the final rudder output.

[0157] In summary, it can be seen that the third hidden layer has the greatest impact on the system output, followed by the second and first hidden layers. Therefore, when the aircraft state changes or during online optimization, focusing on optimizing the parameters of these three layers of the network can change the control performance of the system.

[0158] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. An aircraft attitude control method based on interpretable reinforcement learning, characterized in that: The following steps are involved: Step 1: Establish a Markov decision model for aircraft attitude control and design an aircraft offline attitude controller; Step 2: Based on the trained Markov decision model, a visualization analysis is performed based on the t-SNE algorithm to achieve an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller; Step 3: Based on the SHAP value feature analysis, the influence of multiple hidden layers in the aircraft reinforcement learning controller on the network output is sorted out to achieve the interpretability analysis of the aircraft neural network model.

2. The aircraft attitude control method based on interpretable reinforcement learning according to claim 1, characterized in that: In the step 1, a Markov model including a state set, an action set, a reward function, a state transition probability and a discount factor is established as follows; The state set: The three-channel attitude error, attitude angle, and attitude angular velocity obtained by the aircraft using the sensor are taken as the state, and the state set S is defined as: S = {e1, Θ, ω}; where Θ = [α, β, σ] T is the three-channel attitude angle, where α is the angle of attack, β is the sideslip angle, σ is the roll angle, and ω=[p,q,r] T They are the angular velocities of the three channels: roll, pitch and yaw, e1 = Θ - Θ d is the vehicle attitude tracking error; The action set: Considering the stable tracking control requirements of the aircraft, based on the aircraft attitude control model, the action set includes the three-direction control rudder deviation of the aircraft, thereby controlling the network action set A c Defined as: in δ ψ ,δ γ They are the control rudder deflections in pitch, yaw and roll directions respectively; The reward function establishes a reward and punishment mechanism, including: The error between the current pose and the desired pose: Among them, e1i represents the attitude tracking error in the three directions of roll, pitch and yaw; Rudder Smoothness Bonus: Where X is the input three-dimensional array with n samples and m dimensions, and R smooth The smaller it is, the smoother the data is; Dynamic performance: as shown below: In the formula, the overshoot in three directions is defined, α d ,β d ,σ d is the expected attitude in three directions, and the overshoot in three directions is averaged to get the total overshoot: The adjustment time is defined as the time it takes for a sequence to first enter the 2% error band and remain there, as shown in the following equation: Define the adjustment time in three directions, take the average of the adjustment time in three directions, and get the total adjustment time: The dynamic performance reward is defined as: R D =-0.05·OS avg -0.1T set ; Combining the above three indicators, the single-step decision reward value R of the aircraft is: R=R e +R smooth +R D ; The state transition probability: For the aircraft attitude control problem, the aircraft attitude control model is used as the state transition function; that is, the probability of reaching the next state S' after executing action A under the environment state S; Discount factor: The discount factor γ indicates the importance of future benefits relative to current benefits. When γ = 0, it is equivalent to only considering current benefits without considering future benefits. When γ = 1, future benefits and current benefits are considered equally important. By defining the aircraft control Markov model, the aircraft control process can be re-described as follows: for the state set s(t) consisting of the aircraft attitude angle and angular velocity obtained by the sensor at time t, according to the designed immediate benefit value R i (t), determine the optimal action A under the premise of maximum benefit c (t), and interact with the environment, and obtain the vehicle state s(t+1) at the next moment by the state transfer function, and repeat the process until the desired stable tracking control goal is achieved.

3. The aircraft attitude control method based on interpretable reinforcement learning according to claim 1, characterized in that: In the step 1, under the established aircraft control network framework, the network back propagation algorithm is studied, and aircraft control training is carried out in the constructed simulation training environment to achieve stable tracking control of the aircraft's multi-channel attitude without decoupling; during the training process, the SAC algorithm used has two policy networks and four value networks, the policy network is represented by π, and the value network is described by φ; the value network uses the aircraft state and action information to obtain the evaluation value of the action taken on the current state.

4. The aircraft attitude control method based on interpretable reinforcement learning according to claim 1, characterized in that: In step 1, the strategy is guided and updated during the training phase; a complete training process of the SAC algorithm includes a data collection process and a parameter training process: Data collection process: The aircraft observes the environment to obtain the current state S, sends it to the policy network to output the corresponding operation A, and obtains the reward value R and the next state S′; after executing the above operations, each iteration round will store the data in the following form: (S, A, R, S′); Parameter training process: The SAC algorithm expands the traditional expected cumulative reward maximization objective into an entropy regularization objective by introducing a policy entropy term: In the formula, r(s t ,a t ) is the immediate reward, E π [] represents the expected value under strategy π, is the entropy of the strategy, which measures the randomness of the strategy, α is the entropy temperature, a hyperparameter that controls the importance of entropy, and E at-π [] represents the expected probability distribution of action a according to strategy π, which measures the probability of strategy π in state s t Uncertainty in the distribution of next actions; During the value network training process, the SAC algorithm minimizes the loss function L Q (θ i ) to update the network parameters: In the formula, represents the Q value estimate given by the current value network, θ i Represents the parameters of the current network, It represents sampling state, action, reward value, and next moment state from the experience replay pool. The target value y is generated by the target network and is defined as follows: Where γ is the discount factor, a t+1 ′ is the action sampled from the policy network, is the smaller Q value at the future moment; Two value networks are used at the same time to reduce the over-estimation problem of Q value, and entropy regularization is introduced to balance exploration and utilization. In the training of the policy network, the SAC algorithm updates the policy network parameters by minimizing the following loss function: Where: D is the sampling space of the state, E[g] represents the mean square error function; In addition, the entropy temperature parameter α is updated, and the loss function for optimizing the temperature parameter α is defined as: In the formula, Indicates sampling the action distribution and calculating the expected value of the sampled data, -logπ φ (a t |s t ) indicates action a t The negative of the log probability in the policy distribution, used to measure the uncertainty of the policy, is the target entropy; Since the update of the value network requires the current Q value and the Q' value at the next moment, the target strategy network and the target value network are designed, and the parameters are regularly copied through the soft update method. The update method is: θ′←τθ+(1-τ)θ′ Where θ′ is the parameter of the target policy network, θ is the policy network parameter, are the parameters of the two target value networks, are the parameters of the two value networks respectively, and τ is the soft update coefficient.

5. The aircraft attitude control method based on interpretable reinforcement learning according to claim 1, characterized in that: In step 2, a t-SNE visualization analysis of the network output is performed: when the model is trained, the output of the model is visualized, and the output of the last hidden layer of the evaluation network is extracted. Each data sample generates a corresponding high-dimensional feature vector at each time step, and these vectors represent the model's internal representation of the input data at that time step; these high-dimensional feature vectors are reduced to two-dimensional space; for this purpose, t-SNE is used to map these high-dimensional features to two-dimensional space, maintaining the relative distance between similar data points while separating dissimilar data points; then the t-SNE algorithm is used to reduce the dimension to two-dimensional space, complete the visualization analysis, and realize an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller.

6. The aircraft attitude control method based on interpretable reinforcement learning according to claim 1, characterized in that: In step 3, the process of calculating the SHAP value can be regarded as traversing all possible feature combinations to evaluate the marginal contribution of the features to the model output. In the calculation process, it is necessary to consider the changes in the model output caused by adding features in different orders, so as to obtain the average marginal contribution of the features and finally obtain the SHAP value of the feature. The specific calculation formula is: In the formula, φ i is the SHAP value of feature i, S is the feature subset, N is the set of all features, |S| is the number of features in subset S, |N| is the number of features in the full feature set, |S|! (|N|-|S|-1)! is the product of the number of permutations of subset S and the number of permutations of the remaining features, which is used to calculate the combined weights of feature i at different positions, |N|! is the number of permutations of the full feature set, which is used to normalize the weights, f(S∪{i}) represents the output value of the model when using subset S and feature i together for prediction, and f(S) represents the model prediction value corresponding to the feature subset.

Images