Aircraft Attitude Control Method Based on Explainable Reinforcement Learning

Abstract

The present invention belongs to the technical fields of reinforcement learning and aircraft attitude control, and specifically relates to an aircraft attitude control method based on interpretable reinforcement learning. First, an aircraft attitude controller with strong robustness is designed. By introducing a multi-channel control network and combining a reinforcement learning algorithm, the controller can effectively cope with the challenges brought by external disturbances and system nonlinearities, and significantly improve the adaptability and robustness of the aircraft control system. Secondly, the present invention proposes an interpretability method combined with visual analysis. By using the t-SNE algorithm to perform dimensionality reduction visualization on the input features of the aircraft control system and combining the SHAP values to perform attribution analysis on the input features, the key factors in the model decision-making process can be revealed, helping to understand how the aircraft control system makes decisions based on different input features. The present invention realizes high adaptability and transparency of the aircraft control system in complex environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of reinforcement learning and aircraft attitude control, and particularly relates to an aircraft attitude control method based on explainable reinforcement learning. Background Art

[0002] With the continuous development of aircraft technology, especially the increasing demand for online control in complex environments, ensuring that aircraft can achieve stable performance in dynamically changing environments has become a major challenge in the aerospace field. Although traditional aircraft control methods can ensure the stability of aircraft to a certain extent, they often lack sufficient adaptability and flexibility when faced with online environmental changes, real-time mission adjustments, and sudden interference. In addition, aircraft control algorithms often rely on large amounts of offline data for training, resulting in weak generalization capabilities of the algorithms when faced with new environments or unseen situations. Therefore, improving the online learning capabilities and interpretability of aircraft control algorithms has become a key issue that needs to be urgently addressed in the field of aircraft intelligent control. Summary of the Invention

[0003] The purpose of the present invention is to provide an aircraft attitude control method based on interpretable reinforcement learning, specifically involving the use of a training method based on the Soft Actor-Critic (SAC) reinforcement learning method to achieve flight stability control, and interpretable analysis based on a visualization method. Aiming at the online learning and interpretability requirements of aircraft control, the present invention introduces a reinforcement learning algorithm, combines the control strategy with real-time data, and improves the interpretability of the model through t-SNE visualization and SHAP value analysis, thereby achieving high adaptability and transparency of the aircraft control system in complex environments.

[0004] The technical solutions adopted by the present invention are as follows:

[0005] The aircraft attitude control method based on interpretable reinforcement learning includes the following steps:

[0006] Step 1: Establish a Markov decision model for aircraft attitude control and design an aircraft offline attitude controller;

[0007] Preferably, in step 1, considering the flight characteristics of the aircraft in a large airspace and a wide speed range, and based on the requirements for stable control of the aircraft under the influence of complex external interference, a Markov model including a state set, an action set, a reward function, a state transition probability, and a discount factor is established as follows;

[0008] The state set: The three-channel attitude error, attitude angle, and attitude angular velocity obtained by the aircraft using the sensor are used as the state, and the state set S is defined as: S = {e1, Θ, ω}; where Θ = [α, β, σ] TIt is the three-channel attitude angle, where α is the angle of attack, β is the sideslip angle, σ is the bank angle, and ω = [p, q, r] T They are the angular velocities of the roll, pitch, and yaw channels respectively. e1 = Θ - Θ d is the attitude tracking error of the aircraft;

[0009] The action set: Considering the requirements of aircraft stable tracking control and based on the aircraft attitude control model, the action set includes the control rudder deflections in three directions of the aircraft, and then the control network action set A c is defined as: where δ ψ , δ γ are the control rudder deflections in the pitch, yaw, and roll directions respectively;

[0010] The reward function, based on the requirements of ensuring the steady-state performance and dynamic performance of the aircraft, establishes a reward and punishment mechanism, including:

[0011] The error between the current attitude and the desired attitude: where e1i represents the attitude tracking errors in the roll, pitch, and yaw directions;

[0012] Rudder deflection smoothing reward:

[0013] In the formula, X is the input three-dimensional array with n samples and m dimensions, R smooth The smaller it is, the higher the smoothness of the data; based on this constraint, it is ensured that the change of the rudder deflection is continuous and can better meet the actual engineering requirements;

[0014] Dynamic performance: Mainly consider the overshoot and settling time during the operation of the aircraft as dynamic performance indicators, as shown in the following formula: In the formula, the overshoot in three directions is defined, α d , β d , σ d are the desired attitudes in three directions. The overshoots in three directions are averaged to obtain the total overshoot: The settling time is defined as the time when the sequence first enters the 2% error band and remains in it all the time, as shown in the following formula: The settling times in three directions are defined, and the settling times in three directions are averaged to obtain the total settling time: The dynamic performance reward is defined as:

[0015] R D = -0.05·OS avg - 0.1T set ; Combining the above three indicators, the single-step decision reward value R of the aircraft is:

[0016] R = R e + R smooth + R D ;

[0017] The state transition probability: For the aircraft attitude control problem, the attitude control model of the aircraft is used as the state transition function; that is, the probability of reaching the next state S' after executing the action A under the environmental state S.

[0018] Discount factor: The discount factor γ represents the importance of future reward values relative to the current reward value; when γ = 0, it is equivalent to only considering the current reward and not considering future rewards, and when γ = 1, future rewards and current rewards are regarded as equally important.

[0019] Through the definition of the aircraft control Markov model, the aircraft control process can be redescribed as follows: For the state set s(t) composed of the aircraft attitude angle and angular velocity obtained by the aircraft based on sensors at time t, according to the designed immediate reward value R i (t), the corresponding optimal action A c (t) is determined under the premise of the highest reward, and interacts with the environment to obtain the aircraft state s(t + 1) at the next moment, and repeat this process until the goal of forming the desired stable tracking control is achieved.

[0020] Preferably, in step 1, under the established aircraft control network framework, the network backpropagation algorithm is studied, and aircraft control training is carried out in the built simulation training environment to achieve stable tracking control of the aircraft's multi-channel attitude without decoupling; during the training process, the SAC algorithm used has two policy networks and 4 value networks. The policy network is represented by π, and the value network is described by φ; the value network (only runs in the training phase) uses the aircraft state and action information to obtain the evaluation value of taking the action for the current state.

[0021] Preferably, in step 1, the policy is guided and updated during the training phase; a complete training process of the SAC algorithm includes a data collection process and a parameter training process:

[0022] Data collection process: The aircraft observes the environment to obtain the current state S, sends it into the policy network to output the corresponding operation A, obtains the reward value R and the next state S'; after executing the above operations, each iteration round will store the data in the following form: (S, A, R, S').

[0023] This process requires a large number of rounds to obtain a sufficient number of experience vectors and store them all in the experience pool. This stage does not involve the value evaluation network and only includes the interaction between the policy network and the environment. To avoid the negative impact of policy correlation on the training effect, an on-policy sampling and update mechanism is introduced. By gradually updating the policy during the interaction between the policy network and the environment and dynamically adjusting the policy network parameters, the data of each collection round is made independent and diverse. This policy update method not only improves the coverage and representativeness of the data in the experience pool but also enhances the robustness and generalization ability of model training;

[0024] Parameter training process: The core idea of the SAC algorithm is to expand the traditional expected cumulative reward maximization objective into an entropy-regularized objective by introducing a policy entropy term:

[0025]

[0026] where r(s t ,a t ) is the immediate reward, E π [] represents the expected value under the policy π, is the entropy of the policy, measuring the randomness of the policy, α is the entropy temperature, a hyperparameter controlling the importance of entropy, and E at-π [] represents taking the expectation of the probability distribution of the action a according to the policy π, measuring the uncertainty of the action distribution of the policy π in the state s t ; through this objective function, the policy of SAC can not only effectively utilize the reward signal in the environment but also maintain sufficient randomness to explore possible optimal policies in complex environments;

[0027] During the training of the value network, the SAC algorithm updates the network parameters by minimizing the loss function L Q (θ i ):

[0028]

[0029] where represents the Q-value estimate given by the current value network, θ i represents the parameters of the current network, represents sampling the state, action, reward value, and next-state from the experience replay pool, and the target value y is generated by the target network, and its definition is as follows:

[0030]

[0031] where γ is the discount factor, a t+1 ′ is the action sampled from the policy network, is the smaller Q-value at the future moment;

[0032] The innovation of the SAC algorithm lies in simultaneously using two value networks to reduce the overestimation problem of Q-values, and introducing entropy regularization to balance exploration and exploitation, thereby improving the robustness of the policy. In the training of the policy network, the SAC algorithm updates the policy network parameters by minimizing the following loss function:

[0033]

[0034] where: D is the sampling space of states, and E[g] represents the mean squared error function;

[0035] In addition, the entropy temperature parameter α is updated; the loss function for defining and optimizing the temperature parameter α is:

[0036]

[0037] where, denotes sampling from the action distribution, calculating the expected value of the sampled data, and -logπ φ (a t |s t ) represents the negative value of the log probability of action a t in the policy distribution, which is used to measure the uncertainty of the policy, is the target entropy;

[0038] Since the update of the value network requires obtaining the current Q-value and the next moment Q'-value, which makes the training data show correlation and improves the network convergence speed, a target policy network and a target value network are designed, and the parameters are periodically copied through the soft update method. The update method is:

[0039] θ′←τθ+(1 - τ)θ′

[0040]

[0041] where, θ′ is the parameter of the target policy network, θ is the parameter of the policy network, are the parameters of the two target value networks respectively, are the parameters of the two value networks respectively; τ is the soft update coefficient.

[0042] Step 2: According to the trained Markov decision model, perform visualization analysis based on the t-SNE algorithm to achieve an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller;

[0043] Preferably, in step 2, network output t-SNE visualization analysis is performed: When the model is trained, the output of the model is visualized, and the output of the last hidden layer of the evaluation network is extracted. Each data sample generates a corresponding high-dimensional feature vector at each time step, and these vectors represent the internal representation of the input data by the model at this time step. Since the dimensions of these vectors are usually high, it is very difficult to directly visualize these eigenvalues. Therefore, we need to reduce the dimensions of these high-dimensional feature vectors to a two-dimensional space for subsequent visualization analysis. For this purpose, t-SNE is used to map these high-dimensional features to a two-dimensional space, maintaining the relative distance between similar data points and separating dissimilar data points at the same time. Then, the t-SNE algorithm is used to reduce the dimensions to a two-dimensional space to complete the visualization analysis and achieve an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller.

[0044] Step 3: Based on SHAP value feature analysis, sort out the influence program of multiple hidden layers in the aircraft reinforcement learning controller on the network output, and realize the interpretability analysis of the aircraft neural network model.

[0045] Preferably, in step 3, the process of calculating the SHAP value can be regarded as traversing all possible feature combinations to evaluate the marginal contribution of features to the model output. During the calculation process, it is necessary to consider the change of the model output when features are added in different orders, so as to obtain the average marginal contribution of the features and finally obtain the SHAP value of the feature. The specific calculation formula is:

[0046]

[0047] In the formula, φ i is the SHAP value of feature i, S is the feature subset, N is the set of all features, |S| is the number of features in subset S, |N| is the number of features in the full feature set, |S|!(|N|-|S|-1)! is the product of the permutation number of subset S and the permutation number of the remaining features, which is used to calculate the combination weight of feature i in different positions, |N!| is the permutation number of the full feature set, which is used to normalize the weight, f(S∪{i}) represents the output value of the model when using subset S and feature i together for prediction, and f(S) represents the model prediction value corresponding to the feature subset. By summing all possible feature combinations, the contribution of features is ensured to be comprehensively considered.

[0048] The technical effects achieved by the present invention are:

[0049] The present invention specifically relates to the realization of flight stability control by using a training method based on the Soft Actor-Critic (SAC) reinforcement learning method and performing interpretable analysis based on a visualization method. In response to the online learning and interpretability requirements of aircraft control, by introducing a reinforcement learning algorithm and combining real-time data to optimize the control strategy, and at the same time enhancing the interpretability of the model through t-SNE visualization and SHAP value analysis, the high adaptability and transparency of the aircraft control system in complex environments are achieved.

[0050] In the present invention, considering the external disturbances and model uncertainties faced by the aircraft in complex environments, the present invention aims to propose an interpretable intelligent robust controller design method for aircraft to address the limitations of traditional control methods in practical applications. Specifically, first, for the attitude control problem of the aircraft in complex environments, a highly robust aircraft attitude controller is designed. By introducing a multi-channel control network and combining a reinforcement learning algorithm, the controller can effectively cope with the challenges brought by external disturbances and system nonlinearities, significantly improving the adaptability and robustness of the aircraft control system. Second, based on the t-SNE (t-Distributed Stochastic Neighbor Embedding) algorithm, the high-dimensional data of the aircraft control network is mapped to a two-dimensional space. By analyzing the distribution of different states or actions in the low-dimensional space, the changing trend of the strategy or potential structural features are determined; finally, based on the SHAP (SHAPley Additive Explanations) value, an effective method is provided to analyze and interpret the prediction features of the model, realizing the quantification of the importance of the different layer network features of the aircraft reinforcement learning controller in the decision-making process, thereby achieving the interpretability of the model from a more intuitive perspective.

[0051] In the present invention, considering the external disturbances and model uncertainties of the aircraft in complex operating environments, a robust control strategy based on reinforcement learning is designed. A reward function including the three-channel attitude tracking error of the aircraft is established, and considering the engineering requirements for improving the dynamic performance during the operation process, dynamic performance indicators such as the overshoot of attitude control are further added to the reward function, realizing the steady-state tracking and dynamic performance optimization of the aircraft in complex environments.

[0052] In the present invention, considering the unexplainability problem of the reinforcement learning algorithm in aircraft control, an interpretable analysis framework for the aircraft reinforcement learning controller based on the t-SNE algorithm is designed. By extracting the output of the last hidden layer of the evaluation network of the aircraft reinforcement learning controller, a visual representation of the changing trend of the control strategy or potential structural features is achieved.

[0053] In the present invention, further, a method for explaining the relationship of an aircraft reinforcement learning network based on SHAP value analysis is designed to quantify the importance of different layer network features of the aircraft reinforcement learning controller in the decision-making control process, intuitively show which features contribute the most to the model output, reveal the transparency and logic of the control decision, and thus improve the credibility of the aircraft control system. Description of the Drawings

[0054] Figure 1 is the architecture diagram of the strongly robust aircraft control algorithm in the present invention;

[0055] Figure 2 is the control network reward function curve in Step 1 of the present invention;

[0056] Figure 3 is the completion rate every 50 rounds in Step 1 of the present invention;

[0057] Figure 4 is the number of steps per round in Step 1 of the present invention;

[0058] Figure 5 is the attitude error curve in Step 1 of the present invention and the enlarged view of the corresponding area;

[0059] Figure 6 is the attitude error curve without considering dynamic performance in Step 1 of the present invention;

[0060] Figure 7 is the attitude angular velocity curve in Step 1 of the present invention;

[0061] Figure 8 is the rudder surface deflection angle in Step 1 of the present invention;

[0062] Figure 9 is the visualization diagram of the control network output based on t-SNE in Step 2 of the present invention;

[0063] Figure 10 is the diagram of mapping Q value colors to the attitude error curve in Step 2 of the present invention;

[0064] Figure 11 is the feature analysis diagram based on SHAP value in Step 3 of the present invention. Detailed Embodiments

[0065] In order to make the objectives and advantages of the present invention clearer, the present invention will be specifically described below in conjunction with embodiments. It should be understood that the following text is only used to describe one or several specific implementation manners of the present invention, and does not strictly limit the scope of protection of the specific claims of the present invention.

[0066] The present invention proposes an interpretable intelligent robust controller design method for aircraft, aiming to solve the problems of external interference and model uncertainty faced by aircraft in complex environments and overcome the limitations of traditional control methods. By introducing a multi-channel control network and a reinforcement learning algorithm, an aircraft attitude controller with strong robustness is designed, which can effectively cope with external interference and nonlinear challenges, and significantly improve the adaptability and robustness of the system. At the same time, aiming at the problem of the non-interpretable nature of the reinforcement learning algorithm in applications, methods combining t-SNE visualization and SHAP value feature attribution are proposed respectively, which effectively reveal the decision-making process of the aircraft control model, thus providing a more intuitive and interpretable decision-making basis for the control system.

[0067] As Figures 1-11 shown, this embodiment discloses an aircraft attitude control method based on interpretable reinforcement learning, including the following steps:

[0068] Step 1: Establish a Markov decision model for aircraft attitude control and design an aircraft offline attitude controller;

[0069] Step 2: According to the trained Markov decision model, perform visual analysis based on the t-SNE algorithm to achieve an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller;

[0070] Step 3: Based on SHAP value feature analysis, complete the sorting of the influence of multiple hidden layers in the aircraft reinforcement learning controller on the network output, and achieve the interpretability analysis of the aircraft neural network model.

[0071] Specific implementation of Step 1:

[0072] Establish a Markov decision model for aircraft attitude control and design an aircraft offline attitude controller:

[0073] Considering the flight characteristics of the aircraft in a large airspace and wide speed range, and based on the aircraft stability control requirements under the influence of external complex interference, etc., establish the following Markov model including a state set, an action set, a reward function, a state transition probability, and a discount factor:

[0074] State set:

[0075] Take the three-channel attitude error, attitude angle, attitude angular velocity, etc. obtained by the aircraft using sensors as the state s, then the state set S is defined as:

[0076] S = {e1, Θ, ω} (1)

[0077] where, Θ = [α, β, σ] T is the three-channel attitude angle, where α is the angle of attack, β is the sideslip angle, σ is the bank angle, and ω = [p, q, r]T They are the angular velocities of the roll, pitch, and yaw channels respectively. e1 = Θ - Θ d is the attitude tracking error of the aircraft.

[0078] Action set:

[0079] Considering the requirements of aircraft stable tracking control and based on the aircraft attitude control model, the action set includes the control rudder deflections in three directions of the aircraft. Therefore, the action set A of the control network c is defined as:

[0080]

[0081] where δ ψ , δ γ are the control rudder deflections in the pitch, yaw, and roll directions respectively.

[0082] Reward function:

[0083] To ensure the control performance of the aircraft under strong interference, the selection of the reward function is crucial. When designing the reward function in the present invention, two parts of requirements are comprehensively considered: on the one hand, considering the requirements of the aircraft stable tracking control under the influence of unknown external interference of the aircraft, taking the attitude tracking error of the three channels as the reference content, the smaller the tracking error, the greater the reward, and the larger the tracking error, the greater the punishment, so as to ensure the steady-state control performance of the aircraft under the influence of interference; on the other hand, considering that the overshoot and other dynamic characteristic changes during the operation of the aircraft are also crucial to the control performance of the aircraft, so dynamic characteristic indicators (such as the control overshoot and response time of the aircraft) are further added to the reward function setting. The better the dynamic performance indicators, the greater the reward, so as to ensure the dynamic performance changes during the operation of the aircraft, and further improve the control performance of the aircraft.

[0084] In summary, based on the above two aspects of requirements to ensure the steady-state performance and dynamic performance of the aircraft, a reward and punishment mechanism is established, which mainly includes the following three parts:

[0085] The error between the current attitude and the desired attitude:

[0086]

[0087] In the formula, e 1i represents the attitude tracking errors in the roll, pitch, and yaw directions.

[0088] Rudder deflection smoothing reward:

[0089]

[0090] In the formula, X is the input three-dimensional array with n samples and m dimensions, R smoothThe smaller value indicates higher data smoothness. Based on this constraint, the continuity of rudder deflection change is ensured, which better meets the actual engineering requirements.

[0091] Dynamic performance:

[0092] The overshoot and settling time during the flight of the aircraft are mainly considered as dynamic performance indicators, as shown in the following formula:

[0093]

[0094] In the formula, the overshoot in three directions is defined, α d , β d , σ d are the desired attitudes in three directions. The average of the overshoots in three directions is taken to obtain the total overshoot:

[0095]

[0096] The settling time is defined as the time when the sequence first enters the 2% error band and remains within it, as shown in the following formula: The settling time in three directions is defined, and the average of the settling times in three directions is taken to obtain the total settling time: The dynamic performance reward is defined as:

[0097] R D =-0.05·OS avg -0.1T set (7)

[0098] Combining the above three indicators, the single-step decision reward value R of the aircraft is:

[0099] R = R e +R smooth +R D (8)

[0100] State transition probability:

[0101] For the aircraft attitude control problem, the aircraft attitude control model is used as the state transition function. That is, the probability of reaching the next state S′ after performing the action A under the environmental state S.

[0102] Discount factor:

[0103] The discount factor γ represents the importance of future benefit values relative to the current benefit value. When γ = 0, it is equivalent to only considering the current benefit and not considering the future benefit. When γ = 1, the future benefit and the current benefit are regarded as equally important.

[0104] Through the definition of the Markov model for aircraft control, the aircraft control process can be redescribed as follows: for the state set s(t) composed of the aircraft attitude angle and angular velocity obtained by the aircraft based on sensors at time t, according to the designed immediate reward value R i (t), the corresponding optimal action A c (t) is determined under the premise of the highest reward, and interacts with the environment. The next aircraft state s(t + 1) is obtained by the state transition function, and this process is repeated until the goal of forming the desired stable tracking control is achieved.

[0105] Considering that when the Critic target network in the Deep deterministic policy gradient (DDPG) reinforcement learning algorithm calculates the behavior value of the target action, since it does not interact with the environment, it is easy to overestimate the state value, resulting in poor policy performance. The Twin delayed deep deterministic policy gradient (TD3) algorithm adds a dual Q network to reduce the estimation bias, but the robustness of the policy is poor. To address this problem, the present invention designs a controller based on the Soft Actor-Critic (SAC) reinforcement learning algorithm. SAC is a deep reinforcement learning algorithm based on the maximum entropy reinforcement learning framework. By adding a policy entropy term to the objective function, the learning process not only pursues the maximization of the cumulative reward, but also encourages the policy output to have higher randomness, thereby enhancing the exploration and robustness of the policy. The overall network architecture is as Figure 1 shown

[0106] Under the established aircraft control network framework, the network backpropagation algorithm is studied, and aircraft control training is carried out in the built simulation training environment to achieve stable tracking control of the multi-channel attitude of the aircraft without decoupling. During the training process, the SAC algorithm adopted has two policy networks and four value networks. The policy network is represented by π, and the value network is described by φ. The value network (only runs in the training stage) uses the aircraft state and action information to obtain the evaluation value of taking an action for the current state, so it guides the update of the policy during the training stage. A complete training process of SAC includes a data collection process and a parameter training process:

[0107] Data collection process

[0108] During the data collection process, the aircraft observes the environment to obtain the current state S, which is sent into the policy network to output the corresponding operation A, and the reward value R and the next state S′ are obtained. After performing the above operations, the data will be stored in the following form in each iteration round:

[0109] (S,A,R,S′)(9)

[0110] This process requires a large number of rounds to obtain a sufficient number of experience vectors, which are all stored in the experience pool. This stage does not involve the value evaluation network and only includes the interaction between the policy network and the environment. To avoid the negative impact of policy correlation on the training effect, an on-policy sampling and updating mechanism is introduced. By gradually updating the policy during the interaction between the policy network and the environment, the policy network parameters are dynamically adjusted, making the data of each collection round independent and diverse. This policy update method not only improves the coverage and representativeness of the data in the experience pool but also enhances the robustness and generalization ability of model training.

[0111] Parameter update process:

[0112] The core idea of SAC is to expand the traditional expected cumulative reward maximization objective to an entropy-regularized objective by introducing a policy entropy term:

[0113]

[0114] where r(s t ,a t ) is the immediate reward, E π [] represents the expected value under the policy π,

[0115] is the entropy of the policy, measuring the randomness of the policy, α is the entropy temperature, a hyperparameter controlling the importance of entropy, and E at-π [] represents taking the expectation of the probability distribution of the action a according to the policy π, measuring the uncertainty of the action distribution of the policy π in the state s t . Through this objective function, the policy of SAC can not only effectively utilize the reward signal in the environment but also maintain sufficient randomness to explore possible optimal policies in complex environments.

[0116] During the training of the value network, the SAC algorithm updates the network parameters by minimizing the loss function L Q (θ i ):

[0117]

[0118] where represents the Q-value estimate given by the current value network, θ i represents the parameters of the current network, represents sampling the state, action, reward value, and next-state from the experience replay pool, and the target value y is generated by the target network, and its definition is as follows:

[0119]

[0120] where γ is the discount factor, a t+1 ′ is the action sampled from the policy network, which is the smaller Q value at future time.

[0121] The innovation of the SAC algorithm lies in simultaneously using two value networks to reduce the overestimation problem of Q values, and introducing entropy regularization to balance exploration and exploitation, thereby improving the robustness of the policy. In the training of the policy network, the SAC algorithm updates the policy network parameters by minimizing the following loss function:

[0122]

[0123] where: D is the sampling space of states, and E[g] represents the mean square error function.

[0124] In addition, the entropy temperature parameter α is updated; the loss function for defining and optimizing the temperature parameter α is:

[0125]

[0126] where represents sampling the action distribution, calculating the expected value of the sampled data, and -logπ φ (a t |s t ) represents the negative of the log probability of action a t in the policy distribution, which is used to measure the uncertainty of the policy, is the target entropy.

[0127] Since the update of the value network requires obtaining the current Q value and the Q′ value at the next moment, which makes the training data correlated and improves the network convergence speed, a target policy network and a target value network are designed, and the parameters are periodically copied by the method of soft update. The update method is:

[0128]

[0129] where θ′ are the parameters of the target policy network, θ are the parameters of the policy network, are the parameters of the two target value networks respectively, are the parameters of the two value networks respectively; τ is the soft update coefficient.

[0130] This embodiment conducts simulation verification on step 1:

[0131] Parameter setting;

[0132] Physical parameter setting of the aircraft: the dynamic pressure is q = 5189.62528, the characteristic area of the aircraft is S = 4.17, the characteristic length of the aircraft is L = 4.12, and the system sets the inertia parameter as I xx= 321, I yy = 4785, I zz = 4625, I xz = 0, the initial attitude angle is Θ0 = [α0, β0, σ0] T = [0.17 0.05 0.34] T rad, the initial attitude angular velocity is designed as: ω0 = [p0, q0, r0] T = [0 0 0] T rad. The reference attitude angle Θ d ~ Uniform(-0.5, 0.5) rad means that Θ d is a random number uniformly distributed between -0.5 and 0.5. During training, the control network output rudder deflection range is controlled within (-20°, 20°). The maximum number of time steps for each training episode is set to 3000 steps, and the duration of one time step is 0.005 s. The termination condition for each training episode is: the attitude error of the aircraft in three directions does not exceed 1° or the training steps reach the maximum time step. After multiple parameter adjustments, the training hyperparameters of the reinforcement learning algorithm are set as shown in Table 1.

[0133] Table 1 Hyperparameters of the robust adversarial reinforcement learning algorithm

[0134]

[0135] Analysis of simulation results

[0136] After continuous iterative training, the simulation results are obtained as Figures 2-8 shown. Among them, Figure 2 is the reward value curve obtained by the aircraft control network. It can be seen from the figure that as the number of iterations increases, the reward value becomes larger and larger, indicating that the control network has converged and has good control performance. Figure 3 represents the completion rate of every 50 episodes during the training process, that is, the proportion of episodes in which the steady-state error of each channel does not exceed 1° in 50 episodes. It can be seen that the completion rate is the highest at 98% when the abscissa is 10, and the control network at this time is saved as the final model. Figure 4 is the number of episode steps required to reach the control requirement accuracy during training. As the number of training times increases, the number of episode steps gradually decreases, and only 1400 steps are required to converge under the optimal control network.

[0137] Figure 5 is the attitude error curve for testing using the control network after offline training. It can be seen from the figure that the attitude error tends to be stable within 5 seconds and meets the control accuracy requirements. Subsequently, between 8 - 11 seconds, disturbances are added to the pitch, roll, and yaw channels respectively to verify the robustness of the control network. The disturbance magnitude is [50sin(2t), 50cos(t), -60sin(t)] T, it can be seen that the attitude error always remains stable. Finally, the three-channel tracking errors are (-0.0034, 0.0075, 0.0012) rad, verifying the effectiveness of the strongly robust aircraft attitude control method proposed in the present invention. Compared with the error curve Figure 6 without considering the dynamic performance in Figure 7 , there are significant improvements in both the overshoot and the settling time. Figure 8 It shows that the attitude angular velocity converges within 5 seconds. When an interference occurs at the 8th second, it always maintains a stable state, indicating that the robust control method proposed in the present invention can handle deviation interference well.

[0138] Specific implementation manner of step 2:

[0139] Based on the trained model, perform visual analysis using the t-SNE algorithm to achieve an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller.

[0140] The interpretability of the reinforcement learning model is one of the important factors restricting its application in the aerospace field. To improve the transparency and interpretability of the reinforcement learning model, the present invention conducts visual analysis of the network model from two aspects: First, considering that t-SNE (t-Distributed Stochastic Neighbor Embedding) is an effective visualization tool, by mapping high-dimensional data to two-dimensional or three-dimensional space and preserving the local structure between data points, we can intuitively understand the decision-making process of the model. In reinforcement learning, t-SNE can be used to visualize high-dimensional data such as the policy space, state space, and value function. By analyzing the distribution of different states or actions in the low-dimensional space, it helps us discover the changing trends or potential structural features of the policy. Second, considering that SHAP (SHAPley Additive Explanations) values provide an effective method to analyze and interpret the prediction features of the model, especially for deep reinforcement learning networks. By using SHAP values, the importance of different features in the decision-making process can be quantified, thereby improving the interpretability of the model. During the training process of reinforcement learning, the agent usually makes action decisions based on state features, and SHAP values can help us understand how each state feature affects the final decision result. By visualizing the SHAP values, we can intuitively understand which features contribute the most to the model output, thereby identifying potential influencing factors and further optimizing the design of the model. Therefore, the visualization of the aircraft reinforcement learning controller will be introduced from these two aspects below.

[0141] Visual analysis of network output t-SNE

[0142] t-SNE is a non-linear technique for dimensionality reduction of high-dimensional data, mainly used for visualizing complex high-dimensional data sets. It embeds high-dimensional data into a two- or three-dimensional space, keeping similar points close in the low-dimensional space, thus helping people observe and analyze the distribution and structure of data in the low-dimensional space. The core goal of t-SNE is to preserve the local structure of the data by minimizing the difference between two distributions.

[0143] After the model is trained, visualize the output of the model and extract the output of the last hidden layer of the evaluation network. Each data sample generates a corresponding high-dimensional feature vector at each time step, and these vectors represent the internal representation of the model for the input data at that time step. Since the dimensions of these vectors are usually high, it is very difficult to directly visualize these eigenvalues. Therefore, we need to reduce the dimensionality of these high-dimensional feature vectors to a two-dimensional space for subsequent visual analysis. For this purpose, t-SNE is used to map these high-dimensional features to a two-dimensional space, keeping the relative distances between similar data points while separating dissimilar data points. Then use the t-SNE algorithm to reduce the dimensionality to a two-dimensional space, as Figure 9 shown. There is no interference in the left figure and interference is added in the right figure. One episode includes 3000 time steps. Each point represents a high-dimensional data sample in a time step. The color of the point represents the size of the Q value. Dark colors indicate smaller Q values and light colors represent larger Q values. Through color mapping, we can intuitively see which data points have larger or smaller output values. The clusters in the figure represent the similarity between samples, and the distance between samples reflects their similarity in the high-dimensional space.

[0144] It can be observed from the figure that the steps with high Q values are clustered in the same area in the embedding space, indicating that the model has learned to represent states with similar Q values in the two-dimensional space. The path in the episode is represented by the continuous transition of colors, which indicates that the aircraft has experienced different states in one episode. And there is an obvious separation between the purple and yellow areas, indicating that the model makes a key decision, which is exactly the moment when the attitude error meets the conditions. In addition, in-depth understanding of the structure of the control strategy in the high-dimensional space through t-SNE visualization can help determine the direction of strategy improvement. For example, in the right figure, the points with low Q values are scattered among the high Q value (warm color) states, indicating that the system control error increases after adding perturbations, and then returns to the high Q value level with the adjustment of the control network. Figure 10 To map the color of the Q value to the attitude error curve, as time passes in the whole episode, the attitude error continuously decreases and the Q value increases accordingly. After adding perturbations, the attitude error increases and the corresponding color becomes darker. It shows that after adding perturbations, there is a color change, and the perturbations have an impact on attitude control, and finally the color indicates that stable control under perturbations is achieved.

[0145] Specific implementation of Step 3:

[0146] Based on SHAP value feature analysis, the importance of the influence of multiple hidden layers in the aircraft reinforcement learning controller on the network output (i.e., the rudder deflection in three directions) is sorted out, and the interpretability analysis of the aircraft neural network model is realized.

[0147] As machine learning models become increasingly complex, especially deep learning and ensemble learning models, the "black box" characteristics of the models lead to a lack of transparency in the decision-making process. Therefore, model interpretability methods have gradually become a research hotspot. SHAP value is a feature importance measurement method based on game theory, which can help reveal the decision-making mechanism of the model, thereby improving the interpretability of the model. The theoretical basis of the SHAP value stems from the Shapley value in game theory, which was first proposed by economist Lloyd Shapley to solve the problem of how to reasonably allocate benefits in cooperative games. In game theory, the Shapley value is used to evaluate the contribution of each participant to the overall benefit. In machine learning, features are regarded as "cooperative participants", and the prediction result of the model is the "total benefit", and the SHAP value is used to measure the independent contribution of each feature to the prediction result.

[0148] The process of calculating the SHAP value can be regarded as traversing all possible feature combinations to evaluate the marginal contribution of features to the model output. In the calculation process, it is necessary to consider the change of the model output when features are added in different orders, so that the average marginal contribution of features can be obtained, and finally the SHAP value of the feature can be obtained. The specific calculation formula is as follows:

[0149]

[0150] In the formula, S is the feature subset, N is the set of all features, and f(S) represents the model prediction value corresponding to the feature subset. By summing all possible feature combinations, the contribution of features can be comprehensively considered.

[0151] To explain the contribution of each layer of the network to the model output, visual analysis is carried out based on the SHAP value. The model in this section has 3 continuous action outputs, namely the rudder deflection in three directions, and the input is the attitude angle and angular velocity of the aircraft, with a total of 7 input features. For the explanation, all features are sorted according to their attribution to the action output, and then the two most important features are selected to explain the model prediction. First, the output of each layer of the network is extracted, and the SHAP value is calculated with the final model output. Then, the principal component analysis method is used to extract the most important SHAP value as the SHAP value of this layer of the network. Finally, the re-extracted SHAP values of each layer are concatenated to obtain Figure 11 .

[0152] In this set of SHAP value plots, each plot shows the influence of the outputs of each layer of the deep network (such as L1, L2, L3, L4, L5) on the final three-rudder deflection output of the model. The horizontal axis represents the SHAP value, reflecting the influence (positive or negative) of each feature on the model output; the vertical axis lists the network names, sorted by their importance to the model's influence (the higher the layer, the greater its influence on the output); the magnitude of the feature value is represented by color: red represents a larger feature value, and blue represents a smaller feature value. The distribution of points of different colors shows how the change in the feature value affects the SHAP value.

[0153] From Figure 11 it can be analyzed that:

[0154] From Figure 11 in terms of the structure, L3 appears as the most important feature in all three rudder deflection outputs. Its SHAP value distribution is relatively wide, indicating that its influence on the model prediction is significant and diverse; L2 and L1 also contribute significantly to the model, but their influence degree is lower than that of L3; the influence of L4 and L5 layers is smaller and the distribution is concentrated, indicating that their role in the prediction result is limited.

[0155] From the perspective of the distribution density and variation, L3 has a large number of data points in the entire SHAP value range, indicating that its influence covers most samples, suggesting that the contribution of L3 to the output result is global. The distribution of L3 is not only wide, but also the sample density within the SHAP value range is higher. While the extreme values of L2 have a greater impact, but the overall contribution is less. The length of its distribution is caused by the outliers of a few samples, indicating that most samples are significantly affected by L3 rather than being dominated by a few samples.

[0156] From the color distribution, both red and blue (high and low feature values) in L3 have strong manifestations in both positive and negative SHAP value ranges, indicating that it significantly affects the output at different values. Specifically, the high-value features are concentrated in the negative SHAP value region, indicating that its high-value features reduce the final rudder deflection output, and the low-value features are concentrated in the positive SHAP value region, indicating that its low-value features increase the final rudder deflection output.

[0157] In summary, it can be seen that the third hidden layer has the greatest influence on the system output, followed by the second and first layers. Therefore, during the change of the aircraft state or the online optimization process, focusing on optimizing the network parameters of these three layers can change the control performance of the system.

[0158] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. The structures, devices, and operation methods not specifically described and explained in the present invention are implemented by conventional means in the art without special instructions and limitations.

Claims

1. Aircraft attitude control method based on interpretable reinforcement learning, characterized in that: It includes the following steps: Step 1: Establish a Markov decision model for aircraft attitude control and design an offline attitude controller for the aircraft. In the above Step 1, establish a Markov model including a state set, an action set, a reward function, a state transition probability, and a discount factor as follows: The state set: Taking the three-channel attitude error, attitude angle, and attitude angular velocity obtained by the aircraft using sensors as the state, the state set S is defined as: S = {e1, Θ, ω}; where, Θ = [α, β, σ] T is the three-channel attitude angle, where α is the angle of attack, β is the sideslip angle, σ is the bank angle, and ω = [p, q, r] T are the angular velocities of the roll, pitch, and yaw channels respectively, and e1 = Θ - Θ d is the aircraft attitude tracking error; The action set: Considering the requirements of the aircraft's stable tracking control and based on the aircraft attitude control model, the action set includes the control rudder deflections in three directions of the aircraft, thereby controlling the network action set A c It is defined as: Where δ ψ , δ γ are the control rudder deflections in the pitch, yaw, and roll directions respectively; For the above reward function, establish a reward and punishment mechanism, including: Error between the current attitude and the desired attitude: where e 1i represents the attitude tracking errors in the roll, pitch, and yaw directions; Rudder deflection smoothing reward: Where X is the input three-dimensional array with n samples and m dimensions, and R smooth The smaller it is, the higher the smoothness of the data; Dynamic performance: as shown in the following formula: In the formula, the overshoots in three directions are defined, and α d , β d , σ d are the desired attitudes in three directions. The overshoots in three directions are averaged to obtain the total overshoot: The settling time is defined as the time when the sequence first enters the 2% error band and remains within it, as shown in the following formula: The settling times in three directions are defined, and the settling times in three directions are averaged to obtain the total settling time: The dynamic performance reward is defined as: R D = -0.05·OS avg - 0.1T set ; Considering the above three indicators, the single-step decision reward value R of the aircraft is: R = R e + R smooth + R D ; For the above state transition probability: for the aircraft attitude control problem, regard the attitude control model of the aircraft as the state transition function; that is, the probability of reaching the next moment state S′ after executing the action A under the environmental state S. Discount factor: The discount factor γ represents the importance of the future benefit value relative to the current benefit value; when γ = 0, it is equivalent to only considering the current benefit and not considering the future benefit, and when γ = 1, the future benefit and the current benefit are regarded as equally important. By defining the Markov model of aircraft control, the aircraft control process can be redescribed as follows: for the state set s(t) composed of the aircraft attitude angle and angular velocity obtained by the sensor at time t of the aircraft, according to the designed immediate reward value R i (t), the corresponding optimal action A c (t) is determined on the premise of the highest reward, and interacts with the environment. The aircraft state s(t + 1) at the next moment is obtained by the state transition function, and this process is repeated until the goal of forming the desired stable tracking control is achieved; Step 2: According to the trained Markov decision model, perform visual analysis based on the t-SNE algorithm to realize an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller. Step 3: Based on the SHAP value feature analysis, complete the sorting of the influence of multiple hidden layers in the aircraft reinforcement learning controller on the network output, and realize the interpretability analysis of the aircraft neural network model.

2. The aircraft attitude control method based on interpretable reinforcement learning according to claim 1, wherein: In the above Step 1, under the established aircraft control network framework, study the network backpropagation algorithm, and perform aircraft control training in the built simulation training environment to realize the stable tracking control of the aircraft multi-channel attitude without decoupling; during the training process, the SAC algorithm used has two policy networks and four value networks, the policy network is represented by π, and the value network is described by φ; the value network uses the aircraft state and action information to obtain the evaluation value of taking the action for the current state.

3. The aircraft attitude control method based on interpretable reinforcement learning according to claim 1, characterized in that: In the above Step 1, guide and update the policy during the training stage; a complete training process of the SAC algorithm includes a data collection process and a parameter training process: Data collection process: The aircraft observes the environment to obtain the current state S, sends it into the policy network to output the corresponding operation A, obtains the reward value R and the next state S′; after performing the above operations, each iteration round will store the data in the following form: (S, A, R, S′). Parameter training process: The SAC algorithm expands the traditional expected cumulative reward maximization target into an entropy regularization target by introducing a policy entropy term: where \(r(s t ,a t )\) is the immediate reward, denotes the expected value under policy \(\pi\), is the entropy of the policy, measuring the randomness of the policy, \(\alpha\) is the entropy temperature, a hyperparameter controlling the importance of entropy, \(E at-π []\) denotes taking the expectation with respect to the probability distribution of action \(a\) according to policy \(\pi\), measuring the uncertainty of the action distribution of policy \(\pi\) in state \(s t \). During the process of value network training, the SAC algorithm updates the network parameters by minimizing the loss function L Q (θ i ): where represents the Q-value estimate given by the current value network, and θ i represents the parameters of the current network, represents sampling the state, action, reward value, and next-state from the experience replay pool, and the target value y is generated by the target network, which is defined as follows: where γ is the discount factor, a t+1 ′ is the action sampled from the policy network, is the smaller Q value at future time; At the same time, two value networks are used to reduce the overestimation problem of the Q value, and entropy regularization is introduced to balance exploration and exploitation. In the training of the policy network, the SAC algorithm updates the policy network parameters by minimizing the following loss function: In the formula: D is the sampling space of the state, and E[·] represents the mean square error function; In addition, update the entropy temperature parameter α, and define the loss function for optimizing the temperature parameter α as: wherein, denotes sampling the action distribution, calculating the expected value of the sampled data, and -logπ φ (a t |s t ) represents the negative value of the log probability of action a t in the policy distribution, which is used to measure the uncertainty of the policy, is the target entropy; Since the update of the value network requires obtaining the current Q value and the next moment Q′ value, design a target policy network and a target value network, and regularly copy the parameters by the method of soft update. The update method is: θ′←τθ+(1-τ)θ′ where $\theta'$ are the parameters of the target policy network, $\theta$ are the parameters of the policy network, are the parameters of two target value networks respectively, are the parameters of two value networks respectively, and $\tau$ is the soft update coefficient.

4. The aircraft attitude control method based on interpretable reinforcement learning according to claim 1, characterized in that: In step 2, network output t-SNE visualization analysis is performed: After the model is trained, the output of the model is visualized, and the output of the last hidden layer of the evaluation network is extracted. Each data sample generates a corresponding high-dimensional feature vector at each time step, and these vectors represent the internal representation of the model for the input data at that time step; these high-dimensional feature vectors are reduced to two-dimensional space; for this purpose, t-SNE is used to map these high-dimensional features to two-dimensional space, maintaining the relative distance between similar data points while separating dissimilar data points; then the t-SNE algorithm is used to reduce the dimension to two-dimensional space to complete the visualization analysis and achieve an intuitive representation of the learning performance of the aircraft reinforcement learning neural network controller.

5. The aircraft attitude control method based on interpretable reinforcement learning according to claim 1, characterized in that: In step 3, the process of calculating SHAP values can be regarded as traversing all possible feature combinations to evaluate the marginal contribution of features to the model output; during the calculation process, it is necessary to consider the change in the model output when features are added in different orders, so as to obtain the average marginal contribution of the features and finally obtain the SHAP value of the feature; the specific calculation formula is: Where, φ i is the SHAP value of feature i, S is the feature subset, N is the set of all features, |S| is the number of features in subset S, |N| is the number of features in the full feature set, |S|!(|N| - |S| - 1)! is the product of the number of permutations of subset S and the number of permutations of the remaining features, which is used to calculate the combination weights of feature i in different positions, |N|! is the number of permutations of the full feature set, which is used to normalize the weights, f(S ∪ {i}) represents the output value of the model when using subset S and feature i together for prediction, and f(S) represents the model prediction value corresponding to the feature subset.

Images