Mpc parameter adaptive straight air compound control method based on deep reinforcement learning

Abstract

The application provides an MPC parameter self-adaptive direct-gas composite control method based on deep reinforcement learning, and specifically comprises the following steps: designing a nominal MPC controller according to a nominal dynamics model, and designing nominal values of a Q matrix and an R matrix; constructing an adaptive controller architecture based on the nominal MPC controller and a deep neural network, and adjusting a state weight matrix Q and a control weight matrix R of the MPC controller according to an output of the deep neural network; constructing an aircraft simulation environment containing random bias of dynamics parameters, and designing a reward function; training the deep neural network based on the same strategy reinforcement learning algorithm according to the adaptive controller architecture and the aircraft simulation environment, and obtaining a control strategy for maximizing the flight index. The application can solve the problem that the model predictive control with fixed weight parameters in the prior art cannot be applied to the case where there is large uncertainty in aerodynamic characteristics.

Description

Technical Field

[0001] This invention belongs to the field of aircraft guidance and control technology, and particularly relates to an MPC parameter adaptive direct air composite control method based on deep reinforcement learning. Background Technology

[0002] Aircraft typically use thrust reverser control systems to assist aerodynamic control surfaces in attitude control during flight phases when aerodynamic forces are insufficient. This control method is a combination of direct force and aerodynamic control, essentially a multi-actuator combined and coordinated control system. Existing control methods generally do not treat the multi-actuator composite system as a unified multi-input system, ignoring the interaction between actuators. This leads to certain losses when dissimilar control variables are coordinated, and makes energy control difficult.

[0003] Model predictive control (MPC) uses an explicit multi-input multi-output dynamic model as the basis for prediction. This model naturally describes the coupling relationships between all input and output variables, making MPC one of the effective methods for handling control problems of multi-input multi-output constrained systems. However, when faced with significant uncertainties in aerodynamic characteristics, MPC using fixed-weight parameters will continuously optimize based on a biased model, leading to a decrease in controller performance. At the execution level, the relationship between aerodynamic control and RCS control effectiveness changes with dynamic pressure, and its optimal allocation ratio is time-varying. Fixed control weights cannot adapt to this change, resulting in reduced controller effectiveness. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0005] This invention provides an adaptive direct-air composite control method for MPC parameters based on deep reinforcement learning. This method includes:

[0006] Design a nominal MPC controller based on the nominal dynamic model, and design the nominal values ​​of the Q and R matrices; wherein, in the nominal MPC controller, the angle of attack is selected as the state variable of the controlled object. and pitch angular velocity Select the control variable as the rudder deflection angle. and RCS control quantity The nominal MPC control model is:

[0007]

[0008] In the above formula, This is the predicted value of the state vector at time k+j. This is the reference value at time k+j; Let J be the state weight matrix at time j. , Reflects the importance of the magnitude of the state variable. For matrix Factors on the diagonal; Let J be the control weight matrix at time j. , reflect The allocation of control variables and aerodynamic control variables. For matrix Factors on the diagonal; This represents the control quantity at time k+j-1; Indicates the prediction time domain; This indicates finding in the prediction time domain that... The control quantity sequence is the minimum value; constraint (1) represents the discrete prediction model. These represent the state variables at times k+1 and k, respectively. These represent the control quantity and output quantity at time k, respectively; , , All are state equation matrices; constraints (3) The number of steps taken;

[0009] An adaptive controller architecture is constructed based on a nominal MPC controller and a deep neural network. The state weight matrix Q and control weight matrix R of the MPC controller are adjusted according to the output of the deep neural network. The deep neural network is constructed using an Actor-Critic structure: the network input consists of the state parameters of the controlled object, and the Actor is the scaling factor generated from each factor on the diagonal of the MPC's Q and R matrices. , The neural network adopts a fully connected structure. The output of each layer of the Actor and Critic network is normalized, and the network weights are initialized using an orthogonal initialization method.

[0010] Construct a simulation environment for an aircraft with random biasing dynamic parameters and design a reward function;

[0011] Based on the adaptive controller architecture and the aircraft simulation environment, a deep neural network is trained using a similar policy reinforcement learning algorithm to obtain a control strategy that maximizes flight performance.

[0012] Furthermore, the aircraft simulation environment is a six-degree-of-freedom dynamic model of the aircraft. The model inputs are rudder control signals and RCS control signals, and the output is the aircraft state information.

[0013] Furthermore, the reward function is designed as follows:

[0014]

[0015] in, Normalization refers to controlling the corresponding variable within a certain fixed range of values. This represents the angular error, and the related item is the tracking reward. The relevant item is a stable reward; and This represents the normalized output of the actuator; The integral term represents the error, and the correlation term represents the smoothing reward; parameters to The scale coefficients for tracking reward, stability reward, aerodynamic energy reward, RCS energy reward, and smoothing reward respectively determine the importance percentage of the corresponding reward in the total reward; to The sensitivity coefficients are for tracking reward, stability reward, air rudder energy reward, RCS energy reward, and smoothing reward, respectively, which determine the degree of precision the network needs to control its state or actions to obtain rewards.

[0016] Furthermore, during training, the update period of the MPC controller weight matrix is ​​set to be the same as the RCS control period.

[0017] Furthermore, during training, the MPC controller weight matrix is ​​updated as follows:

[0018] ,

[0019] The MPC controller calculates the control quantity based on the weight matrix and the nominal MPC control model. and The reinforcement learning strategy experience is stored at the beginning and end of each RCS control cycle.

[0020] This invention provides an adaptive direct-aerodynamic composite control method based on deep reinforcement learning (MPC). This method designs a nominal MPC controller and, in scenarios with significant uncertainties in dynamic coefficients, constructs an adaptive controller architecture using a deep neural network. It also builds an aircraft simulation environment with randomly biased dynamic parameters, ultimately obtaining a control strategy that maximizes flight performance. This method is applicable to composite control scenarios with significant uncertainties in dynamic characteristics and where fuel consumption constraints must be considered, thus enhancing system robustness. Compared to existing technologies, this invention addresses the limitation of model predictive control with fixed weight parameters in situations with significant uncertainties in aerodynamic characteristics. Attached Figure Description

[0021] The accompanying drawings, which form part of this specification, are provided to further illustrate embodiments of the invention and, together with the textual description, explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0022] Figure 1 A schematic diagram of a nominal MPC controller provided according to a specific embodiment of the present invention is shown;

[0023] Figure 2 A schematic diagram of an adaptive controller architecture provided according to a specific embodiment of the present invention is shown;

[0024] Figure 3 , Figure 4 , Figure 5 and Figure 6 The dynamic parameters are shown respectively. and The chart compares the performance of adaptive MPC control with that of traditional MPC in terms of angle of attack, pitch rate, RCS output, and aerodynamic control output, assuming nominal values.

[0025] Figure 7 , Figure 8 , Figure 9 and Figure 10 The dynamic parameters are shown respectively. Pulled off by -30% and A comparison chart of adaptive MPC control performance and traditional MPC in terms of angle of attack, pitch rate, RCS output and aerodynamic control output when the yaw rate is -30%;

[0026] Figure 11 , Figure 12 , Figure 13 and Figure 14 The dynamic parameters are shown respectively. Pulled off by 30% and A comparison chart of adaptive MPC control performance and traditional MPC in terms of angle of attack, pitch rate, RCS output and aerodynamic control output when the yaw is increased by 30%;

[0027] Figure 15 , Figure 16 , Figure 17 and Figure 18 The dynamic parameters are shown respectively. Pulled off by -30% and A comparison chart of adaptive MPC control performance and traditional MPC in terms of angle of attack, pitch rate, RCS output and aerodynamic control output when the yaw is increased by 30%;

[0028] Figure 19 , Figure 20 , Figure 21 and Figure 22 The dynamic parameters are shown respectively. Pulled off by 30% and A comparison chart of the adaptive MPC control performance and the traditional MPC in terms of angle of attack, pitch rate, RCS output, and aerodynamic rudder output when the yaw rate is -30%. Detailed Implementation

[0029] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0032] According to a specific embodiment of the present invention, an adaptive direct-air composite control method for MPC parameters based on deep reinforcement learning is provided, the method specifically including:

[0033] Design a nominal MPC controller based on the nominal dynamics model, and design the nominal values ​​of the Q matrix and R matrix;

[0034] Based on the nominal MPC controller and deep neural network, an adaptive controller architecture is constructed, and the state weight matrix Q and control weight matrix R of the MPC controller are adjusted according to the output of the deep neural network.

[0035] Construct a simulation environment for an aircraft with random biasing dynamic parameters and design a reward function;

[0036] Based on the adaptive controller architecture and the aircraft simulation environment, a deep neural network is trained using a similar policy reinforcement learning algorithm to obtain a control strategy that maximizes flight performance.

[0037] This configuration approach provides a deep reinforcement learning-based adaptive direct-air composite control method for MPC parameters. The method designs a nominal MPC controller and, in scenarios with significant uncertainties in dynamic coefficients, constructs an adaptive controller architecture using a deep neural network. It also builds an aircraft simulation environment with randomly biased dynamic parameters, ultimately obtaining a control strategy that maximizes flight performance. This method is applicable to composite control scenarios with significant uncertainties in dynamic characteristics and where fuel consumption constraints must be considered, thus enhancing the system's robustness.

[0038] Deep reinforcement learning technology is suitable for solving complex and difficult-to-model application scenarios. Research on applying this technology to improve the autonomy and intelligence of flight control in the field of aircraft dynamics and control has attracted much attention. Applying this technology to improve the control quality of high-speed aircraft is a new technical approach.

[0039] First, in this invention, a nominal MPC controller is designed based on a nominal dynamic model, and the nominal values ​​of the state weight matrix Q and the control weight matrix R are designed.

[0040] As a specific embodiment of the present invention, the state variable of the controlled object is selected as the angle of attack. and pitch angular velocity Select the control variable as the rudder deflection angle. and RCS control quantity The air rudder control cycle is The RCS control cycle is , .

[0041] Establish the following nominal MPC control model:

[0042]

[0043] In the above formula, This is the predicted value of the state vector at time k+j. This is the reference value at time k+j; Let J be the state weight matrix at time j. , Reflects the importance of the magnitude of the state variable. For matrix Factors on the diagonal; Let J be the control weight matrix at time j. , reflect The allocation of control variables and aerodynamic control variables. For matrix Factors on the diagonal; This represents the control quantity at time k+j-1; Indicates the prediction time domain; This indicates finding in the prediction time domain that... The control quantity sequence is the minimum value.

[0044] Constraint (1) represents the discrete prediction model. These represent the state variables at times k+1 and k, respectively. These represent the control quantity and output quantity at time k, respectively; , , All are state equation matrices.

[0045] ,

[0046] Indicates pitching moment pitch angular velocity The partial derivatives, express angle of attack The partial derivatives, Indicates lift angle of attack The partial derivatives, express right Partial derivatives, express right The partial derivatives, Indicates lift Partial derivative with respect to rudder deflection angle; This represents the moment of inertia of the pitch channel. Indicates quality, Indicates flight speed.

[0047] In constraint (3) The number of steps is the number of runs, indicating that only up to... The controller updates the control input only after the control cycle has elapsed. The control law is obtained by solving the quadratic programming problem above.

[0048] In constraint (1) and constraint (3), “()” represents variables and conditions, and “[]” in constraint (2) and constraint (3) represents array positions and value ranges.

[0049] Furthermore, after completing the design of the nominal MPC controller, an adaptive controller architecture is constructed based on the nominal MPC controller and the deep neural network. The state weight matrix Q and control weight matrix R of the MPC controller are adjusted according to the output of the deep neural network.

[0050] As a specific embodiment of the present invention, a deep neural network is constructed using an Actor-Critic structure. The network input is the state parameters of the controlled object, and the Actor is the scaling factor generated from each factor on the diagonal of the Q and R matrices of the MPC. , The neural network adopts a fully connected structure. The output of each layer of the Actor and Critic networks is normalized, and the network weights are initialized using an orthogonal initialization method.

[0051] Furthermore, after constructing the adaptive controller architecture, a simulation environment for the aircraft with random biasing of dynamic parameters is built, and a reward function is designed.

[0052] In a specific embodiment of the present invention, the aircraft simulation environment is a six-degree-of-freedom dynamic model of the aircraft. The model inputs are rudder control signals and RCS control signals, and the outputs are aircraft state information, mainly including angle information and angular velocity information, to determine the range of parameter deviations within the model. The reward function is designed as follows:

[0053]

[0054] In the reward function, Normalization refers to controlling the corresponding variable within a certain fixed range of values. This represents the angular error, and the related item is the tracking reward. The relevant item is a stable reward; and This represents the normalized output of the actuator; The integral term represents the error, and the correlation term represents the smoothing reward; parameters to The scale coefficients for tracking reward, stability reward, aerodynamic energy reward, RCS energy reward, and smoothing reward respectively determine the importance percentage of the corresponding reward in the total reward; to The sensitivity coefficients are for tracking reward, stability reward, air rudder energy reward, RCS energy reward, and smoothing reward, respectively, which determine the degree of precision the network needs to control its state or actions to obtain rewards.

[0055] Furthermore, after constructing an aircraft simulation environment with random yaw of dynamic parameters, the neural network is trained based on the same-policy reinforcement learning algorithm to obtain a control strategy that maximizes flight performance.

[0056] In a specific embodiment of the present invention, the update period of the MPC controller weight matrix is ​​set to be the same as the RCS control period. During training, the MPC controller weight matrix is ​​updated as follows:

[0057] ,

[0058] The MPC controller calculates the control quantity based on the weight matrix mentioned above and the nominal MPC control model from the preceding steps. and The reinforcement learning strategy experience is stored at the beginning and end of each RCS control cycle.

[0059] This invention is based on an MPC controller and combines the powerful fitting ability of neural networks to dynamically adjust the controller state weights according to the state parameters of the controlled object. Matrix, control weights A matrix is ​​used to enable adaptive control parameters and enhance system robustness.

[0060] The advantages of this invention compared to the prior art are:

[0061] (1) Based on the MPC controller, the present invention regards the attitude control direct air composite control system as a unified multi-input system, uses an explicit multi-output multi-input dynamic model as the prediction basis, describes the coupling relationship between input variables and all output variables, includes equality and inequality constraints in the optimization model, directly considers the interaction between actuators in the online optimization process, reduces the loss when heterogeneous control variables are matched, and improves the composite control performance.

[0062] (2) By combining deep neural networks, a corresponding scaling factor is generated for each factor in the Q and R matrices of the MPC controller. Addressing model uncertainty, the state weight Q matrix is ​​dynamically adjusted, and the optimization objective is reconstructed to proactively compensate for model defects, thereby enhancing the system's robustness without modifying the prediction model itself. The distribution optimization of control quantities among heterogeneous actuators is achieved by adjusting the control weight R matrix.

[0063] (3) This invention utilizes the advantages of MPC controller in handling multi-input multi-output systems and the powerful fitting ability of deep neural networks to improve the adaptive ability of the Q matrix and R matrix of MPC controller in the face of complex uncertainties. It realizes the cross-fusion and complementary advantages of traditional control methods and deep reinforcement learning technology, and provides a new solution for attitude control and direct air composite control scenarios.

[0064] To gain a further understanding of the present invention, the MPC parameter adaptive direct-air composite control method based on deep reinforcement learning will be described in detail below with reference to specific embodiments.

[0065] According to specific embodiments of the present invention, an adaptive direct-air composite control method for MPC parameters based on deep reinforcement learning is provided, such as... Figure 1 As shown, a simulation environment for an aircraft with randomly biased model parameters is constructed. Addressing uncertainty, a deep neural network generates a corresponding scaling factor for each factor in the state weight Q matrix and control weight R matrix of the MPC. Without modifying the prediction model itself, it actively compensates for model bias by dynamically adjusting the Q and R matrices, improving system robustness and providing a novel control scheme for aircraft direct-air composite control scenarios.

[0066] The MPC parameter adaptive direct-air composite control method based on deep reinforcement learning in this embodiment specifically includes the following steps.

[0067] Step 1: Design a nominal MPC controller based on the nominal dynamics model, and design the nominal values ​​of the Q matrix and R matrix.

[0068] This embodiment uses aircraft pitch channel attitude control as a specific example, and the fixed parameter MPC controller is designed as follows: Figure 1 As shown. Pitch rate and angle of attack are selected as state variables. The control quantity is , , The aerodynamic control cycle is 5ms, and the RCS control cycle is 100ms. The MPC controller parameters are shown in the table below.

[0069] Table 1 Fixed Parameter MPC Controller

[0070]

[0071] Step 2: Construct an adaptive controller architecture based on the nominal MPC controller and deep neural network.

[0072] Adaptive parameter MPC controller design, such as Figure 2As shown. The PPO-Clip algorithm was selected as the training algorithm. The Actor and Critic were designed with 3 layers, each with 128 nodes. The Actor input is... , representing angle error, pitch rate, error integral, RCS of the previous cycle, and flight motion pressure, respectively. The Actor output is: , , and , respectively corresponding The sum of the diagonal elements of the matrix The diagonal elements of the matrix are shown in the following formula.

[0073]

[0074] Step 3: Construct an aircraft simulation environment with randomly biased dynamic parameters and design a reward function.

[0075] The aircraft simulation environment is a longitudinal dynamics model of the aircraft, as detailed in state equation matrices A, B, and C in Table 1. and Perform random biasing, with the biasing parameter range being: The reward function is shown in the following formula:

[0076]

[0077] The main parameters are shown in the table below.

[0078] Table 2 Reward Function Parameters

[0079]

[0080] Step four: Train the neural network using a similar policy reinforcement learning algorithm to obtain a control strategy that maximizes flight performance. The specific algorithm training process is as follows:

[0081] 1) Initialize the policy network and value network parameters orthogonally;

[0082] 2) Set the algorithm hyperparameters:

[0083] Table 3 Algorithm Hyperparameters

[0084]

[0085] 3) for episode in range(episodes);

[0086] 4) for t=1,T do;

[0087] 5) Determine whether the RCS control cycle has been reached based on the number of simulation steps. If it is, update the adjustment factor of the MPC controller weighting matrix; otherwise, retain the previous result.

[0088] 6) The MPC controller calculates the control quantity based on the weight matrix and the nominal prediction model. and The environmental model is pushed forward to advance the simulation time. The reinforcement learning strategy experience is stored in each RCS control cycle;

[0089] 7) end for, corresponding to step 4).

[0090] 8) end for, corresponding to step 3).

[0091] In one embodiment of the present invention, the trained controller can achieve stable control under random biasing of dynamic coefficients, and the control effect is as follows: Figures 3 to 22 As shown.

[0092] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A direct-air composite control method for MPC parameters based on deep reinforcement learning, characterized in that, The MPC parameter adaptive direct-air composite control method based on deep reinforcement learning includes: Design a nominal MPC controller based on the nominal dynamic model, and design the nominal values ​​of the Q and R matrices; wherein, in the nominal MPC controller, the angle of attack is selected as the state variable of the controlled object. and pitch angular velocity Select the control variable as the rudder deflection angle. and RCS control quantity The nominal MPC control model is: In the above formula, This is the predicted value of the state vector at time k+j. This is the reference value at time k+j; Let J be the state weight matrix at time j. , Reflects the importance of the magnitude of the state variable. For matrix Factors on the diagonal; Let J be the control weight matrix at time j. , reflect The allocation of control variables and aerodynamic control variables. For matrix Factors on the diagonal; This represents the control quantity at time k+j-1; Indicates the prediction time domain; This indicates finding in the prediction time domain that... The control quantity sequence is the minimum value; constraint (1) represents the discrete prediction model. These represent the state variables at times k+1 and k, respectively. These represent the control quantity and output quantity at time k, respectively; , , All are state equation matrices; constraints (3) The number of steps taken; An adaptive controller architecture is constructed based on a nominal MPC controller and a deep neural network. The state weight matrix Q and control weight matrix R of the MPC controller are adjusted according to the output of the deep neural network. The deep neural network is constructed using an Actor-Critic structure: the network input consists of the state parameters of the controlled object, and the Actor is the scaling factor generated from each factor on the diagonal of the MPC's Q and R matrices. , The neural network adopts a fully connected structure. The output of each layer of the Actor and Critic network is normalized, and the network weights are initialized using an orthogonal initialization method. Construct a simulation environment for an aircraft with random biasing dynamic parameters and design a reward function; Based on the adaptive controller architecture and the aircraft simulation environment, a deep neural network is trained using a similar policy reinforcement learning algorithm to obtain a control strategy that maximizes flight performance.

2. The MPC parameter adaptive direct-air composite control method based on deep reinforcement learning according to claim 1, characterized in that, The aircraft simulation environment is a six-degree-of-freedom dynamic model of the aircraft. The model inputs are rudder control signals and RCS control signals, and the output is aircraft state information.

3. The MPC parameter adaptive direct-air composite control method based on deep reinforcement learning according to claim 1, characterized in that, The reward function is designed as follows: ， in, Normalization refers to controlling the corresponding variable within a certain fixed range of values. This represents the angular error, and the related item is the tracking reward. The relevant item is a stable reward; and This represents the normalized output of the actuator; The integral term represents the error, and the correlation term represents the smoothing reward; parameters to The scale coefficients for tracking reward, stability reward, aerodynamic energy reward, RCS energy reward, and smoothing reward respectively determine the importance percentage of the corresponding reward in the total reward; to The sensitivity coefficients are for tracking reward, stability reward, air rudder energy reward, RCS energy reward, and smoothing reward, respectively, which determine the degree of precision the network needs to control its state or actions to obtain rewards.

4. The MPC parameter adaptive direct-air composite control method based on deep reinforcement learning according to claim 1, characterized in that, During training, the update period of the MPC controller weight matrix is ​​set to be the same as the RCS control period.

5. The MPC parameter adaptive direct-air composite control method based on deep reinforcement learning according to claim 4, characterized in that, During training, the MPC controller weight matrix is ​​updated as follows: ， ， The MPC controller calculates the control quantity based on the weight matrix and the nominal MPC control model. and The reinforcement learning strategy experience is stored at the beginning and end of each RCS control cycle.

Images