A physical constraint layered reinforcement learning hybrid electric aircraft energy management method
By employing a physical constraint hierarchical reinforcement learning method, combined with residual neural networks and robust model predictive control, the energy management problem of hybrid electric propulsion aircraft in dynamic environments was solved, achieving online adaptive optimization and safe control of energy allocation in uncertain environments.
Patent Information
- Application Number
- CN202611049518.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing energy management methods for hybrid electric propulsion aircraft struggle to reliably meet fuel economy optimization, physical consistency maintenance, and hard safety constraints in dynamic and uncertain environments. Traditional logical rules lack adaptability, dynamic programming methods require prior knowledge of flight process information, and model predictive control suffers from decreased control accuracy under unmodeled environmental disturbances.
A hierarchical reinforcement learning approach based on physical constraints is adopted. By compensating for model bias through residual neural networks and combining risk-sensitive reinforcement learning with robust model predictive control, the probability of constraint violation and prediction uncertainty are monitored in real time, and a safety control signal is generated to ensure that energy is allocated within the physical hard constraints.
It achieves adaptive energy management that balances long-term fuel economy and instantaneous safety in dynamic and uncertain environments, improving the energy management safety and reliability of hybrid aircraft in complex environments.
Smart Images

Figure CN123116771A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of energy management technology, specifically to a hybrid electric vehicle energy management method based on physical constraint hierarchical reinforcement learning. Background Technology
[0002] Hybrid electric propulsion aircraft, by integrating traditional fuel power and battery power, can significantly reduce fuel consumption and emissions, and is one of the important development directions in the current aviation field. The emergence of new aircraft configurations such as distributed electric propulsion and vertical takeoff and landing further puts forward higher requirements for the real-time performance, adaptability and safety of energy management systems. As one of the core technologies of hybrid electric propulsion aircraft, the task of energy management strategy is to rationally allocate the proportion of fuel power and electric power use throughout the flight process, so as to achieve comprehensive optimization of fuel economy, range and component life under the premise of meeting flight performance and safety constraints.
[0003] In existing energy management technologies for hybrid electric propulsion aircraft, determining power allocation schemes through preset logic rules or fuzzy logic controllers offers advantages such as simple computation and good real-time performance. However, these methods typically rely on quasi-static assumptions, making it difficult to achieve reasonable power allocation under high-frequency transient conditions such as wind disturbances and ship wakes. Furthermore, fixed logic rules lack adaptability and cannot be adjusted online according to changes in the flight environment and mission requirements, often resulting in fuel economy falling short of theoretical optimality. To address the limitations of rule-based methods, researchers have proposed optimization-based energy management strategies. Dynamic programming methods can obtain globally optimal power allocation schemes, but require complete prior knowledge of the entire flight profile, making real-time application impossible in dynamically changing mission scenarios. Model predictive control, as a rolling time-domain optimization method, can solve optimal control problems in a finite time domain online and explicitly handle states and constraints. However, its performance is highly dependent on model accuracy, and control accuracy significantly decreases when encountering unmodeled environmental disturbances such as ship wakes and urban canyon winds. Moreover, nonlinear model predictive control has a heavy online computational burden, making it difficult to achieve high-frequency real-time control with limited onboard computing resources. Summary of the Invention
[0004] To achieve highly adaptable, physically consistent, and safe energy management in dynamic and uncertain environments, while simultaneously ensuring long-term fuel economy optimization, physical consistency maintenance, and reliable satisfaction of hard safety constraints, this invention proposes a hybrid electric vehicle energy management method based on hierarchical reinforcement learning of physical constraints, comprising the following steps: S1: Obtain real-time information on aircraft motion and energy storage status, environmental disturbance information, and uncertainty covariance estimates used to characterize model mismatch and environmental randomness, and input them into the residual neural network; S2: Obtain the random residual correction amount, represented in the form of a probability distribution, from the output of the residual neural network. The random residual correction amount includes a mean term for compensating for model errors and a prediction uncertainty for quantifying the reliability of the correction. S3: Based on the current state information and the random residual correction, a risk-sensitive reinforcement learning strategy is adopted for high-level planning, with the goal of maximizing the cumulative reward of introducing risk-averse parameters. An adaptive Lagrange multiplier method is used to handle safety constraints, and the output includes decision instructions containing reference trajectories and energy allocation coefficients. S4: Input the decision command into the low-level robust model prediction and control layer, use the prediction uncertainty to perform forward recursion, calculate the constraint violation probability, and track the reference trajectory and generate the initial control signal under the premise of satisfying the physical hard constraints. S5: Construct consistency loss based on the deviation between decision instructions and initial control signals, and use consistency loss to correct the optimization objective of the reinforcement learning strategy; S6: Real-time monitoring of two indicators: constraint violation probability and prediction uncertainty. When either indicator exceeds the safety threshold, a safety control signal is output to ensure the basic stability of the aircraft; otherwise, a preliminary control signal is output.
[0005] This invention achieves adaptive energy management for hybrid electric vehicles in dynamic and uncertain environments by using residual networks to compensate for model biases online and quantify prediction reliability, risk-sensitive reinforcement learning to achieve safe decision-making and long-term energy optimization under uncertainty, robust MPC to ensure the execution of transient hard constraints, consistency loss to reduce planning and execution biases, and a safety backoff mechanism to provide fallback protection for extreme conditions.
[0006] Furthermore, in step S1, the aircraft motion and energy storage status information includes at least: the aircraft's three-dimensional position, velocity, attitude angle, battery state of charge, and remaining fuel mass; The environmental disturbance information includes at least wind field information; The value of the uncertainty covariance estimate increases with the increase of the aircraft velocity norm and with the increase of the environmental disturbance intensity norm.
[0007] Furthermore, in step S2, the residual neural network adopts a physical information neural network structure, and the total loss function during training is composed of a weighted average of a data fitting term, a physical law penalty term, and an uncertainty calibration term, wherein: The data fitting term is the mean square error between the state transition predicted by the residual neural network and the state transition actually observed in the pre-collected samples. The pre-collected samples include the state, action and corresponding state at the next time step in the historical time step. The physical law penalty terms include momentum conservation residual penalty terms, thermodynamic energy balance residual penalty terms, and vorticity conservation residual penalty terms; The uncertainty calibration term is either negative log-likelihood loss or continuous ranking probability fractional loss. The input to the uncertainty calibration term is the statistical distribution of the predicted uncertainty output by the residual neural network and the deviation of the actual state transition in the pre-collected samples from the nominal dynamic model.
[0008] Furthermore, the residual neural network applies the following structural constraints when outputting the mean term: Set an upper limit truncation threshold for the norm of the mean term; Lipschitz regularization is applied to the weights of the residual neural network, which constrains the upper bound of the derivative of the residual correction with respect to the input state.
[0009] Furthermore, in step S3, the risk-sensitive reinforcement learning strategy is constructed as a maximum entropy actor-critic reinforcement learning strategy, whose strategy optimization objective includes an expected cumulative reward term, an entropy regularization term, and a risk penalty term. The risk penalty term is determined based on at least one of the variance of the cumulative return, the prediction uncertainty of the residual neural network output, and the risk of constraint violation based on the prediction uncertainty estimate; The strategy optimization objective is further modified by the adaptive Lagrange multiplier terms corresponding to each security constraint. Each adaptive Lagrange multiplier is updated according to the positive part of the difference between the constraint index value and the corresponding preset tolerance threshold. The decision instructions output by the high-level risk-sensitive reinforcement learning strategy include at least a reference trajectory and an energy allocation coefficient, wherein the energy allocation coefficient is used to allocate power between the fuel power source and the electrical energy storage source.
[0010] Furthermore, in step S3, the safety constraints include: upper and lower bound constraints on the battery state of charge, upper bound constraints on the output power of the power motor, upper bound constraints on the motor temperature, and lower bound constraints on the remaining fuel mass. The energy distribution coefficient is a continuous variable with a value in the closed interval between 0 and 1. A value of 0 corresponds to the pure fuel power mode, and a value of 1 corresponds to the pure battery power mode.
[0011] Furthermore, in step S4, the low-level robust model predictive control adopts chance-constrained model predictive control: The constraint condition for the chance-constrained model predictive control is that the probability of the aircraft's motion and energy storage state falling into a preset safe state set is not lower than a preset probability threshold. The forward recursion is based on the prediction uncertainty output by the residual neural network. The prediction uncertainty is forward recursively calculated through the nominal dynamics model of the aircraft to obtain the probability distribution of the aircraft's motion and energy storage state at each step in the future prediction time domain.
[0012] Furthermore, in step S4, the low-level robust model predictive control adopts pipe model predictive control: The pipe model predictive control uses the prediction uncertainty to construct a worst-case reachable set containing all possible state deviations, and uses the worst-case reachable set as the pipe boundary for following the reference trajectory; The physical hard constraints are the set of safety constraints after reverse contraction based on the pipeline boundary.
[0013] Furthermore, in step S6, the safety threshold includes a constraint violation probability threshold and an uncertainty norm threshold; The safety control signal is generated by a proportional-integral-derivative controller or a preset hover controller; After outputting a safety control signal, the system switches back to the initial control signal only after the constraint violation probability is lower than a preset recovery threshold and the duration exceeds a preset hysteresis delay. The recovery threshold is lower than the constraint violation probability threshold.
[0014] Compared with the prior art, the present invention has at least the following beneficial effects: (1) The present invention proposes a hybrid electric vehicle energy management method based on physical constraint hierarchical reinforcement learning. By embedding physical rules into the learning process of the residual neural network, the physical consistency of the residual correction is guaranteed. At the same time, by using mean term norm truncation and Lipschitz regularization constraints, the sensitivity of the residual correction to input disturbances is controlled within a limited range, thereby reducing the risk of abnormal correction in scenarios outside the training data distribution. (2) The prediction uncertainty is passed down downstream. On the one hand, it serves as the basis for the risk penalty term in the risk-sensitive reinforcement learning strategy, so that the high-level decision-making considers the uncertainty evolution trend in the planning reference trajectory and energy allocation time domain and the future prediction time domain. On the other hand, it is input into the low-level robust model predictive control for forward recursion. Through chance constraints or pipeline constraints, the management of the state distribution boundary is realized, so that the control signal remains feasible within the physical hard constraint boundary. The resulting hierarchical structure of "uncertainty quantification - high-level risk perception planning - low-level constraint-driven execution" allows the same set of residual models to serve the three links of strategy optimization, constraint management and control execution at the same time. The consistency loss feeds back the feasible domain information of the low-level controller to the optimization objective of the high-level strategy, reducing the accumulation of instruction deviation between the planning layer and the execution layer. (3) Under the premise of ensuring physical consistency and hard constraints, the safety and reliability of energy management of hybrid electric vehicles in dynamic environments are improved by systematically transmitting and reusing uncertain information between different levels. Attached Figure Description
[0015] Figure 1 A step diagram illustrating a physical constraint hierarchical reinforcement learning-based hybrid vehicle energy management method. Detailed Implementation
[0016] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.
[0017] The physical constraint hierarchical reinforcement learning-based energy management method for hybrid electric vehicles provided by this invention is applicable to hybrid electric vehicles with distributed electric propulsion systems and vertical takeoff and landing capabilities, especially suitable for hybrid vertical takeoff and landing vehicles performing long-endurance monitoring, search, or transport missions in scenarios with significant wind disturbances and complex airflow conditions, such as marine and urban environments. This invention's method organically integrates physically-based residual learning, risk-sensitive reinforcement learning, and robust model predictive control to achieve online optimization and safe control of vehicle energy allocation in uncertain environments, such as… Figure 1 As shown, this energy management method mainly includes the following steps: S1: Obtain real-time information on aircraft motion and energy storage status, environmental disturbance information, and uncertainty covariance estimates used to characterize model mismatch and environmental randomness, and input them into the residual neural network; S2: Obtain the random residual correction amount, represented in the form of a probability distribution, from the output of the residual neural network. The random residual correction amount includes a mean term for compensating for model errors and a prediction uncertainty for quantifying the reliability of the correction. S3: Based on the current state information and the random residual correction, a risk-sensitive reinforcement learning strategy is adopted for high-level planning, with the goal of maximizing the cumulative reward of introducing risk-averse parameters. An adaptive Lagrange multiplier method is used to handle safety constraints, and the output includes decision instructions containing reference trajectories and energy allocation coefficients. S4: Input the decision command into the low-level robust model prediction and control layer, use the prediction uncertainty to perform forward recursion, calculate the constraint violation probability, and track the reference trajectory and generate the initial control signal under the premise of satisfying the physical hard constraints. S5: Construct consistency loss based on the deviation between decision instructions and initial control signals, and use consistency loss to correct the optimization objective of the reinforcement learning strategy; S6: Real-time monitoring of two indicators: constraint violation probability and prediction uncertainty. When either indicator exceeds the safety threshold, a safety control signal is output to ensure the basic stability of the aircraft; otherwise, a preliminary control signal is output.
[0018] In one specific embodiment of the present invention, the airborne control system executing the above-described energy management method includes a sensor data acquisition unit, a state estimation unit, a residual neural network inference module, a high-level planning module, a low-level control module, and a safety backoff arbitration module. These modules can be deployed on the aircraft's onboard flight computer and interact via a real-time data bus. The sensor data acquisition unit acquires output data from the inertial measurement unit, global positioning system, atmospheric data computer, battery management system, and fuel flow meter. After filtering and fusion by the state estimation unit, a complete state description of the system at the current moment is formed. Simultaneously, the state estimation unit calculates the covariance estimate of the state uncertainty based on the deviation between the historical state sequence and the current observation. This covariance estimate is used to quantify the reliability of the current state estimate and the randomness introduced by sensor noise and data loss.
[0019] Specifically, this invention models the energy management problem of hybrid-electric aircraft as a stochastic constrained Markov decision process. At the beginning of each control cycle, the system first acquires real-time aircraft motion and energy storage state information, environmental disturbance information, and uncertainty covariance estimation, and inputs this information into the residual neural network. Let the system at time... The state is ,in Includes the three-dimensional position of the aircraft ,speed Attitude angle and energy storage status of the hybrid power system, specifically including battery state of charge. and remaining fuel mass ; For measurable environmental disturbances, such as estimated wind field vectors ; Uncertainty covariance estimation characterizes the randomness introduced by sensor noise and state estimation errors. Its value is calculated by the state estimation unit based on sensor noise characteristics and the filtered covariance matrix. The aircraft's three-dimensional position and velocity are provided by the fusion of the Global Positioning System (GPS) and the Inertial Navigation System (INS). The attitude angles are provided by the fusion of the Inertial Measurement Unit (INS) and the magnetometer. The battery state of charge (SOCC) is provided by the Battery Management System (BMS) based on current integration and terminal voltage estimation. The remaining fuel mass is provided by fuel flow meter integration or directly by the fuel quantity sensor. Environmental disturbance information includes at least wind field information, which can be wind vector estimates provided by airborne weather radar or laser Doppler anemometers, or equivalent wind disturbance estimates obtained through inversion of the aircraft's own motion state and aerodynamic model. Uncertainty covariance estimation. The value varies with the aircraft speed norm It increases with the increase of the environmental disturbance intensity norm, and also increases with the increase of the environmental disturbance intensity norm. The uncertainty increases with the increase of the state estimate, meaning that under high-speed maneuvering or strong turbulence conditions, the system imparts a greater uncertainty to the state estimate, thereby adopting a more conservative control strategy in subsequent safety constraint processing.
[0020] The system's control actions are defined as follows ,in The reference trajectory is passed to the lower-level controller. This represents the energy allocation coefficient. The energy allocation coefficient indicates the power distribution ratio between fuel-powered and battery-powered systems; a value of 0 corresponds to pure fuel-powered mode, a value of 1 corresponds to pure battery-powered mode, and intermediate values correspond to hybrid power allocation. Within the above modeling framework, the energy management problem is transformed into a sequential decision-making problem seeking the optimal control strategy while satisfying safety constraints.
[0021] To address the discrepancy between the nominal dynamics model and the actual response of hybrid electric vehicles during flight, this invention introduces a stochastic residual learning mechanism based on a physical information neural network. Nominal dynamics model It describes the kinematics and energy variation laws of an aircraft under ideal conditions, specifically including rigid body kinematic equations, aerodynamic and gravity models, and energy consumption models. The physical state of the aircraft, including its three-dimensional position. ,speed Attitude angle, battery state of charge and remaining fuel mass ; The control inputs include thrust commands for each propulsion motor and power commands for the fuel engine. However, in actual flight, due to factors such as unmodeled aerodynamic effects, battery aging, and environmental disturbances, there is a deviation between the actual system dynamics and the nominal model. Therefore, this invention models the system dynamics as a stochastic differential equation in the following form: , in, For environmental disturbance, This is process noise, used to characterize the amount of residual correction. The high-frequency random disturbances remaining beyond the compensated unmodeled dynamics. Process noise. Follows a zero-mean Gaussian distribution Its covariance matrix According to the state and control input The residual correction varies with the speed of the object, taking a larger value at high speeds or high thrust. The residual correction, output by the residual neural network, is used to compensate for the unmodeled dynamics of the nominal model. To quantify the reliability of the residual estimate, this residual correction is modeled as a probability distribution: , in, This is the mean term, used to compensate for the main direction of model error; To predict uncertainty, this is used to quantify the reliability of the correction. This differs from the aforementioned process noise covariance. Prediction uncertainty The characterization is based on the nominal dynamic model. The uncertainty of the system deviation from the actual dynamics, and the variance of non-high-frequency random disturbances, together constitute the description of the total uncertainty of the system. The probabilistic modeling approach described above allows subsequent low-level robust model predictive control to consider the uncertainty of residual correction during the optimization process, thereby generating control commands with adjustable conservatism. The specific structure of the residual neural network is a multi-layer fully connected network, with the input layer receiving the state... Environmental disturbance and uncertainty covariance The concatenated vector, after being processed by multiple hidden layers, produces the mean term and prediction uncertainty in the output layer. The activation functions of the hidden layers are rectified linear units or their smoothed variants, the mean term of the output layer uses a linear activation function to ensure that the output range is unrestricted, and the prediction uncertainty uses a flexible additive activation function to ensure the positive definiteness of the variance.
[0022] The training of the residual neural network is completed offline, and it is continuously and adaptively updated during flight using data acquired online. The loss function used in the training process is a weighted sum of a data fitting term, a physical law penalty term, and an uncertainty calibration term. The total loss function is expressed as: , in, and These are the weighting coefficients for the physical law penalty term and the uncertainty calibration term, respectively, used to balance the relative contributions of each term to the total loss. Their specific values are determined through cross-validation, with a typical range of 10. -3 Up to 10 0 Data fitting term The mean square error between the residual correction predicted by the residual neural network and the state transition bias actually observed in the pre-collected samples is calculated. Specifically, for the th in the training batch One sample, the network receives the current physical state. Control quantity and environmental disturbances As input, the mean prediction of the output residual correction. Correspondingly, the state transition bias actually observed in the pre-collected samples is the actual change in state from that moment to the next moment. The data fitting term is the square norm of the difference between the network predicted residual and the actual state transition deviation, averaged over batches. Its specific expression is: , in, The training batch size typically ranges from 64 to 256. The pre-collected samples contain the state, actions, and corresponding next-time state at historical moments. These samples can be derived from flight data in a high-fidelity simulation environment or from recorded data in actual flight tests.
[0023] Physical law penalty term The constraint used to ensure that the output of the residual neural network does not violate fundamental physical conservation laws is specifically a weighted sum of the square norms of the momentum conservation residual, the energy balance residual, and the vorticity conservation residual: , in, For momentum conservation residuals, For thermodynamic energy balance residuals, These are the vorticity conservation residuals (based on the vorticity transport equation in fluid mechanics). These residual terms are obtained by substituting the output of the residual neural network into the corresponding partial differential equations to calculate their degree of mismatch.
[0024] Uncertainty calibration item Using negative log-likelihood loss: , in, This represents the deviation of the actual state transitions in the pre-collected samples from the nominal dynamic model. The mean term of the network output and prediction uncertainty The defined Gaussian probability density function. The above calibration terms ensure that the prediction uncertainty of the network output matches the distribution of the actual observed residuals in a statistical sense, rather than simply outputting a fixed or input-independent variance value.
[0025] To further ensure the physical rationality of the residual correction, the residual neural network outputs the mean term. Structural constraints were also imposed. An upper limit truncation threshold was set for the norm of the mean term: , in, A preset cutoff threshold is pre-calibrated based on the aircraft's dynamic characteristics; for example, it can be set as a certain proportion of the maximum thrust or maximum torque. This prevents the residual correction from deviating excessively from the nominal dynamic model, causing the control quantity to exceed the physical limits of the actuator. Simultaneously, Lipschitz regularization is applied to the weights of the residual neural network. This regularization constrains the upper bound of the derivative of the residual correction with respect to the input state; specifically, it constrains the spectral norm of the network weights to not exceed a preset Lipschitz constant. This constraint ensures that when the input state undergoes small changes, the change in the residual correction will not exceed a bounded range, thus guaranteeing the input-output stability of the closed-loop system. This structural constraint is added to the loss function as a regularization term during training, or implemented through explicit projection operations after the network's forward inference.
[0026] At the high-level planning level, a risk-sensitive reinforcement learning strategy is adopted for decision-making based on the current state information and the random residual correction. This risk-sensitive reinforcement learning strategy is constructed as a maximum entropy actor-critic reinforcement learning algorithm. The core difference between this invention and traditional reinforcement learning methods lies in the explicit introduction of a risk penalty term into the policy optimization objective. The policy optimization objective is expressed as: , in, To accumulate discount rewards, used to measure the net benefit of task completion progress and energy consumption; This is the risk aversion parameter; the larger its value, the more the strategy tends to choose action sequences with low return variance and low risk. Through the above objective function, this invention explicitly introduces a risk aversion mechanism into the strategy optimization process, thereby guiding the strategy to avoid control decisions with high uncertainty or high risk during the training phase. Cumulative return The specific definition is: , in, This is a discount factor used to balance the relative weights of near-term and long-term returns; The task's time domain length; This is the immediate reward function. The immediate reward function is designed to include positive mission progress rewards and negative energy consumption penalties. Mission progress rewards are assigned based on how close the aircraft is to the target point, while energy consumption penalties are calculated based on instantaneous fuel consumption rate and battery discharge rate. Risk aversion parameter. The value is preset based on the safety requirements of the mission. For missions performed in high-risk environments such as dense urban areas or ship decks, a larger value is selected. Value; for long-endurance patrol missions conducted in open airspace, a smaller value can be used. value.
[0027] To address safety constraints such as battery state of charge and motor temperature during strategy optimization, this invention employs the adaptive Lagrange multiplier method. Specifically, the safety constraints are expressed as constraint index functions. In the form of, For the first The index value of a safety constraint under the current state and action. This is a preset tolerance threshold for the constraint. In this embodiment, the safety constraints include at least upper and lower bounds for the battery state of charge, upper bounds for the motor output power, upper bounds for the motor temperature, and lower bounds for the remaining fuel mass. The specific values of these constraints are preset based on the aircraft model, mission profile, and battery characteristics. For example, the lower bound for the battery state of charge can be set to 0.2 to ensure sufficient power reserves during landing, the upper bound for the motor temperature is set according to the motor insulation class, and the lower bound for the remaining fuel mass is set according to the amount of fuel required for return. As mentioned above, the high-level strategy optimization objective of this invention is to maximize risk-sensitive cumulative returns. To incorporate penalties for safety constraints into the objective function, this invention employs the Lagrange multiplier method, combining the objective function with the violation penalty terms of each safety constraint into a unified Lagrange function. Specifically, the objective function is used as the principal part of the Lagrange function, and the weighted sum of the differences between each safety constraint index value and the tolerance threshold is used as the constraint penalty term, constructing the complete Lagrange function as follows: , in, For the first The Lagrange multipliers corresponding to each safety constraint. Each Lagrange multiplier The following formula is used for real-time updates: , Update the step size for the multiplier. When the constraint index value Exceeding the tolerance threshold As the Lagrange multiplier increases, it incurs a larger penalty in the policy optimization objective, forcing the policy to adjust in a direction that satisfies the constraints. When the constraint index value falls below the tolerance threshold, the Lagrange multiplier gradually decays to zero. Through this adaptive update mechanism, the policy maximizes the cumulative reward while satisfying the safety constraints.
[0028] To support the training of the aforementioned risk-sensitive strategy, this invention constructs a risk-sensitive commentator network. Unlike traditional reinforcement learning where commentators only estimate expected rewards, the commentator network of this invention estimates the value of state-action pairs as follows: , This value estimation considers both the variance of the reward and the degree of violation of security constraints. During the policy evaluation phase, the critic network updates by minimizing the Bellman error, with its objective calculated from the sampled trajectory under the current policy. In the policy improvement phase, the actor network adjusts its policy parameters based on the value estimate output by the critic, shifting the policy towards higher values. Through this critic-guided policy iteration, the action sequences generated by the actor network not only pursue high expected returns but also avoid getting trapped in control regions with high variance or high constraint violation risk. A maximum entropy regularization term is further incorporated into the policy objective to encourage the policy to explore sufficiently during training, preventing premature convergence to local optima.
[0029] During training, since the reinforcement learning strategy and high-level planning module are primarily trained offline, this invention introduces a consistency learning mechanism to ensure the feasibility of the output decision instructions at the execution level. Specifically, a consistency loss is constructed based on the deviation between the decision instructions and the initial control signal, and this consistency loss is used to correct the optimization objective of the reinforcement learning strategy. The consistency loss function is defined as follows: , in The ideal control quantity obtained by inverse dynamic mapping of the reference command output by the high-level strategy, i.e., based on the reference trajectory. and energy distribution coefficient The desired thrust / torque command is calculated by inverse dynamics model; The consistency loss is the initial control signal actually output by the low-level robust model predictive control after satisfying the physical hard constraints. This consistency loss is added as an additional regularization term to the optimization objective of the high-level reinforcement learning policy, ensuring that the high-level policy, during updates, not only considers maximizing cumulative reward but also whether its output command can be accurately tracked by the low-level controller within the physical constraints. During the training phase, at each policy update, the high-level decision command is input into the low-level robust model predictive controller for solving. The initial control signal obtained from the solution is compared with the original decision command; the consistency loss is the mean square error between the two. After iterative consistency learning training, the output of the high-level policy gradually converges to the feasible region of the low-level model predictive control, thereby reducing tracking losses and energy waste caused by infeasible commands.
[0030] At the lower-level control layer, the decision commands output by the higher-level planning module are input into the lower-level robust model predictive control layer. This lower-level control layer uses the prediction uncertainty for forward recursion to calculate the constraint violation probability, and, under the premise of satisfying the physical hard constraints, tracks the reference trajectory and generates an initial control signal. The lower-level robust model predictive control solves the following finite-time optimal control problem at each control time step: , in, To predict the length of the time domain; The reference trajectory output for high-level strategies at time... The value; It is a positive definite weight matrix, used to measure the penalty for state tracking deviation; The positive definite weight matrix is used to suppress the magnitude or rate of change of the control variable. Solving the above optimization problem is also subject to equality constraints from the system dynamics equations and inequality constraints from physical hard constraints (such as thrust upper and lower limits, control surface deflection angle limits, etc.). By solving the above optimization problem, model predictive control obtains a set of control sequences that minimizes the tracking error while satisfying the system dynamics and physical constraints, and takes the first control variable in the sequence as the initial control signal at the current moment.
[0031] Uncertainty propagation is a core component of low-level robust model predictive control. Forward recursion, based on the prediction uncertainty output by the residual neural network, propagates the uncertainty at the current moment forward to every step in the future prediction time domain through a linearization method of the aircraft's nominal dynamics model. In specific implementation, the Jacobian matrix of the nominal dynamics model with respect to the current state and control variables is first calculated. Then, the contributions of the state covariance from the previous moment, the control uncertainty, and the prediction uncertainty output by the residual neural network at the current moment are added together to obtain the state covariance for the next moment. Through this recursive method, the probability distribution of the aircraft's motion and energy storage state at each step in the future prediction time domain is obtained.
[0032] In one implementation, low-level robust model predictive control employs chance-constrained model predictive control. The constraint condition for chance-constrained model predictive control is that the probability of the aircraft's motion and energy storage state falling into a preset safe state set is not less than a preset probability threshold. , in, For a preset set of safe states, This is a pre-defined probability violation tolerance. The above opportunity constraint means that at each time step in the future prediction time domain ( The probability that the aircraft's motion and energy storage state fall into the safe state set is no less than Based on the probability distribution of each state obtained through forward recursion (i.e., the mean vector and covariance matrix of the state prediction at that moment), the chance constraint is approximately transformed into a deterministic constraint: the mean of the state prediction must lie within the contracted set after shrinking the original set of safe states. The amount of contraction is determined by the covariance matrix and a preset risk tolerance, specifically represented by a confidence ellipsoid related to the covariance matrix. Through this transformation, the original stochastic optimization problem is transformed into an optimization problem with deterministic constraints, which can be efficiently solved on an onboard computer using numerical optimization algorithms such as interior-point methods or sequential quadratic programming. Similarly, when the implementation involves the Pontryagin difference operation of the set, its physical meaning is to shrink the original set of safe constraints inward to offset the effects of uncertainty.
[0033] In another implementation, low-level robust model predictive control employs pipe model predictive control. Pipe model predictive control utilizes predictive uncertainty to construct a worst-case reachable set containing all possible state deviations, and uses this worst-case reachable set as the pipe boundary for following the reference trajectory. The constraints of pipe model predictive control are expressed as follows: , in, This involves using the worst-case reachable set obtained recursively from the prediction uncertainty. Pipe model predictive control first calculates a disturbance-free nominal trajectory based on the nominal dynamics model and the boundary conditions of the prediction uncertainty. Then, it calculates the maximum deviation of the actual state from the nominal trajectory under all possible disturbances, i.e., the pipe radius. The original set of safety constraints is then used. Inward contraction can reach the set The tightened constraint set is then obtained. When solving for the control variables, only the nominal trajectory is required to satisfy the tightened constraints. Therefore, under any bounded disturbance, the actual state will not exceed the original safety constraint boundary. Compared with the chance constraint method, the tube model predictive control provides a more conservative hard safety guarantee. It can guarantee the constraint satisfaction in the worst case without relying on probabilistic assumptions, and is suitable for flight phases with extremely high safety requirements, such as vertical takeoff and landing phases.
[0034] After solving the low-level robust model predictive control, the preliminary control signal is obtained, which includes thrust commands for each propulsion motor and power commands for the fuel engine. Before the control commands are finally output to the actuators, the system performs a safety backoff determination. The system monitors two indicators in real time: constraint violation probability and prediction uncertainty. When either indicator exceeds the safety threshold, it is determined that the current environmental uncertainty is too high or the prediction reliability is insufficient. At this time, a safety control signal that ensures the basic stability of the aircraft is output, instead of the preliminary control signal output by the low-level robust model predictive control. The triggering condition for safety backoff is the decomposition of the following three logical conditions: , in, The probability of constraint violation at future time moments is calculated by the model predicting the control forward propagation. To constrain the violation probability threshold, Let be the norm of the prediction uncertainty output by the residual neural network. The preset uncertainty norm threshold, Predict the value of the optimal control cost function for the current model (finite-time optimal control problem). This is a preset cost threshold. When any of the above conditions are met, the system determines that the current environmental uncertainty is too high or that model predictive control cannot guarantee effective control, and immediately switches to the backup controller to output a safety control signal.
[0035] Safety control signals are generated by a proportional-integral-derivative (PID) controller or a preset hover controller. These controllers are simple in structure, computationally inexpensive, and have rigorously proven performance guarantees, making them suitable as backup control strategies in emergency situations. The PID controller calculates control inputs based on the current attitude and altitude deviations to maintain the aircraft in a safe hover state; the preset hover controller uses pre-calibrated hover throttle and attitude commands, does not rely on real-time optimization calculations, and has extremely high reliability. After outputting the safety control signal, the system enters a fallback state. In fallback mode, the system continuously monitors constraint violation probabilities and prediction uncertainties. Only when the constraint violation probability is... Below the preset recovery threshold Furthermore, the system only switches back from the safety control signal to the initial control signal after the duration exceeds the preset hysteresis delay. This hysteresis-delayed backoff logic prevents the system from frequently switching between the backoff state and the normal state, thus preventing the so-called "flutter" phenomenon.
[0036] In terms of real-time implementation, the method described in this invention employs a multi-rate asynchronous execution architecture. Since the high-level strategy involves planning over a longer timescale and complex neural network inference, its execution frequency is set to 1 to 10 Hz, meaning that a state read, strategy inference, and decision command output are completed every 0.1 to 1 second. The low-level robust model predictive control involves responding to rapid dynamic disturbances and precisely controlling the actuators; its execution frequency is set to 50 to 100 Hz, meaning that a state read, optimization solution, and control output are completed every 0.01 to 0.02 seconds. Through this multi-rate architecture, the computational load of the high-level layer is distributed over longer time intervals, while the low-level layer maintains a high frequency of control updates to ensure control quality. The total control loop delay between the high and low levels is less than 20 ms, meaning the end-to-end delay from sensor data acquisition to control command output to the actuator does not exceed 20 ms. This delay specification meets the bandwidth requirements of vertical takeoff and landing aircraft during hovering and transitional flight phases. To achieve the aforementioned real-time performance, the low-level robust model predictive control employs a real-time iterative sequential quadratic programming algorithm for numerical solution. The sequential quadratic programming algorithm approximates the original nonlinear optimization problem as a quadratic programming subproblem at each sampling time and solves iteratively. The real-time iterative strategy performs only one or a finite number of sequential quadratic programming iterations at each sampling time, rather than solving until complete convergence, thus significantly reducing the single-cycle solution time. The algorithm also employs a warm-start strategy, shifting the optimal control sequence from the previous time step as the initial guess for the current optimization problem, further reducing the number of iterations and keeping the single-cycle solution time within 10ms.
[0037] In the proposed method, the residual neural network and high-level reinforcement learning strategy are trained offline. First, a training scenario set containing diverse wind field disturbances, different task profiles, and various initial states is constructed in a high-fidelity simulation environment. A large number of state transition samples are generated through Monte Carlo sampling. The residual neural network is trained on the sample set using stochastic gradient descent or its adaptive variant (such as the Adam optimizer). During training, the changing trends of the physical law penalty term and uncertainty calibration term are monitored to ensure that the network converges to a parameter configuration that satisfies physical constraints and has good uncertainty calibration capabilities. The high-level reinforcement learning strategy is trained in the simulation environment using a soft actor-critic algorithm. The strategy interacts with the simulation environment to generate trajectory samples at each update, which are stored in an experience replay buffer. The strategy gradient is updated through random sampling batches. During training, the risk aversion parameter is gradually increased. The values are gradually transitioned from initial values to target values to reduce the exploration difficulty in the early stages of training. The trained network weights and policy parameters are deployed to the onboard flight computer. During flight, the system can use online-collected state transition samples to slowly and incrementally update the residual neural network to adapt to long-term, slow drifts such as battery aging and changes in aerodynamic parameters. Online updates are performed at a low frequency, such as once per minute, and the update step size is smaller than that used in offline training to ensure the stability of the online learning process.
[0038] Compared to traditional energy management methods, this invention achieves online adaptive optimization of energy allocation for hybrid electric vehicles in uncertain environments through the aforementioned technical solutions. In the application scenario of vertical takeoff and landing (VTOL) aircraft in marine environments, facing complex wind field disturbances caused by ship deck wakes and ocean-atmospheric boundary layer turbulence, the residual neural network proposed in this invention can learn and compensate for the deviation between the nominal dynamic model and the actual response in real time, while quantifying the uncertainty of this deviation. A high-level risk-sensitive strategy adjusts the power allocation strategy based on the aforementioned uncertainty information, avoiding aggressive battery discharge operations during periods of strong disturbance, thereby reducing the peak battery discharge rate and extending battery cycle life. Low-level robust model predictive control ensures that the aircraft's attitude and altitude remain within the safety envelope under instantaneous strong gusts, preventing loss of control due to actuator saturation or state out-of-bounds conditions. A safety backoff mechanism, as a last line of defense, switches to a simple and reliable backup controller when predictive control cannot provide effective control commands due to computational delays or severe model mismatch, ensuring the basic safety of the aircraft under extreme conditions.
[0039] In another embodiment of the invention, the above method can be further extended to multi-aircraft cooperative scenarios. Each aircraft shares environmental perception information and state estimation results via data link, thereby obtaining a more comprehensive wind field distribution estimate and uncertainty description than single-aircraft perception. Based on the shared information, the residual neural networks of each aircraft can be updated collaboratively, allowing each aircraft to benefit from the observation data of the entire formation and accelerating the adaptation of the residual model to new environments. In formation cooperative scenarios, higher-level strategies also need to consider additional constraints such as formation maintenance and formation changes, which are implemented by adding corresponding constraint terms to the Lagrangian function. Lower-level control needs to coordinate the relative positions between aircraft to avoid collisions, which is achieved by adding a relative position tracking term to the cost function of model predictive control.
[0040] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.
[0041] Furthermore, in this invention, descriptions involving terms such as "first," "second," and "a" are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0042] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0043] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.
Claims
1. A method for energy management of hybrid electric vehicles based on physical constraint hierarchical reinforcement learning, characterized in that, Including the following steps: S1: Obtain real-time information on aircraft motion and energy storage status, environmental disturbance information, and uncertainty covariance estimates used to characterize model mismatch and environmental randomness, and input them into the residual neural network; S2: Obtain the random residual correction amount, expressed in the form of a probability distribution, from the output of the residual neural network. The random residual correction amount includes a mean term for compensating for model errors and a prediction uncertainty for quantifying the reliability of the correction. S3: Based on the current state information and the random residual correction, a risk-sensitive reinforcement learning strategy is adopted for high-level planning, with the goal of maximizing the cumulative reward of introducing the risk aversion parameter. An adaptive Lagrange multiplier method is used to handle safety constraints, and the output includes a decision instruction containing the reference trajectory and energy allocation coefficient. S4: Input the decision command into the low-level robust model prediction and control layer, use the prediction uncertainty to perform forward recursion, calculate the constraint violation probability, and track the reference trajectory and generate the initial control signal under the premise of satisfying the physical hard constraints. S5: Construct consistency loss based on the deviation between decision instructions and initial control signals, and use consistency loss to correct the optimization objective of the reinforcement learning strategy; S6: Real-time monitoring of two indicators: constraint violation probability and prediction uncertainty. When either indicator exceeds the safety threshold, a safety control signal is output to ensure the basic stability of the aircraft; otherwise, a preliminary control signal is output.
2. The hybrid electric vehicle energy management method based on physical constraint hierarchical reinforcement learning as described in claim 1, characterized in that, In step S1, the aircraft motion and energy storage status information includes at least: the aircraft's three-dimensional position, velocity, attitude angle, battery state of charge, and remaining fuel mass; The environmental disturbance information includes at least wind field information; The value of the uncertainty covariance estimate increases with the increase of the aircraft velocity norm and with the increase of the environmental disturbance intensity norm.
3. The hybrid electric vehicle energy management method based on physical constraint hierarchical reinforcement learning as described in claim 1, characterized in that, In step S2, the residual neural network adopts a physical information neural network structure. The total loss function during training is a weighted sum of a data fitting term, a physical law penalty term, and an uncertainty calibration term, wherein: The data fitting term is the mean square error between the state transitions predicted by the residual neural network and the state transitions actually observed in the pre-collected samples. The pre-collected samples include the states and actions at historical moments and the corresponding states at the next moment. The physical law penalty terms include momentum conservation residual penalty terms, thermodynamic energy balance residual penalty terms, and vorticity conservation residual penalty terms; The uncertainty calibration term is either negative log-likelihood loss or continuous ranking probability fractional loss. The input to the uncertainty calibration term is the statistical distribution of the predicted uncertainty output by the residual neural network and the deviation of the actual state transition in the pre-collected samples from the nominal dynamic model.
4. The hybrid electric vehicle energy management method based on physical constraint hierarchical reinforcement learning as described in claim 1, characterized in that, The residual neural network also imposes the following structural constraints when outputting the mean term: Set an upper limit truncation threshold for the norm of the mean term; Lipschitz regularization is applied to the weights of the residual neural network, which constrains the upper bound of the derivative of the residual correction with respect to the input state.
5. The hybrid electric vehicle energy management method based on physical constraint hierarchical reinforcement learning as described in claim 1, characterized in that, In step S3, the risk-sensitive reinforcement learning strategy is constructed as a maximum entropy actor-critic reinforcement learning strategy, whose policy optimization objectives include the expected cumulative reward term, the entropy regularization term, and the risk penalty term. The risk penalty term is determined based on at least one of the variance of the cumulative return, the prediction uncertainty of the residual neural network output, and the risk of constraint violation based on the prediction uncertainty estimate; The strategy optimization objective is further modified by the adaptive Lagrange multiplier terms corresponding to each security constraint. Each adaptive Lagrange multiplier is updated according to the positive part of the difference between the constraint index value and the corresponding preset tolerance threshold. The decision instructions output by the high-level risk-sensitive reinforcement learning strategy include at least a reference trajectory and an energy allocation coefficient, wherein the energy allocation coefficient is used to allocate power between the fuel power source and the electrical energy storage source.
6. The hybrid electric vehicle energy management method based on physical constraint hierarchical reinforcement learning as described in claim 5, characterized in that, In step S3, the safety constraints include: upper and lower bound constraints on the battery state of charge, upper bound constraints on the output power of the power motor, upper bound constraints on the motor temperature, and lower bound constraints on the remaining fuel mass. The energy distribution coefficient is a continuous variable with a value in the closed interval between 0 and 1. A value of 0 corresponds to the pure fuel power mode, and a value of 1 corresponds to the pure battery power mode.
7. The hybrid electric vehicle energy management method based on physical constraint hierarchical reinforcement learning as described in claim 1, characterized in that, In step S4, the low-level robust model predictive control adopts chance-constrained model predictive control: The constraint condition for the chance-constrained model predictive control is that the probability of the aircraft's motion and energy storage state falling into a preset safe state set is not lower than a preset probability threshold. The forward recursion is based on the prediction uncertainty output by the residual neural network. The prediction uncertainty is forward recursively calculated through the nominal dynamics model of the aircraft to obtain the probability distribution of the aircraft's motion and energy storage state at each step in the future prediction time domain.
8. The hybrid electric vehicle energy management method based on physical constraint hierarchical reinforcement learning as described in claim 1, characterized in that, In step S4, the low-level robust model predictive control adopts pipe model predictive control: The pipe model predictive control uses the prediction uncertainty to construct a worst-case reachable set containing all possible state deviations, and uses the worst-case reachable set as the pipe boundary for following the reference trajectory; The physical hard constraints are the set of safety constraints after reverse contraction based on the pipeline boundary.
9. The hybrid electric vehicle energy management method based on physical constraint hierarchical reinforcement learning as described in claim 1, characterized in that, In step S6, the safety threshold includes a constraint violation probability threshold and an uncertainty norm threshold. The safety control signal is generated by a proportional-integral-derivative controller or a preset hover controller; After outputting a safety control signal, the system switches back to the initial control signal only after the constraint violation probability is lower than a preset recovery threshold and the duration exceeds a preset hysteresis delay. The recovery threshold is lower than the constraint violation probability threshold.
