An actuator failure compensation and safety control method based on a double-layer evaluation architecture, a storage medium and a system

Abstract

This disclosure proposes an actuator failure compensation and safety control method, storage medium, and system based on a two-layer evaluation architecture. The bottom layer uses an adaptive model to predict the nominal control output command. Above this, a reinforcement learning model is built to output high-frequency residual compensation commands to eliminate model mismatch. A second evaluation network is independently constructed to assess the future global runaway risk of the vehicle. Lagrange dual optimization is introduced to transform the macro-risk assessment into mathematical constraints to guide the correction of the first strategy network. Continuous flexible degradation is achieved by using a global risk index feedforward to adjust the target cost weight of the predictive control. Discrete quadratic programming command projection is performed by triggering a control barrier function at the limit boundary. This invention overcomes the conservative strategy degradation defects caused by traditional mixed rewards, establishes an absolute safety defense line, and improves the control accuracy and safety robustness of the vehicle under complex nonlinear conditions.

Description

Technical Field

[0001] This disclosure relates to the field of motion control and functional safety technology for autonomous vehicles, and in particular to an actuator failure compensation and safety control method, storage medium and system based on a two-layer evaluation architecture. Background Technology

[0002] In autonomous driving chassis drive-by-wire systems, high-level control algorithms typically rely on high-fidelity simulation environments for initial training and parameter calibration. However, during the migration and deployment to real physical vehicles, significant mismatches often exist between the dynamic response of the real physical system and the nominal prediction model due to unmodeled dynamic characteristics such as tire nonlinear slip and suspension roll, as well as thermal decay, mechanical wear dead zones, or response lags caused by long-term operation of actuators (such as steering motors and braking systems). This model mismatch can easily lead to the failure of the original control strategy and even cause serious driving safety accidents.

[0003] To address the above problems, existing technologies mainly suffer from the following bottlenecks that urgently need to be resolved: First, there is the problem of policy conservatism and degradation caused by a single reward function. Existing technologies often introduce reinforcement learning algorithms for residual compensation control to eliminate the aforementioned model mismatch. However, traditional safety reinforcement learning often adopts a scheme that linearly weights trajectory tracking accuracy with global safety penalties, constructing a single hybrid reward function. This experience-based "reward reshaping" leads to goal conflict for the agent during training: to avoid triggering high safety penalties, the network is prone to getting trapped in local optima, outputting extremely conservative control commands. This prevents the system from effectively releasing and squeezing the potential of the underlying actuators, thus making it difficult to truly bridge the control error gap between simulation and reality.

[0004] Second, there is the issue of misalignment between microscopic residual compensation and macroscopic vehicle safety features. The high-frequency microscopic response lag of the compensation actuator and the macroscopic low-frequency stability ensuring the vehicle's phase plane are inherently misaligned in terms of state-space characteristic scale and evaluation period. Traditional adaptive control algorithms using a single evaluation network architecture struggle to simultaneously and accurately fit both high-frequency residual characteristics and low-frequency stability boundary characteristics. When faced with complex nonlinear conditions, a single network is highly susceptible to generating excessive compensation commands due to conflicts in internal gradient update directions, causing the vehicle to unknowingly approach and exceed the physical safety envelope limit.

[0005] Third, there is the disconnect between flexible degradation and rigid safety constraints. At the online safety control and fault response level, existing degradation strategies mostly employ discrete state machine logic based on fixed physical sensor thresholds. This hard logic switching lacks forward-looking prediction of system degradation trends and can cause drastic jumps in control commands upon triggering, leading to severe vehicle attitude oscillations. On the other hand, while conventional parameter adaptive model predictive control algorithms can achieve a certain degree of compliant degradation by adjusting weights, they lack an absolute safety constraint mechanism based on rigorous mathematical inequalities when encountering extreme conditions such as severe actuator failure. When the optimization solver diverges within the prediction domain, the system cannot forcibly intercept or project dangerous commands, still facing a very high risk of loss of control. Summary of the Invention

[0006] To address the model mismatch error problem encountered during the migration of autonomous vehicle control algorithms from simulation environments to the real physical world, and the control failure problem of actuators under nonlinear faults or performance degradation conditions, this invention aims to solve the following technical bottlenecks: First, existing reinforcement learning methods, when linearly weighting trajectory tracking accuracy and global safety constraints and placing them in the same reward function, are prone to getting trapped in local optima to avoid triggering high safety penalties. This state manifests as the policy network outputting extremely conservative control commands, preventing the system from effectively utilizing the actuator's potential to compensate for the physical mismatch between the simulation and the real environment.

[0007] Second, the high-frequency dynamic response lag of the compensating actuator and the macroscopic phase plane stability of the vehicle differ significantly in state-space characteristics and evaluation period. Traditional single evaluation networks struggle to simultaneously fit high-frequency microscopic residuals and low-frequency macroscopic boundary features, and are prone to generating excessive commands due to gradient update direction conflicts, causing the vehicle to exceed the physical safety envelope.

[0008] Third, existing adaptive control architectures often employ discrete state machine logic based on fixed physical thresholds for safety degradation strategies. This logic can cause drastic changes in control commands when triggered. Furthermore, conventional parameter adaptive algorithms lack a safety constraint mechanism based on strict mathematical inequalities to mitigate extreme failure conditions.

[0009] To address the aforementioned technical problems, this invention proposes an actuator failure compensation and safety control method, storage medium, and system based on a two-layer evaluation architecture. By introducing Lagrange dual optimization constraints and decoupling local dynamic error compensation from global macroscopic safety assessment, precise adaptive control and strict safety boundary constraints are achieved for the underlying nonlinear actuator. The specific technical solution is as follows.

[0010] In a first aspect, this disclosure proposes an actuator failure compensation and safety control method based on a two-layer evaluation architecture, comprising: a reinforcement learning model consisting of a first policy network, a first evaluation network, and a second evaluation network, wherein: the first policy network is configured to acquire basic nominal control commands. residual compensation increment The first evaluation network is configured to combine the state space vector and the residual compensation increment. As input, calculate the value of the local tracking action. The second evaluation network is configured to combine the state space vector and residual compensation instructions. and the state variables characterizing the macroscopic stability boundary of the vehicle As input, calculate the global security risk index. Set global security risk index Less than or equal to the preset physical security threshold To address global security constraints, the process is transformed into a restricted Markov decision process based on the Lagrange multiplier method, and implemented through multiplier updates. and The dynamic balance, thereby enabling the residual compensation increment generated by the first strategy network. It is conducive to security control; utilizing the global risk index Constructing damping weight coefficients Based on damping penalty weights Construct and solve the model predictive control objective function to obtain the current basic nominal control command. Thus, the initial fusion control command is obtained. If the current global risk index Less than or equal to the preset critical threshold The initial fusion control command will then be used as the final command. Otherwise, the initial fusion control command is forcibly projected into the set of actionable domains that ensure absolute physical safety and then passed to the underlying physical executor for execution.

[0011] In a second aspect, a computer-readable storage medium stores a computer program that can be loaded by a processor and execute any of the methods or systems described in this disclosure.

[0012] Thirdly, this disclosure proposes an actuator failure compensation and safety control system based on a two-layer evaluation architecture, including: an actuator failure compensation module configured with a reinforcement learning model consisting of a first policy network, a first evaluation network, and a second evaluation network, wherein: the first policy network is configured to acquire basic nominal control commands. residual compensation increment The first evaluation network is configured to combine the state space vector and the residual compensation increment. As input, calculate the value of the local tracking action. The second evaluation network is configured to combine the state space vector and residual compensation instructions. and the state variables characterizing the macroscopic stability boundary of the vehicle As input, calculate the global security risk index. Set global security risk index Less than or equal to the preset physical security threshold To address global security constraints, the process is transformed into a restricted Markov decision process based on the Lagrange multiplier method, and implemented through multiplier updates. and The dynamic balance, thereby enabling the residual compensation increment generated by the first strategy network. It is beneficial for security control; the initial fusion control command generation module is configured to utilize the global risk index. Constructing damping weight coefficients Based on damping penalty weights Construct and solve the model predictive control objective function to obtain the current basic nominal control command. Thus, the initial fusion control command is obtained. The security control module is configured to adjust the current global risk index. Less than or equal to the preset critical threshold In this case, the initial fusion control command will be used as the final command. Otherwise, the initial fusion control command is forcibly projected into the set of actionable domains that ensure absolute physical safety and then passed to the underlying physical executor for execution.

[0013] The beneficial technical effects of this disclosure are as follows: (1) This invention overcomes the conservative policy degradation defect of traditional single-objective reinforcement learning when dealing with the conflict between safety and tracking accuracy. By constructing an evaluation architecture that decouples the first evaluation network and the second evaluation network, the residual compensation strategy can maximize the approximation of the physical performance limit of the actuator under the theoretical premise of ensuring global safety, thereby significantly improving the generalization accuracy of the algorithm when it is deployed from the simulation environment to real vehicles. (2) This invention proposes a policy evolution framework based on restricted Markov decision process and Lagrange duality mechanism. This framework uses the risk gradient of the second evaluation network to construct strict mathematical constraints, directly intervenes in and guides the weight update of the first policy network. Compared with the traditional empirical reward function reconstruction method, this scheme gives the control system the ability to know and detect the safety boundary autonomously, which greatly enhances the control robustness of the system when dealing with the nonlinear decay characteristics of unknown actuators. (3) This invention establishes a closed-loop safety protection system that runs through the algorithm layer and the execution layer. In the feedforward control stage, the cost function of the predictive control domain is dynamically reconstructed using the global risk index, thereby achieving continuous and smooth adjustment of the vehicle's dynamic response characteristics. In the command issuance stage, discrete constraint barriers are constructed using control obstacle functions, and strict command truncation and projection are implemented through quadratic programming to ensure the absolute functional safety of the control system under extreme and harsh conditions. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A closed-loop architecture diagram of an adaptive control system based on performance integrity assessment in one embodiment. Detailed Implementation

[0016] Terminology: Efficacy Integrity: In this application, it refers to the ability of the system to maintain absolute compliance with the vehicle's physical boundaries through residual compensation and risk feedforward adjustment when actuators (such as steering and brake motors) experience nonlinear decay or partial failure, ensuring that control accuracy and safety constraints do not collapse due to actuator performance degradation.

[0017] To address the model mismatch error problem encountered by autonomous vehicle control algorithms when migrating from simulation environments to the real physical world, as well as the control failure problem when actuators are under nonlinear fault or performance degradation conditions, this disclosure proposes an actuator failure compensation and safety control method based on a two-layer evaluation architecture. By decoupling local dynamic error compensation and global macroscopic safety assessment, it successfully achieves precise adaptive control of the underlying nonlinear actuators, while strictly adhering to and effectively constraining the safety boundary, ensuring the stable operation and reliable performance of the system in complex dynamic environments.

[0018] The following will be combined with the appendix Figure 1 This application provides a clear and complete description of how the technical solution in this case is implemented. Obviously, the described implementation methods are only a part of the implementation methods in this case, not all of them. Based on the implementation methods in this case, all other implementation methods obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0019] Step 1: Construct an adaptive model prediction control layer based on global risk index feedforward. The global risk index is a scalar indicator used to quantify the probability that a vehicle will deviate from a safe and stable region in the future. Specifically, it includes an estimated probability of instability predicted by the second evaluation network based on the current vehicle state. The probability of instability in this application is an indicator used to describe the possibility of autonomous driving losing its stable operating capability under specific conditions. The vehicle state is a combined state including vehicle dynamics state deviations and actuator historical response characteristics, as detailed in the combined state vector below. Introduction to each component.

[0020] The adaptive model predictive control layer employs a parameter-varying model predictive control (Parameter-Varying MPC) architecture. This architecture treats the vehicle as a constrained dynamic system, controlling it according to the current situation in each control cycle. Damping penalty weights in the real-time reconstruction cost function This enables nonlinear flexible adjustment of the control bandwidth. The control cycle refers to the interval at which the control system reads the vehicle's status, calculates, and outputs new control commands (such as throttle, brake, and steering angle).

[0021] The adaptive model predictive control layer constructed in this step serves as the chassis nominal control module, used to process the linearized dynamic equations of the vehicle, and accepts the damping penalty weights output by the upper-layer reinforcement learning network to reconstruct and adjust the model predictive control objective function online.

[0022] The discrete-time state-space evolution equation of the vehicle is defined as follows: , and These are the state transition matrix and control input matrix of the vehicle dynamics system, respectively. Indicates in Always looking towards the future A vehicle discrete state prediction vector with a prediction step size. Indicates in Always looking towards the future The control input vector is calculated using a prediction step size.

[0023] The system inputs the current discrete states of the vehicle and actuators. Compared with reference trajectory By solving the objective function of model predictive control optimization, the theoretical optimal output of the vehicle under the nominal model—the basic nominal control command—is output. The basic nominal control command The subsequent residual compensation instructions will be combined with those output by the first strategy network. Vector superposition is performed to form a preliminary fusion control command. This enables the initial correction of nonlinear errors in the actuator.

[0024] The objective function for model predictive control optimization is shown below, which includes a dynamic damping penalty weight matrix, thus overcoming the limitation of fixed cost weights.

[0025]

[0026] In the formula: This represents the number of prediction time-domain steps for model predictive control, and is a positive integer. Represented by a positive semidefinite matrix Q The quadratic penalty term for the state tracking error is used to constrain the trajectory deviation. Represented by a positive definite matrix R The absolute value of the control input for the weights is a quadratic penalty term used to limit energy consumption or actuator amplitude. The objective function is optimized by model predictive control to make the system follow the desired trajectory. This represents the global risk index (scalar) output by the upper-level second evaluation network based on the overall vehicle status. Represented by a positive definite matrix S Incremental input for weight control The quadratic penalty term is used to smooth out actions; This is the dynamic damping penalty weight matrix output by the upper-layer security assessment network. Each weight value in this matrix increases non-linearly with the increase of the predicted global risk index, and is used to adjust the control input increment in the optimization objective in real time. The severity of the penalty is adjusted to achieve adaptive adjustment of the nominal control bandwidth.

[0027] positive semidefinite matrix QPositive definite matrix R and positive definite matrix S These are hyperparameters calibrated based on the dynamic characteristics of the target vehicle. (Positive semi-definite matrix) Q This represents the tolerance for "state tracking error." For example, if greater emphasis is placed on the accuracy of lateral positioning, then... Q The diagonal elements of the matrix corresponding to lateral deviations are increased. It is positive semi-definite, meaning that some insignificant state errors can be left unpenalized (with a weight of 0). Positive definite matrix. R This represents a constraint on the absolute value of the control input, with the aim of reducing high-load operation or energy consumption of the actuator. It must be positive definite to ensure that the cost function is bounded and that the quadratic programming (QP) problem has a unique optimal solution. Positive definite matrix. S This represents the penalty for the rate of change of control commands. In practical deployments for passenger vehicles, directly limiting severe vibrations in steering wheel angle or brake pedal is crucial to ensuring ride comfort and actuator lifespan.

[0028] The damping weighting coefficient The construction of the damping weight coefficient is not a simple linear scaling mapping, but rather based on a nonlinear smooth function with strict upper and lower bounds to ensure the numerical stability of the underlying quadratic programming solver that predicts the model's control. In this embodiment, the specific formula for constructing the damping weight coefficient is as follows:

[0029] The physical meanings of each parameter in the formula are as follows: This is the basic damping penalty weight, which takes a small positive number. When the vehicle is in a safe operating condition (…), When the baseline weight is relatively low, the system mainly uses this baseline weight to ensure that the control system has sufficient high-frequency response bandwidth and releases the potential of the actuator to complete accurate trajectory tracking. The maximum damping penalty weight is truncated at the boundary, and its value is calibrated based on the upper limit of the Hessian matrix condition number of the underlying solver. When the vehicle faces an extremely high-risk state, the weight is strictly limited to this maximum value to greatly limit the drastic changes in actuator actions, while preventing the algorithm from getting stuck in numerical solution divergence. This is the risk damping activation threshold, representing the turning point for initiating strong damping intervention; is the risk sensitivity shape coefficient, a positive real number, used to adjust the nonlinear surge rate of the damping weight as the risk of runaway increases.

[0030] Constructed using the aforementioned nonlinear functions The system can continuously and smoothly increase the damping penalty on the incremental control input when the vehicle approaches the physical stability boundary, adaptively suppressing high-frequency abrupt changes in underlying execution commands such as steering wheel angle or vehicle acceleration, thereby maintaining a smooth transition of vehicle attitude during the flexible degradation phase.

[0031] Step 2: Construct the first strategy network and the first evaluation network for actuator residual compensation. The first strategy network and the first evaluation network focus on outputting high-frequency compensation commands to eliminate actuator nonlinearity and unmodeled dynamic errors. The first strategy network takes as input a combined state space vector containing vehicle dynamic state deviations and actuator historical response characteristics. :

[0032] in, This indicates the vehicle's current lateral position deviation. This indicates the vehicle's current lateral speed deviation (i.e., the rate of change of lateral position deviation with respect to time). This indicates the vehicle's current heading angle deviation. This indicates the current yaw rate deviation of the vehicle (i.e., the rate of change of the heading angle deviation with respect to time). This indicates the vehicle's current longitudinal speed. Indicates the previous moment The nominal control command issued by the control system to the actuator. Indicates the previous moment The physical state value that the underlying actuator actually responds to and feeds back, this item is compared with... The combined inputs are designed to enable the first policy network to implicitly extract and learn the nonlinear response hysteresis and performance degradation characteristics of the underlying actuators.

[0033] The first-policy network employs a Gaussian stochastic policy and uses reparameterization techniques to output residual compensation increments for the basic nominal control commands. (Local tracking action).

[0034]

[0035] in, and The first policy network is based on the combined state space vector. The output Gaussian distribution means and standard deviation. Indicates that it follows a Gaussian distribution Independent random noise vectors are used for action exploration in the policy network. This represents the Hadamard product operator, which is the element-wise multiplication of a matrix or vector.

[0036] Local reward function of the first evaluation network It only covers high-frequency tracking accuracy loss and control command amplitude penalty, where tracking accuracy loss is measured by the lateral position deviation. deviation from heading angle The penalty is quantified by a quadratic form, and the greater the deviation, the heavier the penalty for the loss of accuracy.

[0037]

[0038] in, , and These are the weights for the lateral error penalty, the heading error penalty, and the residual compensation action amplitude penalty, respectively, set in the local reward function. All are positive real constants. (The text then abruptly shifts to a different topic:) This measure can effectively limit the excessively large high-frequency compensation output of the policy network, preventing the actuator from acting too aggressively.

[0039] The first evaluation network parameters are: It updates the local tracking value estimate by minimizing the mean square error of the Bellman time-series difference. Its loss function is defined as:

[0040] in, This represents the estimated value of the local tracking action calculated by the current first evaluation network. This represents the target value of the local tracking action at the next time step, calculated by the current first evaluation network using a soft update mechanism. This represents the reward time-series discount factor in a Markov decision process, with a value range of [value range missing]. , This indicates that the first policy network is based on the state at the next time step. The next time step residual compensation prediction command from the sampled output. The expectation operator represents the calculation of mathematical expectation based on a batch of data randomly sampled from the experience replay pool D.

[0041] The parameter updates for the first policy network and the first evaluation network are an iterative process. This iterative process does not include any macro-system-level security boundary penalties, allowing the network to focus entirely on maximizing local performance.

[0042] Step 3: Construct a global security risk quantitative assessment system based on the second evaluation network The second evaluation network does not directly participate in the generation of high-frequency motion control commands. Its function is to assess the future global runaway risk level of the system based on the overall vehicle state after the compensation commands are superimposed. The input feature vector of the second evaluation network includes a combined state space vector. Residual compensation instructions and the state variables characterizing the macroscopic stability boundary of the vehicle. Specifically, the state variable It is a set of vectors containing the low-frequency macroscopic dynamic characteristics of the vehicle, specifically including the vehicle's current center of gravity sideslip angle. yaw rate Estimated value of road surface adhesion coefficient and lateral acceleration .

[0043] For example, when a vehicle is cornering at high speed on a low-traction surface, Constructed specifically as vectors The system uses real-time sampling signals from onboard sensors (such as inertial measurement units, IMUs) combined with state observers (such as extended Kalman filters, EKFs) to estimate and obtain the values ​​of each physical quantity in the vector online, and inputs them into the second evaluation network to characterize the vehicle's margin from the physical slip boundary.

[0044] Let the parameters of the second evaluation network be... The second evaluation network outputs a global security risk index. The second evaluation network's global reward function Configured as a nonlinear penalty for severely unstable states:

[0045] This represents the vehicle's center of gravity sideslip angle penalty weighting coefficient, used to limit the vehicle from experiencing severe sideslip. The sideslip angle, representing the current center of mass of the vehicle, is a core physical state quantity characterizing the macroscopic lateral stability boundary of the vehicle. In actual physical systems, the sideslip angle... By fusing the lateral acceleration and yaw rate signals output by the onboard inertial measurement unit (IMU) and the wheel speed sensor signals, the data is estimated in real time using a vehicle dynamics model and a state observer. This represents the weighting coefficient for the penalty of vehicle yaw rate deviation. This represents the absolute value of the deviation between the vehicle's actual yaw rate and the ideal reference yaw rate. This term represents the extreme value penalty, which is essentially a conditionally triggered fixed constant. When the vehicle state exceeds the absolute physical stability boundary or triggers the minimum risk strategy, this term is activated and applies a preset high fixed negative value penalty to the system (e.g., a preset constant C, to forcibly block the strategy network from further exploration into the dangerous state space; within the normal safe driving range, the value of this term is zero).

[0046] Let the parameters of the second evaluation network be... It also uses the time-series difference principle for parameter iteration, and its risk value loss function is defined as:

[0047] This represents the next-time global security risk index estimate calculated by the second evaluation network of the objective. This indicates that the current second evaluation network (network parameters are) is used. The global risk index estimate under the current state combination calculated by forward propagation. This represents the discount factor for reinforcement learning (example value 0.99). This represents the set of state variables that characterize the macroscopic stability boundary of the vehicle at the next moment (including phase plane features such as the predicted centroid sideslip angle and yaw rate).

[0048] Step 4: First-Policy Network Constraint Update Based on Lagrange Duality Mechanism In the parameter optimization phase of the first policy network, this invention transforms the global security constraint into a restricted Markov decision process based on the Lagrange multiplier method. The global security constraint is a soft guidance mechanism: during policy learning, the global risk index output by the second evaluation network... It must be less than or equal to the preset physical security threshold. By utilizing the Lagrange duality mechanism to "guide" the policy network at the parameter update level, the generated policy network is made more efficient. In most operating conditions, the ability to spontaneously avoid hazardous areas is beneficial for safety control.

[0049] Specifically, the global safety constraints transform macroscopic stability indicators such as vehicle center of gravity sideslip angle, yaw rate deviation, and minimum risk strategy penalty term into a rigid constraint region on the policy network's action space. Let the parameters of the first policy network be... The preset absolute security risk tolerance threshold for the system is The system introduces nonnegative dynamic Lagrange multipliers. Construct an unconstrained joint loss function for minimax games:

[0050] in the formula For step two, the first policy network is reparameterized to output residual compensation increments. The mapping function represents the reparameterization output process of the first policy network. Indicates the current combined feature state Output residual compensation increment Under the action, the local tracking action value output by the first evaluation network (i.e., the current state action value estimate in the expectation operator in the formula). With global security risk index Together, they form a dual-dimensional evaluation benchmark of "performance-security", aiming to achieve a dynamic balance between the two through dual updates.

[0051] During model training, the system first updates the Lagrange multipliers along the gradient ascent direction based on the degree of constraint violation. .

[0052]

[0053] Represents Lagrange multipliers scalar learning rate during iterative update process The minimal-maximal joint loss function with respect to Lagrange multipliers The partial derivative (gradient) operator.

[0054] Subsequently, the system updates the parameters of the first policy network along the gradient descent direction. Its deterministic policy gradient expansion is as follows.

[0055]

[0056] In the gradient expansion of the deterministic policy, This indicates that the output value of the value evaluation network corresponds to the input value of the control action. The partial derivative (i.e., action gradient) operator, This indicates the relationship between the policy network action output value and the internal neural network parameters. The partial derivative (i.e., parameter gradient) operator.

[0057] When the system is within the safety envelope ( ), multiplier Converging to zero, the risk penalty gradient of the second evaluation network The isolated policy network parameters are entirely determined by the local performance gradient of the first evaluation network. Guidance. When control commands cause a risk of exceeding limits, the multiplier... The rapid accumulation of gradients in the second evaluation network forces the backpropagation direction of the dominant parameters, compelling the first policy network to correct its output and converge to the boundary of the safe region.

[0058] It is important to note that during offline training, to ensure effective exploration of the policy network in the state space and continuous backpropagation of gradients, Lagrange dual optimization is introduced. Macroscopic risk assessment is used as a soft mathematical constraint to guide the policy network to spontaneously adjust its weights. However, during online control (inference) phases, since the policy network parameters are already fixed, Lagrange dual optimization is no longer performed.

[0059] Step 5: Continuous Feedforward Reconstruction and Discrete Boundary Constraints Based on Global Risk Index While continuous and compliant degradation parameter tuning is performed under normal deviations, the system must ensure absolute physical safety in the face of potential nonlinear failures of the underlying physical actuators (controlled objects) in real vehicles. Therefore, a global risk index output by a second evaluation network is designed to address this issue. Construct a composite safety mechanism combining continuous damping adjustment and discrete boundary interception, when When the vehicle is approaching its absolute physical limits, quadratic programming (QP) is triggered, and rigid motion projection is performed using a control obstacle function. This composite safety mechanism implements a continuous safety feedforward monitoring mechanism.

[0060] First, a continuous model predictive control objective function reconstruction is performed. The system then uses the current global risk index. Mapped to damping weight coefficient The feedforward is then fed into the model predictive control cost function described in step one to achieve a smooth transition in the optimization direction of the predictive control domain. Next, discrete post-processing boundary constraints are executed. The initial fused control command of the system is set as follows: , The basic nominal control command is solved by the adaptive model predictive control layer described in step one. The discrete post-processing boundary constraint is a forced intervention mechanism based on safety verification logic, which is applied when the global risk index is predicted. Exceeding the set critical threshold At this point, the system abandons continuous flexible adjustment and instead uses control barrier functions to construct rigid mathematical inequality constraints. By solving a quadratic programming problem, it forces the fused instructions to optimize and project them into a set of feasible action domains that ensure absolute physical safety, thereby implementing bottom-line protection at the underlying execution level. (Critical threshold) Hard-line interception logic used during the online control phase. This is based on the risk index calculated in real-time. Exceed At this point, the system determines that the current "soft" adaptive adjustment is insufficient to prevent the vehicle from exceeding the physical safety envelope, and that the current flexible adjustment is no longer enough to prevent the vehicle from approaching its physical limits. The system then activates the control barrier function optimizer, revising the original output command... As the optimization baseline (i.e., the "nominal instruction"), it is forcibly projected into an absolutely safe action envelope through quadratic programming to generate the final instruction. And the above-mentioned In the Lagrange duality mechanism used during offline training, the constrained boundary, serving as the optimization objective, aims to guide the first policy network to spontaneously adjust its weights so that the output residual compensation instructions statistically conform to safety expectations. Typically... The set of feasible action domains is a set of control commands.

[0061] When the global risk index of the current vehicle state obtained by the second evaluation network exceeds the preset critical threshold... At this time, the system's underlying hardware controller will activate the control barrier function optimizer. The control barrier function optimizer uses a safety boundary function... Based on rigorous mathematical inequalities, by solving a quadratic programming problem, the fusion instructions are forcibly projected onto the action envelope set representing absolute physical security:

[0062]

[0063] in the formula To characterize the control barrier function of the vehicle dynamics phase plane stability boundary, Let its continuous-time derivative with respect to system state and final control input be . A strictly positive real number used to adjust the speed at which the system converges to the safety boundary, defining the hardness or softness of the safety region. Preset value. The system will use the final command, modified by this strict inequality constraint. It is passed to the underlying physical executor (the controlled object).

[0064] The action envelope set is the boundary of the set of all possible actions that an actuator or system can perform under the premise of satisfying physical constraints, stability constraints, and performance constraints.

[0065] The security boundary function The rigid execution tool of this invention provides a robust safety net for the "global safety constraints" when they fail under extreme failure conditions. It directly defines a positive invariant set and uses mathematical inequalities to forcibly intercept any instructions that might lead to a breach of the "macroscopic boundary".

[0066] The control barrier function: x The comprehensive state vector of a vehicle dynamics system, which includes, but is not limited to, vehicle speed. v Lateral position deviation Heading angle deviation , centroid side slip angle and yaw rate It is the smallest set of parameters that describes the real-time motion and attitude of a vehicle.

[0067] One implementation expression (i.e., a safety boundary based on the centroid sideslip angle) is as follows:

[0068] in It is the centroid sideslip angle The physical limit, It is a preset constant. When At this time, it indicates that the vehicle is in a stable state within the space.

[0069] The state evolution of a vehicle follows the dynamic equations: .in It is a system self-evolution term. It is the direct contribution of control inputs (such as steering angle and braking pressure) to the rate of change of state.

[0070] Derived from the chain rule: For scalar functions Find the total time derivative: Substitute into the dynamic equation ,in and They are respectively h Along the vector field f and g Li Daoshu. Because The explicit includes the control input to be requested. u In QP, this is... Therefore, it can be solved by solving the linear constraints. To reverse engineer the final instruction that meets the safety requirements. 。

[0071] The physical meaning is: in the current state x and control input u The combined effect of these factors determines the "rate of change of distance" between the vehicle and the safety boundary. If the system is simply... x If the function is not configured correctly, then the player can only "watch helplessly as the car slides out of bounds"; precisely because... and u Only by using a hook can the vehicle be stopped just before it crosses the boundary by changing its position. u This generates a reverse "resistance," forcing the vehicle to maintain its state at all times. Within the safe set.

[0072] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0073] For example, an actuator failure compensation and safety control method based on a two-layer evaluation architecture includes: a reinforcement learning model consisting of a first policy network, a first evaluation network, and a second evaluation network, wherein: the first policy network is configured to acquire basic nominal control commands. residual compensation increment The first evaluation network is configured to combine the state space vector and the residual compensation increment. As input, calculate the value of the local tracking action. The second evaluation network is configured to combine the state space vector and residual compensation instructions. and the state variables characterizing the macroscopic stability boundary of the vehicle As input, calculate the global security risk index. Set global security risk index Less than or equal to the preset physical security threshold To address global security constraints, the process is transformed into a restricted Markov decision process based on the Lagrange multiplier method, and implemented through multiplier updates. and The dynamic balance, thereby enabling the residual compensation increment generated by the first strategy network. It is conducive to security control; utilizing the global risk index Constructing damping weight coefficients Based on damping penalty weights Construct and solve the model predictive control objective function to obtain the current basic nominal control command. Thus, the initial fusion control command is obtained. If the current global risk index Less than or equal to the preset critical threshold The initial fusion control command will then be used as the final command. Otherwise, the initial fusion control command is forcibly projected into the set of actionable domains that ensure absolute physical safety and then passed to the underlying physical executor for execution.

[0074] For example, according to the method proposed in this disclosure, a corresponding actuator failure compensation and safety control system based on a two-layer evaluation architecture includes: an actuator failure compensation module, configured with a reinforcement learning model consisting of a first policy network, a first evaluation network, and a second evaluation network, wherein: the first policy network is configured to acquire basic nominal control commands. residual compensation increment The first evaluation network is configured to combine the state space vector and the residual compensation increment. As input, calculate the value of the local tracking action. The second evaluation network is configured to combine the state space vector and residual compensation instructions. and the state variables characterizing the macroscopic stability boundary of the vehicle As input, calculate the global security risk index. Set global security risk index Less than or equal to the preset physical security threshold To address global security constraints, the process is transformed into a restricted Markov decision process based on the Lagrange multiplier method, and implemented through multiplier updates. and The dynamic balance, thereby enabling the residual compensation increment generated by the first strategy network. It is beneficial for security control; the initial fusion control command generation module is configured to utilize the global risk index. Constructing damping weight coefficients Based on damping penalty weights Construct and solve the model predictive control objective function to obtain the current basic nominal control command. Thus, the initial fusion control command is obtained. The security control module is configured to adjust the current global risk index. Less than or equal to the preset critical threshold In this case, the initial fusion control command will be used as the final command. Otherwise, the initial fusion control command is forcibly projected into the set of actionable domains that ensure absolute physical safety and then passed to the underlying physical executor for execution.

[0075] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0076] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0077] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0078] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0079] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0081] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0082] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0083] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0084] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

[0085] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for actuator failure compensation and safety control based on a two-layer evaluation architecture, characterized in that, include: A reinforcement learning model consisting of a first policy network, a first evaluation network, and a second evaluation network, wherein: the first policy network is configured to acquire basic nominal control commands. residual compensation increment The first evaluation network is configured to combine the state space vector and the residual compensation increment. As input, calculate the value of the local tracking action. The second evaluation network is configured to combine the state space vector and residual compensation instructions. and the state variables characterizing the macroscopic stability boundary of the vehicle As input, calculate the global security risk index. Set global security risk index Less than or equal to the preset physical security threshold To address global security constraints, the process is transformed into a restricted Markov decision process based on the Lagrange multiplier method, and implemented through multiplier updates. and The dynamic balance, thereby enabling the residual compensation increment generated by the first strategy network. It is beneficial for safety control; Using the global risk index Constructing damping weight coefficients Based on damping penalty weights Construct and solve the model predictive control objective function to obtain the current basic nominal control command. Thus, the initial fusion control command is obtained. ; If the current global risk index Less than or equal to the preset critical threshold The initial fusion control command will then be used as the final command. Otherwise, the initial fusion control command is forcibly projected into the set of actionable domains that ensure absolute physical safety and then passed to the underlying physical executor for execution.

2. The method according to claim 1, characterized in that, The model predicts the control objective function as follows: In the formula: This represents the number of prediction time-domain steps for model predictive control. Indicates in Always looking towards the future The control input vector is calculated using a prediction step size. Indicates in Always looking towards the future A vehicle discrete state prediction vector with a prediction step size. For reference trajectory, Represented by a positive semi-definite matrix Q The weighted state tracking error quadratic penalty term is used. Represented by a positive definite matrix R The absolute value of the quadratic penalty term is used to control the weights. Represented by a positive definite matrix S Incremental input for weight control The quadratic penalty term, To control the input vector The increment.

3. The method according to claim 1, characterized in that, The initial fusion control commands are forcibly projected into the set of actionable domains that ensure absolute physical safety, specifically: To characterize the control barrier function of the vehicle dynamics phase plane stability boundary, Let its continuous-time derivative with respect to system state and final control input be . It is a pre-defined positive real number.

4. The method according to claim 1, characterized in that, Damping weighting coefficient The construction formula is as follows: In the formula: Based on the basic damping penalty weight, The boundary is truncated to the maximum damping penalty weight. This is the risk damping activation threshold.

5. The method according to claim 1, characterized in that, Residual compensation increment for basic nominal control commands : in, and The first policy network is based on the combined state space vector. The output Gaussian distribution means and standard deviation, Indicates that it follows a Gaussian distribution Independent random noise vectors are used for action exploration in the policy network. This represents the Hadama product operator.

6. The method according to claim 3, characterized in that, ,in It is the centroid sideslip angle The physical limit, It is a preset constant.

7. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 6.

8. An actuator failure compensation and safety control system based on a two-layer evaluation architecture, characterized in that, include: The actuator failure compensation module is configured with a reinforcement learning model consisting of a first policy network, a first evaluation network, and a second evaluation network, wherein: the first policy network is configured to acquire basic nominal control commands. residual compensation increment The first evaluation network is configured to combine the state space vector and the residual compensation increment. As input, calculate the value of the local tracking action. The second evaluation network is configured to combine the state space vector and residual compensation instructions. and the state variables characterizing the macroscopic stability boundary of the vehicle As input, calculate the global security risk index. Set global security risk index Less than or equal to the preset physical security threshold To address global security constraints, the process is transformed into a restricted Markov decision process based on the Lagrange multiplier method, and implemented through multiplier updates. and The dynamic balance, thereby enabling the residual compensation increment generated by the first strategy network. It is beneficial for safety control; The initial fusion control command generation module is configured to utilize the global risk index. Constructing damping weight coefficients Based on damping penalty weights Construct and solve the model predictive control objective function to obtain the current basic nominal control command. Thus, the initial fusion control command is obtained. ; The security control module is configured to adjust the current global risk index. Less than or equal to the preset critical threshold In this case, the initial fusion control command will be used as the final command. Otherwise, the initial fusion control command is forcibly projected into the set of actionable domains that ensure absolute physical safety and then passed to the underlying physical executor for execution.

Images