A vehicle-mounted fail-safe response control method and system based on reinforcement learning

Abstract

The present application relates to the technical field of vehicle control, and particularly relates to a vehicle-mounted fault safety response control method and system based on reinforcement learning, comprising: constructing a boundary dynamic physical feasible region based on physical limit data of an actuator and an effective adhesion coefficient, the dynamic physical feasible region being a hard constraint; under the condition of meeting the dynamic physical feasible region, solving a convex optimization problem with the objective of minimizing the deviation from an original action output, and with a logical inequality constraint group and a corresponding dynamic relaxation penalty weight as a penalty term, projecting and correcting the original action output into a safe corrected action; performing a deterministic compliance check independent of a reinforcement learning strategy network on the safe corrected action, and if the deterministic compliance check is passed, outputting the safe corrected action to the actuator, and if the deterministic compliance check is not passed, switching to a preset safety fallback strategy. The present application provides a solution for continuously online consideration of strict satisfaction of safety regulations and intelligent decision-making.

Description

Technical Field

[0001] This invention relates to the field of vehicle control technology, and in particular to a vehicle fault safety response control method and system based on reinforcement learning. Background Technology

[0002] On-board fault diagnosis is a key technology for ensuring safe vehicle operation. In the field of on-board fault diagnosis and safety control, in order to generate reasonable control commands to ensure vehicle safety when a fault occurs, traditional rule-based methods typically rely on preset fault thresholds and diagnostic fault codes for fault identification and response. With the increasing complexity of vehicle electronic and electrical architectures and the penetration of artificial intelligence technology, reinforcement learning methods are gradually being introduced into the field of on-board fault diagnosis and control. For example, reinforcement learning models are used to extract features from automotive instrument panel images and match fault lights, or a scoring feedback mechanism is introduced to optimize the convergence accuracy of reinforcement learning models in fault diagnosis of new energy vehicles.

[0003] The aforementioned existing technologies have the following technical problems: when the probabilistic decision of reinforcement learning output conflicts with the deterministic constraints required by functional safety specifications, the existing solutions unconditionally discard the reinforcement learning output and revert to a fixed safety action once it is detected that the reinforcement learning output has exceeded the preset safety boundary. This approach causes the decision contribution of the reinforcement learning agent to become zero in critical safety scenarios, and high-end AI computing power fails to produce positive effects at critical safety moments. Summary of the Invention

[0004] The main objective of this invention is to provide a vehicle fault safety response control method and system based on reinforcement learning, aiming to solve the technical problems mentioned in the background art.

[0005] This invention proposes a vehicle-mounted fault-safe response control method based on reinforcement learning, comprising: Obtain the redundancy comparison residual in multi-source sensor data and the verification error rate in communication verification data, and obtain the confidence coefficient to characterize the reliability of sensor signals based on the redundancy comparison residual and the verification error rate; The physical limit data of the actuator and the road surface adhesion coefficient are obtained. The road surface adhesion coefficient is corrected according to the confidence coefficient to obtain the effective adhesion coefficient. Then, a dynamic physical feasible region with adaptive shrinkage of the boundary as the confidence level of the sensor signal decreases is constructed based on the physical limit data of the actuator and the effective adhesion coefficient. The dynamic physical feasible region is a hard constraint. Obtain a set of logical inequality constraints generated by fault tree analysis rules and carrying functional safety level labels, and assign dynamic relaxation penalty weights with priority differences to each constraint in the set of logical inequality constraints. A vehicle operating state vector is constructed based on the multi-source sensor data, and the vehicle operating state vector is input into a reinforcement learning policy network to generate the original action output. Under the condition of satisfying the dynamic physical feasible region, solve a convex optimization problem with the objective of minimizing the deviation from the original action output and with the logical inequality constraint set and the corresponding dynamic relaxation penalty weight as the penalty term, and project and correct the original action output into a safe correction action. The security correction action is subjected to a deterministic compliance check independent of the reinforcement learning policy network. If the deterministic compliance check passes, the security correction action is output to the executor; otherwise, the system switches to a preset security rollback strategy. The degree of deviation between the safety correction action and the original action output is calculated, and the degree of deviation is injected as a negative feedback term into the reward function of the reinforcement learning policy network to drive the reinforcement learning policy network to actively generate the original action output that is closer to the dynamic physical feasible region.

[0006] The present invention also provides a vehicle-mounted fault-tolerant response control system based on reinforcement learning, the system comprising: The vehicle-mounted sensor group includes ASIL-D redundant sensors and vehicle dynamics sensors, used to collect multi-source sensor data, which includes redundancy comparison residuals and communication verification data. The vehicle control unit includes a first processor core and a second processor core; The first processor core is configured to perform the following operations: The confidence coefficient, used to characterize the reliability of the sensor signal, is calculated based on the redundancy ratio residual and the verification error rate in the communication verification data. Based on the physical limit data of the actuator and the road adhesion coefficient estimated based on the vehicle dynamics model, combined with the confidence coefficient, a dynamic physical feasible region whose boundary adaptively shrinks as the confidence level of the sensor signal decreases is constructed as a set of hard constraints. Obtain a set of logical inequality constraints generated by fault tree analysis rules and carrying functional safety level labels, and assign dynamic relaxation penalty weights with priority differences to each constraint according to the functional safety level labels of each constraint and the current vehicle operating status. Under the condition of satisfying the dynamic physical feasible region, a convex optimization problem is solved with the objective of minimizing the deviation from the original action output and with the logical inequality constraint set and the corresponding dynamic relaxation penalty weight as the penalty term. The original action output is projected and corrected into a safe correction action. A fixed number of iterations is used in the solution to ensure that the worst-case execution time is predictable. The second processor core is configured to perform the following operations: A vehicle operating state vector is constructed based on the multi-source sensor data, and the vehicle operating state vector is input into a reinforcement learning policy network to generate the original action output. Calculate the degree of deviation between the safety correction action and the original action output, and inject the degree of deviation as a negative feedback term into the reward function of the reinforcement learning policy network; A safety microcontroller, independent of the vehicle control unit, is communicatively connected to the first processor core and the second processor core via an internal data bus, and is configured to perform the following operations: Receive the security correction action and perform deterministic compliance verification on the security correction action, independent of the reinforcement learning policy network; A hardware gating circuit, connected between the safety microcontroller, the vehicle control unit, and the vehicle drive-by-wire actuator, is configured as follows: When the safety correction action passes the deterministic compliance check, the safety correction action is transmitted to the vehicle drive-by-wire actuator; When the security correction action fails the deterministic compliance check, the output of the security correction action is blocked, and the control signal channel corresponding to the preset security rollback strategy is activated.

[0007] A calibration storage area, connected to the first processor core, is used to store the physical limit data of the actuator.

[0008] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0009] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0010] The beneficial effects of this invention are as follows: This invention uses a confidence coefficient calculated from redundant comparison residuals and verification error rate to correct the road adhesion coefficient, so that the dynamic physical feasible domain adaptively shrinks as the reliability of sensor signals decreases, forming a hard constraint that cannot be relaxed; at the same time, the fault tree analysis rules are encoded into a set of logical inequality constraints with functional safety level labels, and dynamic relaxation penalty weights with priority differences are assigned to each constraint according to the level label and the current vehicle operating state, forming a soft constraint that can be relaxed; on this basis, convex optimization projection is performed with the goal of minimizing the deviation from the original action output, so that the safety correction action, under the premise of strictly satisfying the hard constraints, takes into account the logical rules in the order of safety level, and retains the original decision intention of the agent to the greatest extent. The action after projection correction is output after being verified by an independent deterministic audit unit. If the audit fails, the safety backoff strategy is switched; at the same time, the projection correction amount is injected into the reward function as a negative feedback term, driving the policy network to gradually migrate into the safety domain. Compared to existing technologies where reinforcement learning agents contribute nothing to decision-making in critical security scenarios and high-end AI computing power fails to generate positive utility at security-critical moments, the architecture of this invention provides a solution that balances strict compliance with security regulations with continuous online intelligent decision-making. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a method flow according to an embodiment of this application.

[0012] Figure 2 This is a schematic diagram of the system structure according to an embodiment of this application.

[0013] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0014] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0015] Example 1

[0016] This invention provides a vehicle fault safety response control method based on reinforcement learning. Figure 1 This is a flowchart illustrating a reinforcement learning-based vehicle fail-safe response control method according to an exemplary embodiment. Figure 1 As shown, this method is applied to an onboard fault diagnosis and safety control system, which typically includes multi-source sensors, actuators, vehicle dynamics models, fault tree analysis models, and reinforcement learning policy networks. The method specifically includes the following steps: S1. Obtain the redundancy comparison residual in the multi-source sensor data and the verification error rate in the communication verification data, and calculate the confidence coefficient to characterize the reliability of the sensor signal based on the redundancy comparison residual and the verification error rate.

[0017] In this embodiment, multi-source sensor data refers to signal data collected from various sensors in the vehicle, particularly including dual-channel or multi-channel data provided by redundant sensors that meet ASIL-D standards. Redundancy comparison residual refers to the output difference between multiple sensor channels that perform redundant measurements of the same physical quantity, reflecting the consistency and reliability of the sensor signals. Communication verification data refers to the verification result generated by cyclic redundancy check (CRC) according to the AUTOSAR E2E communication protocol during sensor data transmission; the error rate is the frequency of CRC errors. The confidence coefficient is a continuous value ranging from [0, 1], with a smaller value indicating lower reliability of the sensor signal. In practical applications, the confidence coefficient can be calculated in real time by statistically analyzing the redundancy comparison residual over a period of time, combining it with the communication verification error rate, and using a preset exponential decay function or other mapping relationship. When the sensor signal is highly reliable, the confidence coefficient approaches 1; when the sensor degrades or communication is abnormal, the confidence coefficient approaches 0.

[0018] S2. Obtain the physical limit data of the actuator and the road surface adhesion coefficient estimated based on the vehicle dynamics model, and correct the road surface adhesion coefficient according to the confidence coefficient to obtain the effective adhesion coefficient. Then, construct a dynamic physical feasible region whose boundary adaptively shrinks as the confidence level of the sensor signal decreases, based on the effective adhesion coefficient, as a set of hard constraints.

[0019] In this embodiment, the physical limit data of the actuators refers to the maximum output capabilities calibrated at the factory for actuators such as the braking system, steering system, and drive system. Examples include the maximum hydraulic pressure of the braking system, the maximum motor torque of the steering system, and the maximum output torque of the drive system. This data is typically stored in the calibration storage area of ​​the vehicle control unit. The vehicle dynamics model estimates the road surface adhesion coefficient in real time using estimation algorithms such as Kalman filtering. This coefficient characterizes the frictional characteristics between the tire and the road surface. Multiplying the confidence coefficient by the road surface adhesion coefficient yields the effective adhesion coefficient. Since the confidence coefficient reflects the reliability of the sensor signal, when the sensor signal is unreliable, the effective adhesion coefficient is suppressed, causing the dynamic physical feasible region constructed based on the effective adhesion coefficient to shrink inward, limiting the permissible range of vehicle movement. Specifically, the dynamic physical feasible region can be defined as a continuous and convex set in the acceleration space formed by longitudinal and lateral acceleration, based on the tire friction ellipse model. For example, it can be a domain that satisfies the friction ellipse inequality and related rate constraints. This dynamic physical feasible region serves as a set of unrelaxed hard constraints; any subsequent output action must strictly fall within this set.

[0020] S3. Obtain a group of logical inequality constraints generated by fault tree analysis rule encoding and carrying functional safety level labels, and assign dynamic relaxation penalty weights with priority differences to each constraint according to the functional safety level labels carried by each constraint and the current vehicle operating status.

[0021] In this embodiment, the fault tree analysis rules are fault causal logic extracted from the vehicle fault tree analysis (FTA) model, including AND and OR gate logic relationships between the top event, intermediate events, and bottom event. These logic rules are linearized and encoded to form a set of rules of the form A. i a≤ b i The inequalities are defined such that each constraint corresponds to a set of prohibited action triggering conditions, forming a logical inequality constraint group. Simultaneously, each constraint is assigned a functional safety level label, such as QM, ASIL-A, ASIL-B, ASIL-C, or ASIL-D, to indicate the severity of the hazard involved. When assigning dynamic relaxation penalty weights, on the one hand, different baseline penalty coefficients are determined for each constraint based on the functional safety level label; the higher the safety level, the larger the baseline penalty coefficient. On the other hand, a dynamic adjustment factor is calculated based on the current vehicle operating state, such as vehicle speed measurement and effective adhesion coefficient; the higher the vehicle speed or the lower the effective adhesion coefficient, the larger this adjustment factor. Multiplying the baseline penalty coefficient by the dynamic adjustment factor yields a dynamic relaxation penalty weight with priority differences that changes in real time with the operating conditions. This weight will be used in subsequent convex optimization solutions, ensuring that the penalties for all safety constraints are significantly increased under hazardous conditions, while allowing appropriate relaxation of lower-level constraints when the safety margin is sufficient.

[0022] S4. Construct a vehicle operating state vector based on the multi-source sensor data, input the vehicle operating state vector into the reinforcement learning policy network, and generate the original action output.

[0023] In this embodiment, the vehicle operating state vector is the state space input of the reinforcement learning agent. It integrates key features extracted from multi-source sensor data, including vehicle speed, yaw rate, sensor measurements, fault characteristics, and the aforementioned calculated confidence coefficient and effective adhesion coefficient. The reinforcement learning policy network is a deep neural network that is pre-trained or learned and updated online. It receives the vehicle operating state vector and outputs a probability distribution representing the tendency of various possible diagnostic or control actions. The mathematical expectation or sampling result of this probability distribution serves as the original action output, representing the pure intention of the RL agent without considering safety constraints.

[0024] S5. Under the condition of satisfying the dynamic physical feasible region, solve the convex optimization problem with the objective of minimizing the deviation from the original action output and with the logical inequality constraint set and the corresponding dynamic relaxation penalty weight as the penalty term. The original action output is projected and corrected into a safe correction action. A fixed number of iterations is used in the solution to ensure that the worst-case execution time is predictable.

[0025] In this embodiment, the objective function of the convex optimization problem consists of two terms: the first term is the square of the Euclidean distance between the action to be solved and the original action output, used to preserve the original RL decision preference; the second term is the sum of the products of the violation amounts of each logical inequality constraint after non-negative truncation and the corresponding dynamic relaxation penalty weights, used to penalize violations of the FTA logical rules. The hard constraint condition is that the action to be solved must belong to the dynamic physical feasible region. This convex optimization problem can be solved iteratively using the alternating direction multiplier method or the alternating projection method, with the number of iterations preset to a fixed value, such as 10. This fixed value is calibrated so that the worst-case execution time of the solver is much smaller than the fault tolerance time interval required by the functional safety specification, thereby ensuring the deterministic requirements of vehicle real-time performance. After the solution is completed, a safe correction action is obtained within the hard constraint set, which retains the original decision intent to the greatest extent and prioritizes the satisfaction of high-safety-level logical rules.

[0026] S6. Perform a deterministic compliance check on the security correction action that is independent of the reinforcement learning policy network. If the deterministic compliance check passes, output the security correction action to the executor. If it fails, switch to the preset security rollback strategy.

[0027] In this embodiment, deterministic compliance verification is performed by an independent safety audit unit. This unit does not rely on any learning model and makes judgments solely based on preset deterministic rules. The verification includes: whether the safety correction action meets the physical boundary conditions and logical prohibition rules corresponding to the highest functional safety level label (such as ASIL-D), and whether the magnitude of the action change is within the preset allowable range of change rate. If all verification items pass, compliance is determined, and the safety correction action is transmitted to the vehicle's drive-by-wire actuator for execution. If any item fails, the action output is blocked, and a pre-formally verified physical safety backoff operation is triggered, such as gradual braking to a stop or restricted speed reduction. This mechanism constitutes a two-layer safety architecture of soft projection optimization and hard audit fallback.

[0028] S7. Calculate the degree of deviation between the safety correction action and the original action output, and inject the degree of deviation as a negative feedback term into the reward function of the reinforcement learning policy network to drive the reinforcement learning policy network to actively generate the original action output that is closer to the dynamic physical feasible domain.

[0029] In this embodiment, the degree of deviation can be quantified using the Euclidean distance between the safety correction action and the original action output. This distance characterizes the extent to which the original RL decision is corrected by the safety constraint system. The net reward value is obtained by subtracting the product of this deviation and a preset gain coefficient from the original task reward value, serving as the final reward signal for the reinforcement learning policy network at the current moment. The network parameters are updated using this net reward value through a policy gradient algorithm. The RL agent will gradually learn that actions that deviate from the safety boundary will yield higher rewards, thus intrinsically approaching the safety region during training and online fine-tuning, continuously reducing the degree of correction. This closed-loop feedback gradually internalizes the originally external mandatory safety constraints into the agent's own preferences.

[0030] The technical solution of the above embodiments, through the synergistic effect of the above steps, can solve the problem of unifying the probabilistic decision space of RL and the deterministic logic space of functional safety. First, by using the dynamic physical feasible region corrected by sensor confidence, the hard constraint boundary is adaptively contracted to sensor degradation and changes in operating conditions. Second, by introducing a set of FTA logical inequalities with functional safety level labels and assigning dynamic relaxation penalty weights based on real-time operating conditions, differentiated processing of safety priorities is achieved. On this basis, through convex optimization projection, the original action output of RL is corrected into a safety correction action that satisfies all constraints, ensuring that RL decisions always play a role within the entire safety boundary. Finally, the deterministic auditor provides a formally verifiable safety guarantee, and the closed-loop feedback of the projection correction drives the agent's strategy to evolve towards the interior of the safety domain. In summary, this embodiment, while meeting the ISO26262 functional safety certification requirements, maximizes the contribution of the RL agent in safety-critical scenarios and endows the system with long-term adaptability to dynamic environments.

[0031] In one possible design, step S1, the step of calculating the confidence coefficient used to characterize the reliability of the sensor signal based on the redundancy comparison residual and the verification error rate, includes: S11. Obtain the redundant comparison residual generated by the dual-channel comparison of the ASIL-D redundant sensor, and calculate the residual variance of the redundant comparison residual within the sliding time window. In this embodiment, an ASIL-D redundant sensor refers to a sensor group that meets the ASIL-D functional safety level label requirements of ISO 26262 and performs independent dual-channel measurements of the same physical quantity. The redundancy comparison residual refers to the instantaneous difference between the measurements of the two channels. To smooth instantaneous measurement noise and reflect the trend of sensor degradation, a fixed-length sliding time window is set, and the redundancy comparison residual sequence within this window is continuously acquired. Its statistical variance is then calculated to obtain the residual variance. The larger the residual variance, the worse the consistency between the two redundant channels, and the lower the reliability of the sensor signal.

[0032] The length of the sliding time window is determined based on the signal bandwidth of the ASIL-D redundant sensor and the control cycle of the vehicle control unit. In one embodiment, the length of the sliding time window ranges from 50 to 200 control cycles to achieve a balance between statistical stability and response speed. For example, when the control cycle of the vehicle control unit is 10 ms, the length of the sliding time window can be selected as 100 control cycles, corresponding to a 1-second time window. This length is sufficient to smooth out transient noise and can promptly reflect the degradation trend of the sensor signal. S12. Obtain the CRC check error rate statistically analyzed during the data transmission process of the AUTOSAR E2E communication protocol, and use it as the check error rate; In this embodiment, the AUTOSAR E2E communication protocol provides an end-to-end protection mechanism for data communication between the vehicle-mounted sensors and the control unit. Its cyclic redundancy check (CRC) can detect bit errors during data transmission. The communication controller continuously calculates the ratio of CRC check failures to the total number of transmissions, which is used as the CRC error rate. This CRC error rate directly reflects the health status of the communication link.

[0033] S13. Based on the residual variance and the verification error rate, a value between 0 and 1 is calculated using a preset exponential decay function as the confidence coefficient, wherein the confidence coefficient approaches zero when the ASIL-D redundant sensor signal degrades or communication is abnormal.

[0034] In this embodiment, the residual variance and the verification error rate are used as input variables and substituted into a preset exponential decay function to calculate the confidence coefficient in real time. This exponential decay function can be expressed as:

[0035] Wherein, c(t) is the confidence coefficient, with a value ranging from 0 to 1; σ e 2 (t) represents the residual variance, i.e., the variance of the redundancy comparison residual within the sliding time window, calculated in step S11; r CRC (t) represents the CRC check error rate, which is the CRC check error rate statistically calculated by the AUTOSAR E2E communication protocol and obtained in step S12; α is a first preset constant used to adjust the influence of the residual variance on the confidence coefficient; β is a second preset constant used to adjust the influence of the check error rate on the confidence coefficient; t is the current time. This function ensures that when the sensor signal is normal, both inputs are close to 0, and the confidence coefficient approaches 1; when either input increases, the confidence coefficient decays exponentially, approaching 0, thus mapping the decrease in the confidence level of the sensor signal to a numerical output in a continuous and differentiable manner.

[0036] In one implementation, the values ​​of the first preset constant α and the second preset constant β are both between 0.1 and 10.0, and the specific values ​​are calibrated based on the noise characteristics of the ASIL-D redundant sensor and the historical statistical data of the AUTOSAR E2E communication protocol. Generally, when the typical noise variance of the redundancy comparison residual is large, a smaller α value should be selected to avoid the confidence coefficient being too sensitive to normal noise; when the baseline level of the CRC check error rate of the communication link is high, a smaller β value should be selected to maintain a reasonable confidence range. The specific values ​​of the first preset constant and the second preset constant are determined through sensor calibration experiments and communication tests, and are fixed in the configuration parameters before the vehicle control unit leaves the factory.

[0037] The technical solution described above, by introducing dual indicators—sliding time window residual variance and communication verification error rate—and employing an exponential decay function for fusion calculation, can continuously and in real-time quantify the reliability of sensor signals. Compared to simple threshold judgment, the confidence coefficient provides richer signal quality information, enabling downstream modules to adjust the conservatism of the safety boundary proportionally based on the reliability level, rather than making discrete switches between reliable and unreliable, thereby maximizing system availability while ensuring safety.

[0038] In one possible design, step S2, which involves correcting the road surface adhesion coefficient based on the confidence coefficient to obtain an effective adhesion coefficient, and then constructing a dynamic physically feasible domain whose boundary adaptively shrinks as the confidence level of the sensor signal decreases, includes: S21. Obtain the preset physical limit data of the actuator from the calibration storage area of ​​the vehicle control unit. The physical limit data of the actuator includes the maximum hydraulic pressure value of the braking system, the maximum motor torque value of the steering system, and the maximum output torque value of the drive system. In this embodiment, the calibration storage area of ​​the vehicle control unit is a non-volatile memory inside or outside the MCU, which stores the physical limit values ​​of various actuators calibrated at the time of vehicle delivery. These parameters are unaffected by sensor signal degradation or communication packet loss and are the absolute constraints at the lowest level of constructing the physical boundaries.

[0039] It should be noted that the physical limit data of the actuator reflects the calibrated maximum output capability of the actuator in a good condition. In actual operation, if the actuator suffers partial physical damage or performance degradation (e.g., leakage in the brake hydraulic line leading to a decrease in maximum pressure build-up capability, or a partial short circuit in the motor winding leading to a decrease in maximum output torque), the actual usable output range of the actuator will be less than the factory value fixed in the calibration storage area. For such actuator failure scenarios, this solution addresses them through the following mechanisms: Firstly, the construction of the dynamic physical feasible domain includes the friction ellipse boundary corrected by the effective adhesion coefficient and the upper limit constraint of the actuator's rate of change. These constraints further narrow the allowable range of motion within the actuator's physical limit data. Secondly, the rate of change verification in the deterministic compliance check and the preset backoff operations such as gradual braking in the safety backoff strategy are all adjusted based on the actual feedback of the actuator state at the current moment, rather than relying solely on the factory-calibrated maximum limit value. In extreme scenarios where the actuator fails completely (such as a broken brake line), the safety audit unit determines the violation and triggers the switching of redundant execution channels or an emergency braking request in the safety rollback strategy. This part of the logic falls within the scope of the formal verification of the safety rollback strategy, and will not be elaborated on here.

[0040] S22. Multiply the confidence coefficient by the road surface adhesion coefficient estimated in real time by the vehicle dynamics model through Kalman filtering to obtain the effective adhesion coefficient; In this embodiment, the road adhesion coefficient is estimated online using a Kalman filter algorithm based on vehicle dynamics models combined with signals such as vehicle speed, wheel speed, and yaw rate. This estimate itself depends on sensor input, and its accuracy is affected by the reliability of the sensor signals. Therefore, the confidence coefficient is directly multiplied by the road adhesion coefficient, and the calculation formula is as follows:

[0041] Where, μ eff μ(t) is the effective adhesion coefficient; μ(t) is the road surface adhesion coefficient, which is estimated in real time by the vehicle dynamics model through Kalman filtering; c(t) is the confidence coefficient, which is calculated in step S1. When the confidence coefficient decreases due to unreliable sensor signals, the effective adhesion coefficient decreases accordingly, thereby simulating the effect of road surface slippage at the algorithm level, forcing the subsequent action boundary to contract.

[0042] S23. Based on the ratio of the effective adhesion coefficient to the gravitational acceleration, determine the safe action boundaries of the vehicle in the longitudinal and lateral acceleration dimensions based on the tire friction ellipse model. In this embodiment, the tire friction ellipse model describes the maximum adhesion boundary that the tire can provide in the longitudinal and lateral acceleration planes. The effective adhesion coefficient is multiplied by the gravitational acceleration and used as the semi-axis parameter of the friction ellipse, thus constructing the following friction ellipse inequality:

[0043] Among them, a x For the longitudinal acceleration request; a y For the lateral acceleration request; μ eff (t) is the effective adhesion coefficient, calculated in step S22; g is the gravitational acceleration, a physical constant; a x,max The maximum longitudinal acceleration is determined by the physical limit data of the actuator. When the left-hand side of this inequality is less than or equal to 1, it indicates that the motion vector is within the allowable range of tire adhesion; when it is greater than 1, it indicates that the physical feasible boundary has been exceeded. As the effective adhesion coefficient decreases, the elliptical boundary contracts inward, and the safe motion range narrows.

[0044] S24. The safety action boundary, the physical limit data of the actuator, and the preset upper limit constraint of the actuator action change rate are combined to form the dynamic physical feasible domain that is continuous and convex in the acceleration space.

[0045] In this embodiment, the upper limit constraint on the rate of change of actuator action is used to prevent sudden changes in control commands from causing vehicle instability, and its form is as follows:

[0046] Where a is the action vector, a = [a x ,a y ];R ramp The upper limit constraint on the rate of change of the actuator's motion is a preset constant. Taking the intersection of the three constraints—the friction elliptical boundary, the actuator's physical limit, and the rate limit—constitutes a continuous, closed, and convex dynamic physical feasible region within the acceleration space. This convexity guarantees the global optimality and solution efficiency of subsequent optimization solutions.

[0047] The technical solution of the above embodiment cleverly couples the sensor's health status with the vehicle's dynamic safety boundary in a physical sense by applying the sensor confidence coefficient to the correction of the road surface adhesion coefficient. When the sensor degrades, the confidence coefficient decreases, the effective adhesion coefficient decreases accordingly, the friction ellipse automatically contracts, and the vehicle's allowed range of motion becomes smaller. This allows the system to adapt to information uncertainty in a "conservative driving" manner, maintaining the system's continuous operation capability without sacrificing safety, rather than directly exiting intelligent control.

[0048] In one possible design, step S3, which involves assigning dynamic relaxation penalty weights with priority differences to each constraint based on the functional safety level label carried by each constraint and the current vehicle operating state, includes: S31. Obtain the logical inequality constraint group generated by linearizing the AND and OR logic relationships between the top, middle and bottom events in the fault tree model, where each constraint corresponds to a combination of prohibited action triggering conditions. In this embodiment, the fault tree model is a system-level safety model for whole-vehicle FTA analysis. The top event corresponds to a vehicle-level hazardous event (such as unexpected brake failure), while intermediate and bottom events decompose the fault causes layer by layer. AND and OR gates are used to describe the logical propagation relationships between events. These causal logical relationships are extracted and transformed into linear inequality constraints in the action space. For example, "If the redundant residual of the brake master cylinder pressure sensor exceeds the limit, then increasing the brake pressure is prohibited" can be encoded as a. pressure ≤-0.5; For example, "If the yaw rate of the steering wheel angle sensor and the IMU is inconsistent and exceeds the limit, then large-angle steering is prohibited" can be coded as a steer ≤0.3. The set of all such rules constitutes a set of logical inequality constraints, whose general form is A. i a≤ b i A i Let b be the coefficient vector of the i-th constraint. i Let be the threshold of the i-th constraint, and 'a' be the action vector.

[0049] S32. Obtain the functional safety level label for each constraint in the logical inequality constraint group, and determine a baseline penalty coefficient for each constraint based on the functional safety level label, wherein the constraint corresponding to a higher functional safety level label is assigned a higher baseline penalty coefficient. In this embodiment, the functional safety level label is the ASIL level assessment result based on the ISO 26262 standard for the hazardous events involved in this constraint, including QM, ASIL-A, ASIL-B, ASIL-C, and ASIL-D. ASIL-D corresponds to the highest safety integrity requirement, involving personal safety; QM corresponds to the quality management level, involving only comfort or economy. Baseline penalty coefficient K ASIL (i) Use magnitude difference mapping, for example, ASIL-D is set to 10. 4 Setting ASIL-C to 10³, ASIL-B to 10², and ASIL-A and QM to 10 results in a difference in the magnitude of the penalty for different security level constraints in the optimization objective function.

[0050] S33. Obtain the vehicle speed measurement value and the effective adhesion coefficient at the current moment, and calculate a dynamic adjustment factor that monotonically increases as the vehicle speed measurement value increases or the effective adhesion coefficient decreases. In this embodiment, the vehicle speed measurement value comes from the vehicle bus or wheel speed sensor, and the effective adhesion coefficient is calculated in real time in step S2. Dynamic adjustment factor f(v(t),μ eff (t) is designed as an increasing function of vehicle speed and a decreasing function of the effective adhesion coefficient. For example, f = v(t) / (μ) can be used. eff The expression is in the form of (t) + ε, where ε is a small constant excluding zero, v(t) is the measured vehicle speed, and μ eff (t) represents the effective adhesion coefficient. This significantly increases the adjustment factor under extremely dangerous conditions of high speed and low adhesion, while the adjustment factor approaches its minimum value under safe conditions of low speed and high adhesion.

[0051] The dynamic adjustment factor f(v(t),μ eff The physical meaning of f(t) lies in quantifying the potential danger level of the current operating condition. Increased vehicle speed means increased collision energy, and a decreased effective adhesion coefficient means reduced vehicle handling margin; both require stricter safety constraints. In addition to the above, f=v(t) / (μ) eff In addition to the form (t)+ε), in other embodiments, the dynamic adjustment factor may also be adopted The linear mapping form, or the lookup table function form with vehicle speed and effective adhesion coefficient as inputs, where k 1` k2 is a preset constant. The common characteristic of different function forms is that: when the vehicle speed measurement value increases, the dynamic adjustment factor monotonically increases; when the effective adhesion coefficient decreases, the dynamic adjustment factor monotonically increases.

[0052] S34. Multiply the baseline penalty coefficient of each constraint by the dynamic adjustment factor to obtain the dynamic relaxation penalty weight of that constraint.

[0053] In this embodiment, the formula for calculating the dynamic relaxation penalty weight is:

[0054] Where, λ i (t) represents the dynamic relaxation penalty weight of the i-th constraint; K ASIL (i) is the baseline penalty coefficient for the i-th constraint, determined by the functional safety level label contained in that constraint; f(v(t),μ eff (t) is the dynamic adjustment factor; v(t) is the vehicle speed measurement value, which is the real-time vehicle speed at the current moment; μ eff(t) represents the effective adhesion coefficient, calculated in step S2. This weight combines the static ASIL level priority with the dynamic degree of hazard in the operating condition. This means that even at the same ASIL level, the penalty weight in high-speed, low-adhesion operating conditions will be much higher than that in low-speed, high-adhesion operating conditions; at the same time, under any operating condition, the penalty weight of ASIL-D level constraints is always several orders of magnitude higher than that of ASIL-A / QM level, ensuring the absolute priority of the highest safety level.

[0055] The technical solution described above achieves a dynamic and static management of safety constraint priorities by mapping ASIL level labels and real-time operating condition information together as dynamic relaxation penalty weights. Traditional solutions only process constraints with uniform weights or fixed priorities, failing to distinguish the severity of different ASIL levels or adapt to changes in the driving environment. This embodiment, on the one hand, ensures that high-safety-level constraints are always prioritized in optimization through a quantified benchmark penalty coefficient; on the other hand, it uses a dynamic adjustment factor to automatically tighten all safety constraints under dangerous operating conditions and moderately relax low-level constraints under safe operating conditions to retain more decision-making freedom, thereby achieving a dynamic balance between safety and availability.

[0056] In one possible design, step S5 involves solving a convex optimization problem that aims to minimize the deviation from the original action output and uses the set of logical inequalities and the corresponding dynamic relaxation penalty weights as penalty terms. The original action output is projected and corrected into a safe corrected action. The solution process employs a fixed number of iterations and includes: S51. Construct an objective function, the first term of which is the square of the Euclidean distance between the action to be solved and the original action output, and the second term is the sum of the product of the violation amount of each of the logical inequality constraints after non-negative truncation and the corresponding dynamic relaxation penalty weight. In this embodiment, the objective function takes the following form:

[0057] Where a is the action to be solved (i.e., the action vector); p is the original action output, generated by the reinforcement learning policy network; λ is the square of the Euclidean distance, which is the first term of the objective function, representing the deviation between the action to be solved and the original action output; i (t) is the dynamic relaxation penalty weight of the i-th constraint, which is calculated in step S3; A i a≤ b i Let A be the logical inequality constraint in the i-th row, where A i Let b be the coefficient vector. i The threshold is max(0,A). i·ab i Let be the violation amount of the i-th constraint, which is zero when the constraint is satisfied and positive when it is violated; M is the total number of constraints in the logical inequality constraint group. The first term measures the deviation of the action to be solved from the original action output, and the second term imposes a penalty for violating the logical inequality constraint. Non-negative truncation ensures that a penalty term is generated only when the constraint is violated, and the penalty term is zero when the constraint is satisfied.

[0058] It should be noted that the dynamic physical feasible region Ω phy (t) is always a non-empty set. This is because the dynamic physical feasible region contains at least a zero-action vector (i.e., no acceleration, braking, or steering operation is applied), which satisfies the physical limit data of all the actuators and the upper limit constraint of the actuator's rate of change; simultaneously, due to the effective adhesion coefficient μ... eff Since (t) is always greater than zero, the frictional elliptic inequality always holds at the origin. Therefore, the convex optimization problem always has at least one feasible solution. When the set of logical inequality constraints severely conflicts with the original action output, the safety correction action will degenerate into the action that minimizes the weighted violation of the set of logical inequality constraints within the dynamic physical feasible region. Its safety compliance is ultimately guaranteed by the subsequent deterministic compliance verification.

[0059] S52. The dynamic physical feasible region is used as the hard constraint condition that the action to be solved must satisfy, which together with the objective function constitutes a constrained convex optimization problem. In this embodiment, the complete convex optimization problem is expressed as:

[0060]

[0061] Where a* represents the safety correction action, obtained from this step; Ω phy (t) represents the dynamic physical feasible region, constructed in step S2, and serves as a hard constraint that the action to be solved must strictly satisfy; the meanings of the other symbols are the same as in step S51. Hard constraint a∈Ω phy (t) indicates that the action to be solved must strictly fall within the dynamic physical feasible region. This condition is treated as a non-relaxable constraint in the optimization problem, forming a hierarchical structure with the relaxable logical soft constraints in the objective function.

[0062] S53. The convex optimization problem is solved iteratively using the alternating direction multiplier method or the alternating projection method. The number of iterations is preset to a fixed value, which makes the worst-case execution time of the solver less than the fault tolerance time interval required by the functional safety specification. In this embodiment, the alternating direction multiplier method decomposes the original problem into multiple subproblems and solves them alternately. The alternating projection method uses convex set projection iteration to approximate the optimal solution. The number of iterations is fixed at 10. This value has been calibrated by actual testing on the ARM Cortex-A55 core, ensuring that the total time consumption of a single solution, including all overhead such as matrix multiplication and vector operations, is less than 0.8ms, which is far lower than the fault tolerance time interval specified by ISO26262 (usually less than 100ms), thus ensuring the determinism and predictability of the worst-case execution time. The calibration method for the number of iterations is as follows: First, determine the computational cost of each iteration in the convex optimization problem. In each iteration, the alternating projection method mainly performs one projection operation to the dynamic physical feasible region and several projection operations to the logical inequality constraint group. The computational cost of each projection operation is on the order of the product of the action space dimension and the total number of constraints in the logical inequality constraint group. When the action space dimension ≤ 20 and the total number of constraints ≤ 50, the total number of floating-point operations in a single iteration does not exceed 10. 4 The calibration is performed on a target deployment platform (e.g., ARM Cortex-A55, 1.2GHz) to record the actual time taken for each iteration. Finally, this calibration time is compared with the fault tolerance time interval (10ms to 100ms) required by the functional safety specification to confirm a safety margin of more than an order of magnitude. The fixed number of iterations, 10, is the calibration value chosen to balance solution accuracy and real-time performance under this margin. For different deployment platforms or different constraint sizes, the fixed number of iterations is re-determined using the same calibration process. In other implementations, when the deployment platform has higher computing power or the constraint size is smaller, the fixed number of iterations can be appropriately increased to obtain higher solution accuracy; when the deployment platform has lower computing power or the constraint size is larger, the fixed number of iterations can be appropriately reduced to ensure that the worst-case execution time always meets the fault tolerance time interval requirement. The specific value of the fixed number of iterations is determined through offline calibration and is fixed in the configuration parameters before the vehicle control unit leaves the factory, and is not dynamically changed during operation to ensure the determinism of timing.

[0063] S54. After solving the fixed number of iterations, a safety correction action is obtained that, within the dynamic physical feasible domain, prioritizes satisfying the logical inequality constraint of the higher functional safety level label and maintains the original decision intent to the greatest extent.

[0064] In this embodiment, the solution a* obtained after 10 iterations is the safety correction action. Although it may not have reached the exact optimal solution mathematically, according to the convergence theory of convex set projection, the error between the solution after 10 iterations and the theoretical optimal solution has been limited to a controllable range. Moreover, this solution strictly satisfies all hard constraints, prioritizes soft constraints with high ASIL levels, and is as close as possible to the original action output p. This approximate solution meets engineering safety requirements.

[0065] The technical solution described above integrates the probabilistic output of RL with physical and logical safety constraints within a unified mathematical framework by constructing a convex optimization problem with hierarchical constraints and solving it using a fixed number of iterations. The hard physical constraints guarantee that the absolute safety baseline will never be breached, while the soft logical constraints achieve flexible management of safety priorities through ASIL-aware dynamic weights. The design of a fixed number of iterations rather than a convergence criterion makes the worst-case execution time of the solution process known a priori, a key engineering decision to meet the real-time deterministic requirements of ISO 26262.

[0066] In one possible design, step S6, which involves performing a deterministic compliance check on the security correction action independent of the reinforcement learning policy network, and outputting the security correction action to the executor if the deterministic compliance check passes, and switching to a preset security fallback strategy if it fails, includes: S61. The security correction action output after projection correction is obtained by the security audit unit, which is independent of the reinforcement learning policy network and does not depend on the learning model. In this embodiment, the security audit unit is a formally verified independent hardware or software module. Its design complies with the ISO 26262 requirements for the independence of the safety-critical path and has no shared memory, shared computing resources, or data dependencies with the reinforcement learning policy network. The security audit unit only receives the security correction action a* as input and does not access the internal state, weights, or training data of the RL policy network, ensuring that its judgment results are not affected by any uncertainties or potential failure modes of the learning model.

[0067] S62. The security audit unit performs deterministic boundary compliance verification on the security correction action based on the physical boundary conditions and logical prohibition rules corresponding to the highest preset functional safety level label, and performs change rate verification on the security correction action at the current moment. In this embodiment, the highest functional safety level label refers to ASIL-D level. Its corresponding physical boundary conditions include the absolute upper and lower limits of action defined by the physical limit data of the actuator. The logical prohibition rules include all ASIL-D level logical inequality constraints selected from the fault tree analysis rules. Boundary compliance verification uses deterministic Boolean logic, without involving any probabilistic judgments or fuzzy reasoning. Specifically, it verifies whether each dimension of the safety correction action satisfies the corresponding physical boundary conditions and logical prohibition rules. The rate of change verification calculates the absolute value of the difference between the safety correction action and the action executed at the previous moment, and determines whether it is less than or equal to the preset allowable rate of change range. The calculation method for the rate of change verification can be expressed as:

[0068] Where a*(t) represents the safety correction action at the current moment; a executed (t-1) represents the action executed at the previous moment, i.e., the action actually output to the actuator in the previous control cycle; Δt is the time step of the control cycle; R max The upper limit of the allowable range of the rate of change is a preset constant.

[0069] S63. If the safety correction action passes both the boundary compliance check and the rate of change check, it is determined to be compliant, and the safety correction action is transmitted to the vehicle drive-by-wire actuator to perform the corresponding operation. In this embodiment, the vehicle drive-by-wire actuator includes a brake-by-wire actuator, a steering-by-wire actuator, and a drive-by-wire actuator, which are used to perform braking, steering, and driving operations, respectively. Compliance determination means that a safety correction action is confirmed as safe in three dimensions: physical boundaries, logical rules, and rate of change, and can be directly converted into actuator control commands.

[0070] S64. If any verification item fails, it is determined to be a violation, the output of the security correction action is blocked, and a physical security rollback operation that has been formally verified in advance and does not depend on the agent's decision is triggered as the security rollback strategy.

[0071] In this embodiment, the physical safety rollback operation is a pre-defined sequence of safety operations based on the vehicle's current state and fault type. Examples include gradual braking to a stop, limiting the vehicle speed to a safe range, or switching to a redundant execution channel. The design and verification process of this rollback operation is independent of any learning component and undergoes formal verification based on model checking or theorem proving, ensuring its safety and determinism under all reachable conditions. Simultaneously, violations are recorded in a safety log, including a timestamp, violation type, the value of the safety correction action, and the triggered rollback operation type, for offline analysis and functional safety auditing.

[0072] Specifically, the physical implementation of the triggering is as follows: When the safety audit unit determines a violation, it generates a binary violation indication signal. This signal acts directly on the hardware gating circuit connected to the vehicle drive-by-wire actuator in the form of a hardware interrupt or a GPIO level transition. After receiving the binary violation indication signal, the hardware gating circuit, under the control of its internal hardware logic gates, switches the source of the control signal output to the vehicle drive-by-wire actuator from the first signal channel corresponding to the safety correction action to the second signal channel corresponding to the physical safety rollback operation. The second signal channel carries a fixed control signal sequence pre-stored in a read-only memory, independent of any programmable processor's real-time computation. The delay of the entire triggering and switching process, from the safety audit unit determining the violation to the hardware gating circuit completing the channel switching, is determined by the propagation delay of the hardware circuit, which is on the order of microseconds and much smaller than the fault tolerance time interval. The control signal sequence of the physical safety rollback operation is pre-calculated offline based on the vehicle dynamics model and braking system characteristics, and after formal verification, it is fixed in the read-only memory and cannot be changed during operation. In one embodiment, the control signal sequence for the physical safety rollback operation includes: a braking control command sequence that gradually increases the braking pressure at a preset deceleration slope until the vehicle stops, or a control command sequence that gradually reduces the drive torque to zero at a preset rate while simultaneously applying the parking brake. Both the preset deceleration slope and the preset rate have been proven during the formal verification process not to cause vehicle instability under all reachable conditions.

[0073] The specific method for formal verification includes: modeling the control flow of the physical safety rollback operation as a finite state machine, and expressing the desired safety attributes of the safety rollback strategy (e.g., "under any operating condition, the braking deceleration during rollback never exceeds a preset upper limit" or "no positive driving torque is generated during rollback") as linear time-series logic formulas. An exhaustive search is then performed on all reachable states of the finite state machine using model verification tools (such as NuSMV or UPPAAL) to verify whether the desired safety attributes hold true consistently on all reachable paths. The process and results of the formal verification are stored as technical evidence for functional safety certification.

[0074] In one implementation, the output of the security audit unit is connected to the control signal link of the vehicle's drive-by-wire actuator via a hardware gating circuit. When compliance is determined, the hardware gating circuit activates the control signal channel corresponding to the safety correction action and transmits it to the drive stage of the vehicle's drive-by-wire actuator. When violation is determined, the hardware gating circuit disconnects the control signal channel of the safety correction action and simultaneously activates the preset control signal channel corresponding to the physical safety rollback operation, thereby achieving unconditional switching from the intelligent decision-making channel to the safety rollback channel at the physical level. The hardware gating circuit can be a hardware logic circuit independent of the main processor or a hardware signal routing module integrated into the microcontroller. Its switching behavior is controlled only by the binary judgment signal of the security audit unit and does not depend on any programmable software logic.

[0075] The technical solution of the above embodiments, by adding an independent safety audit unit before the safety correction action is output to the actuator, and using deterministic Boolean logic and formally verified physical safety backoff operations as the last line of defense, achieves a two-layer safety architecture of soft projection optimization and hard audit fallback. This architecture retains the flexibility of convex optimization projection to maximize the contribution of RL decisions within the safety boundary, and meets the stringent formal requirements of ISO 26262 for functional safety certification through a deterministic fallback mechanism.

[0076] It should be further explained that, from a physical implementation perspective, the "soft projection optimization" layer is deployed on the application processor core of the vehicle control unit. Its convex optimization projection solution involves floating-point numerical operations, and the output result is a continuous value safety correction action. This layer seeks an approximate solution that preserves the original decision intent within the safety domain through mathematical optimization. Its calculation results are limited by numerical precision and the number of iterations, falling under the category of "soft" optimization. The "hard audit fallback" layer is deployed on a safety microcontroller independent of the application processor core. Its compliance verification is a deterministic Boolean logic decision, and the output result is a binary compliance / violation signal. This decision signal directly determines the source channel of the control signal at the physical level through the hardware gating circuit. Its operating logic has been formally verified and is unaffected by any software anomalies or failures on the application processor core. The two layers communicate unidirectionally via the internal data bus of the vehicle control unit (transmitting the safety correction action from the "soft projection optimization" layer to the "hard audit fallback" layer). The "hard audit fallback" layer does not feed back any control signals to the "soft projection optimization" layer. The two layers remain independent of each other in terms of physical entity, operating logic, and failure mode, thereby meeting the ISO 26262 requirement for the independence of safety-related functions from non-safety-related functions from interference.

[0077] In one possible design, step S7, which involves calculating the deviation between the safety correction action and the original action output, and injecting the deviation as negative feedback into the reward function of the reinforcement learning policy network, includes: S71. Calculate the Euclidean distance between the safety correction action and the original action output, as a projection correction amount characterizing the degree to which the agent's original decision is corrected by safety constraints; In this embodiment, the formula for calculating the projection correction is:

[0078] Wherein, Δ(t) is the projection correction amount; a*(t) is the safety correction action, which is obtained by solving step S5; p(t) is the original action output, which is generated by the reinforcement learning policy network; The projection correction is the square of the Euclidean distance between the safety correction action and the original action output. A larger projection correction indicates a greater degree of correction by the safety constraint system to the original RL intention, i.e., a greater distance between the agent's current policy and the safety domain; conversely, a smaller projection correction indicates that the agent has learned to generate actions within the safety domain quite well.

[0079] S72. Obtain the original task reward value at the current moment, and subtract the product of the projection correction amount and the preset gain coefficient from the original task reward value to obtain the net reward value. In this embodiment, the original task reward value is a reward signal obtained by the reinforcement learning agent from the environment that is only related to the quality of task completion, such as improved diagnostic accuracy or faster response speed. The formula for calculating the net reward value is:

[0080] Among them, R total (t) represents the net reward value; R task Δ(t) represents the original task reward value, which is the original reward obtained by the agent from the environment at the current moment; Δ(t) represents the projection correction amount, calculated in step S71; γ represents the preset gain coefficient, a positive preset constant used to adjust the intensity of the deviation penalty. This formula subtracts a penalty term proportional to the projection correction amount from the original task reward value, so that original action outputs that are far from the safe domain receive a lower net reward value, while original action outputs that are close to or within the safe domain receive a higher net reward value.

[0081] In one implementation, the preset gain coefficient γ ranges from 0.01 to 1.0, and its specific value is determined based on the original task reward value R. taskThe magnitude of (t) and the desired constraint strength are determined. For example, when the typical magnitude of the original task reward value is 1.0, the preset gain coefficient can be selected as 0.1, so that when the projection correction reaches 1.0, the penalty term contributes 0.1, accounting for approximately 10% of the task reward, thereby achieving a moderate balance between the task optimization objective and the satisfaction of safety constraints. The specific value of the preset gain coefficient is determined through hyperparameter search during the offline training phase.

[0082] S73. The net reward value is used as the final reward signal of the reinforcement learning policy network at the current moment, and is used to calculate the policy gradient and update the network parameters of the reinforcement learning policy network, so that when the reinforcement learning policy network faces similar vehicle running state vectors in the future, it can directly output the original action output that can satisfy all safety constraints without large projection correction.

[0083] In this embodiment, the policy gradient is calculated using a policy gradient theorem commonly used in reinforcement learning, such as the policy gradient calculation methods in the REINFORCE algorithm, the Proximal Policy Optimization (PPO) algorithm, or the Deep Deterministic Policy Gradient (DDPG) algorithm. After calculating the policy gradient, the network parameters are updated using stochastic gradient descent or its variant optimizer, causing the policy network to migrate towards the interior of the safety region in the parameter space. As online fine-tuning continues, the RL agent will gradually develop an intrinsic safety preference, that is, it will actively output actions that satisfy safety constraints, rather than relying on external projection corrections.

[0084] The technical solution of the above embodiments constructs a closed-loop information channel from the security constraint layer back to the agent's policy layer by injecting the projection correction as a negative feedback term into the RL reward function. This closed-loop feedback enables the RL agent to perceive the extent to which its output is corrected by the security boundary and adjust its policy weights accordingly, gradually transforming security constraints from external enforcement to internal preference. This mechanism ensures that the system's security performance is not static but continuously improves with increased online runtime and the accumulation of fine-tuning data, achieving a long-term evolutionary capability of "becoming safer the longer it runs."

[0085] Example 2

[0086] Embodiment 2 of the present invention provides an on-board fault safety response control system based on reinforcement learning. Figure 2 This is a hardware architecture block diagram of a reinforcement learning-based vehicle fail-safe response control system according to an exemplary embodiment. (Refer to...) Figure 2 The system includes an onboard sensor group 01, a vehicle control unit 02, a safety microcontroller 03, a hardware gating circuit 04, and a calibration storage area 05. It should be noted that... Figure 2This is only a schematic diagram of the architecture. In actual implementation, the number of components, connection methods, and integration methods can be adjusted according to the specific vehicle model and electronic and electrical architecture.

[0087] The on-board sensor group 01 includes ASIL-D redundant sensors and vehicle dynamics sensors. The ASIL-D redundant sensors are used to perform dual-channel independent measurements of the same physical quantity, providing redundant measurement signals that can be used to calculate the redundancy comparison residual. The vehicle dynamics sensors include a vehicle speed sensor, a yaw rate sensor, wheel speed sensors, and an inertial measurement unit, used to provide the sensing data required to construct the vehicle's operating state vector and estimate the road adhesion coefficient. The on-board sensor group 01 is connected to the vehicle control unit 02 via an on-board CAN bus or on-board Ethernet communication interface. Sensor data is transmitted to the system memory of the vehicle control unit 02 at preset intervals via direct memory access.

[0088] The vehicle control unit 02 includes a first processor core 021, a second processor core 022, and a neural network accelerator 023. The cores communicate with each other through the internal data bus of the vehicle control unit 02.

[0089] The first processor core 021 is configured to perform the calculation of confidence coefficients, the construction of dynamic physical feasible regions, the acquisition of logical inequality constraint sets and the allocation of dynamic relaxation penalty weights, and the solution of convex optimization projections. In one implementation, the first processor core 021 is an application processor core with ARM Cortex-A55 or equivalent or higher performance. Its algorithm module executes the following processing flow in each control cycle: First, based on the redundancy comparison residuals and verification error rate read from system memory, a confidence coefficient is calculated using an exponential decay function. Second, the physical limit data of the actuator is read from calibration storage area 05, and the road adhesion coefficient, estimated in real-time by the vehicle dynamics model using Kalman filtering, is read from system memory. The effective adhesion coefficient is calculated by combining the confidence coefficient, and a dynamic physical feasible region is constructed. Then, the logical inequality constraint set and its functional safety level label are read from calibration storage area 05 or system memory, and the dynamic relaxation penalty weight is calculated by combining the current vehicle speed measurement and the effective adhesion coefficient. Finally, after receiving the original action output generated by the second processor core 022, the original action output is corrected into a safety correction action by performing a fixed number of iterations of convex optimization projection solution. The specific algorithm for convex optimization projection solution is described in steps S5 and S51 to S54 of Embodiment 1.

[0090] The second processor core 022 is configured to perform inference computation and closed-loop feedback reward computation for the reinforcement learning policy network. The second processor core 022 can be the same type of processor core as the first processor core 021, or it can be a dedicated AI processor core. In one embodiment, the second processor core 022 inputs the constructed vehicle operating state vector into the reinforcement learning policy network deployed in the neural network accelerator 023 to obtain the original action output; simultaneously, the second processor core 022 receives safety correction actions from the first processor core 021, calculates the projection correction amount and injects it into the reward function, uses the net reward value to calculate the policy gradient, and updates the network parameters of the reinforcement learning policy network. The neural network accelerator 023 is a hardware acceleration unit integrated inside the vehicle control unit 02, used to perform matrix operations of the deep neural network.

[0091] The safety microcontroller 03, independent of the vehicle control unit 02, communicates with the first processor core 021 and the second processor core 022 via an internal data bus. The safety microcontroller 03 is an independent microcontroller chip certified with ASIL-D functional safety. The deterministic compliance verification logic running on it has undergone formal verification, and its firmware cannot be modified online after the vehicle leaves the factory. In each control cycle, the safety microcontroller 03 receives safety correction actions output by the first processor core 021, performs boundary compliance checks and rate of change checks on them, and outputs a binary compliance judgment signal. The judgment output terminal of the safety microcontroller 03 is directly connected to the control terminal of the hardware gating circuit 04.

[0092] The hardware gating circuit 04 has its signal input terminals connected to the safety correction action output interface of the vehicle control unit 02 and the safety backoff signal output interface of the safety microcontroller 03, respectively. Its output terminal is connected to the control signal input terminal of the vehicle drive-by-wire actuator, and its control terminal is connected to the judgment output terminal of the safety microcontroller 03. When the safety microcontroller 03 outputs a compliance judgment signal, the hardware gating circuit 04 activates the control signal channel corresponding to the safety correction action; when the safety microcontroller 03 outputs a violation judgment signal, the hardware gating circuit 04 disconnects the control signal channel of the safety correction action and simultaneously activates the control signal channel corresponding to the safety backoff strategy, thus achieving unconditional switching from the intelligent decision-making channel to the safety backoff channel at the physical level. The hardware gating circuit 04 can be a hardware logic circuit independent of each processor or integrated into the hardware signal routing module inside the safety microcontroller 03.

[0093] The calibration storage area 05, connected to the first processor core 021, is a non-volatile memory inside or outside the vehicle control unit 02, used to store the physical limit data of the actuators. The physical limit data of the actuators includes the maximum hydraulic pressure value of the braking system, the maximum motor torque value of the steering system, and the maximum output torque value of the drive system. It is fixed and written during the vehicle's factory calibration and cannot be modified during operation.

[0094] The specific implementation methods of the functions of each component in the above hardware architecture can be found in the specific implementation methods of the corresponding steps in the method embodiment of Example 1, and will not be repeated here.

[0095] Example 3

[0096] Embodiment 3 of the present invention provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0097] In this embodiment, the computer device may be an in-vehicle domain controller, an intelligent driving computing platform, or an embedded safety MCU. The processor may be an ARM Cortex-A series, Cortex-R series, or other processor chips that meet automotive-grade certification.

[0098] Example 4

[0099] Embodiment 4 of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0100] In this embodiment, the computer-readable storage medium can be a non-volatile storage medium, such as a Flash memory, EEPROM, solid-state drive, or other types of readable storage media.

[0101] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A vehicle-mounted fault-tolerant response control method based on reinforcement learning, characterized in that, include: Obtain the redundancy comparison residual in multi-source sensor data and the verification error rate in communication verification data, and obtain the confidence coefficient to characterize the reliability of sensor signals based on the redundancy comparison residual and the verification error rate; The physical limit data of the actuator and the road surface adhesion coefficient are obtained. The road surface adhesion coefficient is corrected according to the confidence coefficient to obtain the effective adhesion coefficient. Then, a dynamic physical feasible region with adaptive shrinkage of the boundary as the confidence level of the sensor signal decreases is constructed based on the physical limit data of the actuator and the effective adhesion coefficient. The dynamic physical feasible region is a hard constraint. Obtain a set of logical inequality constraints generated by fault tree analysis rules and carrying functional safety level labels, and assign dynamic relaxation penalty weights with priority differences to each constraint in the set of logical inequality constraints. A vehicle operating state vector is constructed based on the multi-source sensor data, and the vehicle operating state vector is input into a reinforcement learning policy network to generate the original action output. Under the condition of satisfying the dynamic physical feasible region, solve a convex optimization problem with the objective of minimizing the deviation from the original action output and with the logical inequality constraint set and the corresponding dynamic relaxation penalty weight as the penalty term, and project and correct the original action output into a safe correction action. The security correction action is subjected to a deterministic compliance check independent of the reinforcement learning policy network. If the deterministic compliance check passes, the security correction action is output to the executor; otherwise, the system switches to a preset security rollback strategy. The degree of deviation between the safety correction action and the original action output is calculated, and the degree of deviation is injected as a negative feedback term into the reward function of the reinforcement learning policy network to drive the reinforcement learning policy network to actively generate the original action output that is closer to the dynamic physical feasible region.

2. The vehicle-mounted fault-tolerant response control method based on reinforcement learning according to claim 1, characterized in that, The step of obtaining the confidence coefficient for characterizing the reliability of the sensor signal based on the redundancy comparison residual and the verification error rate includes: Obtain the redundant alignment residual generated by the dual-channel alignment of the ASIL-D redundant sensor, and calculate the residual variance of the redundant alignment residual within the sliding time window; The CRC check error rate is obtained as the check error rate during the data transmission process of the AUTOSAR E2E communication protocol. Based on the residual variance and the verification error rate, a confidence coefficient with a value between 0 and 1 is calculated using a preset exponential decay function, wherein the confidence coefficient approaches 0 when the ASIL-D redundant sensor signal degrades or communication is abnormal.

3. The vehicle-mounted fault-tolerant response control method based on reinforcement learning according to claim 1, characterized in that, The process involves acquiring the physical limit data of the actuator and the road surface adhesion coefficient, correcting the road surface adhesion coefficient based on the confidence coefficient to obtain an effective adhesion coefficient, and then constructing a dynamic physically feasible domain whose boundary adaptively shrinks as the confidence level of the sensor signal decreases, based on the physical limit data of the actuator and the effective adhesion coefficient. This includes: Obtain preset physical limit data of the actuator, which includes the maximum hydraulic pressure value of the braking system, the maximum motor torque value of the steering system, and the maximum output torque value of the drive system; The effective adhesion coefficient is obtained based on the confidence coefficient and the road surface adhesion coefficient. Based on the ratio of the effective adhesion coefficient to the gravitational acceleration, the safe action boundaries of the vehicle in the longitudinal and lateral acceleration dimensions are determined using a tire friction ellipse model. Obtain a preset upper limit constraint on the rate of change of actuator motion, and construct a continuous and convex dynamic physical feasible domain in the acceleration space by combining the safety action boundary, the physical limit data of the actuator, and the upper limit constraint on the rate of change of actuator motion.

4. The vehicle-mounted fault-tolerant response control method based on reinforcement learning according to claim 1, characterized in that, The step of obtaining a set of logical inequality constraints generated by fault tree analysis rules and carrying functional safety level labels, and assigning dynamic relaxation penalty weights with priority differences to each constraint in the set of logical inequality constraints, includes: Obtain the set of logical inequality constraints generated by linearizing the AND and OR logic relationships between the top, middle and bottom events in the fault tree model, where each constraint corresponds to a combination of prohibited action triggering conditions. Obtain the functional safety level label for each constraint in the logical inequality constraint group, and determine a baseline penalty coefficient for each constraint based on the functional safety level label, wherein constraints corresponding to higher functional safety level labels are assigned a higher baseline penalty coefficient. Obtain the vehicle speed measurement value and the effective adhesion coefficient at the current moment, and obtain a dynamic adjustment factor that monotonically increases as the vehicle speed measurement value increases or the effective adhesion coefficient decreases based on the vehicle speed measurement value and the effective adhesion coefficient. The dynamic relaxation penalty weight of each constraint is obtained based on the baseline penalty coefficient and the dynamic adjustment factor.

5. The vehicle-mounted fault-tolerant response control method based on reinforcement learning according to claim 1, characterized in that, The solution is a convex optimization problem that aims to minimize the deviation from the original action output and uses the set of logical inequalities and the corresponding dynamic relaxation penalty weights as penalty terms. It projects and corrects the original action output into a safe correction action, including: Construct an objective function, the first term of which is the square of the Euclidean distance between the action to be solved and the original action output, and the second term is the sum of the products of the violation amount of each of the logical inequality constraints after non-negative truncation and the corresponding dynamic relaxation penalty weight. The dynamic physical feasible region is used as the hard constraint condition that the action to be solved must satisfy, which together with the objective function constitutes a constrained convex optimization problem. The convex optimization problem is solved iteratively using the alternating direction multiplier method. The number of iterations is preset to a fixed value, which ensures that the worst-case execution time of the solver is less than the fault tolerance time interval required by the functional safety specification. After solving for the fixed number of iterations, a safety correction action is obtained that prioritizes satisfying the logical inequality constraint of the higher functional safety level label within the dynamic physical feasible domain, while preserving the original decision intent to the greatest extent.

6. The vehicle-mounted fault-tolerant response control method based on reinforcement learning according to claim 1, characterized in that, The process of performing deterministic compliance verification on the security correction action, independent of the policy network, outputs the result to the executor if the verification passes, and switches to a preset security rollback strategy if it fails, including: The security correction action, after projection correction, is obtained by a security audit unit that is independent of the reinforcement learning policy network and does not depend on the learning model; The security audit unit performs deterministic boundary compliance verification on the security correction action based on the physical boundary conditions and logical prohibition rules corresponding to the preset highest functional safety level label, and performs change rate verification on the security correction action at the current moment. If the safety correction action passes both the boundary compliance check and the rate of change check, it is determined to be compliant, and the safety correction action is transmitted to the vehicle drive-by-wire actuator to perform the corresponding operation. If any verification item fails, it is deemed a violation, the output of the security correction action is blocked, and a pre-formally verified security rollback strategy that does not rely on agent decision-making is triggered.

7. The vehicle-mounted fault-tolerant response control method based on reinforcement learning according to claim 1, characterized in that, The step of calculating the deviation between the safety correction action and the original action output, and injecting the deviation as a negative feedback term into the reward function of the reinforcement learning policy network, includes: Calculate the Euclidean distance between the safety correction action and the original action output to obtain the projection correction amount, which characterizes the degree to which the agent's original decision is corrected by safety constraints. Obtain the original task reward value and preset gain coefficient at the current moment, and obtain the net reward value based on the original task reward value, the projection correction amount and the preset gain coefficient; The net reward value is used as the final reward signal of the reinforcement learning policy network at the current moment. It is used to calculate the policy gradient and update the network parameters of the reinforcement learning policy network, so that the reinforcement learning policy network can directly output the original action output when it faces similar vehicle running state vectors in the future.

8. A vehicle-mounted fault-tolerant response control system based on reinforcement learning, characterized in that, The system includes: The vehicle-mounted sensor group includes ASIL-D redundant sensors and vehicle dynamics sensors, used to collect multi-source sensor data, which includes redundancy comparison residuals and communication verification data. The vehicle control unit includes a first processor core and a second processor core; The first processor core is configured to perform the following operations: The confidence coefficient, used to characterize the reliability of the sensor signal, is calculated based on the redundancy ratio residual and the verification error rate in the communication verification data. Based on the physical limit data of the actuator and the road adhesion coefficient estimated based on the vehicle dynamics model, combined with the confidence coefficient, a dynamic physical feasible region whose boundary adaptively shrinks as the confidence level of the sensor signal decreases is constructed as a set of hard constraints. Obtain a set of logical inequality constraints generated by fault tree analysis rule encoding and carrying functional safety level labels, and assign dynamic relaxation penalty weights with priority differences to each constraint according to the functional safety level labels carried by each constraint and the current vehicle operating status. Under the condition of satisfying the dynamic physical feasible region, a convex optimization problem is solved with the objective of minimizing the deviation from the original action output and with the logical inequality constraint set and the corresponding dynamic relaxation penalty weight as the penalty term. The original action output is projected and corrected into a safe correction action. A fixed number of iterations is used in the solution to ensure that the worst-case execution time is predictable. The second processor core is configured to perform the following operations: A vehicle operating state vector is constructed based on the multi-source sensor data, and the vehicle operating state vector is input into a reinforcement learning policy network to generate the original action output. Calculate the degree of deviation between the safety correction action and the original action output, and inject the degree of deviation as a negative feedback term into the reward function of the reinforcement learning policy network; A safety microcontroller, independent of the vehicle control unit, is communicatively connected to the first processor core and the second processor core via an internal data bus, and is configured to perform the following operations: Receive the security correction action and perform deterministic compliance verification on the security correction action, independent of the reinforcement learning policy network; A hardware gating circuit, connected between the safety microcontroller, the vehicle control unit, and the vehicle drive-by-wire actuator, is configured as follows: When the safety correction action passes the deterministic compliance check, the safety correction action is transmitted to the vehicle drive-by-wire actuator; When the security correction action fails the deterministic compliance check, the output of the security correction action is blocked, and the control signal channel corresponding to the preset security rollback strategy is activated. A calibration storage area, connected to the first processor core, is used to store the physical limit data of the actuator.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images