A fault-tolerant control method and system based on hierarchical reinforcement learning, computer program product

Abstract

This invention relates to a fault-tolerant control method and system based on hierarchical reinforcement learning, and a computer program product, belonging to the field of automatic control technology. In a fault perception and abstraction layer pre-constructed based on a hierarchical reinforcement learning algorithm, fault features are extracted from system state data, and the extracted fault features are dimensionality-reduced to obtain fault coding vectors. In a high-level target decision layer pre-constructed based on the hierarchical reinforcement learning algorithm, a target fault-tolerant strategy for the fault coding vector is selected from a predetermined set of fault-tolerant strategies according to a pre-defined selection logic between the fault coding vector and the fault-tolerant strategy. In a low-level control execution layer pre-constructed based on the hierarchical reinforcement learning algorithm, the SAC algorithm is invoked according to the target fault-tolerant strategy and the fault features to obtain the system's fault-tolerant control actions, thereby controlling the system to execute the fault-tolerant control actions and achieving stable system operation through autonomous learning and hierarchical decision-making.

Description

Technical Field

[0001] This invention relates to a fault-tolerant control method and system based on hierarchical reinforcement learning, as well as a computer program product, belonging to the field of automatic control technology. Background Technology

[0002] In modern complex engineering systems (such as aerospace, advanced manufacturing, energy and power, and intelligent transportation), the safety and reliability of system operation are paramount. During long-term operation, various types of failures inevitably occur, including sensor failure, actuator jamming / saturation, component aging or damage, and external interference exceeding design limits. If these failures are not handled promptly and effectively, they can lead to anything from degraded system performance to catastrophic accidents. Therefore, developing fault-tolerant control technologies that can automatically detect, diagnose, and adjust control strategies after a failure occurs to ensure that the system's basic functions or performance are not lost (or are only acceptablely degraded) has significant theoretical and engineering value.

[0003] Traditional fault-tolerant control methods are mainly divided into two categories: passive fault-tolerant control and active fault-tolerant control. Passive fault-tolerant control: When designing the controller, a limited number of possible fault modes are considered in advance. Through robust control design, the controller is made somewhat "immune" to faults within a specific range. Passive fault-tolerant control does not require online fault diagnosis and has a fast response speed, but its main drawbacks are: strong conservatism, poor adaptability, and weak ability to handle complex faults. Active fault-tolerant control: Typically includes a fault detection and diagnosis module and a controller reconfiguration / adjustment module. When a fault is detected, the controller parameters are adjusted or the control strategy is switched based on the diagnostic information. Active fault-tolerant control can theoretically handle a wider range of faults, but its main drawbacks are: high requirements for fault diagnosis accuracy and real-time performance, strong model dependency, and a limited reconfiguration strategy library.

[0004] Reinforcement learning, as a model-free or data-driven optimization method, learns optimal decision-making strategies (i.e., control laws) through trial and error between an agent and its environment, demonstrating great potential in handling control problems of complex, nonlinear systems. Reinforcement learning agents can learn from experience, adapt to environmental changes, theoretically do not require precise system models, have the potential to handle high-dimensional, nonlinear problems, exhibit strong adaptability, and can theoretically learn optimal or near-optimal strategies.

[0005] However, reinforcement learning still faces key bottlenecks in fault tolerance control (i.e., maintaining stable operation under system failure or anomaly): the curse of dimensionality, low exploration efficiency, insufficient policy transfer and generalization capabilities, and how to ensure safety during the exploration process. Summary of the Invention

[0006] The purpose of this invention is to provide a fault-tolerant control method and system, and computer program product based on hierarchical reinforcement learning, to solve the problems of strong model dependence, weak ability to handle unknown faults, and difficulty in designing high-dimensional systems in existing fault-tolerant control methods, as well as the problems of dimensionality curse, high exploration risk, and poor policy generalization faced by traditional reinforcement learning in fault-tolerant scenarios, so as to achieve stable system operation through autonomous learning and hierarchical decision-making.

[0007] To achieve the above objectives, this invention proposes a fault-tolerant control method based on hierarchical reinforcement learning, which includes the following steps:

[0008] 1) In the fault perception and abstraction layer pre-constructed based on the hierarchical reinforcement learning algorithm, fault features are extracted from the system state data, and the extracted fault features are reduced in dimensionality to obtain the fault coding vector;

[0009] 2) In the high-level target decision layer pre-constructed based on the hierarchical reinforcement learning algorithm, the target fault tolerance strategy for the fault encoding vector is selected from the predetermined set of fault tolerance strategies according to the pre-defined selection logic between the fault encoding vector and the fault tolerance strategy.

[0010] 3) In the underlying control execution layer pre-constructed based on the hierarchical reinforcement learning algorithm, according to the target fault tolerance strategy and the fault characteristics, the SAC algorithm is called to obtain the fault tolerance control action of the system, so as to control the system to execute the fault tolerance control action.

[0011] Furthermore, in the fault perception and abstraction layer, fault features are extracted from the system state data using the following method, and the extracted fault features are then reduced in dimensionality to obtain a fault coding vector:

[0012] A one-dimensional convolutional neural network is invoked to extract spatial features from the system state data, and a long short-term memory network is invoked to extract temporal features from the system state data to obtain fault features; the extracted fault features are then reduced to the fault encoding vector using a fault encoder.

[0013] Furthermore, the high-level target decision layer includes an optimized PPO algorithm, which is invoked to enable the high-level target decision layer to select a target fault-tolerant strategy from a predetermined set of fault-tolerant strategies according to a pre-defined selection logic between fault coding vectors and fault-tolerant strategies.

[0014] Furthermore, the selection logic is as follows:

[0015] When the severity element of the main fault in the fault coding vector takes the value of the preset low-activity range, and the severity elements of all types of sub-faults in the fault coding vector do not exceed the preset stable threshold, there is no single point of over-limit, and the numerical fluctuation is within the allowable range, the tracking fault tolerance strategy is triggered.

[0016] When the severity element of the main fault in the fault coding vector takes the value of the preset medium active range, or when the severity element of any type of sub-fault in the fault coding vector reaches the preset safety limit, the degradation operation fault tolerance strategy is triggered.

[0017] When the main fault severity element in the fault coding vector takes the value of a preset high-activity range, or when a signal is generated to reflect a collision in the physical space, an emergency zero-tolerance strategy is triggered.

[0018] Furthermore, if the fault coding vector is an 8-dimensional fault coding vector, then the selection logic is as follows:

[0019] When z[0] < 0.3 and the z[1:4] norm < 0.2, the tracking fault tolerance strategy is triggered;

[0020] When 0.3≤z[0]<0.7 or z[4]>0.5, the downgraded operation fault tolerance strategy is triggered;

[0021] When z[0]≥0.7 or the collision flag=1, the emergency zero-tolerance strategy is triggered;

[0022] Where z is the 8-dimensional fault coding vector.

[0023] Furthermore, the optimized PPO algorithm also introduces an advantage function that includes fault severity weights, wherein the advantage function is expressed by the following formula:

[0024] Among them, time-series difference residuals ;

[0025] in, To introduce a dominance function that weights the severity of the fault; For time-series difference residuals; Discount factor; Here, T is the GAE dominance function decay coefficient; T is the maximum time step; t is the current time step. This is an instant reward value; The fault encoding vector is w; the learnable weights are w. Let t+1 be the state value function; Let t be the state value function at time t; The system state at time t+1; Let t be the system state at time t.

[0026] Furthermore, in the underlying control execution layer, the SAC algorithm is invoked to perform decision training for each fault-tolerant strategy to obtain the original fault-tolerant control instructions; according to the preset correction rules, the original fault-tolerant control instructions are subjected to real-time safety trimming to obtain the final safety control instructions, so as to obtain the fault-tolerant control actions of the system based on the final safety control instructions.

[0027] Furthermore, it also includes: a hierarchical reward framework pre-built based on a hierarchical reinforcement learning algorithm, wherein the hierarchical reward framework includes a reward function built for the high-level goal decision layer and a reward function built for the low-level control execution layer;

[0028] The reward functions constructed for the high-level target decision-making layer include a task reward function, a fault suppression reward function, a switching reward function, and a security reward function;

[0029] The reward functions constructed for the underlying control execution layer include a tracking accuracy reward function, an energy consumption optimization reward function, a motion smoothing reward function, and a risk penalty function.

[0030] On the other hand, the present invention also proposes a fault-tolerant control system based on hierarchical reinforcement learning, including a processor for executing a computer program to implement the steps of the following method:

[0031] 1) In the fault perception and abstraction layer pre-constructed based on the hierarchical reinforcement learning algorithm, fault features are extracted from the system state data, and the extracted fault features are reduced in dimensionality to obtain the fault coding vector;

[0032] 2) In the high-level target decision layer pre-constructed based on the hierarchical reinforcement learning algorithm, the target fault tolerance strategy for the fault encoding vector is selected from the predetermined set of fault tolerance strategies according to the pre-defined selection logic between the fault encoding vector and the fault tolerance strategy.

[0033] 3) In the underlying control execution layer pre-constructed based on the hierarchical reinforcement learning algorithm, according to the target fault tolerance strategy and the fault characteristics, the SAC algorithm is called to obtain the fault tolerance control action of the system, so as to control the system to execute the fault tolerance control action.

[0034] Furthermore, in the fault perception and abstraction layer, fault features are extracted from the system state data using the following method, and the extracted fault features are then reduced in dimensionality to obtain a fault coding vector:

[0035] A one-dimensional convolutional neural network is invoked to extract spatial features from the system state data, and a long short-term memory network is invoked to extract temporal features from the system state data to obtain fault features; the extracted fault features are then reduced to the fault encoding vector using a fault encoder.

[0036] Furthermore, the high-level target decision layer includes an optimized PPO algorithm, which is invoked to enable the high-level target decision layer to select a target fault-tolerant strategy from a predetermined set of fault-tolerant strategies according to a pre-defined selection logic between fault coding vectors and fault-tolerant strategies.

[0037] Furthermore, the selection logic is as follows:

[0038] When the severity element of the main fault in the fault coding vector takes the value of the preset low-activity range, and the severity elements of all types of sub-faults in the fault coding vector do not exceed the preset stable threshold, there is no single point of over-limit, and the numerical fluctuation is within the allowable range, the tracking fault tolerance strategy is triggered.

[0039] When the severity element of the main fault in the fault coding vector takes the value of the preset medium active range, or when the severity element of any type of sub-fault in the fault coding vector reaches the preset safety limit, the degradation operation fault tolerance strategy is triggered.

[0040] When the main fault severity element in the fault coding vector takes the value of a preset high-activity range, or when a signal is generated to reflect a collision in the physical space, an emergency zero-tolerance strategy is triggered.

[0041] Furthermore, if the fault coding vector is an 8-dimensional fault coding vector, then the selection logic is as follows:

[0042] When z[0] < 0.3 and the z[1:4] norm < 0.2, the tracking fault tolerance strategy is triggered;

[0043] When 0.3≤z[0]<0.7 or z[4]>0.5, the downgraded operation fault tolerance strategy is triggered;

[0044] When z[0]≥0.7 or the collision flag=1, the emergency zero-tolerance strategy is triggered;

[0045] Where z is the 8-dimensional fault coding vector.

[0046] Furthermore, the optimized PPO algorithm also introduces an advantage function that includes fault severity weights, wherein the advantage function is expressed by the following formula:

[0047] Among them, time-series difference residuals ;

[0048] in, To introduce a dominance function that weights the severity of the fault; For time-series difference residuals; Discount factor; Here, T is the GAE dominance function decay coefficient; T is the maximum time step; t is the current time step. This is an instant reward value; The fault encoding vector is w; the learnable weights are w. Let t+1 be the state value function; Let t be the state value function at time t; The system state at time t+1; Let t be the system state at time t.

[0049] Furthermore, in the underlying control execution layer, the SAC algorithm is invoked to perform decision training for each fault-tolerant strategy to obtain the original fault-tolerant control instructions; according to the preset correction rules, the original fault-tolerant control instructions are subjected to real-time safety trimming to obtain the final safety control instructions, so as to obtain the fault-tolerant control actions of the system based on the final safety control instructions.

[0050] Furthermore, it also includes: a hierarchical reward framework pre-built based on a hierarchical reinforcement learning algorithm, wherein the hierarchical reward framework includes a reward function built for the high-level goal decision layer and a reward function built for the low-level control execution layer;

[0051] The reward functions constructed for the high-level target decision-making layer include a task reward function, a fault suppression reward function, a switching reward function, and a security reward function;

[0052] The reward functions constructed for the underlying control execution layer include a tracking accuracy reward function, an energy consumption optimization reward function, a motion smoothing reward function, and a risk penalty function.

[0053] On the other hand, the present invention also proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implements the steps of the fault-tolerant control method based on hierarchical reinforcement learning as described below:

[0054] 1) In the fault perception and abstraction layer pre-constructed based on the hierarchical reinforcement learning algorithm, fault features are extracted from the system state data, and the extracted fault features are reduced in dimensionality to obtain the fault coding vector;

[0055] 2) In the high-level target decision layer pre-constructed based on the hierarchical reinforcement learning algorithm, the target fault tolerance strategy for the fault encoding vector is selected from the predetermined set of fault tolerance strategies according to the pre-defined selection logic between the fault encoding vector and the fault tolerance strategy.

[0056] 3) In the underlying control execution layer pre-constructed based on the hierarchical reinforcement learning algorithm, according to the target fault tolerance strategy and the fault characteristics, the SAC algorithm is called to obtain the fault tolerance control action of the system, so as to control the system to execute the fault tolerance control action.

[0057] Furthermore, in the fault perception and abstraction layer, fault features are extracted from the system state data using the following method, and the extracted fault features are then reduced in dimensionality to obtain a fault coding vector:

[0058] A one-dimensional convolutional neural network is invoked to extract spatial features from the system state data, and a long short-term memory network is invoked to extract temporal features from the system state data to obtain fault features; the extracted fault features are then reduced to the fault encoding vector using a fault encoder.

[0059] Furthermore, the high-level target decision layer includes an optimized PPO algorithm, which is invoked to enable the high-level target decision layer to select a target fault-tolerant strategy from a predetermined set of fault-tolerant strategies according to a pre-defined selection logic between fault coding vectors and fault-tolerant strategies.

[0060] Furthermore, the selection logic is as follows:

[0061] When the severity element of the main fault in the fault coding vector takes the value of the preset low-activity range, and the severity elements of all types of sub-faults in the fault coding vector do not exceed the preset stable threshold, there is no single point of over-limit, and the numerical fluctuation is within the allowable range, the tracking fault tolerance strategy is triggered.

[0062] When the severity element of the main fault in the fault coding vector takes the value of the preset medium active range, or when the severity element of any type of sub-fault in the fault coding vector reaches the preset safety limit, the degradation operation fault tolerance strategy is triggered.

[0063] When the main fault severity element in the fault coding vector takes the value of a preset high-activity range, or when a signal is generated to reflect a collision in the physical space, an emergency zero-tolerance strategy is triggered.

[0064] Furthermore, if the fault coding vector is an 8-dimensional fault coding vector, then the selection logic is as follows:

[0065] When z[0] < 0.3 and the z[1:4] norm < 0.2, the tracking fault tolerance strategy is triggered;

[0066] When 0.3≤z[0]<0.7 or z[4]>0.5, the downgraded operation fault tolerance strategy is triggered;

[0067] When z[0]≥0.7 or the collision flag=1, the emergency zero-tolerance strategy is triggered;

[0068] Where z is the 8-dimensional fault coding vector.

[0069] Furthermore, the optimized PPO algorithm also introduces an advantage function that includes fault severity weights, wherein the advantage function is expressed by the following formula:

[0070] Among them, time-series difference residuals ;

[0071] in, To introduce a dominance function that weights the severity of the fault; For time-series difference residuals; Discount factor; Here, T is the GAE dominance function decay coefficient; T is the maximum time step; t is the current time step. This is an instant reward value; The fault encoding vector is w; the learnable weights are w. Let t+1 be the state value function; Let t be the state value function at time t; The system state at time t+1; Let t be the system state at time t.

[0072] Furthermore, in the underlying control execution layer, the SAC algorithm is invoked to perform decision training for each fault-tolerant strategy to obtain the original fault-tolerant control instructions; according to the preset correction rules, the original fault-tolerant control instructions are subjected to real-time safety trimming to obtain the final safety control instructions, so as to obtain the fault-tolerant control actions of the system based on the final safety control instructions.

[0073] Furthermore, it also includes: a hierarchical reward framework pre-built based on a hierarchical reinforcement learning algorithm, wherein the hierarchical reward framework includes a reward function built for the high-level goal decision layer and a reward function built for the low-level control execution layer;

[0074] The reward functions constructed for the high-level target decision-making layer include a task reward function, a fault suppression reward function, a switching reward function, and a security reward function;

[0075] The reward functions constructed for the underlying control execution layer include a tracking accuracy reward function, an energy consumption optimization reward function, a motion smoothing reward function, and a risk penalty function.

[0076] The beneficial effects of this invention are as follows: In the fault perception and abstraction layer pre-constructed based on the hierarchical reinforcement learning algorithm, fault features are extracted from the system state data, and the extracted fault features are reduced in dimensionality to obtain fault coding vectors; In the high-level target decision layer pre-constructed based on the hierarchical reinforcement learning algorithm, a target fault-tolerant strategy for the fault coding vector is selected from a predetermined set of fault-tolerant strategies according to a pre-defined selection logic between the fault coding vector and the fault-tolerant strategy; In the low-level control execution layer pre-constructed based on the hierarchical reinforcement learning algorithm, the SAC algorithm is invoked according to the target fault-tolerant strategy and the fault features to obtain the fault-tolerant control action of the system, so as to control the system to execute the fault-tolerant control action, thereby achieving stable system operation through autonomous learning and hierarchical decision-making. Attached Figure Description

[0077] Figure 1 This is a flowchart of a fault-tolerant control method based on hierarchical reinforcement learning proposed in this invention;

[0078] Figure 2 This is a schematic diagram of the offline pre-training process of a fault-tolerant control method based on hierarchical reinforcement learning proposed in this invention in a practical application scenario.

[0079] Figure 3 This is a flowchart illustrating the construction process of an online adaptive system architecture in a practical application scenario for a fault-tolerant control method based on hierarchical reinforcement learning proposed in this invention.

[0080] Figure 4 This is a flowchart of a fault-tolerant control method based on hierarchical reinforcement learning proposed in this invention in a practical application scenario. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0082] The inventive concept of this invention lies in addressing the problems of strong model dependence, weak ability to handle unknown faults, and difficulty in designing high-dimensional systems in existing fault-tolerant control methods, as well as the problems of dimensionality curse, high exploration risk, and poor policy generalization faced by traditional reinforcement learning in fault-tolerant scenarios. This invention combines hierarchical reinforcement learning with fault-tolerant control theory to achieve robust control that maintains stable system operation and optimizes performance when a system failure occurs, through autonomous learning and hierarchical decision-making. It is particularly suitable for systems with high-dimensional state spaces, nonlinear characteristics, strong coupling, and complex and variable operating environments, such as unmanned vehicles, industrial robots, aerospace vehicles, and large ship ammunition transport.

[0083] The hierarchical reinforcement learning mentioned above refers to decomposing complex decision-making tasks into multiple layers (e.g., high-level policies are responsible for macro-level goal selection and task decomposition, while low-level policies are responsible for specific action execution), effectively mitigating the curse of dimensionality and improving learning efficiency, policy interpretability, and reusability. This structure is suitable for fault-tolerant control.

[0084] 1) High-level decision-making: can focus on identifying fault states, setting fault tolerance targets (such as switching to safe mode, performance degradation operation, refactoring control structure, etc.) and scheduling subtasks.

[0085] 2) Low-level execution: It can learn the specific control actions to achieve the goal under specific failure modes or subtasks.

[0086] 3) Reusability: Low-level strategies (such as basic stability control) can be reused under different fault scenarios or high-level instructions.

[0087] 4) Efficiency and security: Layered structure can reduce ineffective exploration in high-level policy space, and high-level policies can be designed with safer exploration mechanisms (such as limiting switching between predefined fault-tolerant modes).

[0088] Detailed implementation method 1:

[0089] like Figure 1 The diagram shows a flowchart of a fault-tolerant control method based on hierarchical reinforcement learning proposed in this invention, which includes steps S11, S12, and S13, specifically:

[0090] Step S11: In the fault perception and abstraction layer pre-constructed based on the hierarchical reinforcement learning algorithm, fault features are extracted from the system state data, and the extracted fault features are reduced in dimensionality to obtain a fault coding vector; here, the high-dimensional fault features are compressed into a low-dimensional fault coding vector by an autoencoder.

[0091] Step S12: In the high-level target decision layer pre-constructed based on the hierarchical reinforcement learning algorithm, the target fault tolerance strategy for the fault coding vector is selected from the predetermined set of fault tolerance strategies according to the pre-defined selection logic between the fault coding vector and the fault tolerance strategy; here, the predetermined set of fault tolerance strategies includes the tracking fault tolerance strategy, the degraded operation fault tolerance strategy, and the emergency zero-return fault tolerance strategy.

[0092] In practical applications, taking industrial robots as an example, when the robot's real-time fault coding vector meets the following conditions: the main fault severity element is within a preset low-activity range, and the severity elements of all sub-faults do not exceed preset stable thresholds, there are no single-point exceedances, and the numerical fluctuations are within the allowable range, the system determines it to be a low-risk minor fault state and triggers the tracking fault-tolerance strategy. The strategy maintains the robot's original task, movement speed, and control parameters unchanged; it only activates the background fault tracking module to continuously collect and update the fault coding vector without interfering with the normal control process.

[0093] When the robot's real-time fault coding vector meets either of the following conditions: the severity element of the main fault enters the moderately active range, indicating a moderate anomaly in the core system; or the severity element of any critical sub-fault exceeds the preset safety limit, indicating a partial failure in the auxiliary system, the system determines it to be a controllable risk fault state and triggers a degraded operation fault tolerance strategy. The strategy executes by reducing the robot's movement speed, decreasing the working range, reducing control accuracy requirements, and disabling unnecessary functions; maintaining the execution of core tasks while suppressing fault propagation and continuously optimizing the control strategy.

[0094] When the severity element of the main fault enters the high-activity range in the robot's real-time fault coding vector, it indicates that a serious failure has occurred in the core system; or the robot has experienced a hard collision with the environment or obstacles in physical space, constituting a sudden safety fault. The system determines this to be a high-risk state and immediately triggers the emergency zero-return fault-tolerance strategy. The strategy immediately terminates all current operation instructions; controls each joint of the robot to quickly return to its mechanical origin along a safe path; activates braking protection, cuts off non-core power supply, outputs alarm signals, and awaits manual intervention.

[0095] Step S13: In the underlying control execution layer pre-constructed based on the hierarchical reinforcement learning algorithm, according to the target fault tolerance strategy and the fault characteristics, the SAC algorithm is called to obtain the fault tolerance control action of the system, so as to control the system to execute the fault tolerance control action.

[0096] Meanwhile, in the actual application scenario of this invention, a two-stage training mechanism is adopted, namely, the offline pre-training stage accumulates cross-fault experience in the simulation environment, and the online adaptive stage fine-tunes the strategy in real time in the real system.

[0097] Through the above steps S11-S13, the hierarchical reinforcement learning algorithm is introduced into the fault-tolerant control method, avoiding the problems of dimensionality curse, high exploration risk and poor policy generalization faced by traditional reinforcement learning in fault-tolerant scenarios, and achieving stable system operation through autonomous learning and hierarchical decision-making.

[0098] Method Detailed Implementation 2:

[0099] The following section will explain the fault-tolerant control method based on hierarchical reinforcement learning proposed in this invention, using practical application scenarios as examples.

[0100] The fault-tolerant control method of this invention involves a three-layer decision-making architecture (fault perception and abstraction layer, high-level target decision-making layer, and low-level control execution layer) and a two-stage operation mechanism (offline pre-training stage and online adaptive stage). The following sections will explain each layer of the decision-making architecture and each stage of the operation mechanism.

[0101] The three-tier decision-making architecture divides the task into three levels:

[0102] 1) Fault Perception & Abstraction Layer (FPAL): Real-time analysis of system status, detection of faults, and output of abstract fault codes (i.e., fault code vectors). Abstract fault codes are generated by an autoencoder, compressing high-dimensional fault features into a low-dimensional space.

[0103] 2) High-level Goal Decision Layer (HGDL): Based on the abstract fault codes obtained from the fault perception and abstraction layer, it selects fault-tolerant goals (i.e., fault-tolerant strategies, such as "maintain position tracking", "switch to degraded mode", "initiate emergency braking"). HGDL adopts an option framework, with each fault-tolerant goal corresponding to an "option" (such as Option_Maintain Tracking, Option_Safe Shutdown).

[0104] 3) Low-level Control Execution Layer (LCEL): Receives fault-tolerant objectives from the higher-level objective decision layer and generates specific control actions (i.e., fault-tolerant control actions, such as motor torque action, valve opening action, etc.). The LCEL trains dedicated sub-policies for each fault-tolerant objective's corresponding option and shares basic control skills (such as balance control and trajectory tracking).

[0105] The two-stage operation mechanism includes two training stages of reinforcement learning:

[0106] 1) Offline pre-training phase: Train the basic strategy in the simulation environment and accumulate experience across faults.

[0107] 2) Online adaptive phase: Fine-tune the strategy in the real system to deal with unknown faults in real time.

[0108] Method Detailed Implementation 3:

[0109] Following the specific embodiments described above, in addition to the three-layer decision-making architecture and two-stage operation mechanism, a security constraint embedding mechanism is introduced to improve system security; a risk-sensitive reward function is designed, and a cross-fault knowledge transfer mechanism is formulated. Specifically:

[0110] The safety constraint embedding mechanism is as follows: a safety layer is set up, and dangerous actions are corrected based on preset physical constraints (such as actuator saturation limit and state safety domain) before the bottom control execution layer LCEL outputs an action.

[0111] The risk-sensitive reward function: Design the reward function ,in, This indicates behaviors such as penalizing out-of-bounds states and controlling mutations. By allowing target selection only within a predefined set of safe and fault-tolerant modes in the high-level target decision layer (HGDL), invalid handovers are prohibited, thereby constraining high-level exploration.

[0112] Risk-sensitive reward functions serve as a globally applicable risk constraint framework. Their core function is to define the underlying trade-off logic between "reward" and "risk," and to globally penalize dangerous behaviors across the entire system. This involves designing the reward function. ,in, This indicates behaviors such as penalizing out-of-bounds states and controlling mutations. By allowing target selection only within a predefined set of safety-tolerant modes in the high-level target decision layer (HGDL) and prohibiting invalid switching, high-level exploration is constrained. It is important to note that the multi-dimensional reward functions designed in HGDL (including but not limited to task reward functions (core objectives), fault suppression reward functions, switching reward functions, and safety reward functions) are refined reward decomposition rules specific to the high level. Their core function is to translate the basic framework into specific incentive indicators for high-level decision-making, constraining high-level exploration and guiding optimal target selection. The two (i.e., the risk-sensitive reward function and the multi-dimensional reward function designed in HGDL) are hierarchically complementary. The former (i.e., the risk-sensitive reward function) sets the tone for global risk constraints, while the latter (i.e., the multi-dimensional reward function) implements precise high-level decision-making, jointly achieving the synergistic unity of "risk avoidance" and "multi-objective optimization" in layered fault-tolerant control.

[0113] The cross-fault knowledge transfer mechanism includes: 1) Modular policy reuse: Basic control sub-policies (such as position servoing) in the underlying control execution layer LCEL can be called by multiple higher-level options; 2) Meta-learning initialization: The high-level objective decision layer HGDL is pre-trained using MAML (Model-Agnostic Meta-Learning, whose core goal is to find a set of highly sensitive initial parameters through multi-task pre-training, so that the model can quickly adapt to new tasks with only a small number of gradient updates); 3) Fault pattern clustering: The fault perception and abstraction layer FPAL maps similar faults to the same encoding, triggers the same fault tolerance policy, and reduces redundant learning.

[0114] Through the aforementioned three-layer decision-making architecture, two-stage operation mechanism, security constraint embedding mechanism, risk-sensitive reward function, and cross-fault knowledge transfer mechanism, the following can be achieved:

[0115] 1) Reduce decision-making dimensions: By decomposing tasks in layers, the complex fault recovery process is broken down into high-level goal decisions and low-level control execution, which significantly improves learning efficiency.

[0116] 2) Enhance security: Embed security constraint mechanisms during learning and execution to avoid dangerous exploration in fault states.

[0117] 3) Enhance generalization capability: Through modular strategy design and cross-fault knowledge transfer, achieve rapid adaptation to unknown faults.

[0118] 4) Achieve model-free optimization: Without relying on an accurate mathematical model of the system, it learns the optimal fault-tolerant strategy autonomously through data-driven learning.

[0119] Method Detailed Implementation 4:

[0120] The following section uses an industrial robotic arm control system as an example to explain the fault-tolerant control method based on hierarchical reinforcement learning proposed in this invention.

[0121] The industrial robotic arm control system preferably includes 6 joint motors and position / torque sensors. Possible faults include, but are not limited to, motor failure, encoder drift (encoder drift refers to the gradual deviation between the actual position of the robotic arm and the encoder reading; the trajectory deviates from the preset path during movement, and the repeatability accuracy decreases), and external collisions.

[0122] 1. Construct a three-layer decision-making architecture: namely, construct the Fault Perception and Abstraction Layer (FPAL), the High-Level Goal Decision Layer (HGDL), and the Low-Level Control Execution Layer (LCEL).

[0123] 1) Fault Perception and Abstraction Layer (FPAL): This layer analyzes system status data in real time and uses an autoencoder to compress high-dimensional fault features into low-dimensional abstract fault codes. Specifically, it acquires joint motor data (joint angle q, angular velocity...). Data such as motor current (I) and sensor data (end force F) are used. A one-dimensional convolutional neural network (1D CNN) is called to extract spatial features from all data, and a long short-term memory network (LSTM) is called to extract temporal features from all data, resulting in an 8-dimensional fault coding vector z. This achieves compression of the fully connected layer to 8 dimensions. Where z∈R 8 It includes 4-dimensional fault types and 4-dimensional fault severity.

[0124] In addition, in the fault perception and abstraction layer FPAL, the network structure (1D CNN + LSTM) is implemented through TensorFlow (whose core role is to provide end-to-end support from model development to production deployment); when training data, 500,000 simulation samples of 20 types of faults are injected and 10% Gaussian noise is added; during online updates, incremental training is performed every 24 hours (learning rate 1e-5).

[0125] 2) The High-Level Decision Layer (HGDL), based on the aforementioned low-dimensional abstract fault coding, uses an option framework to select the current fault-tolerance option (i.e., fault-tolerance strategy) from a predefined set of fault-tolerance objectives. Specifically:

[0126] The corresponding fault-tolerance target o (i.e., the current fault-tolerance option) is selected based on the features of the 8-dimensional fault coding vector z. This involves combining the 8-dimensional fault coding vector z with the task objective (e.g., the end-target pose X). d Select the corresponding fault tolerance target o from the predefined set of fault tolerance targets. Each fault tolerance target corresponds to a fault tolerance option. Here, the set of fault tolerance targets is {precise tracking, degraded operation, emergency return to zero}, o∈{precise tracking, degraded operation, emergency return to zero}, and the update frequency is 1 Hz.

[0127] The selection logic for the fault tolerance target o is as follows:

[0128] When z[0] < 0.3 and z[1:4] norm < 0.2, the fault tolerance option of precision tracking is triggered, and the sub-target parameter is tracking error < 1mm; here, z[0] < 0.3 indicates that the first dimension of vector z is in a low-activity state, such as the equipment is running smoothly and there are no abnormal fluctuations; z[1:4] norm < 0.2 indicates that the auxiliary indicators are generally low, and the system is in a "low load" or "stable state". That is, when the main state and auxiliary indicators are at a low level, the system considers the current environment to be safe and can continue tracking operations.

[0129] When 0.3≤z[0]<0.7 or z[4]>0.5, the downgrade operation option is triggered, and the sub-target parameter is tracking error <10mm; here, 0.3≤z[0]<0.7 is compared with the low-activity state of z[0]<0.3, indicating that the main state is in a medium-level jump, and there may be slight abnormalities or the operation strategy needs to be adjusted; z[4]>0.5 indicates that the fifth dimension exceeds the threshold, and it is necessary to actively downgrade to avoid risks. That is, when the system detects potential risks or performance bottlenecks, it automatically switches to "conservative mode" to ensure safety by reducing the running speed, reducing the load or adjusting the parameters.

[0130] When z[0]≥0.7 or the collision flag is 1, the emergency return to zero option is triggered, and the sub-target parameter returns to zero within 5 seconds; here, compared with the previous two low-activity states and medium-level active states, this indicates that the main state is highly active, indicating a serious fault, overload or dangerous state.

[0131] Based on the above fault tolerance target selection logic, an improved PPO (Proximal Policy Optimization) algorithm is used to select the fault tolerance target (such as precise tracking, degraded operation, or emergency zeroing). The PPO algorithm is a reinforcement learning algorithm based on policy gradients. To adapt to fault tolerance decision-making scenarios, the PPO algorithm is improved by modifying the high-level decision output to discrete fault tolerance options (such as precise tracking, degraded operation, or emergency zeroing) instead of continuous actions. Fault severity weights are introduced into the advantage function calculation. Specifically, the advantage function is expressed by the following formula:

[0132] ; ;in, For time-series difference residuals; To introduce a dominance function that weights the severity of the fault; For time-series difference residuals; Discount factor; Here, T is the GAE dominance function decay coefficient; T is the maximum time step; t is the current time step. This is an instant reward value; The fault encoding vector is w; the learnable weights are w. Let t+1 be the state value function; Let t be the state value function at time t; The system state at time t+1; Let t be the system state at time t.

[0133] 3) The underlying control execution layer (LCEL) generates specific control actions based on the selected fault tolerance options and embeds safety constraint mechanisms to correct dangerous actions. Specifically:

[0134] An improved SAC (Soft Actor-Critic) algorithm is used to generate specific control actions. Independent SAC agents are trained for each fault-tolerant option, while a shared target network is used. The target network architecture consists of a Critic network, a 3-layer MLP (256-128-64) and an Actor network, outputting a Gaussian distribution with mean and variance. Key hyperparameters include gamma (discount factor) = 0.99, tau (target network update rate) = 0.005, lr_actor (Actor learning rate) = 3e-4, and lr_critic (Critic learning rate) = 1e-3. The SAC agents are trained to make decisions based on the corresponding fault-tolerant policies and achieve unified updates of the value function and policy transfer through the shared target network, outputting original fault-tolerant control instructions. A safety layer correction rule then performs real-time safety trimming on these original instructions to form the final safety control instructions. The two operate in a sequential, upstream-downstream relationship: the SAC agent and the shared target network are responsible for generating control instructions that meet fault tolerance requirements, while the safety layer's correction rules are responsible for boundary constraints and security hardening of these instructions. First, the agent completes the fault tolerance strategy decision output, and then safety correction ensures the safe and feasible operation of the instructions. These two consecutive steps achieve a coordinated unification of fault-tolerant control and operational safety. Real-time correction is performed through the safety layer, and the correction rules are as follows: , where ε is the dynamic safety margin.

[0135] Fault tolerance option 'o' needs to be considered in conjunction with the current state. (Corresponding to the real-time position / velocity of the robot joint, non-fault characteristics) and combined with the target state (corresponding to the preset motion state of the fault-tolerant option o), the improved SAC algorithm is called to output the joint torque τ∈R6, and the control frequency is 100 Hz, so as to correct the danger by adjusting the joint torque.

[0136] 4) Hardware Interface Specifications: Sensor Data Acquisition: Joint Angle: 17-bit absolute encoder, sampling rate 1kHz; Motor Current: ±50A Hall sensor, accuracy ±0.5%; End-effector Six-Dimensional Force Sensor: Range ±200N, noise <0.1NRMS. Control Command Output: Torque command τ transmitted via EtherCAT bus, cycle 10ms; Safety Layer Hardware Support: FPGA implements real-time motion correction (delay <100μs).

[0137] 2. Reward Function Design and Security Constraints: Different reward functions were designed for the High-Level Goal Decision Layer (HGDL) and the Low-Level Control Execution Layer (LCEL), forming a hierarchical reward framework. HGDL focuses on achieving macro-level goals, while LCEL optimizes local control quality. Specifically:

[0138] 1) High-level target decision layer (HGDL) reward function: In the HGDL reward function, the reward function is mainly divided into task reward (core objective), fault suppression reward, switching reward and security reward.

[0139] a) Task reward function:

[0140] ;in, This represents the proportionality coefficient. Indicates the penalty coefficient. This indicates the actual pose of the robot's end effector. This indicates the desired pose of the robot's end effector. This indicates the real-time angular velocity of the robot's joints. Indicates the safe return-to-zero angular velocity. This indicates the robot's current actual power. This indicates the robot's rated power.

[0141] b) Fault suppression reward function: (Fault coding norm × time, encouraging rapid suppression);

[0142] c) Switching the reward function: ( =0.5, suppressing high-frequency switching);

[0143] in, This indicates switching the penalty coefficient. This represents the fault tolerance strategy at time t. This represents the fault tolerance strategy at time t-1. This indicates the condition for switching strategies.

[0144] d) Security reward function: Continuous safe operation every 10 seconds.

[0145] The above reward configurations, taking a robotic arm as an example: .

[0146] 2) LCEL reward function in the underlying control execution layer: The reward function in LCEL is mainly divided into tracking accuracy reward, energy consumption optimization reward, motion smoothing reward and risk penalty item.

[0147] a) Tracking accuracy reward function: ;in, =20.0, exponential decay;

[0148] in, This represents the reward decay coefficient. This indicates the actual pose of the robot's end effector. This indicates the target pose of the robot's end effector.

[0149] b) Energy consumption optimization reward function: Normalized torque penalty;

[0150] in, Indicates joint torque. This represents the square of the maximum torque.

[0151] c) Motion smoothing reward function: L1 norm suppresses mutations;

[0152] in, This represents the joint torque vector at the current moment; This represents the joint torque vector at the previous moment.

[0153] d) Risk penalty term function: ;

[0154] The above reward function configuration uses a robotic arm as an example: .

[0155] 3) State space protection:

[0156] a) Dynamic shrinkage model of security domain: Among them, the shrinkage rate is This is to reduce the safety domain when the fault persists.

[0157] in, This indicates the current real-time joint angle position of the robot. Indicates the robot's rated nominal position. Indicates the radius of the dynamic security domain. Indicates the initial safety region radius. Represents the contraction rate coefficient. Indicates the duration of the fault.

[0158] b) Implement risk prediction: When Risk_Score > 0.8, force a switch to safe mode.

[0159] 4) Constraints on the exploration process (training safety):

[0160] a) In the LCEL output layer, by exploring the noise variance To constrain motion noise.

[0161] in, This represents the initial basic variance when there are no faults. This represents the fault coding vector.

[0162] b) Implement an experience replay pool mechanism in reinforcement learning to remove experiences that lead to... Empirical samples are used to mask state-restricted areas. Here, q represents the robot's current real-time joint angle position. This represents the set of dynamic safety domains for the robot.

[0163] c) In all network update operations, by setting This is used to prune gradient safety. Represents the gradient operator, This represents the gradient vector of the loss function.

[0164] d) During the offline training phase, fault injection is simulated by training only severe faults (such as motor jamming) in the simulation environment.

[0165] 3. Offline pre-training process, including phased training strategies, fault injection specifications, and performance optimization techniques, refer to the accompanying diagrams in the manual. Figure 2 The offline pre-training process is explained. The offline pre-training phase employs a staged training strategy: first, the FPAL encoder is trained; then, the LCEL executor is trained option-by-option; and finally, the HGDL decision-maker is trained jointly.

[0166] The phased training process mainly includes three phases: FPAL fault encoder pre-training, LCEL low-level actuator training, and HGDL high-level decision-maker training.

[0167] 1) Fault Perception and Abstraction Layer FPAL Fault Encoder Pre-training: Training objective: To learn the mapping from the original state to the low-dimensional fault code. Data generation: 20 basic fault classes are injected with reasonable parameter ranges (mainly including motor power reduction, encoder offset, torque sensor drift, and external collision force). 100,000 samples are collected for each fault. Sensor noise (Gaussian distribution, σ=5% of the range) is added, and the time window length is set to 100ms (10 consecutive states). Network structure and training parameters: Temporal fault features are extracted through two layers of one-dimensional convolution, and then feature dimensionality reduction is completed through three fully connected layers, finally outputting a 16-dimensional low-dimensional fault code. Training uses the Adam optimizer with a learning rate of... The batch size is 512, the loss function is the mean squared error, the maximum number of training epochs is set to 100, and an early stopping strategy and gradient clipping constraints are adopted.

[0168] This ensures that the model converges stably.

[0169] 2) Training of the LCEL (Low-Level Control Execution Layer) actuator: Training mode: From fixed FPAL encoder to independent training of sub-options. The learning stages are divided into primary, intermediate, and advanced levels. In the primary level, the task difficulty is defined as single-joint localization, the fault combination is single fault, the reward function is position error only, and the training step size is set to 500k. In the intermediate level, the task difficulty is defined as multi-joint trajectory tracking, the fault combination is double fault, the reward function is position + energy consumption, and the training step size is set to 800k. In the advanced level, the task difficulty is defined as dynamic obstacle avoidance, the fault combination is random fault, the reward function is the full reward function, and the training step size is set to 1.2M. The SAC algorithm is improved by dynamically associating the entropy coefficient α with the fault code z. According to information theory, the entropy of the optimal policy should satisfy: Where s represents the robot system state, including joint position q and angular velocity q̇.

[0170] Increasing the α value during a failure can preserve more randomness and prevent the strategy from converging prematurely to a dangerous action.

[0171] Parameter sharing mechanism: A hybrid shared-dedicated network structure was designed. The shared layer uses CNN+LSTM to extract spatiotemporal features (shared by all options), while each fault-tolerant option in the dedicated layer has an independent MLP policy head. The policy network can be decomposed into... ,in Indicates a shared feature encoding function. This indicates a policy header specific to the option. Fault codes (z) are dynamically embedded into shared features, enabling lower-level policies to perceive the decision-making intent of higher levels.

[0172] 3) High-level decision-making layer (HGDL) training: PPO policy optimization process: First, sample fault scenarios, then FPAL extracts parameters z, and HGDL generates the corresponding option probabilities. Depending on whether safety masking is performed, if allowed, option o is executed, LCEL performs the corresponding control operation and collects the reward R_high, and PPO is updated every 10 steps; if prohibited, the suboptimal safe option is selected.

[0173] Expert demonstration guidance addresses key issues such as bottlenecks in security exploration, cold start challenges, and policy interpretability. It extracts expert rules from historical fault records and domain knowledge, generating corresponding expert policies (π_expert). A hybrid training reward function is designed for the hybrid training strategy. .

[0174] in, Indicates the original PPO loss. Indicate expert strategy, Indicates the current learning strategy. This represents the KL divergence.

[0175] For motor stall, a step torque of 0 can be injected, with the corresponding expert action being to switch to safe torque mode; for encoder failure, a constant output value can be injected, with the corresponding expert action being to enable redundant sensors; for communication interruption, a 50% random packet loss can be injected, with the corresponding expert action being local buffer control.

[0176] Catastrophic forgetting protection: Catastrophic forgetting refers to the phenomenon where a neural network's performance on older tasks deteriorates sharply as it continues to learn new tasks. Standard Elastic Weight Fixation (EWC) is used for this purpose. ,in For the diagonal elements of the Fisher information matrix, These are the historically optimal parameters. Improved base loss, This represents the fault weighting coefficient. By adjusting the fault perception importance weight, stronger constraints can be imposed on the parameters corresponding to high-frequency / severe faults. In addition, a dynamic regularization parameter was introduced in this experiment. It decreases with training.

[0177] 4. Online Adaptive Deployment: Online adaptive deployment of the hierarchical reinforcement learning fault-tolerant control system, covering its architecture design and real-time optimization strategies. Incremental updates are performed every 10 minutes during the online adaptive phase, with the gradient update steps limited to 50 steps.

[0178] Online adaptive system architecture such as Figure 3 As shown, the core components include a real-time inference pipeline, an online learning module, and a security layer. The real-time inference pipeline includes: FPAL (10ms-level fault code generation, 1D CNN + LSTM), HGDL (1Hz decision frequency, PPO policy network), LCEL (100Hz control frequency, SAC policy network + security layer). The online learning module includes: an experience replay pool, a sliding window storing the latest 10,000 transition samples (s, z, o, a, r, s'), incremental updates, and policy fine-tuning every 10 minutes (limiting gradient steps to ≤50). The security layer includes: a hardware watchdog, independently monitoring joint position / velocity exceeding limits and implementing a fallback policy; when the reward R < -5 for 3 consecutive steps, it forcibly switches to a safe mode.

[0179] In real-time fault-tolerant control systems, traditional fixed resource allocation methods face problems such as resource waste, response latency, and excessive energy consumption. Therefore, it is necessary to design dynamic computing resource allocation, dynamically allocating parameters such as CPU core utilization, GPU stream processor activation ratio, memory bandwidth allocation, and model quantization accuracy. A fault urgency assessment method should also be designed. ,in , This represents the Sigmoid activation function. Indicates learnable weights, This represents the fault coding characteristics, and k represents the slope adjustment coefficient. This represents the function input value. GPU acceleration and CPU vectorization are dynamically allocated based on high / low urgency, with the weight configuration: w=[0.3, 0.2, 0.4, 0.1] (corresponding to motor / sensor / collision / communication faults, slope coefficient k=5.0 (for steep urgency increases). In actual testing on the UR5 robotic arm, the computational load was reduced by 42%, while the emergency fault response latency was shortened to 0.08s.

[0180] This experiment employs a flexible experience replay strategy combined with Prioritized Experience Replay (PER), dynamically adjusting the sampling probability based on sample importance (e.g., TD error). Based on TD error, and considering latent features and option changes, experience replay is flexibly performed, prioritizing the replay of samples with high fault severity and high learning value, and enhancing the learning weights for severe fault samples. Simultaneously, independent sub-buffers are divided according to fault type (e.g., motor failure zone, sensor failure zone), and the sampling ratio for each zone is dynamically allocated according to the fault occurrence frequency, ensuring that rare fault samples are not overwhelmed. Taking LCEL training as an example, the implementation process includes sample storage, priority calculation, flexible sampling, and weight updates. Compared to traditional PER (Prioritized Experience Replay), fault recovery speed is improved by 37% (simulation test data), and the success rate of rare fault handling increases from 52% to 89%.

[0181] An online safety constraint adjustment mechanism was designed, and a layered dynamic safety protection system was constructed. In the action layer (LECL), the adjustment object is the actuator saturation constraint, and the adjustment strategy is to adjust the dynamic safety margin. In the state layer, the adjustment object is the joint safety region, and the adjustment strategy is to adjust the safety domain radius. ,in, , Indicates the initial safety radius. Represents the safety domain shrinkage coefficient. express The fault encoding vector is defined at any given moment. In the decision layer (HGDL), the adjustment object is the option switching rule, and the adjustment strategy is to adjust the urgency threshold.

[0182] Method Detailed Implementation 5:

[0183] like Figure 4The diagram shows a flowchart of a fault-tolerant control method based on hierarchical reinforcement learning proposed in this invention in a practical application scenario. In response to an environment s(t) fault, a fault-tolerant control strategy based on hierarchical reinforcement learning is invoked. The fault perception and abstraction layer (FPAL) outputs a fault encoding vector z, which is then used as input to the high-level target decision layer (HGDL). In the HGDL, the target fault-tolerant option o (i.e., the target fault-tolerant strategy) and the target obtained from o are obtained. In the ground-level control execution layer (LCEL), the system's fault-tolerant control action (i.e., the original action in the diagram) is obtained by combining the target fault-tolerant option o (i.e., the target fault-tolerant strategy), the current state, and the target state. The fault-tolerant control action is then modified through a safe room.

[0184] On the other hand, the present invention also proposes a fault-tolerant control system based on hierarchical reinforcement learning, including a processor for executing a computer program to implement the steps of the fault-tolerant control method based on hierarchical reinforcement learning as described above.

[0185] On the other hand, the present invention also proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implements the steps of the fault-tolerant control method based on hierarchical reinforcement learning as described above. For specific implementation details of the system and the computer program product, please refer to Specific Implementation Details 1-5 of the method, which will not be repeated here.

[0186] In addition, in practical applications, the operating device of this invention must meet the following requirements: a multi-source sensor interface module to collect joint angle, angular velocity, motor current and end force signals.

[0187] The hierarchical processing core should include: an FPGA-implemented real-time safety constraint module with an action correction delay of <100μs; a GPU-accelerated FPAL fault code generation module; a multi-core CPU parallel HGDL decision module and LCEL execution module; and an EtherCAT bus interface that outputs control commands to the actuators at 10ms cycles. The hierarchical processing core employs a dynamic resource allocation strategy: 1) based on fault urgency. ,in , This represents the Sigmoid activation function. Indicates learnable weights, This represents the fault coding characteristics, and k represents the slope adjustment coefficient. Indicates the function input value. Dynamically allocates computing resources; 2) When Urgency(z)>0.8, activates the GPU acceleration channel and allocates more than 80% of CPU core resources.

[0188] In summary, the fault-tolerant control method, system, and computer program product proposed in this invention address the problems of existing fault-tolerant control methods, such as strong model dependence, weak ability to handle unknown faults, and difficulty in designing high-dimensional systems, as well as the problems of dimensionality curse, high exploration risk, and poor policy generalization faced by traditional reinforcement learning in fault-tolerant scenarios. The core of this invention lies in a three-layer decision architecture: 1) Fault Perception and Abstraction Layer (FPAL), which extracts fault features and compresses them into low-dimensional fault codes through 1D-CNN and LSTM networks; 2) High-Level Target Decision Layer (HGDL), which uses an improved PPO algorithm to select fault-tolerant targets (such as precise tracking, degraded operation, and emergency zeroing); 3) Low-Level Control Execution Layer (LCEL), which uses an improved SAC algorithm to generate specific control actions.

[0189] Simultaneously, a two-stage operation mechanism (offline pre-training and online adaptive deployment) and safety constraint mechanisms (such as dynamic safety domains and hardware watchdogs) were designed. The achieved technical effects include: reducing decision-making dimensionality and improving learning efficiency through a layered architecture; embedding safety constraint mechanisms to avoid dangerous exploration; enhancing generalization ability through modular strategies and cross-fault knowledge transfer; and achieving model-free optimization to adapt to complex dynamic systems. Experiments show that this method can reduce fault recovery time to 3.2 seconds and emergency stop trigger rate to 3% on industrial robotic arms, while significantly reducing computational resource consumption.

Claims

1. A fault-tolerant control method based on hierarchical reinforcement learning, characterized in that, The method comprises the following steps: 1) In the fault awareness and abstraction layer constructed in advance based on a hierarchical reinforcement learning algorithm, fault feature extraction is performed on system state data, and the extracted fault features are dimensionally reduced to obtain a fault encoding vector; 2) In the high-level target decision layer constructed in advance based on the hierarchical reinforcement learning algorithm, a target fault tolerance strategy of the fault encoding vector is selected from a predetermined fault tolerance strategy set according to a pre-prepared selection logic between the fault encoding vector and the fault tolerance strategy; 3) In the bottom control execution layer constructed in advance based on the hierarchical reinforcement learning algorithm, a SAC algorithm is called according to the target fault tolerance strategy and the fault features to obtain a fault tolerance control action of the system to control the system to execute the fault tolerance control action.

2. The fault-tolerant control method based on hierarchical reinforcement learning according to claim 1, characterized in that, In the fault awareness and abstraction layer, the system state data is subjected to fault feature extraction and the extracted fault features are dimensionally reduced to obtain a fault encoding vector by the following method: A one-dimensional convolutional neural network is called to extract spatial features of the system state data, and a long short-term memory network is called to extract time sequence features of the system state data to obtain fault features; The extracted fault features are dimensionally reduced to the fault encoding vector by a fault encoder. 3.The hierarchical reinforcement learning based fault-tolerant control method according to claim 1, wherein, The high-level target decision layer comprises an optimized PPO algorithm, which is called to enable the high-level target decision layer to select the target fault tolerance strategy of the fault encoding vector from the predetermined fault tolerance strategy set according to the pre-prepared selection logic between the fault encoding vector and the fault tolerance strategy.

4. The fault-tolerant control method based on hierarchical reinforcement learning according to claim 3, characterized in that, The selection logic is as follows: When the main fault severity element in the fault encoding vector takes a value in a preset low active interval, and each type of sub-fault severity element in the fault encoding vector does not exceed a preset stable threshold, no single point is out of limit and the numerical fluctuation is within an allowable range, a tracking fault tolerance strategy is triggered; When the main fault severity element in the fault encoding vector takes a value in a preset medium active interval, or any one type of sub-fault severity element in the fault encoding vector reaches a preset safety limit, a degraded operation fault tolerance strategy is triggered; When the main fault severity element in the fault encoding vector takes a value in a preset high active interval, or a signal for reflecting a collision in a physical space is generated, an emergency zero-return fault tolerance strategy is triggered.

5. The fault-tolerant control method based on hierarchical reinforcement learning according to claim 4, characterized in that, If the fault encoding vector is an 8-dimensional fault encoding vector, the selection logic is as follows: When z[0]<0.3 and the norm of z[1:4]<0.2, the tracking fault tolerance strategy is triggered; When 0.3≤z[0]<0.7 or z[4]>0.5, the degraded operation fault tolerance strategy is triggered; When z[0]≥0.7 or the collision flag=1, the emergency zero-return fault tolerance strategy is triggered; Wherein, z is the 8-dimensional fault encoding vector.

6. The fault-tolerant control method based on hierarchical reinforcement learning according to claim 3, characterized in that, The optimized PPO algorithm further introduces an advantage function comprising a fault severity weight, wherein the advantage function is represented by the following formula: ； wherein the timing difference residual ; wherein, is the advantage function with the introduction of the fault severity weight; is the time difference residual; is the discount factor; is the GAE advantage function decay coefficient; T is the maximum time step; t is the current time step; is the immediate reward value; is the fault encoding vector; w is the learnable weight; is the state value function at time t+1; is the state value function at time t; is the system state at time t+1; is the system state at time t.

7. The fault-tolerant control method based on hierarchical reinforcement learning according to claim 1, characterized in that, In the bottom control execution layer, a SAC algorithm is called to perform decision training for each fault tolerance strategy to obtain an original fault tolerance control instruction; the original fault tolerance control instruction is subjected to real-time safety clipping according to a preset correction rule to obtain a final safety control instruction, so that the fault tolerance control action of the system is obtained according to the final safety control instruction.

8. The fault-tolerant control method based on hierarchical reinforcement learning according to claim 1, characterized in that, Further comprising: A hierarchical reward framework pre-built based on a hierarchical reinforcement learning algorithm, wherein the hierarchical reward framework includes a reward function built for the high-level goal decision layer and a reward function built for the low-level control execution layer; The reward functions constructed for the high-level target decision-making layer include a task reward function, a fault suppression reward function, a switching reward function, and a security reward function; The reward functions constructed for the underlying control execution layer include a tracking accuracy reward function, an energy consumption optimization reward function, a motion smoothing reward function, and a risk penalty function.

9. A fault-tolerant control system based on hierarchical reinforcement learning, comprising a processor, characterized in that, The processor is used to execute a computer program to implement the steps of the fault-tolerant control method based on hierarchical reinforcement learning as described in any one of claims 1 to 8.

10. A computer program product comprising computer programs / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the fault-tolerant control method based on hierarchical reinforcement learning as described in any one of claims 1-8.

Images