A reinforcement learning fault-tolerant training system oriented towards human feedback

Abstract

This invention belongs to the field of artificial intelligence technology, specifically a reinforcement learning fault-tolerant training system oriented towards human feedback. The system comprises a decoupled recovery engine, a mirror checkpoint engine, and a generation phase checkpoint engine; it supports component-level hot recovery to shorten restart latency after a failure; it utilizes data redundancy between the training and inference engines based on the RLHF paradigm to achieve low-overhead mirror checkpointing; and it sets semantic checkpoints in the generation phase to limit recomputation to sub-iteration ranges, thereby reducing recovery overhead and improving resource utilization. Experimental results show that this invention can reduce interruption time caused by cluster failures, training failures, and suspensions in reinforcement learning jobs based on human feedback, improving cluster training efficiency, and is suitable for large-scale RLHF training under high-performance computing resources.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a reinforcement learning fault-tolerant training system based on human feedback in a large-scale cluster environment. Background Technology

[0002] With the rapid development of artificial intelligence, Large Language Models (LLMs), as the core foundational models of generative artificial intelligence, have made significant progress in the field of natural language processing. Relying on self-supervised pre-training with massive corpora, LLMs have demonstrated outstanding language understanding and generation capabilities, significantly improving the performance of tasks such as machine translation, text generation, question answering systems, and multi-turn dialogues, and continuously driving artificial intelligence from specialized to general-purpose.

[0003] In 2017, Vaswani et al. proposed the Transformer model, which replaced the traditional recurrent / convolutional structure with a self-attention mechanism. This significantly enhanced the ability to model long-range dependencies and brought highly parallel training and inference efficiency, becoming the mainstream architecture of contemporary LLM. Transformer-based models (such as BERT, GPT, Llama, and DeepSeek) rely on self-supervised pre-training on massive corpora, demonstrating excellent language understanding and generation capabilities. They have significantly improved the performance of tasks such as machine translation, text generation, question answering systems, and multi-turn dialogue, and have continuously driven artificial intelligence from specialized to general-purpose.

[0004] To improve the alignment between the response quality of large language models and user expectations, reinforcement learning based on human feedback (RLHF) is typically used for fine-tuning. RLHF, as a key fine-tuning stage, aims to make the model generate content in a way that better conforms to pre-defined interaction logic and response specifications.

[0005] A typical RLHF system consists of multiple models, including an Actor, a Critic, a Reference, and one or more Reward Models. The Actor and Reference are large language models that have been pre-trained or fine-tuned, while the Critic and Reward models can have other structures, typically fine-tuned based on human preference data, and replacing the language modeling head with a scalar output head.

[0006] RLHF training can be divided into two main phases:

[0007] Phase one involves sample generation and sampling (Rollout), where the Actor model in the inference engine generates responses for the cues in the current batch. This process begins with processing the cues in the pre-filling phase to generate the first output token, followed by a decoding phase that progressively and autoregressively generates subsequent tokens. Each cue and its corresponding response together constitute a training sample for the current iteration.

[0008] Phase Two involves multi-model forward propagation and parameter updates (Train). In this phase, the Reference, Reward, and Critic models all perform forward propagation on the generated samples. For each mini-batch, the Actor and Critic models first perform forward computation, then calculate the loss based on the inference results, and update the model parameters through backpropagation. Once the Actor has completed training for all mini-batches, its updated parameters will be used in the next iteration for generating samples in the next round.

[0009] The transition between the two phases relies on a tight parameter synchronization mechanism. Specifically, the Actor model parameters used for inference in the first phase and the Actor model parameters used for training and about to be updated in the second phase must be consistent at the start of the iteration. When switching from phase two to phase one, the updated Actor model parameters must be synchronized back to the inference engine for use in the next iteration. Periodic parameter synchronization is crucial for ensuring the continuity of the training process and data consistency.

[0010] However, as the training scale of large language models continues to expand, the stability and fault tolerance of the system face increasingly severe challenges. Long-running training jobs are highly sensitive to hardware failures and network jitter, making them prone to interruptions and even failures; training cycles that often last for weeks or even months make failures almost inevitable. For example, a 2024 report by Meta AI stated that 446 failures occurred during a 54-day Llama-3 405B training session, mostly triggered by hardware issues such as GPU failures, memory errors, and network interruptions. In large-scale distributed environments, frequent state synchronization between nodes is required; the failure of any GPU or node can halt training until the fault is isolated, repaired, or replaced. When a job involves hundreds or even tens of thousands of GPUs, a single point of failure can cause a system-wide shutdown and significant economic losses. Furthermore, reinforcement learning based on human feedback (RLHF) requires complex interactions and coupling among multiple models, and its algorithmic characteristics amplify these unstable factors, making the risk of training failure even more prominent.

[0011] To mitigate economic losses from training failures or glitches, developers typically introduce backup computing nodes and periodically saved checkpoints to support rapid recovery of training tasks. Because ensuring model state consistency is crucial, building checkpoints often requires short training pauses to create a globally consistent "barrier" to prevent parameter version inconsistencies or state mismatches, thus incurring additional checkpoint overhead. Therefore, existing work primarily focuses on reducing checkpoint overhead, such as employing asynchronous checkpointing processes and incremental checkpoint designs. However, these approaches often fail to yield effective benefits in RLHF scenarios because:

[0012] (1) Repeated calculation: The checkpoint fixes the recovery point at the last checkpoint, which cannot eliminate the "exposed window" loss within the checkpoint interval. That is, the calculation stage that has been completed before the failure still needs to be calculated repeatedly. Since the iteration time of RLHF is often as long as 30 minutes to 1 hour, failures result in a long loss of calculation time, which leads to a large loss.

[0013] (2) Instances need to be rebuilt. When any instance fails, the system needs to delete and reschedule all instances because communication resources cannot be released, resulting in high restart costs. In addition, the RLHF training task involves multiple models, and the restart and loading of the models are more complex than the fine-tuning task of the same level of models. Therefore, the restart overhead of the RLHF training task is significantly greater than that of pre-training and fine-tuning.

[0014] The above mechanism further exposes the following key challenges in the RLHF distributed large model training environment:

[0015] 1. Long recomputation time: Traditional checkpointing schemes cannot reduce the recomputation time after recovery to within one iteration, while the iteration time of RLHF training often reaches the hour level. Compared with fine-tuning and pre-training, such a long iteration time will still result in a long recomputation time even when the checkpoint frequency reaches 1.

[0016] 2. Long restart time: RLHF involves the collaboration of multiple models such as Actor / Ref / Reward / Critic. Restarting requires reloading a large number of weights, restoring the optimizer / stochastic state, and rebuilding the communication domain (such as NCCL process group, RDMA / QP). Due to the limitations of model size and storage I / O, restarting often takes several minutes to several hours, and this pause is difficult to shorten with conventional checkpointing mechanisms.

[0017] 3. Machine failures and training failures: Compared to pre-training / fine-tuning, RLHF's inference-training alternation and cross-engine coupling introduce more failure surfaces; at the same time, the cluster size is larger, and the number of GPUs affected by a single failure and the duration are higher, significantly amplifying the wasted GPU hours and economic costs. Summary of the Invention

[0018] The purpose of this invention is to provide a reinforcement learning fault-tolerant training system oriented towards human feedback. It addresses the issues of rollback and recovery caused by node failures / training failures in the cluster, as well as the decline in resource utilization, and the bottlenecks of low checkpoint efficiency and high recovery overhead. It constructs a fast consistency recovery and decoupled restart mechanism, thereby reducing recalculation and restart costs and improving cluster resource utilization.

[0019] The reinforcement learning fault-tolerant training system for human feedback provided by this invention consists of a decoupled recovery engine, a mirror checkpoint engine, and a generation phase checkpoint engine. Its core features include: supporting component-level hot recovery to shorten restart latency after a failure; utilizing data redundancy between the training and inference engines based on the RLHF paradigm to achieve low-overhead mirror checkpointing; and setting semantic checkpoints in the generation phase to limit recomputation to sub-iteration ranges, thereby reducing recovery overhead and improving resource utilization. Specifically:

[0020] (i) The decoupled recovery engine consists of three types of sub-modules, providing component-level hot recovery for each training component of RLHF, isolating faulty units, and avoiding the overhead of job-level full rescheduling and restart; wherein:

[0021] Submodule 1 is the mapping manager, responsible for maintaining weight mapping; maintaining a sharding mapping table of parameters / states between the inference cluster and the training cluster; after a failure, according to the mapping guidance, the replacement node pulls parameters and optimizer shards from the corresponding memory / storage address and completes the assembly to rebuild the GPU-side training state;

[0022] Specifically, the mapping manager is used to perform redundant mapping of the sharding design between the training cluster and the inference cluster. This design is because there is periodic synchronization between the inference cluster and the training cluster during RLHF training, so they have the same model parameters. Through sharding mapping, the size of checkpoints can be reduced, and high-performance networks can be used to transfer checkpoints during fault recovery to improve recovery efficiency.

[0023] Submodule 2 is the training monitoring engine, used to monitor the training status, collect training status data, and push it to the master node (Rank-0 node). This design avoids the overhead of fault location and diagnosis after a failure. By using the training monitoring engine to report node status periodically, recovery efficiency can be effectively improved. Specifically, it continuously collects the heartbeat, health status, and error codes of each component; once a component failure is detected, it immediately triggers local recovery: isolates the faulty process group, freezes the current training step, saves the minimum necessary context, and coordinates with the mapping manager and node agent to perform recovery to continue running at the nearest semantic boundary.

[0024] Submodule 3 is a node agent responsible for post-fault reconstruction, including rebuilding the communication group and training framework. Since training communication group resources (such as NCCL) cannot be safely released after a failure, a job-level restart of all nodes is required. Through the node agent, a resilient strategy is used to reduce the communication group after a failure, safely decoupling the faulty nodes, enabling node-level restarts for training failures. This design reduces fault recovery from the job level to the node level, minimizing the spread of the fault and effectively reducing restart overhead.

[0025] Furthermore, it connects to the underlying infrastructure, requests alternative nodes and GPU resources, generates deployment specifications based on the current topology, and completes component restart and communication domain initialization.

[0026] The collaboration of the three types of sub-modules can complete component-level hot recovery without interrupting global operations, significantly shortening restart latency and improving resource utilization in fault scenarios.

[0027] (ii) The mirror checkpoint engine is an essential recovery method and a crucial means of fault recovery. Specifically, it is responsible for building backups of the training and inference clusters' training states. This engine relies on the sharding mapping table of the data redundancy and mapping manager to plan and select backup nodes for storing weight parameters and optimizer states. During training, the mirror checkpoint engine senses network idle time and writes checkpoint data in byte form to the corresponding node's memory, thereby constructing mirror checkpoints.

[0028] The mirrored checkpoint engine is designed to achieve efficient and low-cost recovery. This design is based on the data redundancy between the inference and training clusters, which allows for mirrored backups, thus avoiding the need to save the entire training state. By reducing the checkpoint size, the pause overhead associated with saving checkpoints can be minimized. Furthermore, the mirrored checkpoints are primarily stored on the backup nodes as memory checkpoints (i.e., in byte form), and deserialized transmission can be achieved through high-performance networks (such as RDMA), reducing I / O bandwidth bottlenecks during low-recovery periods and improving recovery efficiency.

[0029] (iii) The generation phase checkpoint engine consists of two types of sub-engines, which set semantic breakpoints for the “generation-training” pipeline to reduce recalculation after rollback and restore the accelerated state of the inference side.

[0030] Sub-engine 1 is the Output checkpoint engine. After the inference cluster completes generation, it persists the Rollout results and their metadata. After fault recovery, it directly sends the generated results back to the training engine by sample / batch index, skipping duplicate generation and ensuring order and deduplication (idempotent import). It can converge the rollback recalculation window to the "training side" sub-iteration, significantly eliminating the time overhead of the generation stage.

[0031] Submodule 2 is the KV-Cache checkpoint engine, which selectively snapshots or quickly rebuilds the inference acceleration key-value cache (KV-cache) (e.g., prefix reuse / segmented replay). During job recovery, the KV-cache is rebuilt or preheated based on the session / sample key, reducing prefill latency and allowing direct entry into the decoding stage or shortening prefilling. During the remaining sample inference, the existing accelerated state is restored, improving throughput after recovery and reducing overall latency.

[0032] The two sub-engines work together to ensure that there is no need to regenerate after a rollback, and that the inference performance quickly returns to a steady state.

[0033] While the rollout phase is longer than the training phase, its output size is small, with each batch's output only a few KB. This means storing this output requires almost no additional overhead. Therefore, the system synchronizes the output in the training machine's CPU memory during the training phase and maintains this output in both phases. If a failure occurs during the training phase, the restarted training executor can immediately resume operation based on the output stored in the rollout phase, avoiding time-consuming recalculations during the rollout phase. On the other hand, if a rollout machine fails, the training phase remains unaffected because the output is already stored locally; the recovered rollout executor only needs to start calculating subsequent batches. Although the output checkpointing mechanism prevents duplicate calculations during the rollout phase, it overlooks another important issue: the KV-Cache is also lost when the rollout phase fails. Since the KV-Cache can accelerate subsequent inference speed, the KV-cache checkpointing engine saves the key-value cache to improve inference speed after failure recovery, allowing training to return to a steady state as quickly as possible.

[0034] However, due to the large size of the KV-Cache, significant memory overhead may occur. To balance time efficiency and overhead, the system employs a selective key-value cache checkpointing strategy. We propose a "cache utilization" metric, which refers to the proportion of key-value cache entries hit in subsequent batches. We set a threshold (e.g., 0.1) to balance time efficiency and memory overhead. If cache utilization exceeds the threshold, it indicates that more batches can be stored to accelerate computation; otherwise, storing more batches increases memory overhead without a significant improvement in time efficiency. Here, entries already stored in the key-value cache are not transmitted to the training phase. Furthermore, since all input batches can be preprocessed offline at the beginning of each iteration, the system can identify frequently used key-value entries. Therefore, the system only transmits key-value entries that will be hit in subsequent batches to the training phase. This mechanism only transmits a few percent (e.g., 10%) of the total key-value entries, thus significantly reducing bandwidth and memory overhead.

[0035] Furthermore:

[0036] (i) The decoupled recovery engine is used to manage the model and restore the training configuration during RLHF training. To reduce model loading overhead, this invention uses weight mapping to efficiently utilize redundant parameter slices. To address the inefficiency of restarting all models after a failure, this invention employs a decoupled restart mechanism, restoring only the failed model in real time. To avoid problems caused by outdated communication groups, this invention uses a communication group emptying and reconstruction method to initialize the communication domain.

[0037] Submodule 1 is the mapping manager, which, given the size of the training cluster and inference set and the parallel strategy, guides the recovery nodes to pull checkpoint shards after a failure by maintaining weight mappings. This module determines: (1) the memory mapping of the training GPU weight parameters in the inference GPU; and (2) the memory mapping of the inference GPU weights in the inference GPU under the same stage. The mapping manager uses the following algorithm to determine the available shard partitions.

[0038] According to the recovery model, for each partition Given the availability mask as follows:

[0039] ;

[0040] The availability for the training side is as follows:

[0041] ;

[0042] The set of "task pairs" that need to be assigned values ​​is defined as follows:

[0043] ;

[0044] here For enumeration sequence ,in .

[0045] Having implemented the sharding acquisition algorithm, we define the nearest available inference-side index operator as follows:

[0046] ;

[0047] Using pointer sequences Indicates the current starting selection position and initializes it. ;

[0048] For each step of the sequence The recursive algorithm for updating the inference-side index and pointers of the sliced ​​data is as follows:

[0049] ;

[0050] ;

[0051] From the above, we can obtain the closed form of the index function:

[0052] ;

[0053] Submodule 2 is the training monitoring engine. This engine monitors the heartbeat signals of the components and inserts monitoring probes in front of the cluster. Upon detecting a fault, it promptly isolates and freezes the training process to prevent job-level failures caused by erroneous communication or computation. After detecting a fault, the training monitoring engine, in conjunction with the checkpoint engine and node agents, reconstructs the training parameters, achieving hot recovery, and restarts the training process after recovery.

[0054] Submodule 3 is a node agent. In order to release communication resources and rebuild the communication group, this system designs a node agent to manage the nodes. After a failure occurs, the node agent reduces the communication group to remove the faulty node, then safely releases the communication group resources, applies for alternative resources from the infrastructure and deploys the components, and builds the communication network according to the current topology and parallel strategy to complete the component restart and communication domain initialization.

[0055] (II) In the mirror checkpointing engine, utilizing the data redundancy between the RLHF training cluster and inference cluster under the separate architecture, this invention introduces a mirror recovery mechanism. The training and inference models act as backups for each other, ensuring that the latest model state can be loaded before a failure occurs and effectively reducing bandwidth pressure on the storage server. Based on training execution, the mirror checkpointing engine senses idle bandwidth in the cluster network and, according to the parallel strategy and sharding mapping table, pipelines the optimizer state of the training cluster to the corresponding node memory. The constructed sharding transmission model is as follows.

[0056] Define the timestamp of network idle time According to The model calculates the total amount of checkpoint fragment data that can be transmitted. ,Right now:

[0057] ;

[0058] ;

[0059] This design offers two main benefits: First, it significantly reduces recalculation overhead by recovering from the latest state rather than recalculating after multiple iterations. Second, because it holds the latest state of all models, it only needs to recover the model that failed, thus providing new possibilities for model decoupling in fault recovery.

[0060] (III) The aforementioned checkpoint generation engine, based on the inference engine, includes an Output checkpoint engine and a KV-Cache checkpoint engine to reduce the overhead of inference recalculation after fault recovery. Firstly, during inference, the Rollout results of the request queue are structurally persisted, stored in host memory, and streamed back to the training cluster in batches. In the event of a fault, the training engine directly retrieves and replays the checkpoint, skipping repeated generation and ensuring order and idempotency. Secondly, the KV-Cache checkpoint engine selectively saves the KV-Cache within semantically safe windows (such as pre-filling completion and segmented decoding boundaries), and warms up / rebuilds the inference-side cache after recovery, accelerating the inference of incomplete samples.

[0061] This invention achieves the following functions:

[0062] (1) Checkpoint construction: The system is based on a mirror checkpoint engine, which perceives computing resources, parallel topology and iteration time, automatically constructs a transmission model and generates a mirror checkpoint strategy accordingly. This strategy uses network idle windows for asynchronous dumping and backhaul, realizing low-disturbance and low-overhead checkpoints, significantly reducing checkpoint costs, reducing rollback rounds, and improving cluster utilization and effective training time.

[0063] (2) Training monitoring: The system has a built-in training monitoring engine that uses a lightweight probe and heartbeat mechanism to perceive the task status in real time, quickly locate faults and automatically isolate / empty faulty nodes, and link the recovery process: reinitialize the training context, load necessary checkpoints, and work with the mapping manager to complete data recovery. This mechanism significantly reduces the time for fault detection and handling, reduces interruption costs, and improves cluster utilization and effective training time.

[0064] (3) Decoupled fault recovery: The system adopts component-level decoupling, isolating affected training tasks from healthy components. Only the faulty tasks are restarted / migrated without affecting other modules, which significantly reduces the risk of global interruption and ensures training continuity. Based on the component recovery process, job-level restarts and overall rescheduling can be avoided; the node agent automatically requests alternative nodes from the resource management system or cloud infrastructure, and deploys the components to the alternative nodes in combination with the cluster topology and component dependencies, and completes state reclamation and checkpoint loading to achieve smooth and seamless restart.

[0065] (4) Reduce recalculation overhead: During the generation phase, the system uses the checkpoint engine to persist the Rollout results and KV-cache; after the fault is recovered, the existing results are directly replayed and reused, the inference-side cache is quickly warmed up / rebuilt, the inference of incomplete samples is accelerated, and the high overhead caused by repeated calculation is avoided.

[0066] (5) Improve cluster utilization and reduce recovery costs: The system has real-time fault perception and automatic recovery capabilities, supports rapid recovery and breakpoint resume training, significantly reduces fault interruption and failure restart time, reduces computing power waste and scheduling overhead, and is suitable for large-scale RLHF training under high-performance computing resources. Attached Figure Description

[0067] Figure 1 This is a diagram illustrating the RLHF model configuration and job deployment.

[0068] Figure 2 This is a schematic diagram of the workflow for the RLHF generation and training phases.

[0069] Figure 3 This is the architecture diagram of the RLHF fault-tolerant training system.

[0070] Figure 4 This is a schematic diagram of idle time transmission in a mirror checkpoint sensing network.

[0071] Figure 5 This is a diagram of the mirror checkpoint engine architecture.

[0072] Figure 6 This is a schematic diagram of parameter mapping (fragmented mapping table).

[0073] Figure 7 This is a comparison chart of the average fault recovery time of the fault-tolerant training system and the baseline during synchronous / asynchronous RLHF training. Detailed Implementation

[0074] The design of a reinforcement learning fault-tolerant training system oriented towards human feedback involves the following steps:

[0075] Step 1: When the training framework starts, the decoupled recovery engine of this invention calls the training monitoring and node agent to begin building the GPU environment communication domain for training and inference. This includes Rank mapping, parallel dimension, affinity, and communication topology. Based on the deployment of the training and inference clusters, it calls the mapping manager to implement weight mapping, organizes the sharding mapping table, and pushes it to the image checkpoint engine of each GPU. The training monitoring system of the Rank-i node spawns a heartbeat thread and sends a heartbeat signal to the Rank-0 node. The training monitoring engine of the Rank-0 node, as the root node, starts a diagnostic process to detect and diagnose the training health and fault conditions. This step builds the training monitoring system for the RLHF job, enabling the system to quickly detect and locate faults and reduce downtime due to faults.

[0076] Step 2: The mirror checkpoint of this invention constructs the initial state of the checkpoint in CPU memory. This state is generated based on the selected parallel training strategy and memory optimization scheme, and includes key training information such as initial model weights, optimizer state, and training hyperparameters. Simultaneously, the engine uses the obtained sharding mapping table to achieve fast location and efficient access to parameter shards.

[0077] Step 3: During training, due to the characteristics of RLHF training, the cluster's data stream switches between the inference cluster and the training cluster. The mirror checkpointing engine monitors network traffic and records network idle time between the training and inference clusters. It initiates asynchronous, non-blocking transmission operations through the communication backend to migrate the optimizer state from the training cluster to the inference cluster, achieving mutual backup between the clusters. This process is transparent to the user; the mirror checkpointing engine automatically optimizes the backup process, creates parameter snapshots, and transfers them to the memory of the corresponding backup nodes.

[0078] Step 4: In the Rollout phase, the Output checkpoint engine divides the inference task into Stream-Batch subtasks. Upon completion of each subtask, it saves the corresponding Rollout samples and pushes the output to the training cluster. This step effectively reduces the overhead of re-inference after fault recovery during the training phase, reduces recomputation time, and ensures that training quickly returns to a steady state.

[0079] Step 5: In the Rollout phase, the KV-cache checkpoint engine constructs KV-cache checkpoints based on the storage limit set by the user's hyperparameters, the inference input batch size, and a reasonable save time point. This step effectively accelerates the recovery efficiency after an inference phase failure and reduces recalculation time.

[0080] The specific steps for restoring the training node are as follows:

[0081] Step 1: When a training node fails, the training master detects the failed training node through a survival probe and issues a command to pause training to all nodes participating in the training task (including inference nodes and training nodes), requiring all surviving nodes to transition to a fault recovery state. After the training master stops training, it activates the decoupled recovery engine for fault recovery. Based on the failed node, the master submits the missing model fragments to the recovery engine and requests alternative computing resources from the node agent. After obtaining the alternative resource handles, the fault recovery process begins.

[0082] Step 2: The decoupled fault recovery engine first initializes the initial state of the training engine of the replacement node and cleans up the communication backend context and related resources of the surviving nodes. After the replacement node's engine initialization is complete, the node agent assigns the node's Rank and WorldSize to ensure that the nodes maintain the pre-fault state and reinitializes the communication group to restore the communication backend. After the communication group resumes operation, based on the weight sharding of the replacement node, it requests the required shard metadata information from the mapping manager to pull the corresponding Tensor shards from the CPU memory addresses of the surviving nodes, and performs splicing and splitting according to the parallelism design to complete the restoration of the GPU training state.

[0083] Step 3: After communication and GPU training status are restored, the master controller requires all training nodes to execute the AllGather communication primitive to perform a communication self-check. If the self-check passes, the fault recovery is considered complete. Next, the master controller adjusts the state of all nodes to rollback state and sends rollback commands to the nodes. In rollback state, the inference framework first pulls incomplete samples and loads saved KV-cache checkpoints to accelerate subsequent inference processes. The training framework pulls completed output checkpoints through the checkpoint generation engine, performs recalculation during the training phase, and finally completes the recalculation process of the fault step.

[0084] Step 4: Once the recalculation phase of training is complete, the training engine and inference engine synchronize their parameters. After this phase, the master controller determines the recalculation phase is over, switches all computing nodes to normal training, and resumes the training task, thus restoring training.

[0085] The specific steps for restoring the inference node are as follows:

[0086] Step 1: When an inference node fails, the inference master detects the failed inference node through a survival probe and issues a command to pause training to all nodes participating in the training task (including inference and training nodes), requiring all surviving nodes to transition to a fault recovery state. After the training master stops training, it activates the decoupled recovery engine for fault recovery. Based on the failed node, the master submits the missing model fragments to the recovery engine and requests alternative computing resources from the node agent. After obtaining the alternative resource handles, it begins the fault recovery process.

[0087] Step 2: The decoupled fault recovery engine first initializes the initial state of the training engine of the replacement node and cleans up the communication backend context and related resources of the surviving node. After the replacement node's engine initialization is complete, the decoupled fault recovery engine selects a recovery strategy based on the training configuration. If multiple inference engines exist, the fault recovery engine will shard the node's Rank and WorldSize to ensure that the nodes maintain their pre-fault state and reinitialize the communication group to restore the communication backend. After the communication group is restored, the decoupled fault recovery engine chooses different recovery strategies based on the training configuration. If multiple inference engines exist, since the weights of the inference engines are consistent, the fault recovery engine selects inference engines with the same Rank to synchronize weights for recovery. Otherwise, the decoupled fault recovery engine requests the required shard metadata information from the mapping manager to pull the corresponding Tensor shards from the CPU memory addresses of the surviving nodes, and splices and splits them according to the parallelism design to complete the restoration of the GPU training state of the inference node.

[0088] Step 3: After communication and GPU training status are restored, the master controller requires all training nodes to execute the AllGather communication primitive to perform a communication self-check. If the self-check passes, the fault recovery is considered complete. Next, the master controller adjusts the state of all nodes to rollback mode and sends rollback commands to the nodes. In rollback mode, the inference engine first pulls incomplete samples and loads saved KV-cache checkpoints to accelerate subsequent inference processes. The training engine then pulls completed output checkpoints, performs training recalculation, and finally completes the recalculation process of the fault step.

[0089] Step 4: Once the recalculation is complete, the training engine and inference engine synchronize their parameters. After this stage, the master controller determines that the recalculation phase is over, switches all computing nodes to normal training, and resumes the training task, thus restoring training.

[0090] Experimental Verification. To verify the effectiveness of the proposed fault-tolerant training framework for human feedback-oriented reinforcement learning, a small-scale verification experiment was designed and conducted. The experiment was conducted in an environment consisting of 4 servers, each configured with 8 GPUs (a total of 32 NVIDIA H100 GPUs, interconnected via NVLink, and using InfiniBand for cross-machine communication). The supervised learning model used was QWen-3, the RLHF training framework was OpenRLHF, and the vLLM inference framework and DeepSpeed ​​training framework were used. The training task was deployed through the Ray framework, and the NCCL library was used to implement ensemble communication operations.

[0091] (1) Experimental setup:

[0092] Model selection: QWen-3 14B was used as the training model for synchronous and asynchronous RLHF training.

[0093] Parallelism Strategy: A split deployment is used, with 8 GPUs as the inference cluster, deploying 2 inference engines, and using 4 GPUs for tensor parallelism; 24 GPUs are used as the training cluster, with 8 Actors, 2 data parallel groups, and 4 GPUs for tensor parallelism; 8 Critics, 2 data parallel groups, and 4 GPUs for tensor parallelism; and 4 GPUs each for Reference and Reward.

[0094] Baseline scheme: The traditional checkpoint scheme is adopted, and the checkpoint frequency is set to the optimal frequency.

[0095] Optimization solution: Introduce the fault-tolerant training framework proposed in this invention, and use a decoupled recovery engine, mirror checkpoint engine and generation phase checkpoint engine.

[0096] (2) Performance evaluation indicators

[0097] Training recovery time: The total cost of training a model from the point of interruption until training resumes before the failure occurs.

[0098] (3) Experimental results:

[0099] Figure 7 The diagram compares the recovery time of the present invention with that of the baseline system, demonstrating a significant reduction in mean time to recovery (MTBF). For example, in the RLHF of the 14B model, the baseline checkpointing mechanism wastes 38.8 minutes, while the present invention wastes only 11.7 minutes, a reduction of 69.8%. Furthermore, for the asynchronous RLHF shown in the diagram, the present invention can further reduce the waste time of the 14B model RLHF from 32.1 minutes to 8.2 minutes (a reduction of 74.4%). Simultaneously, the performance improvement consistently exceeds 70% across all model scales, indicating that the present invention performs better in large-scale RLHF tasks. This significant performance improvement is attributed to our decoupled real-time mirror recovery and checkpoint generation mechanism, effectively addressing the cost issues of model restart and recalculation.

[0100] References:

[0101] [1] Meta AI Llam Team. 2024. The Llama 3 Herd of Models. Arxiv e-print, arXiv:2407.21783 (July 2024), arXiv:2407.21783 pages.

[0102] [2] Nvidia NCCL, 2024. https: / / github.com / nvidia / nccl.

[0103] [3] Z. Han, A. You, H. Wang, K. Luo, G. Yang, W. Shi, M. Chen, S.Zhang,Z. Lan, C. Deng, H. Ji, W. Liu, Y. Huang, Y. Zhang, C. Pan, J. Wang,X.Huang, C. Li, and J. Wu. Asyncflow: An asynchronous streaming RL frameworkfor efficient LLM post-training. CoRR, abs / 2507.01663,2025.

[0104] [4] W. Lee, J. Lee, J. Seo, and J. Sim. Infinigen: Efficientgenerative infer-ence of large language models with dynamic KV cachemanagement.In 18th USENIX Symposium on Operating Systems Design and Imple-mentation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, pages 155–172.USENIX Association, 2024.

[0105] [5] M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal,and A. C. Courville. Asynchronous RLHF: faster and more efficient off-policyRL for language models. In The Thirteenth International Conference onLearning Representations, ICLR 2025, Singapore, April 24-28, 2025.OpenReview.net, 2025.

[0106] [6] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H.Lin,and C. Wu. Hybridflow: A flexible and efficient RLHF framework.InProceedings of the Twentieth European Conference on Computer Systems, EuroSys2025, Rotterdam, The Netherlands, 30 March 2025 -3 April 2025, pages 1279–1297. ACM, 2025.

[0107] [7] Y. Zhong, Z. Zhang, X. Song, H. Hu, C. Jin, B. Wu, N. Chen, Y.Chen,Y. Zhou, C. Wan, H. Zhou, Y. Jiang, Y. Zhu, and D. Jiang. Streamrl:Scalable,heterogeneous, and elastic RL for llms with disaggregated streamgeneration. CoRR, abs / 2504.15930, 2025.

[0108] [8] Y. Xiao, Z. Zhou, F. Mao, W. Wu, S. Zhao, L. Ju, L. Liang, X.Zhang,and J. Zhou. Flexrlhf: A flexible placement and parallelism frameworkfor efficient RLHF training. In IEEE International Parallel and Dis-tributedProcessing Symposium, IPDPS 2025, Milano, Italy, June 3-7,2025, pages 358–369. IEEE, 2025.

[0109] [9] J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y.Yang. Safe RLHF: safe reinforcement learning from human feedback. In TheTwelfth International Conference on Learning Representations, ICLR 2024,Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.

[0110]

[10] J. Dong, K. Qian, P. Zhang, Z. Zheng, L. Chen, F. Feng, Y. Xu,Y. Zhu,G. Lu, X. Li, Z. Ren, Z. Wang, B. Luo, P. Zhang, Y. Liu, Y. Chen, Y. Guan, W. Wang, C. Yang, Y. Zhang, M. Yuan, H. Zhao, Y. Li, Z. Zhao, S. Li, X. Zeng, Z. Yao, B. Fu, E. Zha, C. Lin, W. W. Ca, and D. Ca. Evolution ofaegis: Fault diagnosis for AI model training service in production. In 22ndUSENIX Symposium on Networked Systems Design and Implementation, NSDI 2025,Philadelphia, PA, USA, April 28-30, 2025, pages 865–881. USENIX Association,2025。

Claims

1. A human feedback oriented reinforcement learning fault-tolerant training system, characterized in that, It consists of a decoupled recovery engine, a mirror checkpoint engine, and a generation phase checkpoint engine; it supports component-level hot recovery to shorten restart latency after a failure. Based on the RLHF paradigm, low-overhead mirror checkpointing is achieved by leveraging data redundancy between the training and inference engines; and semantic checkpoints are set during the generation phase to limit recomputation to sub-iterations, thereby reducing recovery overhead and improving resource utilization; specifically: (i) The decoupled recovery engine consists of three types of sub-modules, providing component-level hot recovery for each training component of RLHF, isolating faulty units, and avoiding the overhead of job-level full rescheduling and restart; wherein: Submodule 1 is the mapping manager, responsible for maintaining weight mapping; maintaining a sharding mapping table of parameters / states between the inference cluster and the training cluster; after a failure, according to the mapping guidance, the replacement node pulls parameters and optimizer shards from the corresponding memory / storage address and completes the assembly to rebuild the GPU-side training state; Specifically, the mapping manager is used to perform redundant mapping of the sharding design between the training cluster and the inference cluster. Through sharding mapping, the size of checkpoints is reduced, and high-performance networks can be used to transfer checkpoints during fault recovery to improve recovery efficiency. Submodule 2 is the training monitoring engine, which monitors the training status, collects the training status and pushes it to the master node to avoid the overhead of fault location and diagnosis after a failure. The training monitoring engine reports the node status at regular intervals to improve recovery efficiency. Specifically, it continuously collects the heartbeat, health and error codes of each component. Once a component failure is detected, it immediately triggers local recovery: isolates the faulty process group, freezes the current training step, saves the minimum necessary context, and links the mapping manager and node agent to perform recovery so that the training can continue at the nearest semantic boundary. Submodule 3 is a node agent, responsible for reconstruction after fault recovery, including the reconstruction of communication groups and training frameworks. Through the node agent, elastic strategies are used to reduce communication groups after a fault, and safe decoupling is achieved after draining faulty nodes. Node-level restarts can be performed for training faults. This reduces fault recovery from the job level to the node level, minimizing the spread of faults and reducing restart overhead. Furthermore, it interfaces with the underlying infrastructure, requests alternative nodes and GPU resources, generates deployment specifications based on the current topology, and completes component restarts and communication domain initialization. The three types of sub-modules work together to complete component-level hot recovery without interrupting the global operation, significantly shortening restart latency and improving resource utilization in fault scenarios; (ii) The mirror checkpoint engine is an important way to recover from failures. Specifically, it is responsible for building backups of the training state of the training cluster and the inference cluster. The engine relies on the sharding mapping table of the data redundancy and mapping manager to plan and select backup nodes for storing weight parameters and optimizer state. During training, the mirror checkpoint engine senses the network idle time and writes the checkpoint data into the memory of the corresponding node in byte form, thereby realizing the construction of mirror checkpoints. (iii) The generation phase checkpoint engine consists of two types of sub-engines, which set semantic breakpoints for the "generation-training" pipeline to reduce recalculation after rollback and restore the accelerated state of the inference side. Sub-engine 1 is the Output checkpoint engine. After the inference cluster completes the generation, it persists the Rollout results and their metadata. After the fault is recovered, the generated results are directly sent back to the training engine by sample / batch index, skipping repeated generation and ensuring order and deduplication. It can converge the rollback and recalculation window to the "training side" sub-iteration, significantly eliminating the time overhead of the generation stage. Submodule 2 is a KV-Cache checkpoint engine that performs selective snapshots or fast reconstruction of the inference acceleration key-value cache (KV-cache); it rebuilds or warms up the KV-cache by session / sample key during job recovery, reducing prefill latency and allowing direct entry into the decoding stage or shortening the prefill; it restores the existing accelerated state during the remaining sample inference, improving throughput after recovery and reducing overall latency. The two sub-engines work together to ensure that there is no need to regenerate after a rollback, and that the inference performance quickly returns to a steady state.

2. The human feedback oriented reinforcement learning fault-tolerant training system of claim 1, wherein, In the KV-Cache checkpointing engine of the generation phase checkpointing engine, a selective key-value cache checkpointing strategy is adopted, which proposes a "cache utilization" metric: the proportion of key-value cache entries hit in subsequent batches, and sets a threshold to balance time efficiency and memory overhead; if the cache utilization exceeds the threshold, it indicates that more batches can be stored to accelerate the calculation; otherwise, storing more batches will increase memory overhead, while the improvement in time efficiency is not significant. Here, entries already stored in the key-value cache are not transmitted to the training phase; furthermore, since all input batches can be preprocessed offline at the start of each iteration, the system can identify frequently used key-value entries; therefore, the system will only transmit key-value entries that will be hit in subsequent batches to the training phase, thereby greatly reducing bandwidth and memory overhead.

3. The human feedback oriented reinforcement learning fault-tolerant training system of claim 1, wherein, In the decoupled recovery engine, given the size of the training cluster and inference set and the parallel strategy, the mapping manager guides the recovery nodes to pull checkpoint shards after a failure by maintaining weight mappings. This module determines: (1) the memory mapping of training GPU weight parameters in the inference GPU; (2) the memory mapping of inference GPU weights in the inference GPU under the same stage; the mapping manager uses the following algorithm to determine the available shard partitions: According to the recovery model, for each partition Given the availability mask as follows: ； The availability for the training side is as follows: ； The set of "task pairs" that need to be assigned values ​​is defined as follows: ； Here To enumerate the sequence Where ; The algorithm for shard acquisition is implemented, and the nearest available inference-side index operator is defined as follows: ； Using a sequence of pointers Indicates the current starting selection position and initializes ;; For each step in the sequence , the update algorithm for the sliced inference side index and the pointer is defined recursively as follows: ； ； From the above, we can obtain the closed form of the index function: ； The training monitoring engine monitors the heartbeat signals of the components and inserts monitoring probes in front of the cluster. When a fault is detected, it isolates and freezes the training in a timely manner to avoid job-level failures caused by incorrect communication or computation. After a fault is detected, the training monitoring engine will work with the checkpoint engine and node agents to rebuild the training parameters, achieve hot recovery, and restart the training process after recovery. The node agent manages nodes to release communication resources and rebuild the communication group. After a failure occurs, the node agent reduces the communication group to remove the faulty node, then safely releases the communication group resources, applies for alternative resources from the infrastructure and deploys components, and then builds a communication network according to the current topology and parallel strategy to complete component restart and communication domain initialization.

4. The human feedback oriented reinforcement learning fault-tolerant training system of claim 1, wherein, The mirror checkpointing engine utilizes data redundancy between the RLHF training and inference clusters in a separate architecture, introducing a mirror recovery mechanism. The training and inference models act as backups for each other, ensuring the latest model state is loaded before a failure occurs and effectively reducing bandwidth pressure on the storage server. Based on training execution, the mirror checkpointing engine detects idle bandwidth moments in the cluster network and, according to the parallel strategy and sharding mapping table, pipelines the optimizer state of the training cluster to the corresponding node memory. The constructed sharding transfer model is as follows: Define the timestamp of network idle time ,according to The model calculates the total amount of checkpoint fragment data that can be transmitted. ,Right now: ； ； This design offers two main benefits: first, it significantly reduces recalculation overhead by recovering from the latest state rather than recalculating after rolling back multiple iterations; second, because it holds the latest state of all models, it only needs to recover the model that failed, thus providing new possibilities for model decoupling in fault recovery.

5. The reinforcement learning fault-tolerant training system for human feedback as described in claim 1, characterized in that, The generated checkpoint engine, based on the inference engine, includes an Output checkpoint engine and a KV-Cache checkpoint engine to reduce the overhead of inference recalculation after fault recovery. First, during the inference process, the Rollout results of the request queue are structured and persisted, stored in the host memory, and streamed back to the training cluster in batches. When a fault occurs, the training engine directly retrieves and replays the checkpoint, skips repeated generation, and ensures order and idempotency. Second, the KV-Cache checkpoint engine selectively saves the KV-Cache in a semantically safe window, and warms up / rebuilds the inference-side cache after recovery to accelerate the inference of incomplete samples.

Images