An iteration level point-to-point elastic recovery system for distributed training of a large model

Abstract

The application provides an iteration level point-to-point elastic recovery system for large model distributed training, and belongs to the field of large model distributed training.The system comprises a coordination service, a manager and a copy; the coordination service converges local states distributed in processes into a global consistent decision; the manager is a local agent of each process and is located between a training process and the coordination service; the copy is a minimum training unit, each copy has an independent process group and independent communication storage; when training is started, each copy is independently started and registered with the coordination service; in a normal stage, the Megatron original synchronization path is used to complete intra-copy reduction, and cross-copy reduction is completed under the coordination of the manager; after a fault occurs, the coordination service recalculates arbitration according to a heartbeat timeout, and healthy copies continue to promote training; after a fault copy is recovered, the fault copy is recovered through state direct transmission to catch up with parameters and rejoin. The application significantly reduces recovery delay and training progress loss.

Description

Technical Field

[0001] This invention belongs to the field of large-scale distributed training, specifically relating to an iterative point-to-point elastic recovery system for large-scale distributed training. Background Technology

[0002] As the parameter scale of large language models continues to grow, distributed training has become a necessity rather than an optimization. Models such as GPT-4, LLaMA, and Qwen typically require thousands to tens of thousands of GPUs for collaborative training, with training cycles often spanning weeks or even months. At this scale, faults are no longer low-probability anomalies, but rather a normal perturbation in the training process.

[0003] From a system metrics perspective, fault recovery directly determines ETTR (Effective Training Time Ratio). A slow recovery path will increase MTTR, while an excessive rollback range will amplify the effective training loss corresponding to RPO (Recovery Point Objective). In high-failure scenarios, even if the cost of a single failure is low, repeated switching between run-interrupt-recovery will continuously suppress long-term throughput. Therefore, the goal of system recovery should not be limited to simply being able to recover, but should also meet the requirements of fast recovery, minimal rollback, and continuous online operation.

[0004] Traditional checkpoint recovery mechanisms are currently the most common fault-tolerant baselines, but they reveal significant shortcomings in large-scale training. The core problem lies not only in slow recovery, but also in the fact that recovery capability is built upon heavy I / O and long window rollbacks. As model size increases, checkpoint volume grows accordingly. In the event of a failure, the entire state needs to be read from persistent storage, and the recovery process typically takes minutes; during this time, the training process and GPU are essentially in a waiting state. Furthermore, checkpoints are saved at fixed step intervals, meaning that after a failure, one can only roll back to the most recent saved point, requiring recalculation of intermediate training steps. The larger the interval, the higher the effective training loss from a single failure.

[0005] This mechanism also presents a typical dilemma: more frequent saving can reduce rollback losses, but it increases I / O interference during training; sparser saving can reduce daily disturbances, but it amplifies the cost of failures. As the model state continues to grow, the operational costs of storage capacity, write bandwidth, version management, and cleanup strategies will also increase accordingly. More importantly, checkpoint recovery is essentially still a stop-rollback-restart path, which is not friendly to dynamic node online / offline states and is difficult to smoothly take over in scenarios such as preemption, migration, and fault replacement. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by providing an iterative point-to-point elastic recovery system for distributed training of large models. The goal is to achieve iterative elastic recovery without altering the main training logic, while ensuring correct state takeover in hybrid parallel scenarios.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A peer-to-peer elastic recovery system for distributed training of large models includes: a coordination service, a manager, and replicas. The coordination service is a globally unique coordination node that converges the local states scattered across processes into globally consistent decisions. The manager is a local agent for each process, located between the training process and the coordination service. Each replica is the smallest training unit, with an independent process group and independent communication storage. The replica runs the native Megatron training process and maintains a complete TP / PP / EP topology. At training startup, each replica starts independently and registers with the coordination service. During normal operation, replicas use the native Megatron synchronization path to complete intra-replica reductions, while replicas complete cross-replica reductions under the coordination of the manager. After a failure, the coordination service recalculates arbitration based on heartbeat timeouts, and healthy replicas continue training. After the failed replica recovers, it catches up on parameters through direct state transmission and rejoins the system.

[0008] Optionally, an in-replica reduction is performed within the replica to obtain the replica-level average gradient:

[0009] in, Indicates a copy The average gradient, globally, has Each GPU is divided into Each copy contains [number] copies. = / One GPU; record copy Medium GPU j The resulting gradient is .

[0010] Optionally, cross-replica specifications are executed between replicas, with the following result:

[0011] in, This indicates the result of the global specification, which includes two phases: intra-replica specification and cross-replica specification.

[0012] Optionally, the coordination service recalculates arbitration based on heartbeat timeout as follows: the coordination service continuously maintains the heartbeat timestamps of registered replicas and first filters healthy replicas within the timeout window during each round of decision-making; if the number of healthy replicas is lower than the minimum number of replicas, the arbitration in this round fails directly and waits for the member status to recover; the number of healthy replicas participating in the arbitration in this round meets the majority condition.

[0013] Optionally, for newly joined replicas that have not yet stably participated, the coordination service delays arbitration within the join_timeout window; if the replicas that participated in the previous round are still healthy in this round, the results of the previous round are reused; the arbitration results are embedded into the training loop in a specific time sequence.

[0014] Optionally, for events occurring in the first... If the failure occurs before arbitration, the arbitration will directly exclude the faulty copy. If the commit is successful, the loss is 0; if the failure occurs after arbitration but before gradient synchronization, and the cross-replica reduction fails, return False. If the first step is discarded, the loss is 1; if the failure occurs after gradient synchronization but before parameter update, the submission fails. The step is discarded, resulting in a loss of 1.

[0015] Optionally, the coordination service achieves state recovery through point-to-point direct memory transfer between replicas, including state catching up after a faulty replica restarts, cold start synchronization when a new replica is added during expansion, and catching up with lagging replicas caused by network jitter; the coordination service automatically identifies replicas that need to be restored through an arbitration algorithm and carries a restoration instruction in the result.

[0016] Optionally, during the state recovery process, the coordination service identifies the lagging replica and issues a recovery target. The lagging replica triggers a state transfer from the normal replica to the lagging replica through a weight request during the training loop, so that the two replicas are re-aligned in terms of steps and enter the next round of arbitration. Meanwhile, the normal replica continues to advance training without global shutdown or waiting.

[0017] Optionally, it also includes a distributed-trained MoE model, which comprises dense layers and sparse expert layers, with the following parallel dimensions: For dense layers, the data parallelism dimension is defined as Attention_DP:

[0018] in, TP is the vocabulary size, PP is the tensor parallelism, CP is the pipeline parallelism, and CP is the context parallelism. The data parallelism dimension for the expert layer is defined as Expert_DP:

[0019] Where expert_TP is the parallelism of the expert tensor, and EP is the parallelism of the expert.

[0020] Optionally, the intersection of the attention data parallelism and expert data parallelism corresponds to a set of GPUs that simultaneously hold the same dense parameters and the same expert parameters. During fault recovery, the members of the intersection are direct recovery sources for each other, and the ranks within the same intersection are distributed to different replicas.

[0021] The beneficial effects of this invention are: (1) Iterative Arbitration Protocol: Through a per-step should_commit voting mechanism, elastic recovery of single-iteration progress loss is achieved. In case of failure, at most one iteration is lost, which is significantly reduced compared to the hundreds of steps of rollback in traditional checkpoints. The heartbeat detection and arbitration algorithm of the coordination service provide support for the timeliness of fault detection and the consistency of decision-making.

[0022] (2) Point-to-point direct memory transfer: Eliminates storage I / O bottlenecks through rank-to-rank state transfer. The state registration mechanism supports full recovery of the model, optimizer, and learning rate scheduler. Experiments show that direct state transfer reduces recovery latency by about 90% compared to checkpoint loading.

[0023] (3) Two-phase reduction: Cross-replica gradient synchronization is injected through Monkey Patch to achieve non-intrusive integration of the Megatron complex parallel framework. Mathematical proof shows the equivalence between two-phase reduction and global reduction, and experimental results show that the training convergence trend is consistent with the fault-intolerant baseline.

[0024] (4) MoE Dual-Dimensional Perception: Based on intersection analysis, replica partitioning is derived. Mathematical analysis shows that a usable recovery source can be constructed when the power-of-2 GPU and the parallelism condition is met. A dynamic communication group can simultaneously handle gradient synchronization of both dense and expert parameters. Attached Figure Description

[0025] Figure 1 It is the overall architecture of an iterative point-to-point elastic recovery system for distributed training of large models.

[0026] Figure 2 This is a schematic diagram of hierarchical gradient reduction.

[0027] Figure 3 This is a sequence diagram for replica state synchronization.

[0028] Figure 4 It is a parallel dimensional relationship between two types of data.

[0029] Figure 5 This is a topology diagram of an iterative point-to-point elastic recovery system for distributed training of large models. Detailed Implementation

[0030] The invention will now be described in further detail with reference to the accompanying drawings.

[0031] SHIFT, an iterative peer-to-peer elastic recovery system for distributed training of large models, consists of three core components: a coordination service (Lighthouse), a manager, and replicas. Their collaborative relationships are as follows: Figure 1 As shown, the topology is as follows Figure 5 As shown.

[0032] The coordination service is the sole global coordination node, undertaking four types of responsibilities: replica registration and qualification management, heartbeat detection and health view maintenance, step-level arbitration calculation, and recovery source and target pairing scheduling. Its core function is to converge the local states scattered across various processes into globally consistent decisions, preventing split views from arising among replicas in the event of a failure.

[0033] The manager acts as a local agent for each worker process, situated between the training process and the coordination service. It is responsible for initiating and waiting for arbitration, maintaining cross-replica communication groups, managing state serialization / deserialization callbacks, and capturing and reporting runtime errors. Through this encapsulation, the training loop only needs to call a single interface to complete commit decisions, cross-replica synchronization, and state takeover, reducing the complexity of upper-layer code.

[0034] The replica is the smallest training unit. Each replica has an independent process group and independent communication storage, internally running the native Megatron training process and maintaining the complete TP / PP / EP topology. This design first isolates the impact of faults within the replica boundary, and then arbitration determines whether to trigger cross-replica collaboration.

[0035] At the start of training, each replica starts independently and registers with the coordination service. During the normal phase, the native Megatron synchronization path is used within each replica, and cross-replica gradient reduction is completed under the coordination of the manager. After a failure occurs, the coordination service recalculates the arbitration based on the heartbeat timeout, and healthy replicas continue training; after the failed replica recovers, it catches up with the parameters through direct status transmission and rejoins.

[0036] This architecture explicitly separates the control plane and the data plane: the control plane is responsible for member management, arbitration, and recovery scheduling, while the data plane is responsible for gradient reduction and state transmission. The direct benefit of this separation is clearer fault boundaries: control plane jitter will not directly pollute the training path, and instantaneous fluctuations in the data plane will not disrupt the arbitration state machine. For persistent fault scenarios, this layering makes it easier to pinpoint the source of the problem: whether it's inconsistent member views, abnormal communication group construction, or degraded state transmission links, thereby improving observability and maintainability.

[0037] (1) Two-phase reduction and gradient synchronization mechanism The core design of SHIFT is to split global reduction into two stages: intra-replica reduction and cross-replica reduction, which are then injected into the Megatron training process in a non-intrusive manner. This design references the two-stage reduction idea of ​​HSDP (Hybrid Sharded Data Parallel) but is adapted for the complex parallel topology of Megatron. The principle of hierarchical gradient reduction is as follows: Figure 2 As shown.

[0038] Figure 2 This paper demonstrates the gradient reduction process when 16 GPUs are divided into two replicas. The first level of reduction occurs within each replica: the eight GPUs in each replica first perform local reduction to obtain the replica-level average gradient. The second level of reduction occurs between replicas: the average gradients of the two replicas are then reduced across replicas to obtain the global average gradient. This design has two direct benefits: intra-replica reduction reuses the native Megatron implementation, requiring no additional development; cross-replica reduction is implemented through dynamic communication groups, supporting dynamic changes in members. The equivalence between two-phase reduction and global reduction can be formally verified. Let the global gradient be... N Each GPU is divided into R Each copy contains [number] copies. n=N / R One GPU. (Record a copy) i The gradients generated by each GPU are .

[0039] Phase 1 (Intra-replica reduction): The reduction is performed within each replica, resulting in the replica-level average gradient.

[0040] This stage directly reuses Megatron's native distributed data parallel implementation, without the need to add new communication logic within replicas.

[0041] Phase Two (Cross-Replica Reduction): Representative gradients from each replica execute the reduction once more.

[0042] After substituting, we get:

[0043] That is, the result of global reduction. Therefore, two-phase reduction is mathematically equivalent to global reduction, and correctness can be guaranteed using only AVG operations.

[0044] From an engineering perspective, the two-phase design also improves communication organization: Phase 1 prioritizes the use of high-bandwidth interconnects within replicas (such as NVLink) for local aggregation, while Phase 2 only performs cross-replica synchronization at the replica granularity, thereby reducing traffic peaks across node links and generally being more conducive to controlling tail latency. In fault recovery scenarios, this layering also makes it easier to pinpoint the problem to a specific level: whether it is an intra-replica reduction anomaly or a cross-replica communication anomaly.

[0045] On the engineering side, SHIFT needs to find an injection point compatible with Megatron's asynchronous reduction path. Megatron's DistributedDataParallel uses bucket-level asynchronous reduction to overlap communication and computation. Its critical path can be summarized as follows: start_grad_sync is responsible for initiating the asynchronous reduction, and finish_grad_sync is responsible for waiting for the handle to complete and ensuring that the gradient is available.

[0046] SHIFT injects cross-replica reduction after `finish_grad_sync`, not at `start_grad_sync`. This is because when `overlap_grad_reduce=True` (enabled by default), `start_grad_sync` only initiates the operation; the gradient reduction is not yet complete. If cross-replica synchronization is performed by directly reading the gradient at this point, it's easy to read intermediate states and cause data races. Conversely, `finish_grad_sync` blocks and waits for all asynchronous handles to return, ensuring that the reduction within the replica is complete. Only then does cross-replica synchronization satisfy the consistency requirement. In practice, SHIFT modifies the `_ParamAndGradBucketGroup.finish_grad_sync` method using Monkey Patch to achieve non-intrusive integration. Table 1 provides the pseudocode.

[0047] Table 1 Monkey Patch for Hierarchical Gradient Reduction

[0048] This implementation has three engineering features: First, cross-replica reduction is started non-blockingly and waits centrally, preserving inter-bucket concurrency; second, cross-replica reduction is only executed when both the manager and dynamic communication group are ready, facilitating canary launches and problem localization; third, the injection process does not modify the Megatron-LM source code, but rather takes over the target method through runtime patching. In practice, these three points correspond to the three goals of performance, operability, and maintainability, respectively. The line numbers corresponding to Algorithm 1 can be seen as follows: Line 2 calls the original finish_grad_sync to complete the intra-replica reduction; lines 3-10 execute the cross-replica reduction under the condition that the communication group is ready; among them, lines 4-7 initiate non-blocking allreduce bucket by bucket, and lines 8-9 uniformly wait for the handle to complete to ensure that the gradient of this round enters the committable state.

[0049] The communication group upon which the cross-replica protocol relies is not statically allocated, but dynamically updated based on the arbitration result. Cross-replica communication uses a dynamic communication group, the number of which changes dynamically with the arbitration result. When a replica fails, the communication group automatically removes the failed member; when the replica recovers, the communication group automatically adds new members.

[0050] The design of the dynamic communication group follows three principles: First, communication actions rely on arbitration results to ensure that the participant set has converged; second, non-participating members do not enter the current round of the communication group to avoid slowing down healthy members; third, single-member communication anomalies are locally isolated and reported to prevent them from spreading to the entire process. The first two principles guarantee correctness and efficiency, while the third principle ensures that faults will not amplify along the communication group.

[0051] Furthermore, SHIFT employs a deterministic member ordering and round-binding strategy when rebuilding groups: within the same arbitration round, all participants build communication groups in a consistent order. This mechanism reduces the risk of race conditions caused by inconsistent member views and avoids communication anomalies caused by some processes switching to new groups while others are still using the old groups.

[0052] (2) Arbitration Agreement and Iterative Recovery The arbitration protocol is the core mechanism for SHIFT's iterative elastic recovery, ensuring that all healthy replicas reach consensus before each training step begins. The coordination service continuously maintains the heartbeat timestamps of registered replicas and filters healthy replicas within the timeout window during each round of decision-making. If the number of healthy replicas is lower than the minimum number of replicas (min_replicas), the arbitration round fails and waits for member recovery. To avoid dual-master scenarios in network partitions, the system requires that the number of healthy replicas participating in the arbitration round meets a majority condition: |participants|>|registered| / 2. For newly joined but not yet stably participating replicas, the system delays arbitration within the join_timeout window (default 30 seconds) to give them a chance to catch up; if a participant from the previous round remains healthy in this round, the result from the previous round is reused to reduce steady-state jitter. The arbitration result needs to be embedded into the training loop in a specific time sequence to ensure semantic correctness of the commit. The coupling between arbitration and the training loop is as follows: In the `zero_grad` phase, `start_quorum` is initiated asynchronously, allowing arbitration to proceed in parallel with the forward / backward processes; subsequently, `finish_grad_sync` completes two-stage gradient synchronization and awaits the results; before parameter updates, `should_commit` is called to obtain the global commit decision. Each process first completes a local health check, then aggregates the votes to the manager service. Only when all votes pass is `optimizer.step` executed; otherwise, the gradient of the current step is discarded, and the next iteration begins. This allows the commit decision to be made at a unified time, avoiding commit forks between different replicas at the same step. This mechanism directly determines the upper bound of the system's progress loss. In traditional checkpointing schemes, after a failure, a rollback to the nearest checkpoint is required, and the lost training steps are directly related to the checkpoint interval. Assuming a checkpoint interval of K steps, and assuming a uniform distribution of failure times, the expected progress loss is K / 2 steps.

[0053] SHIFT's arbitration mechanism guarantees that if a copy fails in the current iteration, the other copies will remove the faulty member through arbitration at the end of the iteration, and training will continue in the next iteration. Even if arbitration fails in the current iteration due to a fault (such as insufficient participants), the loss is limited to the current iteration.

[0054] Suppose a failure occurs at some point in step t. There are three possible scenarios: If the failure occurs before arbitration, arbitration will directly exclude the faulty replica, and step t can be committed normally with a loss of 0; if the failure occurs after arbitration but before gradient synchronization, cross-replica reduction may fail, should_commit returns False, step t is discarded, and a loss of 1 is incurred; if the failure occurs after gradient synchronization but before parameter update, a step will also be lost due to the failed commit.

[0055] Therefore, the upper bound of the schedule loss for SHIFT is 1 step, which is significantly reduced compared to the K / 2 steps of the traditional scheme (K is usually hundreds to thousands).

[0056] It should be noted that this upper bound applies to a single failure event. If multiple failures occur consecutively, the system may incur multiple single-step losses, but the loss in each round is still limited to a single iteration and will not degenerate into a long-window rollback. In other words, the failure cost mainly increases approximately linearly with the number of failures, and is not further amplified by the checkpoint interval.

[0057] After constraining progress loss, another core issue is how to quickly synchronize the state after a failure. SHIFT achieves state recovery through point-to-point direct memory transfer between replicas, bypassing disk I / O bottlenecks. Direct state transfer covers three typical scenarios: state catching up after a failed replica restarts, cold start synchronization when a new replica is added during expansion, and catching up with lagging replicas caused by network jitter. These three are essentially the same problem: the target replica's progress lags behind the global maximum progress.

[0058] The arbitration algorithm automatically identifies copies that need to be restored (members whose training steps are less than the maximum number of steps) and includes a restoration instruction in the result. This restoration decision does not rely on additional external control logic and can be directly integrated with the training loop. To ensure that any training state, including the model and optimizer, can be restored via a unified path, SHIFT provides a state registration interface, allowing users to register serialization and deserialization callbacks for any state.

[0059] After registration, state transfer is automatically scheduled by SHIFT: the source calls `state_dict` to export the state, and the target calls `load_state_dict` to load it. With a unified interface layer, the model, optimizer, and scheduler states can all be processed along the same recovery path. The recovery process is triggered by the arbitration result, and the core steps are as follows: The system first compares the training steps of each replica, marking members lagging behind the global maximum as recovery targets, and returns three key fields in the arbitration result: `heal`, `recover_src_replica_rank`, and `recover_dst_replica_ranks`. Then, the source calls `state_dict` to collect the complete state and performs chunked transfer, with the path being GPU → Network → GPU, using HTTP or a process group channel based on NCCL / Gloo. The target then receives the state and calls `load_state_dict` sequentially to recover the model, optimizer, and learning rate scheduler states.

[0060] To ensure consistency during recovery, the target system implements a recovery barrier after state loading: it only re-participates in the submission vote if the model parameters, optimizer state, scheduler state, and training steps all pass the consistency check. This mechanism prevents mixed states from entering the training path, i.e., some states have been updated while others remain in the old version.

[0061] The timing diagram of the state synchronization process is as follows: Figure 3 As shown.

[0062] Figure 3 This demonstrates the complete process of replica 1 joining the training as a lagging replica. Replica 0 has completed multiple training steps and remains healthy, while replica 1 has just completed initialization and is lagging behind in steps. The process can be divided into three stages: the first stage forms an arbitration, where the coordination service identifies the lagging member and issues a recovery target; the second stage performs state synchronization, where replica 1 performs operations such as forward propagation, requesting weights, backpropagation, clearing gradients, gradient reduction, decision-making and commit, and optimizer stepping in the training loop, where weight requests trigger state transfer from replica 0 to replica 1; the third stage completes synchronization, the two replicas realign their steps and enter the next round of arbitration. Throughout the process, replica 0 can continuously advance training without requiring a global shutdown. The correctness of the above process is based on the premise that all replicas maintain a symmetrical parallel configuration. SHIFT requires all replicas to share the same parallel configuration, therefore, the same rank number has isomorphic semantics in each replica: the same TP rank corresponds to the same dense parameter shard, the same PP rank corresponds to the same pipeline stage, and the same EP rank corresponds to the same expert shard. Based on this premise, rank-to-rank transfer is structurally feasible: source rank N The status can be directly loaded into the target rank. N No additional mapping or rearrangement is required. As shown in Table 2, point-to-point direct transmission outperforms traditional checkpoint solutions in several key dimensions.

[0063] Table 2 Comparison of Status Transmission Methods

[0064] (3) Dual-dimensional perception and fault domain isolation The MoE model introduces a sparse expert layer in addition to the traditional dense layer, with these two types of layers corresponding to different data parallel partitioning logics. This difference directly affects the correctness of replica partitioning and recovery paths: if recovery is designed based on only a single data parallel view, it is easy to encounter problems where the state source is correct but the reduction range is inconsistent. This embodiment presents a topology analysis and replica partitioning method in the MoE scenario. The MoE model contains both a dense layer and an expert layer. The former mainly follows the conventional TP / PP partitioning, while the latter activates multiple experts through gating and introduces the EP dimension; therefore, the recovery logic must be compatible with both sets of data parallel views simultaneously.

[0065] The parallel dimensions of the two types of layers are defined as follows: Attention Data Parallelism (Attention_DP): This refers to the data parallelism dimension for dense layers, defined as follows:

[0066] in, TP represents the vocabulary size, PP represents the tensor parallelism, CP represents the pipeline parallelism, and CP represents the context parallelism.

[0067] Expert Data Parallelism (Expert_DP): This refers to the data parallelism dimension at the expert level, defined as follows:

[0068] Where expert_TP is the parallelism of the expert tensor, and EP is the parallelism of the expert.

[0069] The two dimensions use different parallel group partitioning logic. In Megatron-LM, dense parameters and expert parameters are managed by independent RankGenerators. The relationship between the two types of parallel dimensions is as follows: Figure 4 As shown.

[0070] Figure 4 Using 16 GPUs as an example, the relationship between attentional data parallelism and expert data parallelism is illustrated. In the attentional data parallelism dimension, the 16 GPUs are divided into 4 groups, each holding the same dense parameters. In the expert data parallelism dimension, they are divided into 8 groups, each holding the same expert parameters. The intersection of these two dimensions corresponds to the set of GPUs simultaneously holding the same dense parameters and the same expert parameters, thus they can serve as complete recovery sources for each other. Taking GPU 2 as an example, its attentional data parallelism group is {0, 2, 4, 6}, and its expert data parallelism group is {2, 6}. The intersection is {2, 6}, meaning GPU 6 can directly take over from GPU 2.

[0071] Table 3 shows four typical parallel configurations that can be constructed in a 16-GPU scenario, and provides the corresponding data parallelism dimension and number of replicas.

[0072] Table 3 Data parallelism dimension and number of replicas in typical parallel configurations

[0073] In Table 3, the number of replicas is taken as the smaller value of Attention_DP and Expert_DP, i.e., the intersection size. This allows for four configurations with 2 / 2 / 4 / 8 replicas on the same 16-GPU cluster, providing a unified and comparable set of configurations for subsequent throughput overhead experiments.

[0074] The core connection between the two parallel dimensions lies in their intersection—the set of ranks that simultaneously hold both types of parameters. The intersection is defined as the set of ranks that simultaneously hold the same dense parameters and the same expert parameters:

[0075] Ranks within the intersection belong to both the same Attention_DP group and the same Expert_DP group, thus sharing both dense parameter slices and expert parameter slices, and consequently, the complete model state. During fault recovery, members of the intersection can serve as direct recovery sources for each other. Based on this property, replica partitioning requires distributing ranks within the same intersection across different replicas to ensure that any rank always has a usable recovery source in other replicas. For example, consider a configuration of 16 GPUs, TP=2, PP=2, EP=4, and expert_TP=1. ,

[0076] Taking rank 2 as an example, its Attention_DP group is {0, 2, 4, 6}, its Expert_DP group is {2, 6}, and their intersection is {2, 6}. Therefore, rank 6 is isomorphic to rank 2 in terms of both dense parameters and expert parameters, and can be directly used as a recovery source.

[0077] Similarly, a pairwise intersection can be obtained, such as {0, 4}, {1, 5}, {2, 6}, {3, 7}, {8, 12}, etc. Based on the principle of assigning members of the same intersection to different replicas, the following partition can be formed: Replica 0 is {0, 1, 2, 3, 8, 9, 10, 11}, and Replica 1 is {4, 5, 6, 7, 12, 13, 14, 15}. This partition satisfies both executability and recoverability constraints. The key question is: can a sufficiently large intersection always be obtained to make the replica partition feasible?

[0078] Theorem: When the number of GPUs is a power of 2, and Attention_DP>1 and Expert_DP>1, the intersection size is ≥2.

[0079] This theorem states that as long as the parallel dimensions of two data points are both greater than 1, each rank will necessarily have a recovery source in other replicas, and replica partitioning is always feasible under this condition.

[0080] If a data parallelism dimension equals 1, it means that there is no data parallelism partitioning for that dimension, and all GPUs hold the same parameter shards. In this case, it is impossible to build multiple independent copies, and SHIFT degenerates into the traditional checkpointing method. However, in large-scale training of large models, data parallelism has become an inevitable choice—when the model size exceeds the memory capacity of a single GPU, tensor parallelism or pipeline parallelism is needed to partition the model; when the dataset size exceeds the processing power of a single machine, data parallelism is needed to accelerate sample consumption. Therefore, in real-world production environments, it is common and reasonable to have two data parallelism dimensions greater than 1. Based on intersection analysis, the copy partitioning algorithm for SHIFT is shown in Table 4.

[0081] The goal of the verification phase is to ensure that non-data parallel groups are not split, meaning that the same TP group, the same PP phase, and the same EP group all fall completely within a single replica. If any constraint is not met, the current partition will be rejected and a recalculation will be triggered. SHIFT uses the same dynamic communication group to handle gradient synchronization of dense parameters and expert parameters simultaneously, and its correctness is guaranteed by the mathematical equivalence of the two-phase reduction. The corresponding line numbers for Algorithm 2 can be mapped as follows: lines 2-6 calculate the intersection group for each rank; lines 7-14 assign ranks to different replicas according to the intersection; lines 15-19 perform replica integrity verification; line 20 returns the replica partition result.

[0082] Table 4. Replica partitioning algorithm for MoE-aware systems

[0083] For dense parameters, the global data parallelism is Attention_DP:

[0084] For the expert parameter, the global data parallelism is Expert_DP:

[0085] The two-phase AVG operation is equivalent to a global AVG for both types of parameters, thus eliminating the need to maintain two separate cross-replica communication channels. The key reason for this is that the number of replicas takes the size of the intersection, and this intersection allows the two-phase reduction to cover the complete data parallel dimension corresponding to the two types of parameters.

[0086] Besides replica partitioning, the replication startup strategy also directly affects the scope of fault propagation. In the traditional joint startup method, all nodes are started via torchrun-nnodes=N and share the same TCP-Store. A failure of the master node (the node with rank 0) will cause the TCPStore to fail, which in turn will cause all worker nodes to fail, which is a typical cascading failure path.

[0087] SHIFT employs an independent replica startup approach, isolating fault domains at the architectural level. The core of this design is decoupling intra-replica execution consistency from inter-replica fault propagation paths: the former is guaranteed by the process group within the replica, while the latter is established on demand by a coordination service and dynamic communication group. The difference between the two startup methods can be intuitively compared through command formats: Command format for joint startup: # All nodes use the same command, distinguished by NODE_RANK torchrun --nnodes=2 --node_rank=$NODE_RANK \ --master_addr=$MASTER_ADDR \ --nproc_per_node=8 train.py The main risk of this model is that master_addr points to a single master node. If the master node fails, TCPStore becomes invalid, and all nodes will be unable to continue to complete distributed coordination and communication.

[0088] Command format for standalone startup: # Replica 0 (Node 0): Start independently REPLICA_GROUP_ID=0 torchrun --nnodes=1 --nproc_per_node=8 train.py # Replica 1 (Node 1): Start independently REPLICA_GROUP_ID=1 torchrun --nnodes=1 --nproc_per_node=8 train.py Each replica is an independent torchrun process group with its own TCPStore. When replica 0 fails, replica 1's TCPStore continues to function normally, allowing healthy replicas to continue training without waiting for a global restart. After independent startup, inter-replica collaboration is achieved through two paths: the control plane provides a unified connection and coordination service, responsible for arbitration and recovery scheduling; the data plane dynamically creates cross-replica communication groups based on the arbitration results, responsible for gradient synchronization and state transmission. These two paths correspond to consistency and performance objectives, respectively: the control plane ensures decision convergence, and the data plane ensures execution efficiency. When PP > 1 and the pipeline stage spans multiple nodes, a single replica itself requires multi-node collaboration. For example, when PP = 2, a replica might simultaneously cover parts of the rank of node 0 and node 1. In this case, a hybrid strategy of joint startup within replicas and independent startup between replicas should be adopted: joint startup within replicas ensures pipeline execution correctness, while independence between replicas avoids cross-replica master-slave coupling. This strategy satisfies three points simultaneously in engineering: pipeline semantics remain unchanged, fault domain boundaries are clear, and replicas can reconnect according to the original topology after recovery.

[0089] Experimental results show that the present invention significantly reduces recovery latency (by approximately 90%) and training progress loss (from hundreds of steps to no more than one step), while keeping training throughput overhead below 1%. Long-term fault injection experiments further demonstrate that even under extreme fault frequencies (one fault per minute for 2 hours), the system maintains training continuity, the overall loss curve converges, and replicas recover and rejoin training within seconds.

[0090] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. An iterative point-to-point elastic recovery system for distributed training of large models, characterized in that, include: Coordination services, managers, and replicas; The coordination service is a globally unique coordination node that converges the local states scattered across various processes into globally consistent decisions. The manager is a local agent for each process, located between the training process and the coordination service. The replica is the smallest training unit, with each replica having an independent process group and independent communication storage. The replica runs the native Megatron training process and maintains the complete TP / PP / EP topology. When training starts, each replica starts independently and registers with the coordination service. During normal operation, the replica completes intra-replica reduction using the native Megatron synchronization path, and the replicas complete cross-replica reduction under the coordination of the manager. After the failure occurred, the coordination service recalculated the arbitration based on the heartbeat timeout, and the healthy copy continued to advance the training; After the faulty copy is restored, the parameters are caught up and re-added via status direct transmission.

2. The iterative point-to-point elastic recovery system for distributed training of large models as described in claim 1, characterized in that: Performing the replica-level reduction within the replica yields the replica-level average gradient: in, Indicates a copy The average gradient, globally, has Each GPU is divided into Each copy contains [number] copies. = / One GPU; record copy Medium GPU j The resulting gradient is .

3. The iterative point-to-point elastic recovery system for distributed training of large models as described in claim 2, characterized in that: The cross-replica contract is executed between replicas, and the result is: in, This indicates the result of the global specification, which includes two phases: intra-replica specification and cross-replica specification.

4. The iterative point-to-point elastic recovery system for distributed training of large models as described in claim 1, characterized in that: The coordination service recalculates arbitration based on heartbeat timeout as follows: The coordination service continuously maintains the heartbeat timestamps of registered replicas and first filters healthy replicas within the timeout window during each round of decision-making; if the number of healthy replicas is lower than the minimum number of replicas, the arbitration for this round fails directly and waits for the member's status to recover. The number of healthy copies participating in this round of arbitration meets the majority requirement.

5. The iterative point-to-point elastic recovery system for distributed training of large models as described in claim 4, characterized in that: For newly joined replicas that have not yet stably participated, the coordination service delays arbitration within the join_timeout window; if the replicas that participated in the previous round are still healthy in this round, the results of the previous round are reused; the arbitration results are embedded into the training loop in a specific time sequence.

6. The iterative point-to-point elastic recovery system for distributed training of large models as described in claim 4, characterized in that: For what happened in the If the failure occurs before arbitration, the arbitration will directly exclude the faulty copy. If the commit is successful, the loss is 0; if the failure occurs after arbitration but before gradient synchronization, and the cross-replica reduction fails, return False. If the first step is discarded, the loss is 1; if the failure occurs after gradient synchronization but before parameter update, the submission fails. The step is discarded, resulting in a loss of 1.

7. The iterative point-to-point elastic recovery system for distributed training of large models as described in claim 4, characterized in that: The coordination service achieves state recovery through point-to-point direct memory transfer between replicas, including state catching up after a faulty replica restarts, cold start synchronization when a new replica is added during expansion, and catching up with lagging replicas caused by network jitter. The coordination service automatically identifies replicas that need to be restored through an arbitration algorithm and includes restoration instructions in the results.

8. The iterative point-to-point elastic recovery system for distributed training of large models as described in claim 7, characterized in that: During the state recovery process, the coordination service identifies the lagging replica and issues a recovery target. The lagging replica triggers a state transfer from the normal replica to the lagging replica through a weight request during the training loop, so that the two replicas re-align their steps and enter the next round of arbitration. Meanwhile, the normal replica continues to advance training without global shutdown or waiting.

9. The iterative point-to-point elastic recovery system for distributed training of large models as described in claim 1, characterized in that: It also includes a distributed-trained MoE model, which consists of dense layers and sparse expert layers, with the following parallel dimensions: For dense layers, the data parallelism dimension is defined as Attention_DP: in, TP is the vocabulary size, PP is the tensor parallelism, CP is the pipeline parallelism, and CP is the context parallelism. The data parallelism dimension for the expert layer is defined as Expert_DP: Where, expert_TP is the parallelism of the expert tensor, and EP is the parallelism of the expert.

10. The iterative point-to-point elastic recovery system for distributed training of large models as described in claim 9, characterized in that: The intersection of the attention data parallelism and expert data parallelism corresponds to a set of GPUs that simultaneously hold the same dense parameters and the same expert parameters. During fault recovery, the members of the intersection are direct recovery sources for each other, and the ranks within the same intersection are distributed to different replicas.

Images