Large model checkpoint disaster recovery system based on network computing and asynchronous checkpoint

Abstract

The present invention discloses a large-model checkpoint disaster recovery system based on network computing and asynchronous checkpoints. The disaster recovery system adopts a hierarchical storage design and includes multiple groups, each group including at least two computing nodes, and the computing nodes in the group save each other's checkpoints; the computing nodes include at least a CPU, memory, a NIC, and multiple GPUs; the most recent checkpoint of each GPU is saved in the CPU memory of the current computing node, and the most recent checkpoint is saved in the CPU memory of adjacent computing nodes in the same group through the network; the checkpoints are saved to the storage nodes by utilizing the characteristics of data parallelism and network computing. The large-model disaster recovery system under the distributed system proposed by the present invention can provide fast fault recovery for large-model training, realize high-frequency checkpoints, reduce the time overhead during training and fault recovery, and does not generate additional training throughput overhead. At the same time, the system is applicable to existing distributed parallel training methods and training frameworks.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning large models, and in particular to a large model checkpoint disaster recovery system based on network computing and asynchronous checkpoints. Background Art

[0002] Deep learning models have demonstrated outstanding performance across a wide range of tasks, including computer vision and natural language processing, and are widely used in image recognition, autonomous driving, machine translation, text generation, medical imaging, financial services, intelligent assistants, and personalized recommendations. Since Google proposed the Transformer architecture and introduced the self-attention mechanism in 2017, large language models based on this architecture have rapidly developed. These models, with their unprecedented language understanding and generation capabilities, have garnered widespread attention from both academia and industry. However, these models often have hundreds of billions of parameters, making training prohibitively expensive. These models require numerous GPUs and weeks or even months of training time. This massive computing system poses significant challenges for reliable training. Specifically, due to the sheer scale and high degree of synchronization required for GPU training, hardware failures are common during training, and even a single GPU failure can disrupt the entire training process, necessitating a restart. Therefore, preserving the current state of the model during training, allowing for recovery when needed, is fundamental and central to large-scale model disaster recovery strategies.

[0003] Model checkpointing is an important component of large model training, but checkpointing is an expensive process because each time the latest weight file is saved as a checkpoint, the training process is blocked. However, not performing checkpoints or reducing the checkpoint frequency will result in significant loss of training progress. At the same time, when software or hardware errors occur and the training process needs to be restarted, all training tasks must stop their process and restart from the last saved checkpoint. Summary of the Invention

[0004] In order to reduce the fault recovery overhead in existing deep learning training, this patent proposes a large model disaster recovery system under a distributed training system. By utilizing network computing and storage-end computing power, it can achieve fast checkpoint storage during the large model training process without the need to explicitly determine the checkpoint save frequency. On the other hand, in order to speed up the recovery speed in the event of a fault, the high bandwidth of the CPU memory (the memory of the computing platform represented by HGX, which is called CPU memory to distinguish it from GPU video memory) is utilized to save the latest checkpoints in the local CPU memory and remote CPU memory.

[0005] The present invention discloses a large-scale checkpoint disaster recovery system based on network computing and asynchronous checkpoints. The disaster recovery system adopts a layered storage design and includes multiple groups. Each group includes at least two computing nodes. The computing nodes in the group save each other's checkpoints.

[0006] The computing node includes at least a CPU, a memory, a network transmission module and multiple GPUs;

[0007] The most recent checkpoint of each GPU is saved in the CPU memory of the current computing node, and the most recent checkpoint is saved in the CPU memory of the adjacent computing node in the same group through the network transmission module.

[0008] Furthermore, when the checkpoint is written to the local CPU memory and the adjacent CPU memory, the write operation is called through an asynchronous programming method after the weight update step of each iteration without blocking the training process.

[0009] Furthermore, when the checkpoint is written to the adjacent CPU memory, the idle time in the training process is utilized to perform checkpoint communication, thereby reducing the impact of checkpoint communication on communication during training.

[0010] The method for confirming the idle time includes the following steps:

[0011] Step 1: During each iteration, hooks are set at the entry and exit of all collective communication operations to record timestamps. The start and end times of each communication operation are recorded, and the communication packet size, latency, and current network bandwidth utilization are also collected.

[0012] Communication operations include gradient synchronization, parameter acquisition, and RDMA calls;

[0013] Step 2: construct a continuous timeline for all recorded communication intervals and mark the communication process as occupied "busy interval";

[0014] Discretize the entire timeline, dividing it into several small time slices. Mark whether there is communication operation in each time slice. Count the distribution and cumulative duration of the "busy interval" in each iteration and calculate the candidate idle time window.

[0015] Step 3: traverse the discretized timeline and merge adjacent time windows according to the "busy interval" or "idle" status to generate a continuous idle time period list;

[0016] An initial minimum idle threshold is set. Only idle segments longer than this threshold are considered to be idle time periods that can be used for checkpoint transmission.

[0017] In step 4, a sliding window is used to count the distribution of the idle time periods detected in the last N iterations, calculate their mean, and use a simple exponential smoothing algorithm to predict the possible length of the idle time period in the next iteration:

[0018] S t =αX t +(1-α)S t-1

[0019] Among them, X t is the length of the idle time period detected in the current iteration, S t is the smoothing value at the current moment, S t-1 is the last smoothed value, α is the smoothing coefficient, which determines the weight distribution between the new observation and the historical smoothed sequence; in the first round, there is no historical data, and the first observation is used for initialization, setting S0 = X0;

[0020] Step 5: When it is detected that the idle interval exceeds the current dynamic threshold, a checkpoint transmission operation is triggered. If the idle interval is not long enough to complete the entire transmission task, the data transmission is split into multiple times and dynamically supplemented using the "partial transmission-feedback-determining idle continuation" method. At the same time, the actual transmission delay and used network resources are monitored online. If a sudden increase in network utilization is detected during the transmission process, the transmission rate is dynamically reduced or the transmission is temporarily suspended to wait for the next idle window.

[0021] Step 6: Dynamically adjust the minimum idle threshold based on the distribution. When the idle time in the past iterations is generally short, the threshold can be lowered. When the idle time is long, the threshold is raised to reserve enough time for the transmission of larger data blocks. When it is detected that the current network utilization rate has decreased, the idle judgment threshold can be appropriately lowered to maximize the use of short idle time.

[0022] Furthermore, the large-scale checkpoint disaster recovery system based on network computing and asynchronous checkpoints further includes a storage node, wherein the storage node includes a plurality of non-volatile storage devices and a relatively small computing power device;

[0023]

[0024] FLOPs 前向传播+反向传播 Represents the floating point calculation amount of forward propagation and back propagation, T 前向传播+反向传播 Represents the time required for forward propagation and back propagation, T 参数更新 Represents the time required for parameter update, and P is calculated 存储 Represents the computing performance required by the storage node's group. The total computing power within the group must not be lower than the calculated result.

[0025] Furthermore, the checkpoint storage process on the storage node utilizes the characteristics of the data parallel strategy and includes the following steps:

[0026] Step 21: Initialization is performed. The storage node prepares the checkpoint data to be used for this training in advance. Based on the adopted data parallel strategy, the storage node distributes the complete model to each computing node according to the predetermined rules.

[0027] Step 22: Each model parallel group calculates the parameter gradient locally;

[0028] Step 23: Perform global gradient aggregation and synchronize it to all nodes;

[0029] Step 24: Model updating phase.

[0030] Furthermore, in step 21, the following steps are further included:

[0031] Step 211: Before starting training, the storage node prepares in advance the checkpoint data, gradient synchronization data, and parameter update data to be used in this training;

[0032] Checkpoint data comes from the archive of the previous training session or is manually configured by the user. Checkpoints typically contain: initial values ​​of model parameters, optimizer state, and global state that needs to be restored.

[0033] Checkpoint data, gradient synchronization data, and parameter update data are classified according to transmission priority. Gradient synchronization and parameter updates must be synchronized in a timely manner and are high-priority tasks, while checkpoint data and backup are low-priority tasks.

[0034] The distributed training process requires checkpoint storage and parallel execution between CPUs, using asynchronous programming to achieve non-blocking operations. A central coordinator is used to centrally manage distributed monitoring data, prediction results, and scheduling policies, and feed this information back to the running distributed training process. This prevents low-priority tasks from being triggered during communication-intensive phases, and allows for rapid checkpoint transmission when there is sufficient idle time.

[0035] Step 212: Based on the adopted data parallel strategy, the storage node distributes the complete model to each computing node according to predetermined rules; each computing node loads a copy of the model in the memory of the local computing device when it starts, and the structure of each copy is completely consistent.

[0036] Furthermore, in step 22, the following steps are also included:

[0037] In step 221, each model parallel group performs forward propagation on its assigned data subset, i.e., the input data is processed layer by layer by the model to calculate the output;

[0038] In the forward propagation calculation, each device uses a locally loaded copy of the parameters, and there is no communication overhead between parallel groups of models;

[0039] Step 222: The model output is compared with the true label, and a local loss value is calculated using a loss function. The loss calculation result provides a numerical basis for back propagation.

[0040] In step 223 , each computing node calculates the gradient on the local data subset based on the local loss value through the back propagation algorithm.

[0041] Furthermore, in step 23, the following steps are further included:

[0042] Step 231: Each computing node first obtains a local gradient, and then needs to integrate the gradients of each node to obtain a global gradient;

[0043] In step 232, the aggregated global gradient is broadcast to all computing nodes to ensure that each computing node has consistent gradient information.

[0044] Step 233: Synchronize the gradient to the storage node for subsequent checkpoint update operations, so that the storage node can obtain the latest global state and provide a basis for recovery and debugging.

[0045] Furthermore, in step 231, a full reduction operation is performed:

[0046] All hosts submit communication data to their connected switches. After receiving the data, the leaf switches use their built-in engines to calculate and process the data, and then submit the resulting data to the spine switches. The spine switches also use their own engines to aggregate the resulting data received from several switches and submit it to the root switch. The root switch performs the final calculations and returns the results to all host nodes.

[0047] All nodes exchange data with each other and jointly complete gradient averaging or accumulation; they participate in some calculations through network switches or dedicated high-performance network communication libraries to further accelerate gradient aggregation.

[0048] Furthermore, in step 24, the following steps are further included:

[0049] Step 241: After receiving the global gradient, each computing node performs parameter update calculation according to a preset optimizer;

[0050] During the update process, the computing node calculates and adjusts the parameter values ​​based on the current parameters, global gradients, and the internal state of the optimizer to keep the model state of each node consistent;

[0051] Step 242: After receiving the global gradient synchronization, the storage node needs to complete a similar update operation;

[0052] The entire parameter matrix or optimizer state is divided into multiple blocks of a certain size. The state of each block is updated on the computing device in sequence. After each block is updated, it is immediately written back to the memory. Space is reserved in the memory to store the updated block. The write-back operation from the memory to the non-volatile storage device is asynchronous to reduce the blocking effect on the parameter update. By not waiting for the data to be written to the storage node, the next fast state update is performed directly. The independent background thread or process uses idle resources to asynchronously write the memory data to the storage device in real time, thereby reducing the delay caused by I / O operations.

[0053] Step 243: After the update is completed, the checkpoint data on the storage node contains the latest model parameters, optimizer state, and global gradient;

[0054] Based on the existing storage nodes, it is expanded to multi-copy checkpoints, storing checkpoint data on multiple physical or logical nodes. This ensures that if a storage node fails and cannot read data, other copies can still ensure the smooth recovery operation.

[0055] Prioritize network bandwidth for high-priority tasks; low-priority tasks use preemptive transmission or background transmission, and allow interruption and retransmission when necessary.

[0056] The beneficial effects achieved by the present invention are:

[0057] The large-model disaster recovery system proposed in this paper, within a distributed system, provides rapid fault recovery for large-model training, implements frequent checkpoints, reduces training and fault recovery time, and does not incur additional training throughput overhead. Furthermore, this system is applicable to existing distributed parallel training methods and frameworks.

[0058] This paper proposes an idle time detection method that allows for more flexible and efficient utilization of idle network bandwidth, achieving a better balance between checkpoint transmission and training computation. Furthermore, this solution uses non-intrusive scheduling to minimize disruption to the training process. It also ensures improved checkpoint transmission reliability through fault tolerance and resource reservation. Furthermore, this method supports various collective communication operations and network protocols, making it applicable to a variety of distributed training frameworks, ensuring scalability and compatibility.

[0059] This paper proposes a data parallel strategy for gradient aggregation that fully utilizes network resources, speeding up data exchange. Training and checkpoint writing to storage nodes are executed in parallel, allowing checkpoints to be written to storage nodes at a minimal computational cost without increasing training time. Furthermore, storage nodes utilize block processing and asynchronous writeback when updating checkpoints, optimizing system performance. These steps enable the system to maintain high performance, high reliability, and low latency in a large-scale distributed training environment, meeting the demands of training complex models and massive amounts of data.

[0060] During the distributed training process, the present invention uses a central coordinator to centrally manage distributed monitoring data, prediction results, and scheduling policies. This information is then fed back to the running distributed training process, preventing low-priority tasks from being triggered during periods of intense communication and enabling rapid checkpoint transmission when there is sufficient idle time. Furthermore, a notification mechanism and reserved bandwidth allow the distributed training system to detect idle time windows and coordinate notifications to background transmission tasks to initiate or pause. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Checkpoint replica placement strategy;

[0062] Figure 2 .Asynchronous storage of local CPU memory checkpoints;

[0063] Figure 3 .Redundant cpu memory checkpoint communication;

[0064] Figure 4 .Remote storage checkpoint architecture. DETAILED DESCRIPTION

[0065] The present invention will be further described below with reference to specific embodiments, and the advantages and features of the present invention will become clearer as the description proceeds. However, these embodiments are merely exemplary and do not constitute any limitation to the scope of the present invention. It should be understood by those skilled in the art that the details and forms of the technical solutions of the present invention may be modified or replaced without departing from the spirit and scope of the present invention, and such modifications and replacements fall within the scope of protection of the present invention.

[0066] The present invention provides a large-model checkpoint disaster recovery system based on network computing and asynchronous checkpoints. This disaster recovery system adopts a layered storage design. A computing node contains multiple GPUs. The checkpoints of the GPUs in the computing node are stored in the CPU memory of the computing node, the CPU memory of adjacent computing nodes in the same group, and the non-volatile storage device of the remote storage node.

[0067] The local CPU memory and adjacent CPU memory only store the most recent checkpoint, which is transparent to the user. Checkpoint preservation and recovery are automatically performed by the system. Remote storage stores user-visible historical checkpoints for fault recovery, model debugging, and model rollout.

[0068] Checkpoints in local CPU memory and neighboring CPU memory differ from checkpoints in remote storage in format, but are essentially identical in content. Checkpoints in memory can be considered copies of GPU memory, while checkpoints in remote storage are saved in a serialized format for convenient storage and compression. During typical training, GPU memory stores (partial) model parameters and optimizer state, while CPU memory stores some parameter settings for the current training. Therefore, after the CPU memory receives a copy of the GPU memory, it obtains all the states required for the current training. The checkpoint in remote storage represents the serialized content of the corresponding data.

[0069] Checkpoints are written to local CPU memory and neighboring CPU memory by the worker agent on the local machine. The placement strategy is determined based on the compute node topology during training initialization. At runtime, the worker agent transfers checkpoints from GPU memory to CPU memory based on the placement strategy.

[0070] The purpose of checkpoints in adjacent CPU memory is to prevent failures in which the checkpoints in the local CPU memory may be unavailable. In this case, the system must rely on checkpoints in remote storage, which incurs significant recovery costs. To improve the probability of successful recovery from CPU memory, this system uses a redundant checkpoint mechanism, storing copies of the local CPU checkpoints in adjacent CPUs.

[0071] Saving copies requires significant overhead, including CPU memory usage and copy transmission overhead. Since failures during training are frequent but not concentrated, this system has chosen a simpler copy measure, namely the grouping of computing nodes (here, combination refers to the way computing nodes are allocated, which is different from the group in the following text). Computing nodes usually contain 8 GPUs, which undertake the main computing tasks in model training. Computing nodes also contain CPU, RAM, NIC and other devices. The computing nodes in the combination save each other's checkpoints. The unit of the minimum combination is 2. When there is an odd number of computing nodes, the last combination is three nodes. For example Figure 1 As shown in the figure, the CPU memory of compute node 01 stores the checkpoints of the GPUs of compute nodes 01 and 02, and the CPU memory of compute node 02 stores the checkpoints of the GPUs of compute nodes 01 and 02. The same applies to the three nodes.

[0072] To further improve write speeds when writing to local CPU memory and adjacent CPU memory, the system adopts a fine-grained pipeline approach. Checkpoint writes refer to copying model parameters from GPU memory to CPU memory. When writing to local CPU memory, performing writes synchronously would significantly block the training process. During large model training, it's obvious that model parameters only change during parameter updates. Forward and backward propagation don't change the model's parameters, so the copying process can be handled asynchronously from the computation process.

[0073] like Figure 2 As shown in the figure, after each iterative weight update step, the write operation is called through asynchronous programming methods such as callback functions, Promise, and async / await methods without blocking the training process.

[0074] The system needs to regulate checkpoint traffic when writing to adjacent CPU memory. This is because large model training, due to the large number of model parameters, often employs distributed training strategies such as data parallelism, model parallelism, pipeline parallelism, and hybrid strategies. These parallel strategies rely on collective communication operations for synchronization, placing significant demands on network bandwidth. Transmitting checkpoints from the local CPU to adjacent CPUs consumes network bandwidth, creating potential competition for network resources, potentially delaying communication operations during training, and hindering computation. To mitigate this issue, the system regulates checkpoint traffic to minimize its impact on training.

[0075] By utilizing the idle time in the training process to perform checkpoint communication, the impact of checkpoint communication on communication during training can be reduced. During the forward and backward propagation of the model, a large number of training communication operations are involved, including gradient synchronization and parameter acquisition. There are idle time points in these operations, and there is no communication when the parameters are updated. Therefore, checkpoint communication operations can be performed at these idle time points. When the idle time is insufficient, the training process will be blocked. Figure 3 shown.

[0076] The confirmation of idle time periods uses an adaptive idle time period detection method, which mainly includes three steps: data collection and time series modeling, idle time period identification and adaptive threshold adjustment, and real-time scheduling feedback.

[0077] Step 1: Network event log collection: During each iteration, hooks are set at the entry and exit of all collective communication operations (such as gradient synchronization, parameter acquisition, and RDMA calls) to record timestamps. The start time t_start and end time t_end of each communication operation are recorded and saved to the log system. Additional information such as communication packet size, latency, and current network bandwidth utilization is also collected.

[0078] Step 2: Time series data construction: Based on all recorded communication intervals, a continuous timeline is constructed, and communication processes are marked as occupied "busy intervals." The entire timeline is discretized, dividing time into several small time slices (for example, a 1-millisecond time window). The presence of communication operations within each time slice is marked. The distribution and cumulative duration of "busy intervals" in each iteration are counted to calculate candidate idle time windows.

[0079] Step 3: Initial idle interval detection: Traverse the discretized timeline and merge adjacent time windows according to the "busy" or "idle" status to generate a list of continuous idle time periods. Set an initial minimum idle threshold (for example, 50ms). Only idle periods exceeding this threshold are considered to be idle time periods that can be used for checkpoint transmission.

[0080] Step 4, time prediction and model update: Using a sliding window, calculate the distribution of the idle time periods detected in the last N (such as 10 or 20) iterations, calculate their mean, and use a simple exponential smoothing algorithm to predict the possible length of the idle time period in the next iteration.

[0081] S t =αX t +(1-α)S t-1

[0082] Among them, X t is the length of the idle time period detected in the current iteration, S t is the smoothing value at the current moment, S t-1 is the last smoothed value, and α is the smoothing coefficient, which determines the weight distribution between the new observation and the historical smoothed sequence. In the first round, there is no historical data, so S0 = X0 can be set, that is, the first observation is used for initialization.

[0083] Step 5: Idle Time Confirmation and Checkpoint Scheduling: When an idle interval is detected exceeding the current dynamic threshold, a checkpoint transmission operation is triggered. If the idle interval is insufficient to complete the entire transmission task, the data transmission is split into multiple parts, dynamically supplemented by a "partial transmission-feedback-determining idle continuation" approach. Simultaneously, actual transmission latency and network resource usage are monitored online. If a sudden increase in network utilization is detected during transmission, the transmission rate is dynamically reduced or the transmission is temporarily suspended until the next idle window.

[0084] Step 6 dynamically adjusts the minimum idle threshold based on the distribution. If idle time has been generally short over the past few iterations, the threshold can be lowered (while ensuring that transmission is not severely interrupted), for example, to 0.8 times the median. If idle time is long, the threshold can be raised to reserve sufficient time for the transmission of larger data blocks. If a decrease in network utilization is detected, the idle threshold can be appropriately lowered to maximize the use of short idle periods.

[0085] This solution allows for more flexible and efficient utilization of idle network bandwidth, achieving a better balance between checkpoint transmission and training computation. Furthermore, this solution uses non-intrusive scheduling to minimize disruption to the training process. It also ensures improved reliability of checkpoint transmission through fault tolerance and resource reservation. Furthermore, this method supports various collective communication operations and network protocols, making it applicable to a variety of distributed training frameworks, ensuring scalability and compatibility.

[0086] In the process of training large models, data parallelism is the most commonly used parallel training solution, because data parallelism can improve the model training speed while having low network overhead. Compared with other parallel strategies that are passively adopted due to the large model parameters, data parallelism is more of an active choice. Common data parallel strategies include DP, DDP, FSDP, ZERO, etc. This system is designed based on the general process of data parallelism, namely data segmentation, model replication, parallel calculation, and gradient aggregation. Among them, the gradient aggregation will aggregate the calculated gradients (full reduction operation). Since the model is replicated on different model parallel groups (the model parallel strategy is implemented between model parallel groups), the model parallel groups perform forward and reverse calculations based on the same model parameters and different data to obtain gradients for aggregation between model parallel groups. The aggregated gradients are then used by the local GPU for local parameter updates.

[0087] like Figure 4As shown, a group consists of multiple compute nodes and leaf switches. Leaf switches are fully connected to compute nodes and spine switches. Storage nodes also have corresponding leaf switches that connect to spine switches. Taking a two-tier fat-tree network topology as an example, a compute node typically contains eight GPUs, which handle the majority of computing tasks during model training. Compute nodes also include CPUs, RAM, NICs, and other devices. The number of leaf and spine switches is determined by the number of switch ports and the network topology.

[0088] Storage nodes contain multiple non-volatile storage devices and less computing power, typically several GPUs, as well as CPUs, RAM, NICs, etc. The number of storage nodes can vary, and the number of storage nodes does not affect the algorithm flow. Multiple storage nodes store complete checkpoint information and are mutually redundant. The computing power of storage nodes does not participate in model training, so choose devices with lower computing power than other computing nodes. The specific selection can be based on the formula:

[0089]

[0090] F:OPs 前向传播+反向传播 Represents the floating point calculation amount of forward propagation and back propagation, T 前向传播+反向传播 Represents the time required for forward propagation and back propagation, T 参数更新 Represents the time required for parameter update, and P is calculated 存储 Represents the required computing performance of the storage node's group. The total computing power within the group must not be lower than the calculated result. If the total computing power is lower than the requirement, remote checkpoints cannot be saved for each iteration. In this case, the checkpoint frequency can be adjusted. For example, saving a checkpoint after every parameter update can reduce the computing power requirement by half. Given that the computing power of the storage node's group is negligible compared to the overall training computing power, reducing the computing power of the storage node's group is not recommended.

[0091] The checkpoint storage process on the storage node utilizes the characteristics of the data parallel strategy. The specific process is as follows:

[0092] Step 21, initialization phase

[0093] Step 211, checkpoint preparation. Before training begins, the storage node prepares the checkpoint data, gradient synchronization data, and parameter update data needed for this training session. Checkpoints can be archived from a previous training session or manually configured by the user. Checkpoints typically include: initial values ​​of model parameters, optimizer state (such as momentum and learning rate decay parameters), and other global states that may need to be restored, such as the global training step count and random seed.

[0094] Checkpoint data, gradient synchronization data, and parameter update data are classified according to transmission priority. Gradient synchronization and parameter update must be synchronized in a timely manner as high-priority tasks, while checkpoint data and backup are low-priority tasks.

[0095] The distributed training process requires checkpoint storage and parallel execution between CPUs, utilizing asynchronous programming (such as async / await, Promises, and callback functions) for non-blocking operation. At this point, an integrator, or central coordinator, centrally manages distributed monitoring data, prediction results, and scheduling policies, feeding this information back to the running distributed training process. This prevents low-priority tasks from being triggered during communication-intensive phases, and allows for rapid checkpoint transmission when there is sufficient idle time.

[0096] In step 212, model distribution, based on the adopted data parallel strategy, the storage node distributes the complete model to each compute node according to predefined rules. Upon startup, each node loads a copy of the model from the memory of its local computing device (e.g., GPU or TPU). Each copy is structurally identical but subsequently processes different data subsets.

[0097] Step 22, parallel computing stage

[0098] In step 221, forward propagation calculations are performed on each model parallel group on their assigned data subset. This means that the input data is processed layer by layer by the model to calculate the output. During this process, each device uses a locally loaded copy of the parameters, eliminating the communication overhead between model parallel groups, as is the case with other parallel strategies.

[0099] Step 222, loss calculation, the model output is compared with the true label, and the local loss value is calculated through the loss function (such as cross entropy, mean square error, etc.). The loss calculation result provides a numerical basis for back propagation.

[0100] Step 223: Backpropagation calculates the gradient. Each compute node uses the backpropagation algorithm to calculate the gradient on the local data subset based on the local loss value. This stage only calculates the parameter gradient locally and does not involve cross-node synchronization. Therefore, the data parallel strategy does not introduce additional network overhead.

[0101] Step 23: Parameter synchronization phase

[0102] Step 231, local to global gradient aggregation, each computing node first obtains a local gradient, and then needs to integrate the gradients of each node to obtain the global gradient. This process usually uses an all-reduce operation: all hosts submit communication data to their respective connected switches. After receiving the data, the leaf switch uses the built-in engine to calculate and process the data, and then submits the result data to the spine switch. The spine switch also uses its own engine to aggregate the result data received from several switches and submits it to the root switch. The root switch performs the final calculation and returns the result to all host nodes. All nodes exchange data with each other and jointly complete the gradient averaging or accumulation. Network switches or dedicated high-performance network communication libraries (such as those based on RDMA) can participate in part of the calculation to further speed up the gradient aggregation.

[0103] Step 232: Synchronize the gradients to all nodes. The aggregated global gradients will be broadcast to all computing nodes to ensure that each node has consistent gradient information, which facilitates subsequent model parameter updates.

[0104] Step 233: Synchronize the gradients to the storage nodes. In addition to being used for local updates, a copy of the global gradients must also be synchronized and stored on the storage nodes. This gradient data is used for subsequent checkpoint updates, allowing the storage nodes to obtain the latest global state and providing a basis for recovery and debugging.

[0105] Step 24, model update phase

[0106] Step 241, parameter update (on the compute node side): After receiving the global gradient, each compute node performs parameter update calculations based on a preset optimizer (such as SGD or Adam). During the update process, the compute node calculates and adjusts parameter values ​​based on the current parameters, the global gradient, and the optimizer's internal state (such as momentum), maintaining a consistent model state at each node.

[0107] Step 242: Update Checkpoint (Storage Node Side). After receiving the global gradient synchronization, the storage node needs to complete similar update operations. First, the checkpoint data (including model parameters and optimizer state) is loaded from persistent storage into memory. Considering the large amount of model parameter data, the update is divided into blocks according to the device's video memory capacity or system memory. The entire parameter matrix or optimizer state is divided into multiple blocks of a certain size. The state update of each block is completed in sequence on the computing device (which may use a GPU or dedicated accelerator). After each block is updated, it is immediately written back to the memory. Space is reserved in the memory to store the updated block. The write-back operation from the memory to the non-volatile storage device is asynchronous to reduce the blocking effect on the parameter update. That is, without waiting for the data to be written to the storage node, the next fast state update is directly performed. Independent background threads or processes use idle resources to asynchronously write the memory data to the storage device in real time, thereby reducing the delay caused by I / O operations.

[0108] Step 243, overall consistency and storage write back, after the update is completed, the checkpoint data on the storage node contains the latest model parameters, optimizer status and global gradient. This ensures that when the system needs to be restored, it can be started from the checkpoint data and continue training without losing progress. Data consistency verification will also be performed at this stage to ensure that no errors occur during the storage and reading of the data. Furthermore, in order to ensure data reliability, based on the existing storage nodes, it can be expanded to multiple copy checkpoints, that is, checkpoint data (including model parameters, optimizer status, global gradients, etc.) are stored on multiple physical or logical nodes. At this time, if a storage node cannot read the data due to a failure, other copies can still ensure the smooth recovery operation. At the same time, the network bandwidth of high-priority tasks is given priority; low-priority tasks use preemptive transmission or background transmission, and allow interruption and retransmission when necessary.

[0109] The above steps fully utilize network resources during gradient aggregation, accelerating data exchange. Training and checkpoint writing to storage nodes are executed in parallel, allowing checkpoints to be written to storage nodes at a minimal computational cost without increasing training time. Furthermore, storage nodes utilize block processing and asynchronous writeback when updating checkpoints, optimizing system performance. These steps enable the system to maintain high performance, high reliability, and low latency in large-scale distributed training environments, meeting the demands of training complex models and massive amounts of data.

[0110] The fault recovery function of this system involves four components: a set of worker agents, a root agent, a distributed key-value store, and an operator.

[0111] The worker agent is responsible for monitoring the health status of the machine it is responsible for, updating it to the distributed key-value storage system, and completing the writing of the checkpoint to the local CPU memory and the memory of the adjacent CPUs.

[0112] The root agent runs on a regular training machine or a dedicated machine. It periodically checks the health of each training machine from the distributed key-value store. If the root agent detects a training machine failure, it takes appropriate action based on the failure type. When a root machine fails, a leader election is performed using the distributed key-value store.

[0113] Cloud operators manage computing resources and replace failed machines with healthy ones when needed.

[0114] This system is based on a hierarchical storage design and takes different measures for different fault types.

[0115] When a software failure occurs, the training process is interrupted, but the hardware is still healthy and the checkpoints stored in the local CPU memory are still accessible, so training can be resumed directly from the local checkpoint.

[0116] When a hardware failure occurs, the faulty machine must be located and replaced. This locating process relies on hardware monitoring tools provided by the hardware vendor and a distributed key-value store system. Specifically, the worker agent and root agent periodically send heartbeat signals to the distributed key-value store. If the distributed key-value store fails to detect a heartbeat signal, the corresponding machine or network is considered faulty. Recovery is then performed based on the grouping process. If a checkpoint replica is available within the group, recovery can be performed directly within the group. If a checkpoint replica is unavailable within the group, recovery must be performed from the storage node.

[0117] When a hardware failure occurs, cloud operators should immediately provide a healthy machine to replace the faulty one. However, this replacement operation causes training to wait, so the training cluster can reserve some spare machines. When a machine experiences a hardware failure, the spare machine can be immediately activated to replace the faulty one for recovery, and tasks on the faulty machine can be assigned to the spare machine to ensure that overall training is not interrupted. At the storage node level, when a storage node fails, the distributed file system within the cluster ensures high-availability and rapid recovery of data.

[0118] The root broker then returns the failed machine and requests another backup machine.

[0119] The above are only specific steps of the present invention and do not constitute any limitation to the scope of protection of the present invention; any technical solutions formed by equivalent transformation or equivalent replacement fall within the scope of protection of the present invention; the parts not elaborated in detail in the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A large-scale checkpoint disaster recovery system based on network computing and asynchronous checkpoints, characterized in that: The disaster recovery system adopts a layered storage design. The disaster recovery system includes multiple groups, each group includes at least two computing nodes, and the computing nodes in the group save each other's checkpoints; The computing node includes at least a CPU, a memory, a network transmission module and multiple GPUs; The most recent checkpoint of each GPU is saved in the CPU memory of the current computing node, and the most recent checkpoint is saved in the CPU memory of the adjacent computing node in the same group through the network transmission module; When the checkpoint is written into the adjacent CPU memory, the idle time in the training process is utilized to perform checkpoint communication, thereby reducing the impact of the checkpoint communication on the communication during training; The method for confirming the idle time includes the following steps: Step 1: During each iteration, hooks are set at the entry and exit of all collective communication operations to record timestamps. The start and end times of each communication operation are recorded, and the communication packet size, latency, and current network bandwidth utilization are also collected. Communication operations include gradient synchronization, parameter acquisition, and RDMA calls; Step 2: Construct a continuous timeline for all recorded communication intervals and mark the communication process as occupied "busy interval"; Discretize the entire timeline, dividing it into several small time slices. Mark whether there is communication operation in each time slice. Count the distribution and cumulative duration of "busy intervals" in each iteration and calculate candidate idle time windows. Step 3: Traverse the discretized timeline and merge adjacent time windows according to the "busy interval" or "idle" status to generate a continuous idle time period list; An initial minimum idle threshold is set. Only idle segments longer than this threshold are considered to be idle time periods that can be used for checkpoint transmission. In step 4, a sliding window is used to count the distribution of the idle time periods detected in the last N iterations, calculate their mean, and use a simple exponential smoothing algorithm to predict the possible length of the idle time period in the next iteration: S y =αX t +(1-α)S t-1 Among them, X t is the length of the idle time period detected in the current iteration, S t is the smoothing value at the current moment, S t-1 is the last smoothed value, α is the smoothing coefficient, which determines the weight distribution between the new observation and the historical smoothed sequence; in the first round, there is no historical data, and the first observation is used for initialization, setting S0 = X0; In step 5, when it is detected that the idle interval exceeds the current dynamic threshold, a checkpoint transmission operation is triggered. If the idle interval is not long enough to complete the entire transmission task, the data transmission is split into multiple times and dynamically supplemented using the "partial transmission-feedback-determining idle continuation" method. At the same time, the actual transmission delay and used network resources are monitored online. If a sudden increase in network utilization is detected during the transmission process, the transmission rate is dynamically reduced or the transmission is temporarily suspended to wait for the next idle window. Step 6: Dynamically adjust the minimum idle threshold based on the distribution. When the idle time in the past iterations is generally short, the threshold can be lowered. When the idle time is long, the threshold is raised to reserve enough time for the transmission of larger data blocks. When it is detected that the current network utilization rate has decreased, the idle judgment threshold can be appropriately lowered to maximize the use of short idle time.

2. The large-scale checkpoint disaster recovery system based on network computing and asynchronous checkpoints according to claim 1 is characterized in that: When the checkpoint is written to the local CPU memory and the adjacent CPU memory, the write operation is called through an asynchronous programming method after the weight update step of each iteration without blocking the training process.

3. The large-scale checkpoint disaster recovery system based on network computing and asynchronous checkpoints according to claim 1 is characterized in that: The large-model checkpoint disaster recovery system based on network computing and asynchronous checkpoints further includes a storage node, wherein the storage node includes a plurality of non-volatile storage devices and a device with less computing power; FLOPs 前向传播+反向传播 Represents the floating point calculation amount of forward propagation and back propagation, T 前向传播+反向传播 Represents the time required for forward propagation and back propagation, T 参数更新 Represents the time required for parameter update, and P is calculated 存储 Represents the computing performance required by the storage node's group. The total computing power within the group must not be lower than the calculated result.

4. The large-scale checkpoint disaster recovery system based on network computing and asynchronous checkpoints according to claim 3 is characterized in that: The checkpoint storage process on the storage node utilizes the characteristics of the data parallel strategy and includes the following steps: Step 21: Initialization is performed. The storage node prepares the checkpoint data to be used for this training in advance. Based on the adopted data parallel strategy, the storage node distributes the complete model to each computing node according to the predetermined rules. Step 22: Each model parallel group calculates the parameter gradient locally; Step 23: Perform global gradient aggregation and synchronize it to all nodes; Step 24, model updating phase.

5. The large-scale checkpoint disaster recovery system based on network computing and asynchronous checkpoints according to claim 4 is characterized in that: In step 21, it further includes: Step 211: Before starting training, the storage node prepares in advance the checkpoint data, gradient synchronization data, and parameter update data to be used in this training; Checkpoint data comes from the archive of the previous training session or is manually configured by the user. Checkpoints typically contain: initial values ​​of model parameters, optimizer state, and global state that needs to be restored. Checkpoint data, gradient synchronization data, and parameter update data are classified according to transmission priority. Gradient synchronization and parameter updates must be synchronized in a timely manner and are high-priority tasks, while checkpoint data and backup are low-priority tasks. The distributed training process requires checkpoint storage and parallel execution between CPUs, using asynchronous programming to achieve non-blocking operations. A central coordinator is used to centrally manage distributed monitoring data, prediction results, and scheduling policies, and feed this information back to the running distributed training process. This prevents low-priority tasks from being triggered during communication-intensive phases, and allows for rapid checkpoint transmission when there is sufficient idle time. Step 212: Based on the adopted data parallel strategy, the storage node distributes the complete model to each computing node according to predetermined rules; each computing node loads a copy of the model in the memory of the local computing device when it starts, and the structure of each copy is completely consistent.

6. The large-scale checkpoint disaster recovery system based on network computing and asynchronous checkpoints according to claim 4 is characterized in that: In step 22, it further includes: In step 221, each model parallel group performs forward propagation on its assigned data subset, i.e., the input data is processed layer by layer by the model to calculate the output; In the forward propagation calculation, each device uses a locally loaded copy of the parameters, and there is no communication overhead between parallel groups of models; Step 222: The model output is compared with the true label, and a local loss value is calculated using a loss function. The loss calculation result provides a numerical basis for back propagation. In step 223 , each computing node calculates the gradient on the local data subset based on the local loss value through the back propagation algorithm.

7. The large-scale checkpoint disaster recovery system based on network computing and asynchronous checkpoints according to claim 4 is characterized in that: In step 23, the method further includes: Step 231: Each computing node first obtains a local gradient, and then needs to integrate the gradients of each node to obtain a global gradient; In step 232, the aggregated global gradient is broadcast to all computing nodes to ensure that each computing node has consistent gradient information. Step 233: Synchronize the gradient to the storage node for subsequent checkpoint update operations, so that the storage node can obtain the latest global state and provide a basis for recovery and debugging.

8. The large-scale checkpoint disaster recovery system based on network computing and asynchronous checkpoints according to claim 7, characterized in that: In step 231, a full reduction operation is performed: All hosts submit communication data to their connected switches. After receiving the data, the leaf switches use their built-in engines to calculate and process the data, and then submit the resulting data to the spine switches. The spine switches also use their own engines to aggregate the resulting data received from several switches and submit it to the root switch. The root switch performs the final calculations and returns the results to all host nodes. All nodes exchange data with each other and jointly complete gradient averaging or accumulation; they participate in some calculations through network switches or dedicated high-performance network communication libraries to further accelerate gradient aggregation.

9. The large-scale checkpoint disaster recovery system based on network computing and asynchronous checkpoints according to claim 4, characterized in that: In step 24, it also includes: Step 241: After receiving the global gradient, each computing node performs parameter update calculation according to a preset optimizer; During the update process, the computing node calculates and adjusts the parameter values ​​based on the current parameters, global gradients, and the internal state of the optimizer to keep the model state of each node consistent; Step 242: After receiving the global gradient synchronization, the storage node needs to complete a similar update operation; The entire parameter matrix or optimizer state is divided into multiple blocks of a certain size. The state of each block is updated on the computing device in sequence. After each block is updated, it is immediately written back to the memory. Space is reserved in the memory to store the updated block. The write-back operation from the memory to the non-volatile storage device is asynchronous to reduce the blocking effect on the parameter update. By not waiting for the data to be written to the storage node, the state update of the next block is carried out directly. The independent background thread or process uses idle resources to asynchronously write the memory data to the storage device in real time, thereby reducing the delay caused by I / O operations. Step 243: After the update is completed, the checkpoint data on the storage node contains the latest model parameters, optimizer state, and global gradient; Based on the existing storage nodes, it is expanded to multi-copy checkpoints, storing checkpoint data on multiple physical or logical nodes. This ensures that if a storage node fails and cannot read data, other copies can still ensure the smooth recovery operation. Prioritize network bandwidth for high-priority tasks; low-priority tasks use preemptive transmission or background transmission, and allow interruption and retransmission when necessary.

Images