Deep learning fault tolerance-oriented weak consistency check point storage and recovery method

Abstract

The invention discloses a weak consistency check point storage and recovery method for deep learning fault tolerance, which relates to the technical field of deep learning fault tolerance and comprises a step check point storage mechanism based on parameter consistency controllability and a fault recovery mechanism based on multi-check-point cascade index. According to the step check point storage mechanism, coupling dependence of a check point version and iteration steps is relieved, the storage process of model parameters is divided into a plurality of sub-processes, asynchronous execution is carried out in a plurality of continuous iteration steps, check points comprise parameters of different iteration steps, and weak consistency snapshots are formed. According to the multi-check-point cascade indexing mechanism, historical check points are used for constructing hybrid restart points, parameter blocks are rapidly positioned through timestamp indexes, and redundant data are deleted on the basis of redundancy assessment. According to the method, the storage and recovery overhead of the check points in the training process is remarkably reduced, and the training efficiency and the resource utilization rate are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning fault tolerance, and in particular to a weak consistency checkpoint preservation and recovery method for deep learning fault tolerance. Background Art

[0002] In recent years, deep learning models have demonstrated remarkable performance breakthroughs in fields such as computer vision and natural language processing. The capabilities of large models are proportional to their scale, and the trend towards model scaling has become an important paradigm for current large model training. However, large model training faces severe system reliability challenges. Modern LLMs training requires the coordination of tens of thousands of GPUs for months of distributed computing. Due to unpredictable reasons such as hardware failures, system problems, or network errors, the training process may be interrupted at any time, resulting in data loss. Restarting training not only wastes a lot of time, but may also result in the loss of all training results before the interruption point, significantly impacting the reliability and efficiency of deep learning applications. Moreover, the inherent characteristics of this large-scale training system cause failures to exhibit a power-law distribution. The core problem of existing fault recovery mechanisms stems from the inefficiency of traditional consistency checkpoints. Current mainstream frameworks use synchronous periodic checkpoint strategies such as Nebula and PyTorch's native checkpointing tool to ensure parameter consistency. Their workflow includes: 1) regularly serializing and storing distributed training status (including model parameters, optimizer status, and training metadata) in a remote persistent storage system; 2) rolling back the training status through the most recent checkpoint when a failure occurs. While this mechanism ensures the reliability of the training process, it also introduces significant time and space overhead challenges.

[0003] Efficient checkpointing is increasingly crucial for deep learning research. Without a fault-tolerant training framework, developing deeper and larger models inevitably leads to brittleness. Common failures include exploding gradients, division by zero, and dead activations. This necessitates the creation of breakpoints, similar to those used when debugging computer programs, allowing researchers to easily jump back to the state of the model before it crashed during training. Frequent checkpointing allows failed training processes to be restarted as quickly as possible and also serves as a breakpoint for debugging models. To date, standard checkpointing practices have been relatively primitive. The most common approach is to directly save the model state, which depends on the robustness of the backend system to prevent this operation from becoming a bottleneck. The most obvious technical challenge is that deep models are typically large, so frequent checkpointing can impose an unmanageable burden on I / O and storage, even on modern distributed platforms. On the other hand, reducing the amount of data saved with each checkpoint can also reduce overhead, such as by prioritizing checkpoints to only save parameters whose values ​​have changed significantly since the previous checkpoint. Non-redundant checkpointing stores checkpoints in CPU memory. This requires processors to encode checkpoints using parity, allowing for recalculation when a processor fails. However, encoding and decoding checkpoints for large model training is prohibitively expensive. FTCCharm++ stores two checkpoint copies on two processors for fault tolerance, but lacks analysis of optimal checkpoint placement. Several distributed systems have been proposed for data placement in the cloud. For example, CRUSH distributes data replicas evenly across machines to maintain statistically balanced utilization of storage and bandwidth resources; Volley develops automated technology for placing application data across data centers; Bamboo uses redundant computing to provide elasticity and fast recovery for training large DNN models on preemptible instances; Varuna also supports large model training on preemptible instances, but it requires users to manage checkpoints, such as frequency and storage, for fault recovery; the Gemini framework believes that the availability of CPU main memory is higher than that of GPU video memory, so it saves checkpoints directly to main memory rather than persistent storage media, thereby reducing the time overhead of backup; DataStates-LLM, as the current latest framework, introduces a lazy asynchronous multi-level approach that keeps the tensors that make up the model and optimizer state shards unchanged for extended periods of time, making it possible to copy their contents in the background with minimal interference during training.

[0004] Therefore, it is urgent to study a weak consistency checkpoint preservation and recovery method for deep learning fault tolerance. Summary of the Invention

[0005] To address the above technical issues, the present invention discloses a weak consistency checkpoint preservation and recovery method for deep learning fault tolerance. This method, based on the weak consistency checkpoint technology for deep learning fault tolerance, provides a more flexible and efficient technology for recovering DNN training when it faces interruptions, thereby improving the stability and efficiency of deep learning systems. Due to the strict constraints of existing consistency checkpoints, which require saving all parameters at the same time, the optimization space for checkpoint access overhead is limited. Under the premise of allowing for errors, a weak consistency checkpoint technology is designed to quickly save model checkpoints and jointly restore checkpoint states, reducing time and space overhead.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A weakly consistent checkpoint preservation and recovery method for deep learning fault tolerance includes: a stride checkpoint preservation mechanism based on controllable parameter consistency, and a fault recovery mechanism based on multi-checkpoint cascade indexing. The stride checkpoint preservation mechanism removes the coupling dependency between checkpoint versions and iteration steps, divides the model parameter preservation process into several sub-processes, and executes them asynchronously in multiple consecutive iteration steps, so that the checkpoint contains parameters from different iteration steps, forming a weakly consistent snapshot. The multi-checkpoint cascade indexing mechanism uses historical checkpoints to construct hybrid restart points, quickly locates parameter blocks through timestamp indexing, and deletes redundant data based on redundancy assessment.

[0008] Optionally, the stride checkpoint saving mechanism includes: dividing the model parameters into several subsets by layer or block, and saving each part of the parameters in sequence in multiple consecutive iteration steps to form a complete checkpoint; wherein, each iteration step saves part of the parameters, and the final checkpoint is composed of the parameters of multiple iteration steps.

[0009] Optionally, for heterogeneous stride checkpoints, to address the bucket wait effect, optimizations are performed at both the horizontal and vertical levels:

[0010] Horizontally between different computing nodes, each node is allowed to complete cold backup storage of local parameters at different iteration steps;

[0011] Vertically, within a node, parameter preservation is allowed across iterations. If the checkpoint save of all local parameters is not completed within the node, the calculation of the next iteration step is entered and the parameters are updated. The parameters that have not yet been saved are replaced with the updated versions.

[0012] Optionally, the stride checkpoint saving mechanism further includes a weak consistency error metric, including:

[0013] Assign a checkpoint version number to each node and calculate the version number difference. The formula is: ij =|δ i -δj |;δ i is the checkpoint version number of the i-th node, δ j is the checkpoint version number of the jth node, δ ij is the version number difference between node i and node j;

[0014] Define the weak consistency score as the maximum version number difference between all nodes, the maximum version number difference E max The formula is: w is the number of nodes;

[0015] When E max >∈, the system believes that there is an excessively large weak consistency error, which affects the training effect of the model, and adjusts the checkpoint saving strategy, including starting local checkpoint saving or adjusting the saving priority.

[0016] Optionally, the priority adjustment is to adjust the checkpoint saving frequency or postpone the checkpoint saving time according to the delay of the node, specifically: according to the total checkpoint overhead time T ckpt , total training time T t , storage write time T s And the number of nodes w, dynamically adjust the consistency threshold, the formula is:

[0017]

[0018] Where, Reflects the impact of checkpoint overhead on training. If the checkpoint time is much longer than the training time, the ratio is close to 1, which means that frequent storage will affect the training progress. Otherwise, it means that the storage operation has little impact on training. γ is the impact strength, 1.0≤γ≤2.0; Reflects the impact of storage speed. When storage writing is slow, T s becomes larger, causing ∈ to increase and relax the weak consistency threshold; It is used to measure the synchronization impact of multi-node training to prevent the checkpoint writing speed of some nodes from slowing down the overall training progress. If the checkpoint time of all nodes is close, then T ckpt,i / w is approximately equal to T ckpt , which is close to a constant.

[0019] Optionally, the multi-checkpoint cascading index mechanism includes:

[0020] Construct a bitmap index matrix M, where rows correspond to checkpoints, columns correspond to parameter blocks, and matrix elements M ij Indicates block BLK j At checkpoint ck i whether it exists in

[0021] By traversing the matrix to locate the parameter block required for recovery, the parameters in the most recent checkpoint are obtained.

[0022] Optionally, a checkpoint pruning strategy is also included, specifically:

[0023] Each parameter block is subdivided into sub-blocks, a meta-index is established, and the preservation status of each sub-block is recorded;

[0024] Calculate the redundancy ρ of the same parameter block in different checkpoints. The formula is: The number of intersection sub-blocks is the number of sub-blocks stored in the meta-indexes corresponding to the two checkpoints, and the number of union sub-blocks is the total number of sub-blocks stored in the meta-indexes corresponding to the two checkpoints.

[0025] When ρ>∈, delete the old version parameter block.

[0026] Optionally, the multi-checkpoint cascade indexing mechanism further includes timeliness evaluation, which performs a weighted evaluation on different versions of the parameter block. The formula is:

[0027] V BLK =α·S x +(1-α)·S y ;

[0028] Where S x 、S y Checkpoint ck x and ck y The number of iteration steps is α∈[0,1], and α∈[0,1] is the weight factor.

[0029] The beneficial effects of the present invention are:

[0030] Based on an in-depth analysis of large-scale training environments and model characteristics, this paper identifies bottlenecks in traditional consistency checkpointing technology and proposes a checkpoint optimization solution based on weak consistency, effectively reducing the checkpoint saving and recovery overhead during training. Specifically:

[0031] (1) A stride checkpoint mechanism based on controllable parameter consistency. In response to the problem that many existing checkpoint preservation methods mainly rely on periodic preservation and fixed-step checkpoint strategies, a stride checkpoint mechanism that can relax consistency constraints is proposed. The stride checkpoint mechanism is designed to decouple the checkpoint version and the iteration step to eliminate bucket waiting. By forming snapshots in a stride-by-step preservation mode, the mechanism demonstrates better training efficiency in practical applications. In addition, in-depth analysis under the weak consistency error metric shows that the mechanism has little impact on the convergence of the model after resuming training, and the final model accuracy loss is controlled at an extremely low level, which can fully guarantee the reliability of the training process.

[0032] (2) A high-quality fault recovery mechanism based on multi-checkpoint cascade indexing. Aiming at the problem existing in the existing priority checkpoint method, that is, the priority of all parameters will change dynamically during the iterative training process, resulting in them being scattered and saved in different priority checkpoints. The present invention proposes a checkpoint recovery mechanism that makes full use of its historical checkpoints. First, a hybrid restart point is constructed by cascading multiple versions of data to improve its coverage of parameters and optimize recovery efficiency. Secondly, a timestamp-based cascade index is designed to support the rapid extraction of required data from multi-version cold backup checkpoints to construct restart points. At the same time, it can guide the deletion of redundant data between multiple checkpoints to release storage space in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the stride checkpoint mechanism of the present invention;

[0034] Figure 2 The traditional consistency checkpoint (A) and the weak consistency stride checkpoint (B) of the present invention;

[0035] Figure 3 A schematic diagram of constructing a restart point by cascading multiple priority checkpoints according to an embodiment of the present invention;

[0036] Figure 4 This is a diagram of the cascaded multi-priority checkpoint architecture of the present invention;

[0037] Figure 5 A schematic diagram of index pruning according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] The research content of this invention focuses on two main areas: the first is a stride checkpoint mechanism based on controllable parameter consistency. Checkpoint storage overhead can be reduced through techniques such as prioritizing important data, compression, adjusting backup frequency, and asynchronous storage. All of these approaches require that data within a checkpoint belong to the same iteration step to ensure parameter version consistency. However, heterogeneous software and hardware inevitably lead to differences in backup times across nodes. Strong consistency constraints can trigger a bucket effect and waste computing power. Considering the self-adjustment and fault tolerance capabilities of iterative model training, this invention relaxes consistency constraints and proposes a stride checkpoint mechanism with weak consistency, i.e., controllable parameter consistency. The second is high-quality fault recovery based on cascaded indexing of multiple priority checkpoints. Unlike existing priority checkpointing approaches that frequently save highly changing parameters to achieve partial checkpoint recovery, the priority checkpoint strategy, like the full checkpoint strategy, writes the same amount of data to persistent storage at each constant number of iterations. While this effectively improves recovery speed, it does not significantly reduce I / O overhead. Therefore, it is necessary to design a mechanism based on the joint operation of multiple priority checkpoints and an efficient checkpoint indexing method to reduce the checkpoint recovery overhead.

[0040] A weakly consistent checkpoint preservation and recovery method for deep learning fault tolerance includes: a stride checkpoint preservation mechanism based on controllable parameter consistency, and a fault recovery mechanism based on multi-checkpoint cascade indexing. The stride checkpoint preservation mechanism removes the coupling dependency between checkpoint versions and iteration steps, divides the model parameter preservation process into several sub-processes, and executes them asynchronously in multiple consecutive iteration steps, so that the checkpoint contains parameters from different iteration steps, forming a weakly consistent snapshot. The multi-checkpoint cascade indexing mechanism uses historical checkpoints to construct hybrid restart points, quickly locates parameter blocks through timestamp indexing, and deletes redundant data based on redundancy assessment.

[0041] 1. A stride checkpoint mechanism based on controllable parameter consistency

[0042] The stride checkpoint preservation mechanism based on controllable parameter consistency is described from the following two aspects: 1) Stride checkpoint mechanism: Remove the coupling dependency between the checkpoint version and the iteration step to eliminate the bucket wait. The core idea is to complete a complete model state save within a span of multiple iteration steps, so that the checkpoint version no longer strictly corresponds to a single iteration step, thereby minimizing the interference of the preservation overhead on training. Compared with traditional strong consistency checkpoints, stride checkpoints are weak consistency snapshots: that is, the various parameters contained in the checkpoint may come from different training moments. 2) Weak consistency error metric: In order to avoid introducing large model errors due to relaxing consistency constraints, explore the metric of weak consistency errors of data versions, reduce errors through load migration, priority threshold adjustment, etc., and achieve a dynamic balance between consistency constraints and the elimination of the bucket effect.

[0043] Optionally, the stride checkpoint saving mechanism includes: dividing the model parameters into several subsets by layer or block, and saving each part of the parameters in sequence in multiple consecutive iteration steps to form a complete checkpoint; wherein, each iteration step saves part of the parameters, and the final checkpoint is composed of the parameters of multiple iteration steps.

[0044] Common deep learning training, such as PyTorch checkpoint settings, is performed synchronously, and training is paused after the checkpoint operation is completed. However, synchronous checkpointing introduces long checkpoint pauses, which will lead to greater runtime overhead if executed frequently. For example, considering a strategy of checking every three iterations, the model state is written to disk after the weight update phase, which updates the weights based on the gradients calculated in the backward pass. The stride checkpoint mechanism proposed in the present invention fully explores and utilizes the data dependencies between model parameters in distributed training to achieve more efficient weak consistency checkpoint preservation. The checkpoint preservation phase of the stride checkpoint does not have the waiting block of the snapshot phase during the model training process. The number of parameters required for the snapshot is calculated using the corresponding snapshot interface. Through a full stride, for example, in the first checkpoint preservation process, some parameters θ1 can be saved after the first iteration step, and the remaining parameters θ2 can be saved after the second iteration step, thereby forming a complete checkpoint ck1=θ1∪θ2; then the complete checkpoint snapshot is persistently stored. At the same time, a snapshot of a complete checkpoint may be composed of parameters from two or more different iteration steps. Through Figure 1 This fully strided checkpoint design can achieve the same progress as trouble-free training after completing the second checkpoint save. That is, when the model training is completed, the overall training time cost is no longer affected by the blocking cost of checkpoint saving.

[0045] In the traditional mechanism, the kth checkpoint saves the model state corresponding to the kNth step (assuming it is saved every N steps); while the stride checkpoint allows a checkpoint process to span multiple training steps, so it is no longer limited to the state of a single step. Specifically, the stride checkpoint divides the entire model parameter saving process into several fine-grained sub-processes and inserts these sub-processes into consecutive training iterations for asynchronous execution. For example, the model parameters are divided into P1, P2, ..., P by layer or block. m In a checkpoint cycle, the training is not stopped at once to save all parts. Instead, m consecutive iterations are used to save P1 in the first step, P2 in the second step, ..., and P in the mth step. m In this way, a complete snapshot of the model state is gradually completed over a span of m training steps, and the training computation is almost uninterrupted during this process. Figure 1 This process is vividly demonstrated: while the training thread is performing normal forward / reverse calculations, the dedicated checkpoint thread asynchronously writes some parameters of the previous moment to the storage. m When the save is completed, a complete version of the checkpoint is generated. Since the parameters of each part of the stride checkpoint are not extracted at the same time, the generated checkpoint is weakly consistent as a whole, that is, some parameters are already the values ​​of later iterations, while others are still the values ​​of earlier iterations. The parameter vector θ(t) represents the state of the model parameters at the end of the training step t. Assume that a stride checkpoint involves an iteration step with a span of K, starting from the tth step and completing at the t+K-1th step. Then the model parameters saved by this checkpoint can be expressed as:

[0046]

[0047] Among them, θ i (t i ) indicates that the i-th part of the model parameters is in the training step t i For strong consistency checkpoints, all t i are equal (both are checkpoint trigger step t), so Strictly corresponds to the overall model of a certain step. In the stride checkpoint, each t i Can be different, covering different time points from t to t+K-1.

[0048] Optionally, for heterogeneous stride checkpoints, to address the bucket wait effect, optimizations are performed at both the horizontal and vertical levels:

[0049] Horizontally between different computing nodes, each node is allowed to complete cold backup storage of local parameters at different iteration steps;

[0050] Vertically, within a node, parameter preservation is allowed across iterations. If the checkpoint save of all local parameters is not completed within the node, the calculation of the next iteration step is entered and the parameters are updated. The parameters that have not yet been saved are replaced with the updated versions.

[0051] In order to deal with the bucket effect caused by the backup delay caused by the heterogeneity of hardware and software in multiple nodes, the core design idea is to build heterogeneous stride checkpoints. The existing checkpoint saving mechanism requires that the saved content comes from the same iteration step, but due to the heterogeneity of hardware such as computing power, network, storage, and computing load, the saving delay of different nodes varies and changes dynamically. Figure 2 As shown in Figure A, at the iteration step S t When saving checkpoint ck1, nodes w1 and w3 are completed but w2 is delayed, resulting in a waste of computing power. t+n When saving checkpoint ck2, w1 becomes a bottleneck. To solve the bucket waiting effect, optimization is carried out from both horizontal and vertical levels. Horizontally, between different computing nodes, the consistency constraint of the checkpoint content belonging to the same iteration step is lifted, allowing each node to complete the cold backup of local parameters at different iteration steps; vertically, within the node, parameter preservation is allowed across iteration steps, that is, if all nodes have completed the calculation work of the current iteration step, and a node w i Not yet completed all local parameters θ i If the checkpoint is saved, the parameters are updated normally and the calculation of the next iteration step is entered. The parameters that have not been saved are directly replaced by the updated version. Figure 2 As shown in Figure B, w1 has a large delay in the iteration step S t Not finished at the end Complete preservation, enter the iteration step S t+1 back, Updated to Therefore, the parameter version that has not been saved is changed to w2 directly gives up on S t Save the data and postpone it to S t+n Save locally. w3 can save local data frequently due to its low latency, i.e. t and S t+n Therefore, the checkpoint Obviously, it is composed of local parameters saved by different nodes at different iteration steps.

[0052] Optionally, the stride checkpoint saving mechanism further includes a weak consistency error metric, including:

[0053] Assign a checkpoint version number to each node and calculate the version number difference. The formula is: ij =|δ i -δ j |;δi is the checkpoint version number of the i-th node, δ j is the checkpoint version number of the jth node, δ ij is the version number difference between node i and node j;

[0054] Define the weak consistency score as the maximum version number difference between all nodes, the maximum version number difference E max The formula is: w is the number of nodes;

[0055] When E max >∈, the system believes that there is an excessively large weak consistency error, which affects the training effect of the model, and adjusts the checkpoint saving strategy, including starting local checkpoint saving or adjusting the saving priority.

[0056] The difference in checkpoint parameter versions will obviously introduce weak consistency errors when restarting due to a fault. Although the self-correction ability of the iterative calculation itself can alleviate its impact, excessive weak consistency errors still seriously affect the convergence speed. To solve this problem, we first need to clarify how to measure the version differences between different nodes and within nodes. An intuitive measurement method is to characterize the iteration step of the model by the version number. The version number represents the progress of the model parameter update, and the difference in model versions on different nodes is the embodiment of weak consistency errors. For checkpoints that are being saved, the current iteration version number can be used directly as an identifier; for checkpoints that have not yet been saved, the most recently saved version number is used. For stride checkpoints, such as The larger iteration step t+1 is taken as its version number, that is, the version number of the stride checkpoint will be identified by the larger iteration step to maintain version consistency.

[0057] Optionally, the priority is adjusted to adjust the frequency of checkpoint saving or postpone the timing of checkpoint saving according to the node's latency. Specifically, at the beginning of the training task, the checkpoint analyzes several specific indicators that affect the training execution through the corresponding interface of the data iterator: the total checkpoint overhead time T ckpt , total training time T t , storage write time T s Based on the above weak consistency strategy, the following formula for weak consistency error ∈ is obtained:

[0058]

[0059] in, Reflects the impact of checkpoint overhead on training. If the checkpoint time is much longer than the training time, the ratio is close to 1, which means that frequent storage will affect the training progress. Otherwise, it means that the storage operation has little impact on training. γ is the impact strength, 1.0≤γ≤2.0; Reflects the impact of storage speed. When storage writing is slow, Ts becomes larger, causing ∈ to increase and relax the weak consistency threshold; It is used to measure the synchronization impact of multi-node training to prevent the checkpoint writing speed of some nodes from slowing down the overall training progress. If the checkpoint time of all nodes is close, then T ckpt,i / w is approximately equal to T ckpt , which is close to a constant.

[0060] In order to alleviate excessive weak consistency errors, the system can adjust the checkpoint saving strategy according to the current weak consistency score. If the error is too large, the training process can be optimized in the following ways. Start a new local checkpoint save: When it is detected that the weak consistency error of a node exceeds the threshold ∈, the node needs to start a new local checkpoint save operation to ensure that the locally saved parameters are consistent with the global model. Priority adjustment: For nodes with large delays, the time overhead of local checkpoint saving can be reduced through the priority adjustment mechanism. Specifically, the frequency of checkpoint saving can be adjusted or the timing of checkpoint saving can be postponed according to the node's delay to avoid excessive resource consumption.

[0061] The weak consistency error metric is designed to effectively control the inconsistency of parameter versions during the step checkpoint saving process, and experimentally sets the maximum version difference threshold E max The checkpoint frequency is used as the control basis for parameter version differences to effectively constrain the parameter version difference error introduced by stride saving.

[0062] 2. High-quality fault recovery mechanism based on multi-checkpoint cascading index

[0063] The stride checkpointing mechanism, based on controllable parameter consistency, focuses on the runtime overhead of checkpointing during deep learning training, effectively reducing this overhead and improving overall checkpoint save / restore efficiency. However, for models with large parameter sizes, while storage speeds have improved, the increasing number and frequency of checkpoints means that reading checkpoints during failure recovery incurs significant I / O overhead, impacting recovery efficiency. Securing fast recovery is also a key challenge for checkpoint failure recovery. Prioritized checkpointing is a common solution for reducing backup size, often used during the checkpoint save phase. Its key design principle is to only record parameter values ​​that have significantly changed since the last checkpoint, thereby reducing the amount of backup data. This approach offers significant advantages in reducing storage space, especially when a large number of parameters are updated at each checkpoint during training. However, prioritized checkpointing has significant limitations during failure recovery. When a failure occurs, the system typically rolls back to the most recent priority checkpoint and restarts the computation. Given that all parameters are constantly changing during iterative training, the content stored in each priority checkpoint is not exactly the same. Obviously, the priority checkpoint closest to the failure point cannot cover all parameters. Especially when model parameters change significantly during training, the most recent checkpoint may not store all necessary parameters. Furthermore, the priority checkpoint design ignores information from historical checkpoints and relies on the most recent checkpoint for recovery. This approach results in a prohibitively high recovery cost, especially in large-scale training tasks, where frequent failure recovery can waste significant computing resources and further burden the system.

[0064] To address these issues, this paper proposes a high-quality fault recovery strategy using a multi-checkpoint cascade index. This strategy leverages these historical checkpoints to optimize the recovery process in two ways: 1) Cascading multi-version priority checkpoints constructs hybrid restart points, improving parameter coverage and optimizing recovery efficiency; 2) Designing a timestamp-based multi-version checkpoint cascade index allows for rapid extraction of required data from multi-version cold backup checkpoints for restart point construction, while also guiding the deletion of redundant data between checkpoints to free up storage space.

[0065] Prioritized checkpointing can reduce the amount of data saved per checkpoint and lower overhead. At system startup, an initial checkpoint is saved to persistent storage. As training progresses, the system updates portions of the checkpoint whenever a certain number of iterations is reached or other saving conditions are met. Specifically, checkpoints do not save all parameters at each time, but instead selectively save parameters that have changed significantly. This selective saving not only reduces I / O operations but also optimizes storage space usage. Because checkpoints are continuously updated, a checkpoint at a given moment may contain a mixture of parameters from different iterations. This mixture means that a single checkpoint may store values ​​for different subsets of model parameters from multiple iterations. To effectively manage these parameters, the prioritized checkpoint design allows for policy-controlled selection of parameter subsets. For example, priority can be determined based on the frequency of parameter updates or the magnitude of change. The core idea of ​​the prioritized checkpointing policy is to determine which parameters to save based on the degree of change since the last save. Its goal is to minimize the perturbations caused by failures. In the event of a failure, restoring a checkpoint reduces the amount of recalculation required. A single prioritized checkpoint may not cover all parameters, especially in models with a large number of parameters. To improve parameter coverage, this section proposes a scheme for cascading multiple historical checkpoints to construct hybrid restart points. By cascading multiple historical checkpoints, it is possible to ensure that all necessary parameters can be obtained during recovery without relying on the integrity of a single checkpoint.

[0066] Optionally, the multi-checkpoint cascading index mechanism includes:

[0067] Construct a bitmap index matrix M, where rows correspond to checkpoints, columns correspond to parameter blocks, and matrix elements M ij Indicates block BLK j At checkpoint ck i whether it exists in

[0068] By traversing the matrix to locate the parameter block required for recovery, the parameters in the most recent checkpoint are obtained.

[0069] like Figure 3As shown, the contents recorded in the priority checkpoints ck1, ck2, ck3, and ck4 are different from each other, but none of them can cover all parameters θ. Taking the rollback to ck4 as an example, it is obvious that only the parameters in BK1 are covered, and the rest (BLK2-BLK4) can only be initialized to the original values, which introduces a lot of redundant calculations in retraining. In fact, this part of the content may have been saved in the other checkpoints. For example, the contents of BLK2-BLK4 exist in ck1-ck3 respectively. Although the parameter versions provided in these checkpoints are older, they can still significantly reduce redundant calculations compared to the original initial values. Therefore, all historical checkpoints are cascaded to build a restart point to improve parameter coverage. In this example, a restart point covering all parameters can be built through ck1-ck4.

[0070] The cascading multi-priority checkpoint mechanism proposed in this paper is a novel strategy for integrating historical information and building recovery integrity. Cascading refers to the fact that during the save phase, instead of relying on a single checkpoint to independently cover all parameters, different parameter versions are sequentially combined over time, leveraging the complementarity of multiple historical checkpoints to construct a logically complete set of checkpoints. This strategy inherits the storage space-saving advantages of priority checkpoints while also addressing coverage gaps during recovery through the cascaded utilization of historical information. Essentially, it transforms checkpoint save and recovery from the traditional "single-version full coverage" model to a "multiple-version on-demand splicing" model, significantly improving flexibility and recovery accuracy while tolerating loose consistency. The introduction of the cascading strategy not only improves the spatiotemporal locality of parameter access, but also reduces the data scan scope during recovery by cascading hotspot parameters from multiple adjacent checkpoints, thereby shortening access paths and I / O latency. It also provides a structural basis for subsequent parameter redundancy identification and storage reclamation. Because parameter blocks form a mapping relationship between multiple versions, the system can automatically analyze which parameter versions have been fully covered by subsequent checkpoints based on this mapping, implementing a more intelligent space reclamation mechanism. This structural optimization provides the technical foundation for subsequent index-driven recovery and dynamic pruning. However, cascading checkpoints still face two challenges in their implementation: First, the distribution of different parameters across checkpoints is poorly localized, and searching all historical checkpoints in each cold backup for a specific parameter incurs expensive I / O overhead. Second, checkpoint storage space consumption increases. Unlike traditional checkpoint strategies, which delete older versions after saving new ones, the cascading strategy requires retaining all historical checkpoints.

[0071] To solve the above problems, a timestamp-based multi-version checkpoint cascade index is designed to improve retrieval speed, accurately identify redundant data and delete it. The core idea of ​​cascade indexing is to quickly locate the parameters that need to be restored by dividing all parameters into blocks and recording the saved status of each block in different checkpoints in units of blocks. Each checkpoint corresponds to a time point, and each block corresponds to a spatial area of ​​the model parameters. By establishing a bitmap index matrix M, the temporal and spatial distribution information of the parameters can be recorded. Assume that the system divides all model parameters into N blocks, where each block BLK x Contains several parameters. At different checkpoints ck i In the example, each block may store different versions of data. A matrix M is used to record the saved state of each block at different checkpoints. The rows of the matrix M represent different checkpoints ck1, ck2, ..., ck k , columns represent different parameter blocks BLK1, BLK2,…, BLK N , where M ij Indicates block BLK j At checkpoint ck i For example, a specific entry M in the matrix M 41 = 1 indicates that BLK1 exists in checkpoint ck4. By traversing this matrix, the blocks required for recovery can be quickly identified. For example, if the recovery point requires BLK1, the system simply searches the column corresponding to BLK1 in the matrix and finds the checkpoint closest to the failure time to quickly locate and load the corresponding data. Unlike existing work that frequently saves parameters with significant changes, the prioritized checkpoint strategy does not write a fixed amount of data during each fixed number of iterations. Instead, it selectively saves parameters with significant changes based on their change relative to the previous checkpoint in each iteration. Specifically, at the end of each iteration, the system evaluates the magnitude of each parameter change. Parameters with significant changes are marked as requiring preservation, and only these parameters are written to persistent storage. Furthermore, to further optimize the recovery process, the system creates an index table during each iteration. This index table records which parameters have significantly changed in the current iteration compared to the last saved checkpoint. This indexing mechanism allows the system to quickly locate parameters requiring update in the next iteration based on the "proximity principle," saving only those parameters with significant changes. At the same time, the system will create a unique bucket for deduplication, which will retain the index position corresponding to each round number to ensure the rapid positioning and deduplication of parameter versions. Figure 4 The cascaded multi-priority checkpoint framework is shown, with u0 indicating that the initial checkpoint is empty, and u tIndicates the storage status of the current checkpoint parameters. It can be seen from the figure that the composition of the checkpoint at a certain moment is sparse, that is, the checkpoint parameters are distributed in some positions, and the remaining positions store a small amount of parameters that cannot affect the overall model training quality. Figure 4 It can be seen that The parameter position of the priority checkpoint at time t is represented by the t parameter. The framework mainly performs two steps: 1) establishing the priority checkpoint index when writing, and 2) cascading multiple priority checkpoints for joint recovery when reading, and finally forming a complete checkpoint.

[0072] Optionally, a checkpoint pruning strategy is also included, specifically:

[0073] Each parameter block is subdivided into sub-blocks, a meta-index is established, and the preservation status of each sub-block is recorded;

[0074] Calculate the redundancy ρ of the same parameter block in different checkpoints. The formula is: The number of intersection sub-blocks is the number of sub-blocks stored in the meta-indexes corresponding to the two checkpoints, and the number of union sub-blocks is the total number of sub-blocks stored in the meta-indexes corresponding to the two checkpoints.

[0075] When ρ>∈, delete the old version parameter block.

[0076] Although the above indexing method can efficiently locate parameter blocks, its granularity is relatively coarse and it is difficult to reflect the specific changes in the parameters within each block. For example, block BLK1 is saved in both checkpoints ck2 and ck4, but the parameters they save may have large redundancy. If you only rely on block-level indexes, you cannot judge the specific impact of these redundancies. In order to accurately evaluate redundant data and decide whether to delete old versions, the parameters within the block are further fine-grained and a meta-index is established. The role of the meta-index is to record the preservation status of each sub-block in each block, that is, whether each sub-block has parameters saved in the current checkpoint. In this way, it is possible to accurately determine whether the blocks in the two checkpoints have high redundancy. For example, suppose the meta-index of BLK1 in ck4 is Indicates that the block is divided into four sub-blocks, where sub-blocks 1 and 2 have parameters stored, while sub-blocks 3 and 4 do not store any data. Similarly, assuming that the meta index of BLK1 in ck2 is Indicates that there is data stored in sub-blocks 1 and 3, but not in sub-blocks 2 and 4. By comparing these meta-indices, the redundancy of BLK1 in ck2 and ck4 can be quickly calculated. If the redundancy is high (for example, the overlap exceeds a given threshold ∈), the old version can be deleted to free up storage space.

[0077] Redundancy evaluation is to determine which checkpoint versions can be deleted by calculating the overlap of parameters saved in different checkpoints. Assume that the meta-indexes of blocks BLK1 corresponding to checkpoints ck4 and ck2 are and The redundancy ρ is calculated by the formula. For example, Figure 5 In the example, and The intersection of is sub-block 1, and the number of sub-blocks in the intersection is 1; the number of sub-blocks in the union is 3 (sub-blocks 1, 2, and 3), so ρ = 1 / 3. If ρ exceeds the preset threshold ∈, the BLK1 version in checkpoint ck2 can be deleted. Otherwise, an appropriate version is selected based on timeliness and coverage.

[0078] Optionally, the multi-checkpoint cascade indexing mechanism further includes timeliness evaluation, which performs a weighted evaluation on different versions of the parameter block. The formula is:

[0079] V BLK =α·S x +(1-α)·S y ;

[0080] Where S x 、S y Checkpoint ck x and ck y The number of iteration steps is α∈[0,1], and α∈[0,1] is the weight factor.

[0081] Timeliness evaluation refers to the weighted evaluation of the importance of each block in different checkpoints, usually by the number of checkpoint iterations S x Assume that the blocks BLK1 corresponding to checkpoints ck4 and ck3 have different timeliness, and their evaluation values ​​V BLK1 Calculated by the following formula:

[0082]

[0083] Here, α is a weighting factor, and S4 and S3 are the number of iterations of checkpoints ck4 and ck3, respectively. This evaluation value measures the quality of different block versions and helps the system select the optimal version for restart point construction. For example, if α is 50%, the evaluation of block BLK1 will be affected by 50% S4 and 50% S3 backup blocks. Through timestamp-based cascading indexing, the system can quickly locate required data blocks across multiple historical checkpoints, avoiding repeated reading of redundant data. Furthermore, through redundancy assessment and timeliness weighting, it is possible to accurately identify and delete checkpoint data with high redundancy, effectively freeing up storage space.

[0084] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A weak consistency checkpoint storage and recovery method for deep learning fault tolerance, characterized by: include: A stride checkpoint saving mechanism based on controllable parameter consistency and a fault recovery mechanism based on multi-checkpoint cascade indexing; wherein, the stride checkpoint saving mechanism removes the coupling dependency between the checkpoint version and the iteration step, divides the model parameter saving process into several sub-processes, and executes them asynchronously in multiple consecutive iteration steps, so that the checkpoint contains parameters of different iteration steps, forming a weak consistency snapshot; the multi-checkpoint cascade indexing mechanism uses historical checkpoints to construct hybrid restart points, quickly locates parameter blocks through timestamp indexing, and deletes redundant data based on redundancy assessment.

2. A weak consistency checkpoint storage and recovery method for deep learning fault tolerance according to claim 1, characterized in that: The stride checkpoint saving mechanism includes: dividing the model parameters into several subsets by layer or block, and saving each part of the parameters in sequence in multiple consecutive iteration steps to form a complete checkpoint; wherein, each iteration step saves part of the parameters, and the final checkpoint is composed of the parameters of multiple iteration steps.

3. A weak consistency checkpoint storage and recovery method for deep learning fault tolerance according to claim 2, characterized in that: To address the bucket wait effect for heterogeneous stride checkpoints, we optimize both horizontally and vertically: Horizontally between different computing nodes, each node is allowed to complete cold backup storage of local parameters at different iteration steps; Vertically, within a node, parameter preservation is allowed across iterations. If the checkpoint save of all local parameters is not completed within the node, the calculation of the next iteration step is entered and the parameters are updated. The parameters that have not yet been saved are replaced with the updated versions.

4. A method for saving and restoring weak consistency checkpoints for deep learning fault tolerance according to claim 3, characterized in that: The stride checkpoint saving mechanism also includes weak consistency error metrics, including: Assign a checkpoint version number to each node and calculate the version number difference. The formula is: ij =|δ i -δ j |;δ i is the checkpoint version number of the i-th node, δ j is the checkpoint version number of the j-th node; Define the weak consistency score as the maximum version number difference between all nodes, the maximum version number difference E max The formula is: w is the number of nodes; When E max >∈, the checkpoint saving strategy is adjusted, including starting local checkpoint saving or adjusting the saving priority.

5. A method for saving and restoring weak consistency checkpoints for deep learning fault tolerance according to claim 4, characterized in that: Priority adjustment is to adjust the frequency of checkpoint saving or postpone the timing of checkpoint saving according to the delay of the node, specifically: according to the total checkpoint overhead time T ckpt , total training time T t , storage write time T s And the number of nodes w, dynamically adjust the consistency threshold, the formula is: Where, Reflects the impact of checkpoint overhead on training. If the checkpoint time is much longer than the training time, the ratio is close to 1, which means that frequent storage will affect the training progress. Otherwise, it means that the storage operation has little impact on training. γ is the impact strength, 1.0≤γ≤2.0; Reflects the impact of storage speed. When storage writing is slow, T s becomes larger, causing ∈ to increase and relax the weak consistency threshold; It is used to measure the synchronization effect during multi-node training to prevent the checkpoint writing speed of some nodes from slowing down the overall training progress. If the checkpoint time of all nodes is close, then T ckpt,i / w is approximately equal to T ckpt , which is close to a constant.

6. A weak consistency checkpoint storage and recovery method for deep learning fault tolerance according to claim 1, characterized in that: The multi-checkpoint cascading index mechanism includes: Construct a bitmap index matrix M, where rows correspond to checkpoints, columns correspond to parameter blocks, and matrix elements M ij Indicates block BLK j At checkpoint ck i whether it exists in By traversing the matrix to locate the parameter block required for recovery, the parameters in the most recent checkpoint are obtained.

7. A method for saving and restoring weak consistency checkpoints for deep learning fault tolerance according to claim 6, characterized in that: It also includes checkpoint pruning strategies, specifically: Each parameter block is subdivided into sub-blocks, a meta-index is established, and the preservation status of each sub-block is recorded; Calculate the redundancy ρ of the same parameter block in different checkpoints. The formula is: The number of intersection sub-blocks is the number of sub-blocks stored in the meta-indexes corresponding to the two checkpoints, and the number of union sub-blocks is the total number of sub-blocks stored in the meta-indexes corresponding to the two checkpoints. When ρ>∈, delete the old version parameter block.

8. A method for saving and restoring weak consistency checkpoints for deep learning fault tolerance according to claim 6, characterized in that: The multi-checkpoint cascade indexing mechanism also includes timeliness evaluation, which performs weighted evaluation on different versions of parameter blocks. The formula is: V BLK =α·S x +(1-α)·S y ; Where S x 、S y Checkpoint ck x and ck y The number of iteration steps is α∈[0,1], and α∈[0,1] is the weight factor.

Images