Distributed training memory checkpoint recovery method and system based on locality-aware erasure code

Abstract

The application discloses a kind of based on locality perception erasure code distributed training memory checkpoint recovery method and system, mainly solve the problem of large network transmission overhead, low recovery efficiency of existing distributed training checkpoint recovery, its scheme is: the training node in distributed training cluster is divided into several node groups, each group contains pre-set quantity node;Checkpoint data is divided into multiple data blocks as source block and stored in local on each node;Coding task allocation relationship and cross-node striping distribution of source block are constructed in each node group, determine the source block set corresponding to each coding task according to mutual orthogonal latin square;Each node executes exclusive-or operation on source block to generate check block, and save in local, form locality-aware erasure code protection mechanism;When node failure, checkpoint data is read from local memory of failure node and recovered to training equipment preferentially.The application can realize local recovery, reduce cross-node transmission, improve fault tolerance and efficiency, and can be used for large-scale deep language model training.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and specifically relates to a distributed training memory checkpoint recovery method and system, which can be used for large-scale deep language model training. Background Technology

[0002] With the rapid development of large-scale deep language models, distributed training has become the primary method for model training. Because the training process is typically lengthy and involves a large number of computing nodes, software and hardware failures are prone to occur during training, leading to training interruptions and significant waste of computing resources. Therefore, a checkpointing mechanism is needed to periodically save the training state so that training can be quickly resumed after a failure.

[0003] In existing technologies, traditional checkpointing schemes typically store model parameters, optimizer states, and other data in remote storage. While this approach achieves training state persistence, in large-scale distributed training scenarios, the large volume of checkpoint data makes the recovery process susceptible to limitations in network bandwidth and storage I / O performance, resulting in high recovery latency. To improve recovery efficiency, existing technologies have proposed in-memory checkpointing schemes, storing checkpoint data in the node host's memory and achieving fault tolerance through cross-node replication. This scheme leverages the high bandwidth of memory to improve checkpoint read / write efficiency, but replication incurs significant memory overhead and has limited fault tolerance. To address the issues of high memory overhead and limited fault tolerance in replication, existing technologies have further proposed using erasure coding for redundant protection of in-memory checkpoints to improve fault tolerance with lower storage overhead. However, existing erasure coding schemes typically do not adequately consider the access locality problem in distributed training fault recovery. In software fault recovery scenarios, checkpoint data often needs to be obtained from remote nodes, resulting in high network transmission overhead and affecting recovery speed. In hardware fault recovery scenarios, erasure coding recovery usually requires reading data from multiple nodes and reconstructing it, which also brings high network traffic and recovery overhead.

[0004] Patent document CN202510760701.3 discloses a data recovery method, apparatus, computer equipment, computer-readable storage medium, and computer program product. It employs a grouping and verification mechanism, achieving data recovery by decoding the target verification data block and the target storage data block. While this method can improve recovery efficiency under ideal conditions, in practical distributed checkpoint applications, it consumes significant network resources and lacks adaptability to hotspot data access, significantly diminishing its practical application value.

[0005] In summary, existing technologies still struggle to simultaneously achieve high fault tolerance, low storage overhead, and fast recovery performance, especially in meeting the practical needs for localized checkpoint recovery and low network overhead recovery in large-scale distributed training scenarios. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the prior art by proposing a distributed training memory checkpoint recovery method and system based on locality-aware erasure coding, so as to reduce cross-node data transmission overhead during fault recovery, reduce network recovery traffic during fault recovery, and improve the recovery efficiency and reliability of the distributed training system.

[0007] The technical approach to achieving the objective of this invention is as follows: by using locality-aware erasure coding based on orthogonal Latin square OLS and node-aware placement, the cross-node data transmission overhead and network recovery traffic during the fault recovery process are reduced; by utilizing the OLS attribute reuse algorithm on the surviving nodes to read the surviving common block information, the fault tolerance and recovery reliability of the distributed training system are improved.

[0008] Based on the above ideas, the technical solution of the present invention includes the following:

[0009] 1. A distributed training memory checkpoint recovery method based on locality-aware erasure coding, characterized in that it includes:

[0010] (1) Divide multiple training nodes in the distributed training cluster into several node groups, each node group including a preset number of training nodes, as the basic unit for local encoding and recovery;

[0011] (2) On each training node, the training checkpoint data to be saved is divided into multiple data blocks, and these data blocks are used as source data blocks for erasure coding and stored in the memory of the corresponding training node.

[0012] (3) Construct the coding task allocation relationship and cross-node striping distribution relationship of the source data blocks corresponding to the training nodes in each node group, and determine the source data block set corresponding to each coding task based on the preset mutually orthogonal Latin square.

[0013] (4) Each training node performs an XOR operation on the source data blocks in the source data block set according to the corresponding coding task to generate a check block, and saves it in the local memory of the corresponding training node to form a locality-aware erasure coding protection structure and a memory checkpoint based on erasure coding protection.

[0014] (5) When a training node is detected to be faulty, the checkpoint data required for recovery is read from the local memory of the faulty node first, and then restored to the corresponding training device to realize localized recovery of node faults and reduce cross-node data transmission overhead.

[0015] (6) After the checkpoint recovery is completed, the recovered model parameters, optimizer state and related training state are reloaded into the corresponding training environment and the distributed training task continues to be executed.

[0016] Furthermore, in step (3), the encoding task allocation relationship corresponding to the training node and the cross-node bar of the source data block are constructed.

[0017] The banding distribution relationship, its implementation includes:

[0018] Set two within each node group mutually orthogonal Latin squares and ;

[0019] For the A mutually orthogonal Latin square The first node in the node group Each training node performs its corresponding encoding task. That is, coding task The source data blocks are obtained through mutually orthogonal Latin squares. Data blocks from multiple different training nodes within a node group are selected across nodes to form a cross-node striped distribution. , where n is the number of training nodes.

[0020] Furthermore, in step (3), the set of source data blocks corresponding to each encoding task is determined according to a preset mutually orthogonal Latin square.

[0021] Its implementation includes:

[0022] For coding tasks Read mutually orthogonal Latin squares The The element value in row +1 and column j is used as the target index, where k = n - 2;

[0023] exist In the first k rows, select all cells whose element values ​​are the same as the target index. Based on the source data block number corresponding to the row number and the training node where the source data block corresponding to the column number is located, determine the corresponding source data block set. ,in It is the kth data block.

[0024] Furthermore, in step (4), each training node, according to its corresponding encoding task, processes the source data blocks in the source data block set.

[0025] The data block is XORed to generate a check block, the implementation of which includes:

[0026] For the source data block set Perform an XOR operation on each source data block to generate a corresponding check block. ,in ;

[0027] Each training node generates at least two verification blocks, corresponding to... and The encoding task is performed, and the verification block is stored in the local memory of the corresponding training node.

[0028] 2. A distributed training memory checkpoint recovery system based on locality-aware erasure coding, characterized in that it comprises:

[0029] The node grouping module is used to divide multiple training nodes in a distributed training cluster into several node groups. Each node group contains n training nodes, where n=4 or n is a prime number greater than 4.

[0030] The data partitioning module is used to divide the training checkpoint data to be saved into multiple data blocks on each training node and store them in the memory of the corresponding training node.

[0031] The encoding construction module is used to construct the cross-node striped distribution relationship of source data blocks and the encoding task allocation relationship corresponding to the training nodes within each node group, and to determine the source data block set corresponding to each encoding task based on the preset mutually orthogonal Latin square.

[0032] The verification generation module is used to enable each training node to perform an XOR operation on the source data blocks in the source data block set according to the corresponding encoding task to generate a verification block, and store it in the local memory of the corresponding training node.

[0033] The fault recovery module is used to read the checkpoint data required for recovery from the local memory of the faulty node when a fault is detected, and then restore it to the corresponding training device to ensure that subsequent training can continue.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] Firstly, by introducing a locality-aware erasure coding construction method and a node-aware data placement method, this invention can improve access locality during software fault recovery, enabling the data required for recovery to be obtained first from the local node, thereby effectively reducing cross-node communication overhead and improving recovery speed.

[0036] Secondly, because the present invention uses erasure coding redundancy protection instead of simple copying, it can provide higher fault tolerance with lower storage overhead, thereby improving the reliability of the distributed training system.

[0037] Third, because the present invention adopts hardware fault recovery based on shared source blocks, it can reduce repeated reading of the same source data blocks during the fault recovery process, thereby reducing network recovery traffic and recovery latency.

[0038] Fourth, by utilizing mutually orthogonal Latin squares to construct the relationship between the coding task and the stripe distribution, this invention can achieve a balanced distribution of checkpoint data and coding tasks among multiple training nodes, thereby improving the parallelism and overall execution efficiency of the checkpoint saving stage.

[0039] Simulation results show that the present invention is applicable to large-scale distributed training scenarios and can meet the requirements for high reliability, low redundancy and fast recovery during large model training. Attached Figure Description

[0040] Figure 1 This is a flowchart of the distributed training memory checkpoint recovery method based on locality-aware erasure coding according to the present invention.

[0041] Figure 2 This is a schematic diagram of training node grouping and locality-aware coding in the method of the present invention;

[0042] Figure 3 This is a block diagram of the distributed training memory checkpoint recovery system based on locality-aware erasure coding of the present invention.

[0043] Figure 4 This is a comparison chart of the checkpoint saving performance of the present invention and four existing checkpoint methods on models of different scales;

[0044] Figure 5 This is a comparison chart of the software fault recovery time of the present invention and four existing checkpoint methods on different scale models;

[0045] Figure 6 This is a comparison chart of hardware fault recovery time of the present invention and four existing checkpoint methods on different scale models;

[0046] Figure 7 This is a comparison chart showing the time wasted per iteration between the present invention and four existing checkpoint methods on models of different scales. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0048] Example 1: Distributed training memory checkpoint recovery method based on locality-aware erasure coding.

[0049] Reference Figure 1 The implementation steps of this example include the following:

[0050] Step 1. Locality-aware coding.

[0051] To overcome the difficulty of simultaneously achieving high fault tolerance, low storage overhead, and fast recovery performance in existing technologies, this step improves the fault tolerance and recovery reliability of the distributed training system by using locality-aware erasure coding based on orthogonal Latin squares and node-aware placement.

[0052] The orthogonal Latin square is a special mathematical structure that is widely used in combinatorics. This example uses this special mathematical structure for locality-aware encoding.

[0053] Reference Figure 2 The implementation of this step is as follows:

[0054] 1.1) Grouping of training nodes:

[0055] The training nodes are divided into multiple node groups, each containing n training nodes, and each node group executes independently.

[0056] Checkpoint fragmentation, encoding, and recovery operations, where n=4 or n is a prime number greater than 4;

[0057] 1.2) Checkpoint segmentation:

[0058] Each training node divides the training checkpoint data to be saved into multiple data blocks;

[0059] Based on the number of training nodes n in each node group, each training checkpoint to be saved is divided into n-2 numbers.

[0060] According to the block And stored in the CPU memory of the corresponding training node, where Indicates the training node number. This indicates the data block number in the training node;

[0061] 1.3) Constructing mutually orthogonal Latin squares:

[0062] Two nodes are set within each node group. mutually orthogonal Latin squares and ;

[0063] For the A mutually orthogonal Latin square The first node in the node group Each training node performs its corresponding encoding task. Encoding

[0064] Task The source data blocks are obtained through mutually orthogonal Latin squares. Data blocks from multiple different training nodes within a node group are selected across nodes to form a cross-node striped distribution. n is the number of training nodes;

[0065] 1.4) Selecting the source data block set based on mutually orthogonal Latin squares:

[0066] For coding tasks Read mutually orthogonal Latin squares The The element value in row +1 and column j is used as the target index.

[0067] Where k = n - 2;

[0068] exist In the first k rows, select all cells whose element values ​​are the same as the target index, and then sort them according to the row number of that cell.

[0069] The corresponding source data block number and column number are used to determine the training node where the source data block is located, thus identifying the corresponding set of source data blocks. ,in It is the kth data block;

[0070] 1.5) Perform an XOR operation on the source data block to generate a check block:

[0071] For the source data block set Perform an XOR operation on each source data block to generate a corresponding checksum.

[0072] piece Where k = n - 2;

[0073] Each training node generates at least two verification blocks, corresponding to... and The encoding task is performed, and the verification block is stored in the local memory of the corresponding training node.

[0074] Step 2, fault detection on training nodes.

[0075] Each training node works together to execute training tasks;

[0076] A worker agent runs on each training node to monitor the health status of the node and periodically send heartbeat messages to the master node;

[0077] A root agent is run on the main training node to receive heartbeats, detect heartbeat timeouts, and trigger recovery procedures. If the root agent does not receive a heartbeat from a node within a set time, it is determined that the node has failed.

[0078] Step 3. Determine the type of fault.

[0079] 3.1) The root agent classifies the fault as a software fault or a hardware fault based on the network reachability after the heartbeat timeout, the error logs reported by the node, and the hardware health status.

[0080] 3.2) Based on the above classification criteria, the fault type shall be determined according to the following rules:

[0081] If the node is still accessible via the network but the training process has exited, or if the error log indicates a software malfunction, such as a CUDA error, then it is determined to be a software failure, and step 4 is executed.

[0082] If a node is completely inaccessible from the network, or if hardware monitoring detects hardware anomalies such as GPU damage or network interruption, it is determined to be a single-node hardware failure, and step 5 is executed.

[0083] If no fault is detected, return to step 2 to continue training.

[0084] Step 4. Software fault recovery.

[0085] Restart the training process of the failed node and retain its original rank ID in the cluster. This rank ID is assigned a globally unique integer number to each training process in distributed training and is used by the communication library to identify the process, divide the training data into shards, and index checkpoint data.

[0086] Based on the locality-aware coding construction adopted in step 1, all the original data blocks required by the faulty node are kept in the node's own CPU memory. Without any cross-node network reading, all data blocks can be read directly from the local CPU memory.

[0087] Data blocks are copied from the local CPU memory of the faulty node to its GPU memory via the PCIe link at a speed three times faster than the cross-node network bandwidth, completing the software fault recovery and significantly shortening the recovery time.

[0088] Step 5. Hardware failure recovery.

[0089] When a single-node hardware failure occurs, the failed node needs to be replaced, and the lost data blocks and checksum blocks of that node need to be reconstructed. Existing methods for hardware failure recovery include Gemini, ECCheck, and NaiveEC. This example employs, but is not limited to, a shared block recovery algorithm, utilizing mutually orthogonal Latin squares. and To minimize cross-node reads, the specific implementation is as follows:

[0090] 5.1) Identify the blocks to be recovered: The faulty node has a total of k+2 blocks that need to be recovered, including k=n-2 data blocks. } and 2 check blocks };

[0091] 5.2) Indexing source blocks and deduplicating merges: using mutually orthogonal Latin squares and The source blocks required for the check block and data block indexes are respectively, and after deduplication and merging, the smallest set of source blocks L is obtained;

[0092] 5.3) Remote source block reading: Read all source blocks in L from other surviving nodes in the group and merge them through sharing so that the total number of reads is less than other existing methods;

[0093] 5.4) XOR Decoding: Using lightweight XOR operations to decode and recover all data blocks and parity blocks of the faulty node;

[0094] 5.5) CPU-to-GPU copy: The recovered data blocks are copied to the GPU memory of the replacement node via the PCIe link. At the same time, all recovered data blocks and parity blocks are also kept in the CPU memory to maintain subsequent fault tolerance.

[0095] Step 6. Continue training.

[0096] The recovered model parameters, optimizer state, and related training state are reloaded into the corresponding training environment to continue the distributed training task. The training process resumes from the most recent checkpoint without rolling back to an earlier iteration.

[0097] It should be noted that the step numbers in this example and the claims are only for the purpose of clearly and completely describing the embodiments of the present invention and for ease of understanding, and their order is not limited.

[0098] Example 2: Distributed training memory checkpoint recovery system based on locality-aware erasure coding.

[0099] Reference Figure 3 This example includes: a node grouping module 1, a data partitioning module 2, an encoding construction module 3, a verification generation module 4, and a fault recovery module 5. The encoding construction module 3 includes a Latin square construction submodule 31, an encoding task allocation submodule 32, and a source block selection submodule 33. The verification generation module 4 includes an XOR operation submodule 41 and a verification storage submodule 42. The fault recovery module 5 includes a software fault recovery submodule 51 and a hardware fault recovery submodule 52.

[0100] The working principle of the entire system is as follows:

[0101] The node grouping module 1 divides all training nodes in the distributed training cluster into several node groups.

[0102] Each node group contains n training nodes, where n=4 or n is a prime number greater than 4.

[0103] The data partitioning module 2 is used to further divide the training checkpoint data to be saved on each training node into n-2 data blocks within each node group divided by the node grouping module, and store them in the memory of that training node.

[0104] The encoding construction module 3 is used to construct the cross-node striped distribution relationship of source data blocks and the encoding task allocation relationship corresponding to the training nodes within each node group, and to determine the source data block set corresponding to each encoding task based on a preset mutually orthogonal Latin square. The Latin square construction submodule 31 generates two... mutually orthogonal Latin squares and The coding task allocation submodule 32 is used for each Latin square. Each node below Assign a task code This indicates that a check block is generated on node j, where , The source block selection submodule 33 reads the index according to the indexing rules of the orthogonal Latin square. The The element value in row +1, column j is used as the target index, and... In the first k rows, select all cells whose element values ​​are the same as the target index, use the row number of the cell as the data block number, and the column number as the node number, thereby determining the source data block set. This forms a cross-node striped distribution relationship, where k=n-2.

[0105] The verification generation module 4 is used to utilize the aforementioned source block set Generate a verification block, in which the XOR operation submodule 41 performs a check on each Perform an XOR operation on all source blocks in the array to obtain the check block. ;Verification storage submodule 42 will verify the block The data is saved to the local CPU memory of the corresponding training node j, and together with the original data blocks of that node, it forms a locality-aware erasure code.

[0106] The fault recovery module 5 continuously monitors the health status of each node. When a training node fault is detected, it prioritizes reading the checkpoint data required for recovery from the faulty node's local memory and then restores it to the corresponding training device to ensure the continuation of subsequent training, for example, through a heartbeat mechanism. When a node fault is detected, different recovery processes are executed according to the fault type. Specifically, when a software fault is detected, the software fault recovery submodule 51, based on locality-aware coding, directly reads all the original data blocks from the faulty node's local CPU memory and then copies the data blocks from the CPU memory to the GPU memory via the PCIe link to complete the localized recovery of the software fault. When a hardware fault is detected, the hardware fault recovery submodule 52 executes a shared block recovery algorithm to determine the block to be recovered, including k data blocks and 2 check blocks, using mutually orthogonal Latin squares. and The required source blocks are indexed separately, deduplicated, and merged to obtain the minimum source block set L. All source blocks in L are read from other surviving nodes in the group, and XOR decoding is performed to recover all data blocks and parity blocks of the faulty node. The recovered data blocks are then copied to the GPU memory of the replacement node through the PCIe link and kept in the CPU memory to maintain subsequent fault tolerance.

[0107] It should be noted that the above functional modules can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as a program instruction product. A program instruction product includes one or a set of program instructions. When the program instructions are loaded and executed on a computer, the described process or function is generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable and writable storage medium, or transferred from one computer's readable and writable storage medium to another.

[0108] In this embodiment, the direct coupling or communication connection between the modules can be achieved through indirect coupling or communication connection via interfaces, devices, or modules. The functional modules and sub-modules in this embodiment can dynamically reside within a single processing unit, or each module can exist physically independently, or two or more modules can dynamically reside within a single processing unit. When these dynamic components are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable and writable storage medium. This storage medium can be a memory, disk, or optical disc, etc.

[0109] The effects of this invention can be further illustrated by the following simulation experiments:

[0110] I. Simulation Conditions

[0111] The simulation experiments were conducted on a distributed training framework based on Megatron-LM, where:

[0112] The hardware environment consists of a cluster with four training nodes, each equipped with an NVIDIA A10 GPU (24GB VRAM) and 64GB of system memory, and the nodes are interconnected via a 32Gbps network.

[0113] The software environment includes Ubuntu 20.04, CUDA 12.4, and PyTorch 2.6.

[0114] Three representative deep learning models were used: GPT-2, BERT, and OPT, with three parameter scales set: 335M, 1.5B, and 2.7B respectively.

[0115] The training dataset is the WikiText corpus.

[0116] II. Simulation Content

[0117] Simulation 1: Under the above simulation conditions, the present invention and the existing copying memory checkpointing method Gemini are compared respectively.

[0118] The performance of saving checkpoints during training of models of different sizes is simulated using the erasure coding memory checkpointing method ECCheck, the traditional erasure coding method NaiveEC, and the remote storage method CheckFreq. The results are as follows: Figure 4 As shown.

[0119] from Figure 4 It is evident that the present invention and existing memory checkpointing schemes are significantly superior to the remote storage method CheckFreq;

[0120] The checkpoint saving time of this invention is comparable to ECCheck, but slightly longer than Gemini and NaiveEC. This is because this invention introduces an encoding structure based on mutually orthogonal Latin squares during the saving phase, incurring a certain saving time overhead in exchange for stronger fault tolerance and faster recovery speed. Moreover, this overhead is within the design expectations and is still much lower than remote storage methods.

[0121] Simulation 2: Under the above simulation conditions, the present invention and the existing copying memory checkpointing method Gemini are compared respectively.

[0122] Simulations were performed on the recovery time of the erasure coding memory checkpoint method ECCheck, the traditional erasure coding method NaiveEC, and the remote storage method CheckFreq after software failures during training of models of different scales. The results are as follows: Figure 5 As shown.

[0123] from Figure 5As can be seen, the software fault recovery time of this invention is comparable to that of Gemini based on replication, and is significantly better than NaiveEC, ECCheck, and the remote storage method CheckFreq. This is because this invention uses a locality-aware coding structure, where all the original data blocks required during a software fault are retained in the CPU memory of the faulty node itself. The recovery process does not require any cross-node network reads; it can be directly copied from the CPU to the GPU via the PCIe link.

[0124] As the model size increases from 335M to 2.7B, the recovery times of CheckFreq, NaiveEC, and ECCheck all increase significantly, while the recovery times of this invention and Gemini increase slowly, mainly depending on the PCIe copy bandwidth and the amount of checkpoint data. Therefore, the advantages of this invention are more prominent in large-scale models. The simulation results show that this invention is far superior to existing erasure coding memory checkpointing schemes in terms of software fault recovery performance, comparable to the optimal copying scheme, and has higher storage efficiency and fault tolerance.

[0125] Simulation 3: Under the above simulation conditions, the present invention and the existing copying memory checkpointing method Gemini were compared respectively.

[0126] The recovery time of the erasure coding memory checkpointing method ECCheck, the traditional erasure coding method NaiveEC, and the remote storage method CheckFreq were simulated after a single-node hardware failure during training of models of different scales. The results are as follows: Figure 6 As shown.

[0127] from Figure 6 As can be seen, the hardware fault recovery time of this invention is reduced by 1.9 to 2.2 times compared to CheckFreq, improved by 1.3 to 1.5 times compared to NaiveEC, and comparable to ECCheck recovery performance. This is because this invention employs a shared block recovery algorithm, utilizing mutually orthogonal Latin square indexes of source blocks and deduplicating and merging them, reducing the number of source blocks read across nodes, thereby reducing network traffic and recovery overhead. As the model size increases, the recovery time of all methods increases, but the recovery time of this invention remains lower than that of CheckFreq and NaiveEC, and remains at the same level as ECCheck. The simulation results demonstrate that this invention outperforms remote storage and traditional erasure coding schemes in hardware fault recovery performance, is on par with the best existing erasure coding memory checking scheme, and has significant advantages in other aspects such as software fault recovery and fault tolerance.

[0128] Simulation 4: Under the above simulation conditions, the present invention and the existing copying memory checkpointing method Gemini are compared respectively.

[0129] Simulations of the erasure coding memory checkpointing method ECCheck, the traditional erasure coding method NaiveEC, and the remote storage method CheckFreq are performed, considering the time wasted in a single iteration when training models of different sizes and the occurrence of faults. The results are as follows: Figure 7 As shown.

[0130] from Figure 7 As can be seen, the time wasted in this invention is significantly lower than that of existing erasure coding schemes. It reduces time by 52.1%-61.0% compared to NaiveEC and by 53.2%-62.5% compared to ECCheck. This is because this invention employs a locality-aware coding structure, which allows direct reading of data blocks from local CPU memory during software failures, completely avoiding cross-node network transmission and achieving localized rapid recovery. Simultaneously, it effectively reduces recovery traffic during hardware failures by utilizing a shared block recovery algorithm. Simulation results demonstrate that this invention, through locality-aware coding and a shared block recovery mechanism, can shorten recovery time in both software and hardware failure scenarios, thereby reducing wasted time in combined failure scenarios and improving the overall efficiency of distributed training.

[0131] In summary, this invention achieves both localized rapid recovery of software faults and low-overhead recovery of hardware faults under acceptable checkpoint storage overhead. Its overall efficiency is significantly better than existing erasure coding schemes, and it has fault tolerance and storage efficiency comparable to or even better than replication schemes. It provides a high-performance and highly reliable memory checkpoint recovery scheme for large-scale distributed training.

Claims

1. A distributed training memory checkpoint recovery method based on locality-aware erasure coding, characterized in that, include: (1) Divide multiple training nodes in the distributed training cluster into several node groups, each node group including a preset number of training nodes, as the basic unit for local encoding and recovery; (2) On each training node, the training checkpoint data to be saved is divided into multiple data blocks, and these data blocks are used as source data blocks for erasure coding and stored in the memory of the corresponding training node. (3) Construct the coding task allocation relationship and cross-node striping distribution relationship of the source data blocks corresponding to the training nodes in each node group, and determine the source data block set corresponding to each coding task based on the preset mutually orthogonal Latin square. (4) Each training node performs an XOR operation on the source data blocks in the source data block set according to the corresponding coding task to generate a check block, and saves it in the local memory of the corresponding training node to form a locality-aware erasure coding protection structure and a memory checkpoint based on erasure coding protection. (5) When a training node is detected to be faulty, the checkpoint data required for recovery is read from the local memory of the faulty node first, and then restored to the corresponding training device to realize localized recovery of node faults and reduce cross-node data transmission overhead. (6) After the checkpoint recovery is completed, the recovered model parameters, optimizer state and related training state are reloaded into the corresponding training environment and the distributed training task continues to be executed.

2. The method according to claim 1, characterized in that: In (1), several node groups are divided according to the method that each node group contains n training nodes. Each node group independently performs checkpoint slicing, encoding and recovery operations, where n=4 or n is a prime number greater than 4. In step (2), the training checkpoint data to be saved is divided into multiple data blocks at each training node. This is based on the number of training nodes n in each node group, dividing each training checkpoint to be saved into n-2 data blocks. ,in Indicates the training node number. This indicates the data block number in the training node.

3. The method according to claim 1, characterized in that, In (3), the encoding task corresponding to the training node is constructed. The implementation of the task allocation relationship and the cross-node striping distribution relationship of source data blocks includes: Set two within each node group mutually orthogonal Latin squares and ; For the A mutually orthogonal Latin square The first node in the node group Each training node performs its corresponding encoding task. That is, coding task The source data blocks are obtained through mutually orthogonal Latin squares. Data blocks from multiple different training nodes within a node group are selected across nodes to form a cross-node striped distribution. , where n is the number of training nodes.

4. The method according to claim 1, characterized in that, In (3), the pre-set orthogonal Latin square is used. Define the set of source data blocks corresponding to each encoding task, the implementation of which includes: For coding tasks Read mutually orthogonal Latin squares The The element value in row +1 and column j is used as the target index, where ; exist In the first k rows, select all cells whose element values ​​are the same as the target index. Based on the source data block number corresponding to the row number and the training node where the source data block corresponding to the column number is located, determine the corresponding source data block set. ,in It is the kth data block.

5. The method according to claim 1, characterized in that, In (4), each training node is based on its corresponding encoding. The task is to perform an XOR operation on the source data blocks in the source data block set to generate a check block, the implementation of which includes: For the source data block set Perform an XOR operation on each source data block to generate a corresponding check block. ,in ; Each training node generates at least two verification blocks, corresponding to... and The encoding task is performed, and the verification block is stored in the local memory of the corresponding training node.

6. The method according to claim 1, characterized in that: In step (5), restoring the fault checkpoint data to the corresponding training device means restarting the device when a software fault is detected. In the corresponding training process, the required data blocks are read directly from the local CPU memory of the faulty node, and the data blocks are restored to the GPU through the CPU-to-GPU link to complete the localized recovery. In step (6), the recovered model parameters, optimizer state and related training state are reloaded into the corresponding training environment. This involves reorganizing the recovered data blocks into complete checkpoint data, loading them into the memory of the corresponding training device, restoring the model parameters and optimizer state, and continuing to execute the distributed training task.

7. A distributed training memory checkpoint recovery system based on locality-aware erasure coding, characterized in that, include: The node grouping module is used to divide multiple training nodes in a distributed training cluster into several node groups, each containing n training nodes. Or n is a prime number greater than 4; The data partitioning module is used to divide the training checkpoint data to be saved into multiple data blocks on each training node and store them in the memory of the corresponding training node. The encoding construction module is used to construct the cross-node striped distribution relationship of source data blocks and the encoding task allocation relationship corresponding to the training nodes within each node group, and to determine the source data block set corresponding to each encoding task based on the preset mutually orthogonal Latin square. The verification generation module is used to enable each training node to perform an XOR operation on the source data blocks in the source data block set according to the corresponding encoding task to generate a verification block, and store it in the local memory of the corresponding training node. The fault recovery module is used to read the checkpoint data required for recovery from the local memory of the faulty node when a fault is detected, and then restore it to the corresponding training device to ensure that subsequent training can continue.

8. The method according to claim 1, characterized in that, The encoding construction module includes: Latin square construction submodule, used to set two within each node group mutually orthogonal Latin squares and ; The coding task assignment submodule is used for each mutually orthogonal Latin square. Each training node below Assign a task Service Code This task specifies that a verification block should be generated on this node, where , n is the number of training nodes; The source block selection submodule is used for each encoding task. Read The The element value in row +1, column j is used as the target index. In the first k rows, select all cells whose element values ​​are the same as the target index, and determine the source data block set based on the row and column information of these cells. This forms a cross-node striped distribution relationship, in which .

9. The method according to claim 1, characterized in that, The verification generation module includes: XOR operation submodule: used for each encoding task The corresponding source data block set Perform an XOR operation on each source data block to generate a corresponding check block. ; Verification storage submodule: used to store the generated verification blocks Saved to the local CPU memory of the corresponding training node j, and together with the original data block of that node, constitutes a locality-aware erasure code.

10. The method according to claim 1, characterized in that, The fault recovery module includes: The software fault recovery submodule is used to restart the training process of the faulty node and retain its original rank ID when a software fault is detected. It also reads all the original data blocks directly from the local CPU memory of the faulty node based on locality-aware coding and copies the data blocks from the CPU memory to the GPU memory through the PCIe link to complete the localized recovery of the software fault. Hardware Fault Recovery Submodule: When a single-node hardware fault is detected, this module executes a shared block recovery algorithm to determine the blocks to be recovered, including k data blocks and 2 parity blocks, using mutually orthogonal Latin squares. and The required source blocks are indexed separately, deduplicated, and merged to obtain the minimum source block set L. All source blocks in L are read from other surviving nodes in the group, and XOR decoding is performed to recover all data blocks and parity blocks of the faulty node. The recovered data blocks are then copied to the GPU memory of the replacement node through the PCIe link and kept in the CPU memory to maintain subsequent fault tolerance.

Images