Assembly line parallel fault tolerance method based on erasure codes and related device
By using the pipeline parallel fault tolerance method of erasure coding in distributed deep learning training, redundant blocks are initialized and updated, the storage and computing overhead problems in large-scale neural network training are solved, and efficient failure recovery and load balancing are achieved.
Patent Information
- Application Number
- CN202510605950.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-12
- Publication Date
- 2025-08-15
AI Technical Summary
There are problems of large storage overhead and low efficiency in existing distributed deep learning training, especially when large-scale neural network training, the memory demand is high, and frequent I/O operations affect performance.
Using the pipeline parallel fault tolerance method based on erasure coding, the redundant block is initialized before training and updated during training, and the parameters of the failed node are restored by using existing nodes to reduce storage and calculation overhead.
Pipeline parallel training with low redundant storage overhead and high recovery efficiency is realized, reducing GPU memory consumption, improving the balance of computing and memory load and failure recovery efficiency.
Smart Images
Figure CN120492220A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a pipeline parallel fault-tolerant method and related devices based on erasure codes. Background Art
[0002] In recent years, deep learning has played a crucial role in multiple fields, including image processing, natural language processing, and autonomous driving. This progress is driven by the continuous advancement of AI models, with increasing model and hardware iterations being key. Large-scale neural networks with billions of parameters, such as GPT and the LLaMA family, are driving advances in natural language processing and computer vision, pushing the boundaries of AI cognition. However, due to their large number of parameters, these models pose significant challenges to computing resources, particularly graphics memory. To overcome these challenges and facilitate efficient training of large models, distributed deep learning techniques—particularly sharded parameter parallelism and pipeline parallelism—have become widely adopted solutions. While distributed training techniques can alleviate storage limitations, they also present significant reliability issues, including node failures, network outages, data corruption, and software errors. The Top500 supercomputer list reveals a key issue: despite nearly doubling computing performance annually, the mean time between failures (MTBF) of the entire system has plummeted to single-digit hours. This inverse relationship between scale and stability is particularly detrimental for distributed training frameworks, as frequent node failures significantly increase system instability. The decline in system reliability further highlights the importance of fault-tolerant mechanisms that support data reconstruction. Therefore, designing efficient and reliable distributed training fault-tolerant storage technology has become a core challenge that needs to be overcome.
[0003] In recent years, distributed training fault tolerance methods can be roughly divided into three categories: (1) Checkpoint and restart technology periodically saves the system state (such as model parameters and optimizer state) to persistent storage. When a failure occurs, the system can roll back to the most recent checkpoint, thereby reducing data loss and avoiding retraining from the beginning. It is simple to implement and can effectively save training progress. However, checkpoint technology has significant performance bottlenecks in storage and I / O operations. Large-scale GPU-based training has extremely high requirements for video memory. Checkpoint technology needs to frequently transfer data from GPU video memory to disk. Frequent I / O operations seriously affect performance and latency. (2) Encoded computing divides data into multiple redundant blocks and uses encoding technology to distribute computing tasks to enhance fault tolerance. Even if some nodes fail, the system can use the remaining nodes to reconstruct the complete computing results. Although the encoded computing method can provide efficient fault recovery in some scenarios, it also introduces a lot of storage and computing overhead. The encoding process itself and the additional redundancy will increase the computing overhead of the system. In addition, some computing tasks are difficult to effectively split or encode, which makes encoded computing unsuitable in some cases. (3) The multi-replica fault-tolerance mechanism maintains model consistency by deploying identical copies of the model on computing nodes and periodically synchronizing parameters. When a computing node in the system encounters a hardware failure or network partition, the model parameters can be recovered from the replica. Multi-replica technology brings a large amount of storage overhead, and frequent parameter synchronization between replicas will introduce communication bottlenecks, which will have a negative impact on training throughput. Summary of the Invention
[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a pipeline parallel fault-tolerant method and related devices based on erasure codes to solve the problem of large storage overhead and low storage efficiency in the distributed storage process in the prior art.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] A pipeline parallel fault tolerance method based on erasure codes includes the following steps:
[0007] S1, processes the model parameters of each GPU, generates a data block, divides the data block into sub-blocks, generates a redundant block for each sub-block, and generates additional redundant blocks based on all redundant blocks; the number of sub-blocks obtained by the GPU is equal;
[0008] S2: Initialize the redundant blocks and the fault-tolerant layer redundant blocks, obtain data to train the GPU, update the redundant blocks through gradients during training, and obtain additional redundant blocks based on the updated redundant blocks; repeat the GPU training, redundant block updating, and additional redundant block acquisition process until the training is completed;
[0009] During the execution of S2 and S3, if a GPU fails, the extra redundant blocks of the faulty GPU are restored based on the extra redundant blocks of the normal GPU; based on the restored extra redundant blocks and the redundant blocks of the normal GPU, the redundant blocks of the faulty GPU are restored, and then the sub-blocks of the model parameters are restored.
[0010] A further improvement of the present invention is:
[0011] Preferably, in S1, the process of processing the model parameters of each GPU is to flatten, concatenate and fill the model parameters of each GPU with blank characters to obtain data blocks of equal length.
[0012] Preferably, in S1, the redundant block is obtained by performing element-level addition operation on the sub-blocks; and the additional redundant block is obtained by performing element-level addition operation on the redundant block.
[0013] Preferably, in S2, the training of the GPU is to perform forward calculation and back propagation on the GPU, and update the model parameters while updating the redundant blocks.
[0014] Preferably, in S2, the process of updating the redundant block is: for the i-th redundant block (1≤i≤n-1), the gradients corresponding to the i-th sub-block on all GPUs are accumulated, and the summed gradients are multiplied by the negative learning rate η to update the redundant block.
[0015] Preferably, during the execution of S2 and S3, if a GPU fails, the formula for restoring the extra redundant blocks of the failed GPU based on the extra redundant blocks of other GPUs is:
[0016] er j =er n -er1-er2-…-er j-1 -er j+1 -…-er (n-1)
[0017] Among them, er j are the redundant blocks corresponding to the faulty GPU, er1, er2…er n Represents redundant blocks for otherwise healthy GPUs.
[0018] Preferably, the sub-block recovery formula of the model parameters is:
[0019] p j,k =er k -p i,1 -…-p i,(j-1) -p i,(j+1) -…-p i,(n-1)
[0020] Among them, p j,kis the jth recovered sub-block on the kth GPU, erk is the redundant block on the kth GPU, pi1 is the i-th data sub-block on the first data block, pi,(j-1) is the i-th data sub-block on the j-1th GPU, pi,(j+1) is the i-th data sub-block on the j+1th GPU, and pi(n-1) is the i-th data sub-block on the n-1th GPU.
[0021] A pipeline parallel fault-tolerant device based on erasure codes, comprising:
[0022] The partitioning unit processes the model parameters of each GPU, generates a data block, divides the data block into sub-blocks, generates a redundant block for each sub-block, and generates additional redundant blocks based on all redundant blocks; the GPUs obtain an equal number of sub-blocks;
[0023] The update unit initializes the redundant blocks and the fault-tolerant layer redundant blocks, obtains data to train the GPU, updates the redundant blocks through gradients during training, obtains additional redundant blocks based on the updated redundant blocks, and repeats the GPU training, redundant block updating, and additional redundant block acquisition process until the training is completed.
[0024] During the execution of the fault unit, the partition unit, and the update unit, if a GPU fails, the extra redundant blocks of the faulty GPU are restored based on the extra redundant blocks of the normal GPU; the redundant blocks of the faulty GPU are restored based on the restored extra redundant blocks and the redundant blocks of the normal GPU, and then the sub-blocks of the model parameters are restored.
[0025] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the pipeline parallel fault tolerance method based on erasure codes as described in any one of the above items is implemented.
[0026] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the pipeline parallel fault tolerance method based on erasure codes as described in any one of the above items.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] The present invention discloses a pipeline parallel fault-tolerant method based on erasure codes. The execution process of ECFTP includes three stages: redundant block initialization, redundant block update and error recovery. In traditional pipeline training, data is processed in sequence by multiple GPUs. After each GPU completes forward calculation and back propagation, the parameters are updated through the all-reduce operation, and the cycle repeats until the end of training. ECFTP adds three new steps on the basis of traditional pipeline training: first, a double encoding strategy is used to initialize redundant blocks before training begins, and the initialization encoding redundant blocks is only performed once; second, redundant block parameters are updated before the all-reduce operation to ensure that the redundant blocks and model parameters are updated synchronously, so as to recover after a failure occurs; finally, when a node fails, a recovery process is executed based on the data blocks and redundant blocks of the existing nodes, and the complete model parameters are restored through the existing data blocks and redundant blocks. This method can solve the problems caused by the combination of pipeline parallelism and erasure codes, so as to achieve a pipeline parallel training fault-tolerant strategy with low redundant storage overhead and high recovery efficiency. The method of the present invention also has the following advantages:
[0029] (1) A pipeline fault tolerance mechanism based on erasure codes is proposed, which uses the data of existing nodes to recover lost information. This method avoids the overhead of storing the entire model and significantly reduces the consumption of GPU memory.
[0030] (2) A double encoding strategy is proposed to solve the problem of resource imbalance. By re-encoding and distributing redundant blocks to various nodes, this method ensures a balance between computation and memory load.
[0031] (3) A gradient-based dynamic update mechanism is designed to improve the efficiency of redundant block updates. By directly incorporating gradient information into the encoding process, this mechanism eliminates the need for frequent full encoding and decoding operations, thereby reducing computational overhead and improving update efficiency.
[0032] Furthermore, redundant blocks are generated based on the parameters of the pipeline's parallel nodes and stored directly in GPU memory. During each batch of gradient updates, the parameters of the redundant blocks are synchronously updated. By adopting a double encoding strategy to distribute redundant blocks, ECFTP ensures that even if a single node loses data, the parameters of the failed node can be recovered using the data from the remaining nodes. Furthermore, redundant blocks are updated in situ using gradients, avoiding repeated encoding of the redundant blocks. This method of storing and updating redundant blocks in GPU memory significantly reduces recovery time and computational waste in the event of a failure. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 This is a framework diagram of the pipeline parallel fault-tolerant strategy based on erasure codes of the present invention;
[0034] Figure 2 A schematic diagram of an embodiment of the present invention; DETAILED DESCRIPTION
[0035] Hereinafter, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Thus, a feature identified as "first," "second," "third," or "fourth" may explicitly or implicitly include one or more of such features.
[0036] The co-shooting method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.
[0037] It should be noted that the terms "first," "second," and the like in the description and drawings of the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.
[0038] The present invention discloses a pipeline parallel fault tolerance method based on erasure code, which includes the following steps:
[0039] S1, processes the model parameters of each GPU, generates a data block, divides the data block into sub-blocks, generates a redundant block for each sub-block, and generates additional redundant blocks based on all redundant blocks; the number of sub-blocks obtained by the GPU is equal;
[0040] S2: Initialize the redundant blocks and the fault-tolerant layer redundant blocks, obtain data to train the GPU, update the redundant blocks through gradients during training, and obtain additional redundant blocks based on the updated redundant blocks; repeat the GPU training, redundant block updating, and additional redundant block acquisition process until the training is completed;
[0041] During the execution of S2 and S3, if a GPU fails, the extra redundant blocks of the faulty GPU are restored based on the extra redundant blocks of the normal GPU; based on the restored extra redundant blocks and the redundant blocks of the normal GPU, the redundant blocks of the faulty GPU are restored, and then the sub-blocks of the model parameters are restored.
[0042] The present invention's method, based on an erasure-coding-based fault-tolerant strategy for pipeline parallel training (ECFTP), directly utilizes existing model parameters on each GPU to generate redundant blocks, thereby reducing memory overhead. In pipeline parallel training, model parameters are distributed across multiple GPUs. Specifically, the model parameters on each GPU are treated as data blocks, which are then encoded to generate redundant blocks.
[0043] In some embodiments of the present invention, in S1, the redundant block is initialized by double encoding, and the specific steps of double encoding include data block processing, first encoding, and second encoding.
[0044] Specifically, data block processing involves parameter normalization and segmentation. First, the model parameters on each GPU are flattened, concatenated, and padded with blank characters to equal length to form data blocks (p i ). Then, these data blocks are segmented. For a system with n GPUs, each data block p i (i=1,2,…,n) is divided into n-1 sub-blocks, expressed as:
[0045] p i ={p i1 ,p i2 ,…,p i(n-1)} (1)
[0046] This segmentation ensures that all GPUs process and store an equal number of sub-blocks, laying the foundation for load balancing.
[0047] Specifically, in the first encoding stage, redundant blocks are generated by performing element-wise addition on all sub-blocks of all GPUs. Specifically, for each sub-block index k (1≤k≤n-1), a small redundant block er is calculated. k for:
[0048] er k =p 1k +p 2k +…+p nk (2)
[0049] A total of n-1 redundant blocks are generated; each redundant block er k It is then assigned to the kth GPU to complete the first encoding of the redundant blocks and ensure that all GPUs participate equally in storing these redundant blocks.
[0050] Specifically, the second encoding phase focuses on creating additional redundant blocks n , which acts as an additional fault-tolerance layer. This module is calculated by element-wise addition of the first n-1 small redundant blocks:
[0051] er n =er1+er2+…+er (n-1) (3)
[0052] This encoding step ensures that any GPU failure can be recovered through distributed redundant blocks, thereby improving the fault tolerance of the system.
[0053] Through the dual encoding process described above, each GPU is responsible for encoding and decoding a redundant block of equal length. This layout strategy ensures that if any single GPU fails, data from the remaining GPUs can be used for recovery. The dual encoding strategy achieves a balanced distribution of memory and computational load, minimizes system bottlenecks, and provides a robust fault recovery mechanism.
[0054] In some embodiments of the present invention, in S2, the algorithm for updating redundant blocks using gradients is implemented. Specifically, the redundant blocks are initialized; data is then repeatedly forward-computed and backward-propagated across multiple GPUs; the redundant blocks are updated using gradients; and parameters are updated using an all-reduce operation, repeating this cycle until training is complete. If a failure occurs during training, a recovery process is executed.
[0055] During the gradient update process, for n GPUs p1,p2,p3,…,p n , assuming redundant blocks er1,er2,…,er n-1 has been generated. During the gradient synchronization phase, the gradient vectors of each GPU are flattened. Subsequently, as the model parameters are updated, the following update operations are performed on the redundant blocks:
[0056] For the i-th redundant block (1≤i≤n-1), the gradients of the corresponding i-th sub-blocks on all GPUs are accumulated and multiplied by the negative learning rate η to update the redundant block:
[0057]
[0058] Where η is the learning rate, represents the gradient of the i-th sub-block in the j-th GPU.
[0059] This gradient-based update method ensures that the updated redundant blocks are consistent with the redundant blocks generated by the double coding method.
[0060] The following is er i The mathematical proof that can be calculated in the above manner shows that the update of the model parameters can be expressed as:
[0061]
[0062] where j = 1, 2,..., n; i = 1, 2,..., n - 1, and P j,i represents the updated parameters. The updated redundant blocks can be expressed as:
[0063]
[0064] where, P j,i represents the parameters before the update, and er i represents the redundant blocks after the update.
[0065] In this process, ECFTP introduces a gradient-based method to update the redundant blocks. This method avoids repeated encoding and allows the redundant blocks to be updated synchronously without waiting for the model parameters to be fully updated, thus improving efficiency.
[0066] In some embodiments of the present invention, during the execution of S1 and S2, for the fault recovery algorithm, the process of generating redundant blocks using the double coding method is completely reversible, which enables data recovery in case of GPU failure. When GPU j (0 < j < n) fails, the recovery process includes two key steps. First, by utilizing the n - 1 additional redundant blocks of the n - 1 normal GPUs, the additional redundant block er j corresponding to the faulty GPU is reconstructed. This reconstruction process is achieved through the following formula:
[0067] er j = er n - er1 - er2 - … - er j-1 - er j+1 - … - er (n-1) (7)
[0068] Once the redundant block er j is recovered, the second step involves recovering the parameter sub-blocks related to GPU j. For each sub-block p jk (0 < k < n), the recovery process is completed by using the corresponding additional redundant block er k and the parameter sub-blocks from the normal GPUs. This recovery process is expressed as:
[0069] p j,k = er k - pi,1 -…-p i,(j-1) -p i,(j+1) -…-p i,(n-1) (8)
[0070] Recover all sub-blocks p by iterative j,1 ,p j,2 ,…,p j,(n-1) , the complete model parameters of the faulty GPU j are reconstructed.
[0071] A second aspect of the present invention discloses a pipeline parallel fault-tolerant device based on erasure codes, comprising:
[0072] The partitioning unit processes the model parameters of each GPU, generates a data block, divides the data block into sub-blocks, generates a redundant block for each sub-block, and generates additional redundant blocks based on all redundant blocks; the GPUs obtain an equal number of sub-blocks;
[0073] The update unit initializes the redundant blocks and the fault-tolerant layer redundant blocks, obtains data to train the GPU, updates the redundant blocks through gradients during training, obtains additional redundant blocks based on the updated redundant blocks, and repeats the GPU training, redundant block updating, and additional redundant block acquisition process until the training is completed.
[0074] During the execution of the fault unit, the partition unit, and the update unit, if a GPU fails, the extra redundant blocks of the faulty GPU are restored based on the extra redundant blocks of the normal GPU; the redundant blocks of the faulty GPU are restored based on the restored extra redundant blocks and the redundant blocks of the normal GPU, and then the sub-blocks of the model parameters are restored.
[0075] The following is further described with reference to specific embodiments.
[0076] See also Figure 2 , an example of a 4GPU setup is shown below.
[0077] First, the GPU parameter data is processed as follows: on GPU 0, the model parameters are flattened, concatenated, and padded to equal length to form data blocks P1, P2, and P3.
[0078] Similarly, data blocks P4, P5, and P6 are generated on GPU 1; data blocks P7, P8, and P9 are created on GPU 2; and data block P is generated on GPU 3. 10 ,P 11 ,P 12 In the first encoding stage, the redundant block er1 is calculated as er1=P1+P4+P7+P 10Redundant blocks er2 and er3 are generated in a similar manner. In the second encoding phase, redundant block er4 is obtained by er4 = er1 + er2 + er3. This completes the dual encoding of the 4-GPU system.
[0079] Updating redundant block er2 in a 4-GPU setup. For a 4-GPU configuration, assume that redundant block er2 needs to be updated.
[0080] First, the sum of the gradients of the second sub-block on all GPUs is calculated, and then er2 is updated as follows:
[0081]
[0082] Assume that GPU 2 fails, resulting in the loss of parameters P7, P8, P9 and redundant block er3. The recovery process includes two steps. First, restore er3 by calculating er3 = er4 - er1 - er2. Then, use P7 = er1 - P1 - P4 - P 10 Reconstruct P7. Similarly, restore P8 and P9. Finally, perform the reverse steps of the data processing to restore the whole.
[0083] The third aspect of the present invention discloses a computer device, which is a computer terminal device including a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions; the processor described in this embodiment can be used for A pipeline parallel fault-tolerant method based on erasure codes is implemented, comprising the following steps: S1, processing model parameters of each GPU to generate a data block, dividing the data block into sub-blocks, generating a redundant block for each sub-block, and generating additional redundant blocks based on all redundant blocks; the GPUs obtain an equal number of sub-blocks; S2, initializing the redundant blocks and the redundant blocks of the fault-tolerant layer, acquiring data to train the GPUs, updating the redundant blocks through gradients during the training process, and obtaining additional redundant blocks based on the updated redundant blocks; repeating the GPU training, redundant block updating, and additional redundant block obtaining processes until the training is completed; during the execution of S2 and S3, if a GPU fails, the additional redundant blocks of the failed GPU are restored based on the additional redundant blocks of a normal GPU; the redundant blocks of the failed GPU are restored based on the restored additional redundant blocks and the redundant blocks of the normal GPU, and then the sub-blocks of the model parameters are restored.
[0084] The fourth aspect of the present invention discloses a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement a pipeline parallel fault-tolerant method based on erasure codes in the above embodiment, including the following steps: S1, processing the model parameters of each GPU, generating a data block, dividing the data block into sub-blocks, generating a redundant block for each sub-block, and generating additional redundant blocks based on all redundant blocks; the GPU obtains an equal number of sub-blocks; S2, initializing the redundant blocks and the fault-tolerant layer redundant blocks, obtaining data to train the GPU, updating the redundant blocks through gradients during the training process, and obtaining additional redundant blocks based on the updated redundant blocks; repeating the GPU training, updating the redundant blocks, and obtaining the additional redundant blocks until the training is completed; during the execution of S2 and S3, if the GPU fails, the additional redundant blocks of the faulty GPU are restored based on the additional redundant blocks of the normal GPU; the redundant blocks of the faulty GPU are restored based on the restored additional redundant blocks and the redundant blocks of the normal GPU, and then the sub-blocks of the model parameters are restored.
[0085] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0086] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.
[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A pipeline parallel fault-tolerant method based on erasure codes, characterized in that: The following steps are involved: S1, processes the model parameters of each GPU, generates a data block, divides the data block into sub-blocks, generates a redundant block for each sub-block, and generates additional redundant blocks based on all redundant blocks; the number of sub-blocks obtained by the GPU is equal; S2: Initialize the redundant blocks and the fault-tolerant layer redundant blocks, obtain data to train the GPU, update the redundant blocks through gradients during training, and obtain additional redundant blocks based on the updated redundant blocks; repeat the GPU training, redundant block updating, and additional redundant block acquisition process until the training is completed; During the execution of S2 and S3, if a GPU fails, the extra redundant blocks of the faulty GPU are restored based on the extra redundant blocks of the normal GPU; based on the restored extra redundant blocks and the redundant blocks of the normal GPU, the redundant blocks of the faulty GPU are restored, and then the sub-blocks of the model parameters are restored.
2. The pipeline parallel fault tolerance method based on erasure code according to claim 1, characterized in that: In S1, the process of processing the model parameters of each GPU is to flatten, concatenate and fill the model parameters of each GPU with blank characters to obtain data blocks of equal length.
3. The pipeline parallel fault tolerance method based on erasure code according to claim 2, characterized in that: In S1, the redundant block is obtained by performing element-level addition operation on the sub-blocks; the additional redundant block is obtained by performing element-level addition operation on the redundant block.
4. The pipeline parallel fault tolerance method based on erasure code according to claim 1, characterized in that: In S2, the training of the GPU is to perform forward calculation and back propagation on the GPU, and the model parameters are updated while updating the redundant blocks.
5. The pipeline parallel fault tolerance method based on erasure code according to claim 1, characterized in that: In S2, the process of updating the redundant block is as follows: for the i-th redundant block (1≤i≤n-1), the gradients corresponding to the i-th sub-block on all GPUs are accumulated, and the summed gradients are multiplied by the negative learning rate η to update the redundant block.
6. The pipeline parallel fault tolerance method based on erasure coding according to claim 1, characterized in that: During the execution of S2 and S3, if a GPU fails, the formula for recovering the extra redundant blocks of the failed GPU based on the extra redundant blocks of other GPUs is: is j =is n -er1-er2-…-er j-1 -is j+1 -…-is (n-1) Among them, er j are the redundant blocks corresponding to the faulty GPU, er1, er2…er n Represents redundant blocks for otherwise healthy GPUs.
7. The pipeline parallel fault tolerance method based on erasure coding according to claim 1, characterized in that: The sub-block recovery formula of the model parameters is: p j,k =is k -p i,1 -…-p i,(j-1) -p i,(j+1) -…-p i,(n-1) Among them, p j,k is the jth recovered sub-block on the kth GPU, erk is the redundant block on the kth GPU, pi1 is the i-th data sub-block on the first data block, pi,(j-1) is the i-th data sub-block on the j-1th GPU, pi,(j+1) is the i-th data sub-block on the j+1th GPU, and pi(n-1) is the i-th data sub-block on the n-1th GPU.
8. A pipeline parallel fault-tolerant device based on erasure code, characterized in that: include: The partitioning unit processes the model parameters of each GPU, generates a data block, divides the data block into sub-blocks, generates a redundant block for each sub-block, and generates additional redundant blocks based on all redundant blocks; the GPUs obtain an equal number of sub-blocks; The update unit initializes the redundant blocks and the fault-tolerant layer redundant blocks, obtains data to train the GPU, updates the redundant blocks through gradients during training, obtains additional redundant blocks based on the updated redundant blocks, and repeats the GPU training, redundant block updating, and additional redundant block acquisition process until the training is completed. During the execution of the fault unit, the partition unit, and the update unit, if a GPU fails, the extra redundant blocks of the faulty GPU are restored based on the extra redundant blocks of the normal GPU; the redundant blocks of the faulty GPU are restored based on the restored extra redundant blocks and the redundant blocks of the normal GPU, and then the sub-blocks of the model parameters are restored.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the pipeline parallel fault-tolerant method based on erasure code is implemented as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the pipeline parallel fault tolerance method based on erasure code is implemented as described in any one of claims 1 to 7.