Large model assembly line parallel training fault tolerance method and related equipment
By assigning forward and reverse backup groups to each GPU during the training process of large-scale deep learning models, and synchronizing the activation of tensor and gradient information in advance, the problem of frequent training interrupts is solved, and more efficient fault tolerance and training efficiency is achieved.
Patent Information
- Application Number
- CN202411998158.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-02
AI Technical Summary
During the training of large-scale deep learning models, frequent interruptions caused users to frequently restart and adjust jobs manually, which increased the operation and maintenance burden and cost, and the existing fault-tolerant solutions were not effective.
By assigning forward and reverse standby groups to each GPU cluster, the tensor and gradient information are activated in advance to these virtual CPUs, ensuring that computing tasks can be seamlessly replaced during failover.
Reduces potential error risks, maintains consistency and accuracy of the training process, reduces downtime, and improves overall training efficiency.
Smart Images
Figure CN119917168A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of cloud computing technology, and in particular to a large model pipeline parallel training fault-tolerant method and related equipment. Background Art
[0002] In the training process of large-scale deep learning models, pipeline parallelism is an effective strategy that can not only significantly improve training efficiency, but also effectively overcome the problem of memory limitations of a single device. By segmenting the model and processing it in parallel on multiple computing nodes, pipeline parallelism fully utilizes hardware resources and greatly shortens training time. With the increasing popularity of large model applications, especially in distributed training environments, the importance of pipeline parallelism has become increasingly prominent.
[0003] However, the training of large-scale deep learning models faces many challenges. Due to factors such as the huge number of model parameters, complex data processing, and long training cycles, the high frequency of interruptions during training has become particularly prominent. According to statistics, such training jobs fail almost every day, which brings a heavy operation and maintenance burden to users. Frequent manual restarts and adjustments are required, resulting in a sharp increase in costs. In order to meet these challenges, it is urgent to introduce an efficient fault-tolerant mechanism to reduce the impact of error interruptions.
[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0005] The present disclosure provides a large model pipeline parallel training fault-tolerant method and related equipment, which at least to some extent overcomes the problem that the existing fault-tolerant schemes are not effective, such as the frequent checkpoints in the traditional checkpoint recovery scheme that damage the overall MFU of the system, the frequent checkpoints that result in the loss of a large number of work iterations when reloading from the checkpoints, and the problem that the recovery scheme based on the backup GPU (Graphics Processing Unit) wastes expensive GPU computing resources.
[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0007] According to one aspect of the present disclosure, a large model pipeline parallel training fault tolerance method is provided, comprising:
[0008] In the initialization phase, a forward standby group and a reverse standby group are allocated to each GPU in the GPU cluster; wherein the forward standby group includes a plurality of virtual CPUs, and the plurality of virtual CPUs (Central Processing Units) include a forward main CPU; the reverse standby group includes a plurality of virtual CPUs, and the plurality of virtual CPUs include a reverse main CPU;
[0009] In the forward phase, after the GPU completes the calculation of one layer, it sends the generated activation tensor to the forward main CPU of the forward standby group corresponding to the GPU, so that the forward main CPU distributes the activation tensor to other virtual CPUs in the forward standby group;
[0010] In the forward phase, after the GPU completes the calculation of one layer, it also sends the generated activation tensor to the reverse main CPU of the reverse standby group corresponding to the GPU, so that the reverse main CPU sends the activation tensor distribution to other virtual CPUs in the reverse standby group;
[0011] In the reverse stage, after the GPU completes the reverse propagation of the current pipeline stage, it sends the gradients of each model parameter in this stage to the reverse main CPU of the reverse backup group, so that the reverse main CPU sends the gradient distribution of each model parameter to other virtual CPUs in the reverse backup group to calculate and update the model parameters, and forwards the updated model parameters to the forward main CPU.
[0012] In one embodiment of the present disclosure, in the reverse phase, after the GPU completes the reverse propagation of the current pipeline phase, it sends the gradients of each model in this phase to the reverse main CPU of the reverse standby group, including:
[0013] In the reverse phase, after the GPU completes the back propagation of the current pipeline phase, it symbolizes the gradients of the model parameters in this phase to obtain the symbol information corresponding to the gradient of each parameter. The symbol information is used to indicate the update direction of the parameter.
[0014] The sign information corresponding to the gradient of each model parameter is sent to the reverse main CPU of the reverse standby group.
[0015] In one embodiment of the present disclosure, during the initialization phase, the method further includes: sending the optimizer update strategy for each parameter to the reverse main CPU of the reverse standby group;
[0016] In the reverse phase, after receiving the gradient of the model parameters, the virtual CPUs in the reverse standby group update the optimizer state according to the optimizer update strategy corresponding to the model parameters.
[0017] In one embodiment of the present disclosure, the method further includes:
[0018] In the forward phase, when a GPU node error causes training interruption, the GPU is switched to the forward standby group corresponding to the GPU for recovery;
[0019] After switching to the forward standby group, the forward main CPU of the forward standby group divides the activation tensor of the previous layer into multiple subsets and assigns them to multiple virtual CPUs in the forward standby group for forward calculation processing, summarizes the processing results of multiple virtual CPUs, obtains the activation tensor of the current layer, and sends the activation tensor of the current layer to the GPU of the next layer, and sends the activation tensor of the current layer to the reverse main CPU of the reverse standby group.
[0020] In one embodiment of the present disclosure, during the initialization phase, the method further includes: sending the sparsification setting set to the forward master CPU;
[0021] After switching to the forward standby group, the forward master CPU of the forward standby group also sets the parameter weights in the sparsification setting set to 0.
[0022] In one embodiment of the present disclosure, the method further includes:
[0023] In the reverse phase, if a GPU node error causes training interruption, switch to the reverse standby group corresponding to the GPU for recovery;
[0024] After switching to the reverse standby group, the reverse main CPU of the reverse standby group coordinates other virtual CPUs in the reverse standby group to complete the reverse calculation task, obtains the gradients of each model parameter in this stage, and sends the gradients of each model parameter in this stage to the GPU node of the previous stage on the pipeline.
[0025] In one embodiment of the present disclosure, during the initialization phase, the method further includes: sending the sparsification setting set to the reverse master CPU;
[0026] After switching to the reverse standby group, the reverse main CPU of the reverse standby group also sets the attributes of the parameters in the sparsification setting set to not calculate gradients.
[0027] In one embodiment of the present disclosure, the forward main CPU is the virtual CPU in the forward standby group with the fewest network hops from the current GPU; the number of virtual CPUs in the forward standby group is greater than or equal to the ratio of the computing power of a single GPU to the computing power of a single CPU;
[0028] The reverse main CPU is the virtual CPU in the reverse standby group with the least network hops from the current GPU; the number of virtual CPUs in the reverse standby group is greater than or equal to the ratio of the computing power of a single GPU to the computing power of a single CPU.
[0029] According to another aspect of the present disclosure, an electronic device is provided, including: a memory for storing instructions; and a processor for calling the instructions stored in the memory to implement the above-mentioned large model pipeline parallel training fault-tolerant method.
[0030] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the above-mentioned large model pipeline parallel training fault-tolerant method is implemented.
[0031] According to another aspect of the present disclosure, a computer program product is provided, which stores instructions, and when the instructions are executed by a computer, the computer implements the above-mentioned large model pipeline parallel training fault-tolerant method.
[0032] According to yet another aspect of the present disclosure, there is provided a chip, comprising at least one processor and an interface;
[0033] An interface for providing program instructions or data to at least one processor;
[0034] At least one processor is used to execute program instructions to implement the above-mentioned large model pipeline parallel training fault-tolerant method.
[0035] The large model pipeline parallel training fault-tolerant method and related equipment provided by the embodiments of the present disclosure ensure that these virtual CPUs always have the latest status by synchronizing the activation tensor and gradient information to the forward standby group and the reverse standby group in advance, and can maintain the consistency and accuracy of the training process even in the event of fault switching, thereby reducing the potential risk of errors; after the back propagation is completed, the updated model parameters are promptly passed to the forward main CPU, ensuring the consistency of parameters on all computing nodes and maintaining the consistency and accuracy of training; because the activation tensor and gradient information have been synchronized to the forward standby group and the reverse standby group, even in the event of a GPU failure, the virtual CPU can seamlessly take over the computing task, reducing downtime and improving overall training efficiency; activation synchronization occurs between the GPU and the virtual CPU, using a non-shared network port, which will not interfere with the current communication traffic mainly between the GPUs, thereby avoiding the impact on training performance, cleverly separating different types of communication flows, and ensuring efficient operation of the system.
[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0038] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without any creative work.
[0039] Figure 1 A flow chart of a large model pipeline parallel training fault tolerance method in an embodiment of the present disclosure is shown;
[0040] Figure 2 A schematic diagram of the architecture of a large model pipeline parallel training fault-tolerant system in an embodiment of the present disclosure is shown;
[0041] Figure 3 A forward calculation flow chart according to an embodiment of the present disclosure is shown;
[0042] Figure 4 A reverse propagation flow chart according to an embodiment of the present disclosure is shown;
[0043] Figure 5 A forward calculation interrupt processing flow chart according to an embodiment of the present disclosure is shown;
[0044] Figure 6 A forward flow chart of a backup takeover in an embodiment of the present disclosure is shown;
[0045] Figure 7 Another backup takeover forward flow chart in an embodiment of the present disclosure is shown;
[0046] Figure 8 A reverse calculation interrupt processing flow chart according to an embodiment of the present disclosure is shown;
[0047] Fig. 9 A reverse flow chart of backup takeover in an embodiment of the present disclosure is shown;
[0048] Fig.10 Another backup takeover reverse flow chart in the embodiment of the present disclosure is shown;
[0049] Fig.11 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0051] For ease of understanding, the following first explains the relevant concepts involved in the present disclosure as follows:
[0052] Fault tolerance for large model training: Errors or interruptions are inevitable during large model training, which may be caused by hardware failure, network problems, memory overflow or other unexpected situations. In order to effectively handle these situations, there must be an automatic error recovery mechanism, that is, a fault tolerance mechanism, such as a checkpoint solution.
[0053] Pipeline parallel training of large models: Pipeline parallelism in large model training is a technique used to optimize the efficiency of deep learning model training, especially for models with a very large number of parameters, because these models often exceed the memory capacity of a single computing device. Pipeline parallelism distributes different parts of the model (usually different layers) to multiple computing devices so that multiple input batches can be processed simultaneously, thereby increasing the training speed. Pipeline parallelism divides a large model into multiple consecutive sub-models (usually layers of a deep network), each of which is assigned to a computing device. The input data is divided into multiple small batches, usually called "time steps", and processed sequentially at each time step. Data flows between sub-models. Parameter updates and gradient calculations between sub-models are coordinated to ensure correctness and consistency. Pipeline parallelism allows calculations to be performed on multiple devices at the same time, making full use of computing resources. By distributing model layers to different devices, large models that exceed the memory limit of a single GPU can be trained. However, it involves communication between multiple devices, and improper coordination may lead to reduced efficiency. If the calculation time of some layers (sub-models) is long, it may cause bottlenecks in the entire pipeline, limiting the speed increase.
[0054] Large model training context recovery: In large model training, the training context refers to the current state of model training, which can be used to resume the training process at any time when an error occurs during training interruption. The context usually includes model parameters and optimizer status.
[0055] Model parameters: Contains all model weights and biases.
[0056] Optimizer state: If an optimizer (such as Adam, SGD, etc.) is used, the context also includes the optimizer state, such as momentum, learning rate schedules, accumulated information of gradients, etc. This is very important for maintaining the stability of the training process.
[0057] Activation: refers to some temporary tensors calculated during the forward propagation of training, which will be used for the calculation of the back propagation. If the temporary tensors can be cached after the forward calculation, the calculation of the back propagation can be accelerated. However, the number of activations is huge, and it will take up a lot of storage space.
[0058] 1BitAdam: Reduces storage and computational overhead while maintaining model performance by using a single-bit representation. In this approach, weights or activation values are compressed into a single bit, usually 0 or 1. This quantization can significantly reduce the model's memory footprint and computational requirements.
[0059] Dynamic Sparse Training (DST): is a method to dynamically adjust the sparse structure of a neural network during training. Unlike static sparse training, dynamic sparse training continuously changes the sparse pattern of the network during training in order to find a better sparse structure, thereby significantly reducing computational and storage overhead while maintaining high accuracy.
[0060] DSR (Dynamic Sparse Reparameterization) is a representative method of dynamic sparse training, which allows sparse models to be directly constructed when training deep neural networks without the need to pre-train a large dense model. This method is achieved by dynamically reallocating non-zero parameters, thus overcoming some limitations of previous techniques, such as high computational costs and the need to manually configure the number of free parameters allocated to each layer.
[0061] In large model training, pipeline parallelism is an effective solution to improve efficiency and overcome the memory limitations of a single device. Although its implementation complexity is relatively high, if it can be properly utilized when dealing with large-scale calculations, it can significantly shorten the training time. With the popularity of large models, pipeline parallelism has become increasingly important, especially in distributed training scenarios.
[0062] However, in the training process of large-scale deep learning models, due to factors such as huge resources, complex data processing, and long running time, there may be high-frequency interruption problems. Large-scale deep learning models usually require a large amount of computing resources, including high-performance GPU or TPU clusters. The more training nodes there are, the longer it takes, and the greater the probability of node failure during training. Large-scale training jobs will inevitably encounter node failures. According to an observation by a large model training manufacturer, the failure rate of a single card is about 8% within a month, so the failure rate of a single card in a day is about 0.27%. For a thousand-card training job, the probability of card failure causing training failure within a day is as high as 93%. Therefore, training jobs fail almost every day. After the job fails, users need to restart the job manually, which has high operation and maintenance costs. If the user does not restart in time, the interval will cause the GPU card to be idle, wasting expensive computing resources. In order to meet these challenges, fault-tolerant technology is adopted to reduce the impact of error interruptions. The most commonly used fault-tolerant technologies are checkpoint-based recovery solutions and backup GPU-based recovery solutions.
[0063] Checkpoint technology is already a key fault tolerance and recovery method in large model training, but it also faces a series of problems and challenges. Frequent checkpoints impair the overall MFU of the system. The cluster needs to be constantly paused to save its current training context to persistent memory. In addition, when reloading from a checkpoint, it is usually saved only once every 100 iterations. This means that you can only lose up to 99 useful work steps. On a cluster of 100,000, if each iteration takes 2 seconds, you will lose up to 229 GPU days of work due to a failure at the 99th iteration. Another way to recover from failures is to have the spare node copy from other GPUs only via RDMA on the backend network. Due to the high bandwidth of the backend GPU network, latency can be reduced. With this method, at most 1 iteration will be lost. But the disadvantages of the backup solution are also obvious. Expensive GPU nodes are used as backup devices, which is a huge waste of computing resources.
[0064] There must be some CPU resources in the same data center outside the GPU cluster for training large models. One idea is to use the CPU as a fault-tolerant backup node. However, there are still two obvious obstacles to the use of CPU. The first is the computing power gap between the CPU and the GPU, which will cause the CPU to be unable to keep up with the GPU when joining the training, slowing down the entire training process. The second is the bandwidth problem between the CPU and the GPU. The PCIe bandwidth between the CPU and the GPU is low, which has become a new bottleneck when the state of the large model is restored. Although in other distributed training scenarios, such as data parallelism, some new training technologies, such as two-level data parallelism, can be used to alleviate the limitations of CPU applications, the bottleneck problem is more serious and more difficult to alleviate for pipeline parallelism. In pipeline parallelism, when a certain stage of CPU execution is due to insufficient processing power or long computing latency, the overall performance of the entire pipeline will be limited. This bottleneck will affect training efficiency, reduce the benefits of parallel processing, and may lead to waste of system resources. In other words, pipeline parallelism is particularly sensitive to bottleneck problems compared to other parallel training forms. When the CPU is used as a fault-tolerant backup for the GPU, it is likely to cause adjacent GPU nodes to wait, thereby causing the entire pipeline to pause. The reason for the CPU backup processing bottleneck is not only the computing power gap with the GPU, but also the communication capability gap between the CPU and GPU and even between CPUs, especially the latter. As a result, the idea of stacking multiple CPU computing nodes for parallel processing does not work.
[0065] In summary, if CPU computing power is to be used as backup computing power in a fault-tolerant system for parallel training of large model pipelines, it faces severe bottleneck problems. To this end, the disclosed embodiment proposes a new solution. First, the calculations in the pipeline stage are split, and micro batch level data parallelism is performed within a CPU node group. At the same time, the forward and reverse calculations are respectively assigned to two different CPU groups, which makes up for the lack of CPU computing power and avoids the bidirectional bottleneck of the pipeline. In terms of processing delay, dynamic sparsity is used to reduce the computing demand; in terms of communication delay, 1BitAdam is used to compress the total amount of data transmission, which overcomes the bandwidth barrier of the CPU and reduces the impact of communication delay. The combined use of the two technologies alleviates the two major bottleneck problems of the CPU as a fault-tolerant backup computing power.
[0066] The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.
[0067] It should be noted that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.
[0068] Figure 1A flow chart of a large model pipeline parallel training fault tolerance method in an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the large model pipeline parallel training fault-tolerant method provided in the embodiment of the present disclosure includes S101-S104.
[0069] In S101, during the initialization phase, a forward standby group and a reverse standby group are allocated to each GPU in the GPU cluster; wherein the forward standby group includes multiple virtual CPUs (vCPUs), and the multiple virtual CPUs include a forward main CPU; the reverse standby group includes multiple virtual CPUs, and the multiple virtual CPUs include a reverse main CPU.
[0070] In some embodiments, the GPU cluster may also be referred to as the main training cluster, the forward standby group and the reverse standby group belong to the standby cluster, and the standby cluster is a CPU cluster. The GPUs in the GPU cluster may also be referred to as GPU nodes or GPU training nodes. The forward main CPU and the reverse main CPU are both master vCPUs, which are used to manage other virtual CPUs in the group.
[0071] In some embodiments, the forward main CPU is the virtual CPU in the forward standby group with the fewest network hops from the current GPU; the number of virtual CPUs in the forward standby group is greater than or equal to the ratio of the computing power of a single GPU to the computing power of a single CPU; the reverse main CPU is the virtual CPU in the reverse standby group with the fewest network hops from the current GPU; the number of virtual CPUs in the reverse standby group is greater than or equal to the ratio of the computing power of a single GPU to the computing power of a single CPU.
[0072] In addition, in the embodiment of the present disclosure, the main CPU (the forward main CPU and the reverse main CPU) is used as the root, and the group of virtual CPU nodes is planned in a tree network.
[0073] In one embodiment, a training GPU cluster and a backup CPU cluster are included, the former is composed of multiple GPU main training nodes, and the latter is multiple containerized virtual CPU nodes vCPU, including a forward backup group and a reverse backup group. Each GPU node in the training cluster corresponds to two backup groups in the backup cluster (i.e., a forward backup group and a reverse backup group). There can be N vCPUs in the forward backup group, and there can also be N vCPUs (virtual CPUs) in the reverse backup group. As an example, the number N satisfies: N>=single GPU computing power / single CPU computing power.
[0074] Each backup group has a master CPU (master vCPU), which is responsible for communicating with the corresponding GPU node and summarizing the results and gradients of data parallelism within the group.
[0075] In addition, you can initialize the dynamic sparse training of the model, select the initialization strategy, start applying sparsity according to the initialization strategy, determine which parameters will be sparse, and add them to the sparse setting (sparse_set) set. And send the sparse setting results to the dual CPU group corresponding to each GPU node. Select the adjustment strategy scheme of the single-bit adaptive optimizer.
[0076] In S102, in the forward phase, after the GPU completes the calculation of one layer, it sends the generated activation tensor to the forward main CPU of the forward standby group corresponding to the GPU, so that the forward main CPU sends the activation tensor distribution to other virtual CPUs in the forward standby group.
[0077] In S103, in the forward phase, after the GPU completes the calculation of one layer, it also sends the generated activation tensor to the reverse main CPU of the reverse standby group corresponding to the GPU, so that the reverse main CPU sends the activation tensor distribution to other virtual CPUs in the reverse standby group.
[0078] In S104, in the reverse stage, after the GPU completes the reverse propagation of the current pipeline stage, the gradients of the model parameters in this stage are sent to the reverse main CPU of the reverse backup group, so that the reverse main CPU sends the gradient distribution of each model parameter to other virtual CPUs in the reverse backup group to calculate and update the model parameters, and forward the updated model parameters to the forward main CPU.
[0079] In the above solution, the CPU group (including the forward standby group and the reverse standby group) is used as the backup computing power for GPU failure. Compared with the solution of using the GPU group as a backup, it can reduce the use of GPU resources and has lower costs.
[0080] The computing power of a single CPU is not as good as that of a single GPU. Therefore, in the embodiment of the present disclosure, a CPU group is used to compensate for this, that is, a forward standby group and a reverse standby group are used to perform forward and reverse calculations respectively, and data parallel training is performed within the group to compensate for this.
[0081] After the forward and reverse calculations of each stage of the pipeline parallelism on the GPU are completed, the activation tensors and gradients will be sent to the GPUs of the forward and reverse adjacent pipeline stages respectively. In the disclosed embodiment, the same content will also be sent to the backup CPU group (i.e., the forward backup group and the reverse backup group) of the target GPU, thereby avoiding the waste of iterative calculations in the checkpoint scheme, because the updates of each iteration have been sent to the backup node.
[0082] In some embodiments, the execution subject of the above-mentioned large model pipeline parallel training fault-tolerant method can be a large model pipeline parallel training fault-tolerant system, such as Figure 2As shown, the large model pipeline parallel training fault-tolerant system may include a main training cluster and a backup cluster, wherein the main training cluster includes multiple GPU training nodes.
[0083] In some embodiments, in the reverse stage, after the GPU completes the reverse propagation of the current pipeline stage, the gradient of each model in this stage is sent to the reverse main CPU of the reverse backup group, including: in the reverse stage, after the GPU completes the reverse propagation of the current pipeline stage, the gradient of each model parameter in this stage is symbolized to obtain the symbol information corresponding to the gradient of each parameter, and the symbol information is used to indicate the update direction of the parameter; the symbol information corresponding to the gradient of each model parameter is sent to the reverse main CPU of the reverse backup group.
[0084] In some embodiments, during the initialization phase, the method further includes: sending the optimizer update strategy of each parameter to the reverse main CPU of the reverse standby group; in the reverse phase, after receiving the gradient of the model parameter, the virtual CPU in the reverse standby group updates the optimizer state according to the optimizer update strategy corresponding to the model parameter.
[0085] The process of context synchronization during training in the disclosed embodiment is divided into two stages: forward and reverse:
[0086] like Figure 3 As shown in the figure, in the forward stage, after the forward GPU completes the calculation of one layer, it sends the generated activation tensor (newtensor) to the master CPU (master vCPU) of the two CPU backup groups (forward backup group and reverse backup group), namely the forward master CPU and the reverse master CPU, which then distribute and send it to other vCPUs in the group. The purpose of this step is to synchronize activation to avoid recalculation after interruption. At the current pipeline stage, the traffic of GPU at this time is mainly between GPU and GPU, and the network port is not shared with CPU, so activation synchronization will not affect training performance.
[0087] The steps of context synchronization in the reverse phase during training are as follows Figure 4As shown in the figure, after each computing node GPU completes the back propagation of the current pipeline stage, it sends the gradients of the parameters of the model block layer in this stage to the master CPU (mastervCPU) of the reverse standby group. In this way, the standby group also has the latest training results of the current iteration. The reverse standby group uses the received gradient to update the parameters of the current layer of the model, and forwards the new parameters to the forward standby group, so that both standby groups get the latest training results of the model. Because the size of the gradient is proportional to the number of parameters, the pressure of its synchronization on the network cannot be underestimated. Here we use the single-bit compression method in the single-bit adaptive optimizer. Specifically, the calculated gradient is processed by symbolization (sign), and only its sign information is retained. Therefore, each component of the gradient can only be +1 or -1, so that one bit can be used to represent the update direction of each parameter. Since each gradient component is compressed into one bit, the overall amount of data transmitted is significantly reduced. After this round of iteration is completed, the single-bit adaptive optimizer performs adaptive adjustments according to the preset strategy. Another thing that needs to be processed is the optimizer state synchronization. Because the total size of the optimizer state is twice the size of the gradient, its synchronization puts greater pressure on the network. We choose not to perform explicit transmission but to use the replay method. During the initialization phase, the optimizer state update strategy for each parameter has been sent to the reverse calculation backup vCPU group. After receiving the gradient of each parameter, the latter updates the new parameter according to the pre-made strategy, and also updates each optimizer state according to the corresponding strategy. Such local replay avoids a large amount of data transmission between the GPU and the CPU.
[0088] The disclosed embodiment divides fault tolerance into two stages, namely the forward calculation stage and the reverse propagation stage. If an error interrupt occurs in the forward calculation stage, the context that needs to be saved and restored is the model weight, which has been synchronized to the backup CPU node after the previous iteration, so no communication is required. If the interruption occurs in the reverse propagation stage, because the gradient and activation have been sent to the backup CPU node group in the forward stage, the optimizer state is updated by the CPU's self-replay. Therefore, there is no obvious communication burden when the interruption occurs.
[0089] In some embodiments, during the forward phase, when an error occurs in a GPU node causing training interruption, the GPU is switched to the forward standby group corresponding to the GPU for recovery; after switching to the forward standby group, the forward main CPU of the forward standby group divides the activation tensor of the previous layer into multiple subsets, and assigns them to multiple virtual CPUs in the forward standby group for forward calculation processing, aggregates the processing results of multiple virtual CPUs, obtains the activation tensor of the current layer, and sends the activation tensor of the current layer to the GPU of the next layer, and sends the activation tensor of the current layer to the reverse main CPU of the reverse standby group.
[0090] In some embodiments, during the initialization phase, the sparsification setting set is also sent to the forward master CPU; after switching to the forward standby group, the forward master CPU of the forward standby group also sets the parameter weights in the sparsification setting set to 0.
[0091] like Figure 5 As shown in the figure, if an error occurs during the forward training process, the training will be interrupted. Since the standby vCPU nodes in the standby group have obtained the latest model weights and the latest layer activations, they only need to perform calculations based on the current activation inputs for the model layers that have not been completed in the current stage. The only special thing is that the activations will be divided among the vCPU nodes in the forward standby group, and the forward calculations will be performed on the divided input data in a data-parallel manner. The difference from the calculation of the main node is that the activation values of the last layer obtained after this data parallelization will be summarized by the master CPU (master vCPU), which will then send the summarized results to the master GPU node of the next stage of the pipeline, while the activation values of the newly executed layers will be sent by the forward vCPU group responsible for the calculations to the corresponding nodes in the reverse vCPU group. Because the activations are reused, the standby cluster theoretically only needs to bear half of the pipeline stage workload of the faulty GPU on average, so it will not significantly affect the overall completion time of the currently interrupted iteration.
[0092] After the GPU work is completely handed over to the backup CPU group, the latter begins to take on normal training tasks. The problem at this time is how to keep up with the GPU processing speed of the adjacent stages to avoid becoming the bottleneck stage of the entire pipeline. This is especially important for pipeline parallelism, because the speed of pipeline parallel training depends on the slowest pipeline stage. Therefore, in the post-recovery training stage, we use the dynamic sparse training method for the model that the backup CPU group is responsible for, such as Figure 6 and Figure 7 As shown, the specific process includes S701-S704.
[0093] In S701, the parameter weights in the sparse setting set (sparse_set) are set to 0;
[0094] In S702, the forward master CPU (master vCPU) sends the output activation (i.e., activation tensor) sent by the GPU in the previous stage of the pipeline to the vCPUs in the group in a data parallel manner;
[0095] In S703, each vCPU uses the segmented activation data as input and performs forward calculations;
[0096] In S704, the forward master CPU (master vCPU) collects the output of each vCPU in the group, summarizes it and sends it to the GPU of the next stage of the pipeline as the activation output of this stage.
[0097] In some embodiments, in the reverse phase, when an error occurs in a GPU node causing training interruption, the reverse standby group corresponding to the GPU is switched to recover; after switching to the reverse standby group, the reverse main CPU of the reverse standby group enables the reverse main CPU to coordinate other virtual CPUs in the reverse standby group to complete the reverse calculation task, obtain the gradients of each model parameter in this phase, and send the gradients of each model parameter in this phase to the GPU node in the previous phase of the pipeline.
[0098] In some embodiments, during the initialization phase, the sparsification setting set is also sent to the reverse main CPU; after switching to the reverse standby group, the reverse main CPU of the reverse standby group also sets the attributes of the parameters in the sparsification setting set to not calculate gradients.
[0099] like Figure 8 As shown in the figure, if the training interruption occurs in the reverse propagation stage, because the standby group vCPU has obtained the gradient calculation that has been completed by the GPU in this iteration, it only needs to continue the reverse calculation of the unfinished model layer. The activation of the unfinished forward calculation has also been pre-stored, so each vCPU only needs to perform reverse calculation in the normal form of data parallel back propagation. After the calculation is completed, the master CPU (master vCPU) of the reverse standby group summarizes the gradient and sends it to the GPU node responsible for the previous stage of the pipeline. The local model parameter update is also completed in a distributed manner after the master CPU (master vCPU) distributes the summarized gradient to the vCPU nodes.
[0100] like Fig. 9 and Fig.10 As shown, the reverse calculation phase of the backup group after recovery includes S1001-S1004.
[0101] In S1001, the nograd (do not calculate gradient) attribute is set for the parameters in the sparse setting set (sparse_set) to avoid gradient calculation for them.
[0102] In S1002, the reverse master CPU (master vCPU) splits the gradient sent by the GPU in the next stage of the pipeline and sends it to the vCPUs in the group in a data parallel manner;
[0103] In S1003, each vCPU uses the segmentation gradient as input and performs reverse calculation;
[0104] In S1004, the reverse master CPU (master vCPU) collects the output of each vCPU in the group, summarizes it and sends it to the GPU of the previous stage of the pipeline as the gradient output of this stage.
[0105] The disclosed embodiment utilizes CPU computing power to form a dual standby node group to undertake the training tasks of failed GPUs, and uses pipeline stage data parallelism and forward and reverse calculation separation to make up for the lack of computing power. In the intelligent computing data center, CPU computing resources are relatively idle. According to the computing power gap, the pipeline parallel stage training tasks undertaken by a GPU are divided into two levels and handed over to multiple CPUs to complete, which not only makes up for the lack of computing power, but also improves the resource utilization of GPUs and CPUs. At the same time, based on the computing characteristics of data parallelism, the master node vCPU is responsible for distributing gradients and activations. On the one hand, the weight context is backed up to the standby node in advance, thereby ensuring that the loss of training interruption is reduced to within one iteration; on the other hand, the data traffic mainly occurs between the master node GPU and the backup cluster master CPU (master vCPU), reducing the impact on network performance.
[0106] In the loss recovery within an iteration, a flexible context recovery method is used for the training results within the iteration. Gradient synchronization adopts the method of single-bit adaptive optimizer, which is compressed into single bit, greatly reducing the amount of transmitted data; the activation transmission is carried out in layers, the order of magnitude is within o (number of model layers), the total amount is small, and the network pressure caused by centralized transmission is avoided; for the optimizer state with a large total amount, the reverse standby cluster replay method is adopted. The reverse standby vCPU replays the changes of the optimizer state according to the initial optimizer settings and the real-time gradient information to obtain the latest optimizer state, avoiding the transmission of the optimizer state and saving a lot of bandwidth.
[0107] In order to avoid the backup group becoming a bottleneck in the pipeline stage when taking over execution after error recovery is completed, the embodiment of the present disclosure adopts the latest training technology - dynamic sparse training - for the model layer responsible for the backup group, which reduces the computing power requirements to a certain extent without reducing the training accuracy and convergence speed.
[0108] Although the GPUs in the main training cluster need to perform additional communication with the backup group to transmit gradients, it uses single-bit compression and there is no obvious increase in communication volume. Moreover, the communication with the backup group's master CPU (master vCPU) does not occupy the communication bandwidth before the normal training GPU, and can overlap with the calculation, so no additional calculation and communication delays are introduced.
[0109] The master-slave switching of the disclosed embodiment is very simple. When the fault is eliminated and a new GPU is added to the training, the switch from the backup to the master device only needs to run one iteration of the new GPU as the backup node master CPU (master vCPU) to complete the seamless switch from temporary fault tolerance to normal training.
[0110] The disclosed embodiments can give full play to the computing power and network transmission capabilities of the intelligent computing center in heterogeneous scenarios in terms of hardware. It is suitable for the intelligent computing transformation of existing heterogeneous computing power data centers, and is also suitable for future computing power upgrades, enhancing the autonomous controllability of AI computing power. In the current resource pool, there is often a situation where the GPU is busy and the CPU is idle. Our solution can not only prevent expensive GPU resources from being wasted on backup, but also make full use of CPU resources and improve resource utilization efficiency. At the same time, it greatly optimizes network transmission requirements, and does not have excessive requirements on the bandwidth and latency of network equipment, thereby improving the utilization rate of network equipment.
[0111] In the embodiments of the present disclosure, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The concepts of "first", "second" and the like mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0112] The term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0113] Furthermore, although the steps of the method in the present disclosure are described in a particular order in the drawings, this does not require or imply that the steps must be performed in this particular order or that all the steps shown must be performed to achieve the desired results.
[0114] In some embodiments, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0115] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory.
[0116] In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0117] Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0118] Refer to the following Fig.11 To describe the electronic device provided by the embodiment of the present disclosure. Fig.11 The electronic device 1100 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0119] Fig.11 FIG. 1 is a schematic diagram showing the architecture of an electronic device 1100 provided by an embodiment of the present disclosure. Fig.11 As shown, the electronic device 1100 includes but is not limited to: at least one processor 1110 and at least one memory 1120 .
[0120] The memory 1120 is used to store instructions.
[0121] In some embodiments, the memory 1120 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 11201 and / or a cache memory unit 11202 , and may further include a read-only memory unit (ROM) 11203 .
[0122] In some embodiments, memory 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0123] In some embodiments, the memory 1120 may store an operating system, which may be a real-time operating system (RTX), LINUX, UNIX, WINDOWS, or OS X.
[0124] In some embodiments, data may also be stored in the memory 1120 .
[0125] As an example, the processor 1110 may read data stored in the memory 1120 , where the data may be stored at the same storage address as the instruction, or the data may be stored at a different storage address than the instruction.
[0126] The processor 1110 is used to call the instructions stored in the memory 1120 to implement the steps described in the "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processor 1110 can execute the steps of the above-mentioned large model pipeline parallel training fault-tolerant method embodiment.
[0127] It should be noted that the processor 1110 may be a general-purpose processor or a special-purpose processor. The processor 1110 may include one or more processing cores, and the processor 1110 executes various functional applications and data processing by running instructions.
[0128] In some embodiments, processor 1110 may include a central processing unit (CPU) and / or a baseband processor.
[0129] In some embodiments, the processor 1110 may determine an instruction according to the priority identifier and / or function category information carried in each control instruction.
[0130] In the present disclosure, the processor 1110 and the memory 1120 may be provided separately or integrated together.
[0131] As an example, the processor 1110 and the memory 1120 may be integrated on a single board or a system on chip (SOC).
[0132] like Fig.11 As shown, the electronic device 1100 is in the form of a general-purpose computing device. The electronic device 1100 may further include a bus 1130 .
[0133] The bus 1130 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.
[0134] The electronic device 1100 may also communicate with one or more external devices 1140 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 1100, and / or any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 1150.
[0135] Furthermore, the electronic device 1100 can also communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through the network adapter 1160 .
[0136] like Fig.11 As shown, the network adapter 1160 communicates with other modules of the electronic device 1100 via the bus 1130 .
[0137] It should be understood that although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0138] It is to be understood that the structure shown in the embodiment of the present disclosure does not constitute a specific limitation on the electronic device 1100. In other embodiments of the present disclosure, the electronic device 1100 may include: Fig.11 More or fewer components may be shown, or some components may be combined or separated, or the components may be arranged differently. Fig.11 The components shown may be implemented in hardware, software or a combination of software and hardware.
[0139] The present disclosure also provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the large model pipeline parallel training fault-tolerant method described in the above method embodiment is implemented.
[0140] The computer-readable storage medium in the embodiments of the present disclosure is a computer instruction that can be sent, propagated or transmitted for use by or in conjunction with an instruction execution system, apparatus or device.
[0141] As one example, computer readable storage media are non-volatile storage media.
[0142] In some embodiments, more specific examples of computer-readable storage media in the present disclosure may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, a USB flash drive, a mobile hard disk, or any suitable combination of the foregoing.
[0143] In the embodiments of the present disclosure, the computer-readable storage medium may include a data signal propagated in a baseband or as a part of a carrier wave, in which computer instructions (readable program codes) are carried.
[0144] Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
[0145] In some examples, computing instructions contained on a computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0146] The embodiments of the present disclosure also provide a computer program product, which stores instructions. When the instructions are executed by a computer, the computer implements the large model pipeline parallel training fault-tolerant method described in the above method embodiment.
[0147] The above instructions may be program codes. In specific implementation, the program codes may be written in any combination of one or more programming languages.
[0148] The programming language includes object-oriented programming languages, such as Java, C++, etc., and also includes conventional procedural programming languages, such as "C" language or similar programming languages.
[0149] The program code may execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.
[0150] Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0151] The embodiment of the present disclosure also provides a chip, including at least one processor and an interface;
[0152] An interface for providing program instructions or data to at least one processor;
[0153] At least one processor is used to execute program instructions to implement the large model pipeline parallel training fault-tolerant method described in the above method embodiment.
[0154] In some embodiments, the chip may further include a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.
[0155] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "systems".
[0156] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein.
[0157] The present disclosure is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.
Claims
1. A fault-tolerant method for parallel training of a large model pipeline, characterized in that: include: In the initialization phase, a forward standby group and a reverse standby group are allocated to each GPU in the GPU cluster; wherein the forward standby group includes a plurality of virtual CPUs, and the plurality of virtual CPUs include a forward main CPU; the reverse standby group includes a plurality of virtual CPUs, and the plurality of virtual CPUs include a reverse main CPU; In the forward phase, after the GPU completes the calculation of one layer, it sends the generated activation tensor to the forward main CPU of the forward standby group corresponding to the GPU, so that the forward main CPU distributes the activation tensor to other virtual CPUs in the forward standby group; In the forward phase, after the GPU completes the calculation of one layer, it also sends the generated activation tensor to the reverse main CPU of the reverse standby group corresponding to the GPU, so that the reverse main CPU sends the activation tensor distribution to other virtual CPUs in the reverse standby group; In the reverse stage, after the GPU completes the reverse propagation of the current pipeline stage, it sends the gradients of each model parameter in this stage to the reverse main CPU of the reverse backup group, so that the reverse main CPU sends the gradient distribution of each model parameter to other virtual CPUs in the reverse backup group to calculate and update the model parameters, and forwards the updated model parameters to the forward main CPU.
2. The method according to claim 1, characterized in that In the reverse phase, after the GPU completes the reverse propagation of the current pipeline phase, it sends the gradients of each model in this phase to the reverse main CPU of the reverse standby group, including: In the reverse phase, after the GPU completes the back propagation of the current pipeline phase, it symbolizes the gradients of the model parameters in this phase to obtain the symbol information corresponding to the gradient of each parameter, and the symbol information is used to indicate the update direction of the parameter; The sign information corresponding to the gradient of each model parameter is sent to the reverse main CPU of the reverse standby group.
3. The method according to claim 2, characterized in that In the initialization phase, the method further includes: sending the optimizer update strategy for each parameter to the reverse main CPU of the reverse standby group; In the reverse phase, after receiving the gradient of the model parameters, the virtual CPUs in the reverse standby group update the optimizer state according to the optimizer update strategy corresponding to the model parameters.
4. The method according to claim 1, characterized in that: The method further comprises: In the forward phase, when a GPU node error occurs and the training is interrupted, the forward standby group corresponding to the GPU is switched to recover; After switching to the forward standby group, the forward main CPU of the forward standby group divides the activation tensor of the previous layer into multiple subsets, and allocates them to multiple virtual CPUs in the forward standby group for forward calculation processing, summarizes the processing results of multiple virtual CPUs, obtains the activation tensor of the current layer, and sends the activation tensor of the current layer to the GPU of the next layer, and sends the activation tensor of the current layer to the reverse main CPU of the reverse standby group.
5. The method according to claim 4, characterized in that In the initialization phase, the method further includes: sending the set of sparsification settings to the forward master CPU; After switching to the forward standby group, the forward master CPU of the forward standby group also sets the parameter weights in the sparsification setting set to 0.
6. The method according to claim 1, characterized in that The method further comprises: In the reverse phase, when an error occurs in a GPU node and the training is interrupted, the reverse standby group corresponding to the GPU is switched to recover; After switching to the reverse standby group, the reverse main CPU of the reverse standby group coordinates the other virtual CPUs in the reverse standby group to complete the reverse calculation task, obtains the gradients of each model parameter in this stage, and sends the gradients of each model parameter in this stage to the GPU node of the previous stage on the pipeline.
7. The method according to claim 6, characterized in that In the initialization phase, the method further includes: sending the sparsification setting set to the reverse main CPU; After switching to the reverse standby group, the reverse main CPU of the reverse standby group further sets the attributes of the parameters in the sparsification setting set to not calculate gradients.
8. The method according to any one of claims 1 to 7, characterized in that: The forward main CPU is the virtual CPU in the forward standby group with the least number of network hops from the current GPU; the number of virtual CPUs in the forward standby group is greater than or equal to the ratio of the computing power of a single GPU to the computing power of a single CPU; The reverse main CPU is the virtual CPU in the reverse standby group with the least network hops from the current GPU; the number of virtual CPUs in the reverse standby group is greater than or equal to the ratio of the computing power of a single GPU to the computing power of a single CPU.
9. An electronic device, characterized in that: include: A memory for storing instructions; A processor is used to call the instructions stored in the memory to implement the large model pipeline parallel training fault-tolerant method as described in any one of claims 1-8.
10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by the processor, the large model pipeline parallel training fault-tolerant method described in any one of claims 1-8 is implemented.