Pipelined checkpointing method and operating system therefor

Abstract

The application belongs to the technical field of computers, and discloses a pipeline checkpoint operation method and an operation system thereof, which comprises the following steps: all computing nodes are synchronously started for checkpoint operation, and the checkpoint operation of the corresponding computing node is performed in the training idle time period of the computing node; wherein the training idle time period of the computing node is the time period when the computing node waits for the feedback of training data from other computing nodes; the checkpoint operation comprises three tasks of serialization, encoding and distribution, the serialization is to gradually convert the current state data of the corresponding computing node into a storable byte stream, the encoding is to encode the serialized data, and the distribution is to distribute the encoded data to a storage node for storage, each task of the checkpoint operation is responsible by different computing resources, and different tasks are executed in parallel in a pipeline mode. Through the above mode, the checkpoint can be effectively saved without affecting the training efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computers, and more particularly, to a pipeline checkpoint operation method and an operation system thereof. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning, deep neural networks (DNNs) have achieved great success in natural language processing, computer vision, and speech recognition. However, the scale of modern deep neural network models continues to expand, with more and more model parameters and more complex structures making it extremely difficult to train these models. Typically, these models contain billions or even hundreds of billions of parameters, which exceed the storage and processing capacity of a single computing node. In order to effectively train these large-scale models, distributed training has become an indispensable means, and multiple high-performance computing nodes (such as GPUs or TPUs) are connected through a network to collaboratively complete the model training task.

[0003] Pipeline parallel training is a widely used strategy that divides a large model into multiple consecutive stages, each handled by a different computing node. These nodes perform forward and backward propagation of the model in a pipeline manner, thereby improving the efficiency of training.

[0004] In large-scale distributed systems, the training process not only consumes a lot of resources, but also takes a very long time. In this case, system failures become inevitable. These failures can be caused by a variety of reasons, including hardware failure, software error, network interruption, or even unexpected interruption of computing resources (such as preemption of instances in cloud services). Once a failure occurs, the model state and intermediate results accumulated during the training process may be lost, forcing the training task to start from scratch, resulting in a large waste of computing resources and time loss. In order to alleviate this impact, checkpoint (PipeCheck) technology has emerged, which periodically saves the current state data of the system (including model parameters, optimizer state, etc.), and in the event of a failure, the training can continue from the nearest checkpoint, thereby reducing the loss of training progress.

[0005] Traditional checkpointing techniques usually pause training at the end of each training iteration or training cycle to save checkpoint data. These methods are widely used in deep learning frameworks such as PyTorch and TensorFlow, however, their application has significant limitations. Traditional checkpoint operations usually require pausing training, which significantly increases training time when dealing with large models. Although there are some improved methods, such as adaptively adjusting checkpoint frequency, or modifying the training framework to reduce the disturbance of checkpoint operations to training, most of these methods are designed for data parallel systems, and their effectiveness is still limited for large models with pipeline parallel training. SUMMARY

[0006] In order to overcome the above defects of the prior art or improve the prior art, the present application provides a pipeline checkpoint operation method and an operation system thereof, which aims to effectively save checkpoints without affecting training efficiency.

[0007] To achieve the above object, the present application provides a pipeline checkpoint operation method for checkpoint operation in a neural network distributed pipeline parallel training process jointly performed by multiple computing nodes, which comprises:

[0008] All computing nodes synchronously start the checkpoint operation, and perform checkpoint operation of the corresponding computing node in the training idle time period of the computing node; wherein,

[0009] The training idle time period of the computing node is the time period when the computing node waits for feedback of training data from other computing nodes during forward propagation and backward propagation of the neural network;

[0010] The checkpoint operation comprises three tasks of serialization, encoding and distribution, the serialization is to gradually convert the current state data of the corresponding computing node into a storable byte stream, the encoding is to encode the serialized data, and the distribution is to distribute the encoded data to a storage node for storage, each task of the checkpoint operation is responsible by different computing resources, and different tasks are executed in parallel in a pipeline manner.

[0011] Optionally, each task of the checkpoint operation is responsible by different computing resources, comprising: the CPU is responsible for the serialization task, and two different GPUs are responsible for the encoding task and the distribution task respectively.

[0012] Optionally, the method introduces a priority flow control mechanism to realize network resource allocation of training data and checkpoint data, the priority flow control mechanism comprises: the priority of the training data is higher than that of the checkpoint data, and when there are training data and checkpoint data to be transmitted at the same time, the training data is preferentially transmitted.

[0013] Optionally, the way of preferentially transmitting the training data in the priority flow control mechanism comprises: allocating the training data to a high-priority queue, allocating the checkpoint data to a low-priority queue, preferentially transmitting data in the high-priority queue, and when the data in the high-priority queue is all transmitted, transmitting the data in the low-priority queue.

[0014] Optionally, the encoding processing of the serialized data comprises: using erasure code technology to encode the serialized data.

[0015] Optionally, the serialized data is encoded by using an erasure code technology to obtain a plurality of block data, the block data including data blocks and redundant blocks, and each block data is attached with a version number and then distributed on the local storage node and other storage nodes;

[0016] When the checkpoint data of a new round is stored, a version number of each new block data is generated according to the checkpoint round;

[0017] When the computing node is recovered by using the checkpoint data, the block data is selected according to the version number, so as to ensure that the version numbers of the selected block data on all computing nodes are consistent.

[0018] Optionally, each block data stored on the local storage node is also attached with a check value;

[0019] When the checkpoint data of a new round is stored, a version number of each new block data is generated according to the checkpoint round, and each block data stored on the local storage node is attached with a check value, the check value being calculated according to the block data and the version number thereof;

[0020] When the computing node is recovered by using the checkpoint data, the block data is selected according to the following conditions:

[0021] Condition one, the version numbers of the selected block data on all computing nodes are consistent;

[0022] Condition two, the check value of the block data stored on the local storage node is consistent with the check copy value of the block data, the check copy value of the block data being calculated according to the copy data block of the block data and the version number thereof, the calculation method of the check copy value being the same as that of the check value, and the copy data block of the block data being a reconstructed data obtained by reconstructing the block data according to the block data stored on other storage nodes.

[0023] The application also provides a pipeline checkpoint operation system, which is used for checkpoint operation in a neural network distributed pipeline parallel training process jointly performed by a plurality of computing nodes, and includes:

[0024] A synchronization module is configured to synchronize the start of the checkpoint operation with other computing nodes;

[0025] A serialization module is configured to perform a checkpoint serialization task in a training idle time period of the computing node, the serialization being a step-by-step conversion of the current state data of the corresponding computing node into a storable byte stream;

[0026] An encoding module is configured to perform a checkpoint encoding task in a training idle time period of the computing node, the encoding being an encoding processing of the serialized data.

[0027] a distribution module configured to perform a distribution task of the checkpoint during a training idle period of the computing node, the distribution being to distribute the encoded data to the storage node for storage;

[0028] wherein the serialization module, the encoding module and the distribution module are on different computing resources, and the different tasks are executed in parallel through a pipeline manner.

[0029] Optionally, the system further comprises a priority flow control module configured to make a priority of the training data higher than a priority of the checkpoint data, and when there is training data and checkpoint data to be transmitted at the same time, the training data is preferentially transmitted.

[0030] Optionally, the encoding module is configured to encode the serialized data using an erasure code technology to obtain a plurality of block data, the block data comprising data blocks and redundant blocks, and to attach a version number to each of the block data and attach a check value to each of the block data stored on the local storage node; and whenever a new round of checkpoint data is stored, a version number of each new block data is generated according to a checkpoint round, and a check value is attached to each of the block data stored on the local storage node, the check value being calculated according to the block data and the version number thereof.

[0031] The system further comprises a fault recovery module configured to select the block data when the computing node is recovered using the checkpoint data according to the following conditions:

[0032] Condition one, the version numbers of the selected block data on all the computing nodes are consistent;

[0033] Condition two, the check value of the block data stored on the local storage node is consistent with a check copy value of the block data, the check copy value being calculated according to a copy data block of the block data and a version number thereof, and the check copy value being calculated in the same way as the check value, and the copy data block of the block data being a reconstructed data obtained by reconstructing the block data using block data stored on other storage nodes.

[0034] Overall, compared with the prior art, the above technical solutions conceived by the present application mainly have the following beneficial effects:

[0035] 1.The present application can find that the serialization, encoding and distribution of checkpoints can be independently operated by analyzing the checkpoint operation and the computing resources of the computing nodes, and only interact with each other through data transmission linkage, and the computing nodes are usually equipped with different computing resources, based on the independence of the operation point task and the flexibility of the computing resource allocation, the present application proposes a pipeline checkpoint operation method, which splits the whole process of checkpoint into three independent tasks of serialization, encoding and distribution, each task corresponds to a computing resource, and makes these tasks parallel processing in the training idle time period. In this way, the present application can fully utilize the system resources in the checkpoint operation process, reduce the interference to the training process, and complete the checkpoint operation as quickly as possible in the training idle time period, so as to effectively save the checkpoint without affecting the training efficiency.

[0036] 2.Further, the present application also introduces a priority flow control mechanism to realize the network resource allocation of training data and checkpoint data, which can ensure that the training data flow is prior to the checkpoint data flow in network transmission, thereby minimizing the network congestion and training delay caused by checkpoint operation.

[0037] 3.Further, the present application also uses erasure code technology to encode the serialized data, compared with the traditional checkpoint data replication method, the erasure code technology greatly reduces the occupation of storage and network bandwidth while ensuring the reliability of the system.

[0038] 4.Further, the present application also introduces a version number mechanism based on the erasure code technology, which can ensure that the version numbers of the selected block data during fault recovery are consistent, and can effectively deal with the problem of inconsistent data during fault recovery caused by dynamic changes of memory data.

[0039] 5.Further, the present application also introduces a check value mechanism based on the erasure code technology, which can ensure the accuracy of fault recovery using checkpoint data, and realizes an efficient erasure code fault-tolerant mechanism based on memory. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is the overall framework diagram of executing checkpoint operation in the distributed pipeline parallel training process in an embodiment of the present application;

[0041] Figure 2 is a resource idle time diagram in pipeline parallel training in an embodiment;

[0042] Figure 3 is a system flow scheduling framework diagram in an embodiment of the present application;

[0043] Figure 4 is the influence of checkpoint operation on throughput in distributed training;

[0044] Figure 5 is a comparison chart of fault recovery time of traditional checkpoint and the present application;

[0045] Figure 6 is a comparison chart of end-to-end training duration of different checkpoint mechanisms under specific interference;

[0046] Figure 7 is a comparison chart of the influence of model training accuracy using different checkpoint mechanisms under specific interference. DETAILED DESCRIPTION

[0047] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0048] Embodiment 1

[0049] As Figure 1 is the overall framework diagram of performing checkpoint operation in the process of distributed pipeline parallel training of neural network in an embodiment of the present application. The pipeline parallel training technology divides a large model into multiple continuous stages, each stage is processed by different computing nodes (such as servers), and these computing nodes perform forward propagation and backward propagation of the model in a pipeline manner in parallel. Figure 2 is a schematic diagram of resource idle time in pipeline parallel training in an embodiment. When data is transmitted from one stage to the next stage, some devices may need to wait for data to arrive before continuing to work, resulting in these devices being in an idle state. For example, if the calculation of a certain stage is completed faster, and the next stage is not ready to receive data, the devices of the previous stage will be idle, forming "bubbles". In addition, if the calculation amount of each stage is unbalanced, some devices may also be unable to continue to perform new work while waiting for other devices to complete the task, thereby further aggravating the phenomenon of idle time. Therefore, in pipeline parallel training, there will be "idle bubbles" in the computing nodes, which appear in the process that the nodes wait for other nodes to complete their tasks, and are the training idle time period in the training process.

[0050] Based on the above analysis of the pipeline parallel training process, the present application proposes a checkpoint operation method. At each computing node, by effectively utilizing the training idle time period analyzed above to perform checkpoint operation, the checkpoint can be effectively saved without affecting the normal training process.

[0051] The checkpoint operation method is a pipeline checkpoint operation method, comprising:

[0052] All the computing nodes synchronously start the checkpoint operation, and perform the checkpoint operation of the corresponding computing node in a training idle time period of the computing node; wherein the training idle time period of the computing node is a time period during which the computing node waits for feedback of training data from other computing nodes during forward propagation and reverse propagation of the neural network; the checkpoint operation comprises three tasks of serialization, encoding and distribution, the serialization is to gradually convert current state data of the corresponding computing node into a storable byte stream, the encoding is to encode the serialized data, and the distribution is to distribute the encoded data to a storage node for storage, each task of the checkpoint operation is responsible by a different processor, and different tasks are executed in parallel in a pipeline manner.

[0053] It should be noted that after synchronously starting the checkpoint operation of each computing node, the checkpoint operation of each computing node is actually performed in the training idle time period thereof.

[0054] Specifically, the checkpoint operation emphasizes, on one hand, dividing the process thereof into multiple tasks and performing the checkpoint operation in a pipeline manner, and on the other hand, effectively utilizing the training idle time period, and the two aspects are combined with each other to effectively save the checkpoint without affecting the normal training process. The two aspects are described in detail as follows.

[0055] In the training process of a large-scale deep neural network, the checkpoint operation (i.e. saving the current model state, optimizer state and other necessary system states) is essential for recovery in case of system failure, and thus, the checkpoint operation is a core link for ensuring recovery after system failure. However, the traditional checkpoint operation is usually a serial process, and all related tasks need to be executed sequentially. This way not only takes a long time, but also forces other tasks of the system to be suspended during the execution of the checkpoint operation, resulting in a significant decrease in training efficiency. The core idea of the pipeline checkpoint operation proposed in the present application is to divide the whole process of the checkpoint into multiple independent sub-tasks, and make these tasks be processed in parallel in the training idle time period. In this way, the present application can make full use of system resources during the checkpoint operation process, while reducing the interference with the training process, and complete the checkpoint operation as quickly as possible in the training idle time period.

[0056] Through analysis of the checkpoint operation and the computing resources of the computing nodes, it is found that the serialization, encoding and distribution of the checkpoint can be independently run, and only interact with each other through data transmission, and the computing nodes are usually equipped with multi-core CPUs, GPUs and high-speed storage devices. Based on the independence of the checkpoint operation tasks and the flexibility of the allocation of computing resources, the checkpoint operation is divided into the following key tasks:

[0057] (1) Serialization: Convert the current state data of the system (such as model parameters, optimizer state, random seed, etc.) into a storable byte stream.

[0058] (2) Encoding: Encode the serialized data.

[0059] (3) Distribution: Distribute the encoded data to multiple nodes or storage devices, ensuring that the data can be recovered when a node fails.

[0060] Moreover, the present application assigns the above tasks to different computing resources through a task scheduling mechanism, each task corresponding to a computing resource. Serialization operations can be performed on one CPU core, while encoding and data distribution can be performed on different CPU cores or GPUs. Each task can be executed independently, ensuring that each subtask can be executed as much as possible in parallel, so that more checkpoint operations can be completed as much as possible during the idle time period of training, thereby improving overall efficiency. When the serialization task completes processing of part of the data, this part of the data is immediately passed to the next subtask for encoding without waiting for the entire serialization process to complete. The encoded data block can immediately begin distribution, forming a "pipeline" mode in which data flows between subtasks. Through this "generate and process" pipeline processing, the present application can complete checkpoint operations in a shorter period of time and reduce the waiting time between tasks.

[0061] Experimental results show that, compared with traditional synchronous checkpoint operations, the pipeline checkpoint operation method proposed by the present application can significantly reduce the time overhead of checkpoint operations by embedding checkpoint operations in idle time periods of training, decomposing checkpoint tasks, and executing tasks in a pipeline manner. The method supports higher checkpoint frequencies and reduces interruptions to the training process, providing strong support for efficient and stable checkpoint operations in large-scale deep neural network training, while providing new ideas for the optimal use of system resources.

[0062] Since in distributed deep learning training, different parts of the model are processed in parallel on multiple computing nodes, these nodes need to exchange data (such as activation values and gradients) through the network. At the same time, in order to ensure the fault tolerance of the training process, checkpoint data also needs to be transmitted between different nodes through the network, for example, checkpoint data on a server is partially distributed to other servers for storage, when the server fails, the checkpoint data on the server can be constructed through the checkpoint data stored on other servers, thereby improving the fault tolerance of the system. However, the network bandwidth is limited, when the checkpoint data and the training data need to be transmitted at the same time, network congestion is easily caused, thereby causing the delay of the training process. The traditional checkpoint operation may compete with the training data for network resources, thereby affecting the training efficiency.

[0063] In order to solve the above problems, in some specific embodiments, a priority flow control mechanism is introduced to realize the network resource allocation of the training data and the checkpoint data, and the priority flow control mechanism includes: making the priority of the training data higher than the priority of the checkpoint data, when the training data and the checkpoint data to be transmitted exist at the same time, the training data is preferentially transmitted.

[0064] Specifically, the introduced priority flow control mechanism can effectively reduce the network bandwidth competition of the checkpoint operation in the deep neural network (DNN) training process. By introducing the priority flow control mechanism in the kernel network stack of the operating system of the computing node, it can be ensured that the training data flow is prior to the checkpoint data flow in network transmission, thereby minimizing the network congestion and training delay caused by the checkpoint operation.

[0065] By implementing the priority flow control in the kernel network stack of the operating system, the network competition problem between the checkpoint operation and the training data flow is solved. The priority flow control mechanism ensures that the priority of the training data flow in network transmission is higher than that of the checkpoint data flow, thereby avoiding the interference of the checkpoint operation on the training process.

[0066] Specifically, as Figure 3 As shown in Fig. 1, which is a system flow scheduling framework diagram in an embodiment of the present application, the implementation of the priority flow control mechanism includes the following key steps:

[0067] (1) Flow classification and labeling: first, in the application layer, all data packets to be sent are classified and labeled. Training data packets and checkpoint data packets are assigned different priority labels. Among them, the training data packets are assigned a higher priority, and the checkpoint data packets are assigned a lower priority. These priority labels can be automatically generated based on user-defined rules or according to the type of data packet (for example, using the TOS field in the IP packet).

[0068] (2) Priority queues in the kernel network stack: In the kernel network stack of the operating system, PipeCheck configures multiple queues for different priority levels of data packets. The network stack assigns data packets to the corresponding queue based on their priority tags. High-priority training data packets are placed in high-priority queues, while low-priority checkpoint data packets are placed in low-priority queues.

[0069] (3) Queue scheduling and data packet transmission: The network stack processes data packets in the order of queue priority and transmits them to the ring buffer of the network interface controller (NIC). High-priority data packets are given priority over low-priority data packets during transmission, ensuring that training data can be sent to the network first, while checkpoint data is only sent when there is no training data to transmit.

[0070] (4) Hardware acceleration and optimization: In some cases, the scheduling operations of priority traffic control may introduce additional CPU overhead. To reduce this overhead, PipeCheck can utilize the hardware acceleration features of certain network interface cards (NICs) to offload priority scheduling tasks to the NIC hardware layer for processing. This not only reduces the burden on the CPU but also reduces the latency of data packet transmission. For more complex network configurations, such as using InfiniBand adapters, PipeCheck can also directly hand over scheduling tasks to NIC hardware through transport layer APIs (such as rdma_set_option).

[0071] To further improve the fault tolerance of the checkpoint system, in an embodiment, the serialized data is encoded, including: using erasure coding technology to encode the serialized data. Erasure coding is a widely used technology in data storage and transmission, which is used to recover the original data through redundant information in the case of data loss or damage. One of the most commonly used erasure codes is Reed-Solomon code, which can distribute data blocks and redundant blocks among multiple servers, so that complete data can still be recovered when a certain number of servers fail. In this embodiment, erasure coding technology is applied to the encoding and decoding process of checkpoint data to improve the fault tolerance of the system and reduce the storage space requirement. Compared with traditional checkpoint data replication methods, erasure coding technology greatly reduces the occupation of storage and network bandwidth while ensuring system reliability.

[0072] The specific application and advantages of erasure coding technology in PipeCheck are described in detail as follows:

[0073] In performing checkpoint operations, PipeCheck first serializes the system state (including model parameters, optimizer state, etc.) into a set of data chunks. These data chunks are then encoded with an erasure code technique such as Reed-Solomon code to generate corresponding redundancy chunks. The encoded data chunks and redundancy chunks are distributed across multiple servers to improve fault tolerance of the data. For example, in a typical scenario where checkpoint data needs to be saved across 16 servers with a maximum of 4 servers allowed to fail, PipeCheck only needs to store redundancy data equivalent to 1.33 times the size of the original data, instead of the 5 times storage space required in traditional replication methods. When the system detects that some servers have failed, PipeCheck recovers the lost data by collecting data chunks and redundancy chunks from surviving servers and using the decoding algorithm of the erasure code. This process ensures that even in the case of multi-node failure, the system can recover from the most recent checkpoint and continue the training process.

[0074] Since traditional erasure code mechanisms are usually applied to relatively stable storage spaces such as disks, the checkpoint data in the present application is usually stored in memory space, and memory data is dynamic and prone to inconsistency or errors when reading data during fault recovery. It is difficult to effectively detect and correct transient errors or illegal modifications of memory data using traditional erasure code mechanisms.

[0075] To solve the above problems, in some specific embodiments, the following improvements are made to the traditional erasure code mechanism:

[0076] The serialized data is encoded using erasure code technology to obtain a plurality of block data, the block data includes data chunks and redundancy chunks, and each block data is attached with a version number and distributed in local storage nodes and other storage nodes; when storing the checkpoint data of a new round, the version number of each new block data is generated according to the checkpoint round; when recovering the computing nodes using the checkpoint data, the block data is selected based on the version number to ensure that the version numbers of the selected block data on all computing nodes are consistent.

[0077] In this way, by introducing the version number mechanism, the version numbers of the selected block data during fault recovery are ensured to be consistent, which can effectively deal with the problem of possible inconsistency of data during fault recovery caused by dynamic changes of memory data.

[0078] Furthermore, in some embodiments, in addition to introducing a version number mechanism, a checksum is added to each block of data stored on the local storage node; when storing a new round of checkpoint data, a version number for each new block of data is generated according to the checkpoint round, and a checksum is added to each block of data stored on the local storage node. The checksum is calculated based on the block data and its version number; when restoring a compute node using checkpoint data, the block data is selected according to the following conditions: Condition 1: The version numbers of the selected block data on all compute nodes are consistent; Condition 2: The checksum of the block data stored on the local storage node is consistent with the checksum copy value of the block data. The checksum copy value of the block data is calculated based on the copy data block of the block data and its version number, and the method of calculating the checksum copy value is the same as the method of calculating the checksum. The copy data block of the block data is the reconstructed data obtained by reconstructing the block data based on the block data stored on other storage nodes.

[0079] In condition two, if the data is tampered with or momentarily corrupted, the reconstructed data will differ from the original data. Since the method for calculating the verification copy value is the same as the method for calculating the verification value, the calculated results will also differ if the reconstructed data differs from the original data. Therefore, the correctness of the data can be further verified based on the verification value.

[0080] The following example describes the algorithm process:

[0081] (1) Data writing process: The data to be checked is divided into data blocks D1, D2, ..., D... n and redundant blocks P1, P2, ..., P n Add a version number V to each data block and redundant block. i And check value H i The verification value can be a hash value, and the data is written into memory to record version number update operations;

[0082] (2) Data update process: Generate new version number V new And recalculate the updated checksum H. new Updates are synchronized to relevant nodes or memory modules through consistent broadcasting.

[0083] (3) Fault recovery process: According to version number V i Find the latest consistent set of data blocks D′1, D′2, ..., D′ n The system uses checksums to verify the correctness of the data, and finally uses erasure coding decoding algorithms to reconstruct lost or redundant data blocks and perform fault recovery.

[0084] (4) Fault detection and verification: Periodically compare the verification value H iConsistency check is performed on the data block and the redundancy block in the memory. If a data block error is detected, a fault recovery process is triggered.

[0085] In this way, the version number mechanism can ensure that the selected version number is consistent, the check value mechanism can ensure that the read data is correct, and the combination of the version number mechanism and the check mechanism can ensure the accuracy of the fault recovery using checkpoint data, implement an efficient erasure code fault-tolerant mechanism based on memory, and be applicable to high-performance computing, distributed storage systems and memory-intensive application scenarios.

[0086] Embodiment 2

[0087] The application also relates to a pipeline checkpoint operation system for performing a checkpoint operation in a neural network distributed pipeline parallel training process jointly performed by a plurality of computing nodes, the system being arranged on the computing nodes and comprising:

[0088] a synchronization module for synchronously starting the checkpoint operation with other computing nodes;

[0089] a serialization module for performing a serialization task of the checkpoint in a training idle time period of the computing node, the serialization being step-by-step conversion of current state data of the corresponding computing node into a storable byte stream;

[0090] an encoding module for performing an encoding task of the checkpoint in the training idle time period of the computing node, the encoding being encoding processing of the serialized data;

[0091] a distribution module for performing a distribution task of the checkpoint in the training idle time period of the computing node, the distribution being distribution of the encoded data to a storage node for storage;

[0092] The serialization module, the encoding module and the distribution module are on different computing resources, and different tasks are executed in parallel in a pipeline manner.

[0093] In an embodiment, the system further comprises a priority flow control module for making the priority of the training data higher than the priority of the checkpoint data, and when there are training data and checkpoint data to be transmitted at the same time, the training data is preferentially transmitted.

[0094] In an embodiment, the encoding module is specifically configured to: encode the serialized data using an erasure code technology to obtain a plurality of block data, the block data comprising data blocks and redundant blocks, and attach a version number to each of the block data and attach a check value to each of the block data stored on the local storage node; whenever a new round of checkpoint data is stored, generate a version number of each new block data according to the checkpoint round, and attach a check value to each of the block data stored on the local storage node, the check value being calculated according to the block data and the version number thereof.

[0095] Further, the system further comprises a fault recovery module configured to select the block data according to the following conditions when recovering the computing nodes using the checkpoint data:

[0096] Condition one, the version numbers of the selected block data on all computing nodes are consistent;

[0097] Condition two, the check value of the block data stored on the local storage node is consistent with the check copy value of the block data, the check copy value of the block data being calculated according to the replica data block of the block data and the version number thereof, and the check copy value being calculated in the same way as the check value, and the replica data block of the block data being the reconstructed data obtained by reconstructing the block data according to the block data stored on other storage nodes.

[0098] It can be understood that the system further comprises a monitoring module and a storage module:

[0099] The monitoring module is configured to monitor the training progress and identify the training idle time period;

[0100] The storage module is configured to store the data of the checkpoint.

[0101] The above modules work together to ensure that the checkpoint operation is efficiently completed without interfering with the main training process, and the training progress is quickly recovered when a fault occurs in the system.

[0102] The specific process is as follows:

[0103] The monitoring module monitors the state of the computing nodes in real time during the pipeline parallel training process, and automatically identifies the training idle time period (idle bubble) in the training. When a node is in a state of waiting for other nodes to complete the calculation or data transmission, the monitoring module marks this period as an idle bubble, and immediately triggers the serialization module, the encoding module, and the distribution module to process the checkpoint tasks in a pipeline manner. For example, the data serialization task can be executed on one CPU core, while the encoding task can be performed in parallel on another core or GPU. The pipeline processing mode ensures seamless connection between the tasks, thereby greatly shortening the total time of the checkpoint operation.

[0104] During the pipeline processing, the encoding module uses erasure code technology to encode the serialized data, generating redundant data blocks. These data blocks are passed to the distributed storage module and distributed to multiple servers or storage nodes. The application of erasure code technology greatly reduces the storage requirement, while improving the fault tolerance of data, so that even if some nodes fail, the system can recover the original data through the remaining data blocks. In addition, through data version control and checksum mechanism, the consistency and correctness of the data blocks called during fault recovery are ensured, ensuring that the system can accurately identify and recover the latest checkpoint data when a fault occurs.

[0105] To prevent the interference of checkpoint data transmission on the training main process, the system introduces a priority flow control mechanism. This mechanism is integrated into the kernel network stack of the operating system, which classifies and schedules data packets by priority to ensure that training data has higher priority than checkpoint data in network transmission. Specifically, the network stack will first process high-priority training data packets, while checkpoint data packets will be transmitted when network resources are idle.

[0106] When the system detects a fault, the fault recovery module will quickly start the recovery operation. First, the system checks all the data blocks stored on the servers to ensure their consistency. If inconsistencies are found, the system will decode and recover the missing data blocks through erasure code technology. After recovery, the system will continue training from the latest checkpoint, minimizing the impact of faults on training progress. Throughout the recovery process, data version control and checksum mechanisms ensure data integrity, avoiding recovery failures due to partial data block loss or damage.

[0107] As Figure 4 The figure shows the impact of checkpoint operations on throughput in distributed training, where the horizontal axis represents the checkpoint period. The longer the checkpoint period, the longer the interval between the previous checkpoint operation and the next checkpoint operation, the more training idle time the computing node can use, and the less interference the computing node can directly use the training idle time to complete the checkpoint operation.

[0108] As Figure 5 The figure shows the comparison of fault recovery time between traditional checkpoint (Strawman) and the present invention (PipeCheck). Experiments show that when processing deep neural network models of different sizes (such as BERT-LARGE, GPT-2 and Llama-2), the PipeCheck system can effectively reduce the time occupied by checkpoint operations, thereby significantly reducing the interference with the training process.

[0109] As Figure 6The figure shows the comparison of end-to-end training duration of different checkpoint mechanisms under certain interference. It can be seen from the figure that, when a fault occurs in the process of training a deep neural network, the system can quickly recover the training progress using the PipeCheck scheme of the present application. Compared with the traditional checkpoint (Strawman) method, the PipeCheck shortens the recovery time to 6% to 30% of the traditional method. This means that even in a complex large-scale distributed training environment, the present application can significantly reduce the training interruption caused by faults and ensure the smooth progress of the training process.

[0110] As shown in Figure 7 The figure shows the comparison of the influence of different checkpoint mechanisms on model training accuracy under certain interference. The present application improves the convergence speed of the overall training by reducing the delay in the training process. Figure 7 It is shown that in the training of BERT-LARGE and GPT-2 models, the training convergence speed using the PipeCheck system is 1.8 to 2.5 times faster than the traditional method. This result proves that the present application not only performs well in reducing the overhead of checkpoint operation and recovery time, but also can effectively improve the final effect of training.

[0111] It can be understood that the pipeline checkpoint operation system introduced above can be used to implement the pipeline checkpoint operation method introduced in Embodiment 1, wherein the related functional modules are used for the corresponding steps in the method, and the specific can be referred to the above introduction, which will not be repeated here.

[0112] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered as the scope of the present application. It should be noted that the "in an embodiment of the present application", "for example", "such as" and the like are intended to illustrate the present application, but not to limit the present application.

[0113] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application.

Claims

1. A pipeline checkpointing operation method for performing checkpointing operations during parallel training of a distributed pipeline of a neural network jointly executed by multiple computing nodes, characterized in that, The method comprises: All computing nodes synchronously start a checkpoint operation, and perform the checkpoint operation of the corresponding computing node in a training idle time period of the computing node; wherein, The training idle time period of the computing node is a time period during which the computing node waits for feedback of training data from other computing nodes during forward propagation and reverse propagation of the neural network; The checkpoint operation comprises three tasks of serialization, encoding and distribution, the serialization is stepwise conversion of current state data of the corresponding computing node into a storable byte stream, the encoding is encoding processing of the serialized data, and the distribution is distribution of the encoded data to a storage node for storage, each task of the checkpoint operation is responsible for by different computing resources, and different tasks are executed in parallel in a pipeline manner; the encoding processing of the serialized data comprises: encoding processing of the serialized data by using an erasure code technology to obtain a plurality of block data, the block data comprises data blocks and redundant blocks, and each block data is distributed on a local storage node and other storage nodes after being attached with a version number; when storing new round checkpoint data, a version number of each new block data is generated according to a checkpoint round; when the computing node is recovered by using the checkpoint data, the block data is selected according to the version number, so that the version numbers of the selected block data on all computing nodes are consistent; The method further introduces a priority flow control mechanism to realize network resource allocation of training data and checkpoint data, the priority flow control mechanism comprises: the priority of the training data is higher than that of the checkpoint data, and when there are training data and checkpoint data to be transmitted at the same time, the training data is preferentially transmitted.

2. The pipelined checkpointing method of claim 1, wherein, Each task of the checkpoint operation is responsible for by different computing resources, comprising: the CPU is responsible for the serialization task, and two different GPUs are responsible for the encoding task and the distribution task respectively.

3. The pipelined checkpointing method of claim 1, wherein, The priority flow control mechanism comprises: the training data is allocated to a high-priority queue, the checkpoint data is allocated to a low-priority queue, data in the high-priority queue is preferentially transmitted, and when the data in the high-priority queue is completely transmitted, the data in the low-priority queue is transmitted.

4. The pipelined checkpointing method of claim 1, wherein, A check value is further attached to each block data stored on the local storage node; Each time new round checkpoint data is stored, a version number of each new block data is generated according to a checkpoint round, and a check value is attached to each block data stored on the local storage node, the check value is calculated according to the block data and the version number thereof; Each time the computing node is recovered by using the checkpoint data, the block data is selected according to the following conditions: Condition one, the version numbers of the selected block data on all computing nodes are consistent; The second condition is that the check value of the block data stored in the local storage node is consistent with the check copy value of the block data, the check copy value of the block data is calculated according to the copy data block and the version number of the block data, the calculation method of the check copy value is the same as the calculation method of the check value, and the copy data block of the block data is obtained by reconstructing the block data according to the block data stored in other storage nodes.

5. A pipelined checkpointing operation system for checkpointing operation in a neural network distributed pipeline parallel training process jointly performed by a plurality of computing nodes, characterized in that, The system comprises: A synchronization module is configured to synchronize the starting of the checkpoint operation with other computing nodes; A serialization module is configured to perform a serialization task of the checkpoint during a training idle period of the computing node, the serialization being a step-by-step conversion of the current state data of the corresponding computing node into a storable byte stream; An encoding module is configured to perform an encoding task of the checkpoint during the training idle period of the computing node, the encoding being an encoding processing of the serialized data; the encoding processing of the serialized data comprises: using an erasure code technology to perform the encoding processing of the serialized data to obtain a plurality of block data, the block data comprising data blocks and redundancy blocks, and each block data being distributed on a local storage node and other storage nodes after being attached with a version number; when storing a new round of checkpoint data, generating the version number of each new block data according to the checkpoint round; when recovering the computing node by using the checkpoint data, selecting the block data according to the version number to ensure that the version numbers of the selected block data on all computing nodes are consistent; A distribution module is configured to perform a distribution task of the checkpoint during the training idle period of the computing node, the distribution being a distribution of the encoded data to the storage nodes for storage; A priority flow control module is configured to make the priority of the training data higher than the priority of the checkpoint data, and when there are both training data and checkpoint data to be transmitted at the same time, the training data is preferentially transmitted; The serialization module, the encoding module, and the distribution module are located on different computing resources, and different tasks are executed in parallel through a pipeline manner.

6. The pipelined checkpointed operating system of claim 5, wherein, The encoding module is configured to use an erasure code technology to perform the encoding processing of the serialized data to obtain a plurality of block data, the block data comprising data blocks and redundancy blocks, and each block data being attached with a version number and a check value; Each time a new round of checkpoint data is stored, the version number of each new block data is generated according to the checkpoint round, and each block data stored in the local storage node is attached with a check value, the check value being calculated according to the block data and the version number thereof; The system further comprises a fault recovery module configured to select the block data according to the following conditions when the computing node is recovered by using the checkpoint data: Condition one: the version numbers of the selected block data on all computing nodes are consistent; Condition two, the check value of the block data stored in the local storage node is consistent with the check copy value of the block data, the check copy value of the block data is calculated according to the copy data block and the version number of the block data, and the calculation method of the check copy value is the same as that of the check value, and the copy data block of the block data is obtained by reconstructing the block data according to the block data stored in other storage nodes.

Images