Assembly line check point operation method and operation system thereof

Abstract

The invention belongs to the related technical field of computers, and discloses a pipeline check point operation method and an operation system thereof.The method comprises the steps that all computing nodes synchronously start check point operation, and the check point operation of the corresponding computing nodes is executed in the training idle time period of the computing nodes; wherein the training idle time period of the computing node is a time period when the computing node waits for other computing nodes to feed back training data; the check point operation comprises three tasks of serialization, coding and distribution, the serialization is to gradually convert the current state data of the corresponding computing node into a storable byte stream, the coding is to code the serialized data, and the distribution is to distribute the coded data to the storage node for storage; each task of the check point operation is responsible for different computing resources, and the different tasks are executed in parallel in an assembly line mode. Through the mode, the check points can be effectively stored on the premise of not influencing the training efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of computer-related technologies, and more specifically, relates to a pipeline checkpoint operation method and an operating system thereof. Background Art

[0002] With the rapid development of artificial intelligence and deep learning, deep neural networks (DNNs) have achieved great success in fields such as natural language processing, computer vision, and speech recognition. However, the scale of modern deep neural network models continues to expand, and the increasing number of model parameters and more complex structures make it extremely difficult to train these models. Typically, these models contain billions or even trillions of parameters, which exceeds the storage and processing capabilities of a single computing node. In order to effectively train these large-scale models, distributed training has become an indispensable means, where multiple high-performance computing nodes (such as GPUs or TPUs) are connected through a network to collaboratively complete model training tasks.

[0003] Pipeline parallel training is a widely used strategy that divides a large model into multiple consecutive stages, each of which is processed by a different computing node. These nodes perform the forward and backward propagation of the model in parallel in a pipeline manner, thereby improving the efficiency of training.

[0004] In large-scale distributed systems, the training process not only consumes huge resources, but also often takes a very long time to train. In this case, system failures become inevitable. These failures may stem from a variety of reasons, including hardware failures, software errors, network interruptions, and even unexpected interruptions of computing resources (such as preempted instances in cloud services). Once a failure occurs, the model state and intermediate results accumulated during the training process may be lost, forcing the training task to start from scratch, resulting in a large waste of computing resources and time loss. In order to mitigate this impact, the checkpoint (PipeCheck) technology came into being. By periodically saving the current state data of the system (including model parameters, optimizer status, etc.), training can be continued from the most recent checkpoint in the event of a failure, thereby reducing the loss of training progress.

[0005] Traditional checkpointing techniques usually pause training at the end of each training iteration or training cycle to save checkpoint data. These methods are widely used in deep learning frameworks such as PyTorch and TensorFlow, but their application has significant limitations. Traditional checkpoint operations usually require pausing training, which significantly increases training time when dealing with large models. Although there are some improvement methods, such as adaptively adjusting the checkpoint frequency or modifying the training framework to reduce the interference of checkpoint operations on training, these methods are mostly designed for data parallel systems and their effects are still limited for large models trained in pipeline parallelism. Summary of the invention

[0006] In view of the above defects or improvement needs of the prior art, the present invention provides a pipeline checkpoint operation method and an operating system thereof, the purpose of which is to effectively save checkpoints without affecting the training efficiency.

[0007] To achieve the above object, the present invention provides a pipeline checkpoint operation method for performing a checkpoint operation in a neural network distributed pipeline parallel training process jointly executed by multiple computing nodes, which comprises:

[0008] All computing nodes start the checkpoint operation synchronously, and perform the checkpoint operation of the corresponding computing nodes during the training idle time period of the computing nodes;

[0009] The training idle time period of a computing node is the time period during which the computing node waits for other computing nodes to feed back training data during the forward propagation and backward propagation of the neural network;

[0010] The checkpoint operation includes three tasks: serialization, encoding and distribution. The serialization is to gradually convert the current state data of the corresponding computing node into a storable byte stream, the encoding is to encode the serialized data, and the distribution is to distribute the encoded data to the storage node for storage. Each task of the checkpoint operation is handled by different computing resources, and different tasks are executed in parallel in a pipeline manner.

[0011] Optionally, each task of the checkpoint operation is handled by different computing resources, including: a CPU is responsible for serialization tasks, and two different GPUs are responsible for encoding tasks and distribution tasks respectively.

[0012] Optionally, the method introduces a priority flow control mechanism to implement network resource allocation for training data and checkpoint data, and the priority flow control mechanism includes: making the priority of training data higher than the priority of checkpoint data, and when there are training data and checkpoint data to be transmitted at the same time, the training data is transmitted first.

[0013] Optionally, the method of implementing priority transmission of training data in the priority flow control mechanism includes: allocating training data to a high priority queue, allocating checkpoint data to a low priority queue, prioritizing transmission of data in the high priority queue, and when all data in the high priority queue has been transmitted, transmitting data in the low priority queue.

[0014] Optionally, encoding the serialized data includes: encoding the serialized data using erasure coding technology.

[0015] Optionally, the serialized data is encoded using erasure coding technology to obtain a plurality of block data, wherein the block data includes a data block and a redundant block, and a version number is added to each of the block data and then distributedly stored on a local storage node and other storage nodes;

[0016] When storing a new round of checkpoint data, a version number for each new block data is generated according to the checkpoint round;

[0017] When using the checkpoint data to restore the computing node, the block data is selected based on the version number to ensure that the version numbers of the block data selected on all computing nodes are consistent.

[0018] Optionally, a checksum is also added to each block of data stored on the local storage node;

[0019] Whenever a new round of checkpoint data is stored, a version number of each new block data is generated according to the checkpoint round, and a check value is added to each block data stored on the local storage node. The check value is calculated based on the block data and its version number.

[0020] Whenever a compute node is restored using checkpoint data, block data is selected according to the following conditions:

[0021] Condition 1: The version number of the block data selected on all computing nodes is consistent;

[0022] Condition 2: The check value of the block data stored on the local storage node is consistent with the check copy value of the block data. The check copy value of the block data is calculated based on the copy data block of the block data and its version number, and the method for calculating the check copy value is the same as the method for calculating the check value. The copy data block of the block data is the reconstructed data obtained by reconstructing the block data based on the block data stored on other storage nodes.

[0023] The present invention also provides a pipeline checkpoint operating system for performing checkpoint operations in a neural network distributed pipeline parallel training process jointly executed by multiple computing nodes, which includes:

[0024] A synchronization module, used to start the checkpoint operation synchronously with other computing nodes;

[0025] A serialization module is used to perform a checkpoint serialization task during the idle training period of the computing node, wherein the serialization is to gradually convert the current state data of the corresponding computing node into a storable byte stream;

[0026] An encoding module is used to perform the encoding task of the checkpoint during the idle training time period of the computing node, wherein the encoding is to encode the serialized data;

[0027] A distribution module is used to perform a checkpoint distribution task during a training idle period of a computing node, wherein the distribution is to distribute the encoded data to a storage node for storage;

[0028] Among them, the serialization module, encoding module and distribution module are located on different computing resources, and different tasks are executed in parallel through pipeline mode.

[0029] Optionally, the system further comprises a priority flow control module, which is used to make the priority of training data higher than the priority of checkpoint data, and when there are both training data and checkpoint data to be transmitted, the training data is transmitted first.

[0030] Optionally, the encoding module is used to: use erasure coding technology to encode the serialized data to obtain multiple block data, the block data includes data blocks and redundant blocks, and add a version number to each block data, and add a check value to each block data stored on the local storage node; whenever a new round of checkpoint data is stored, a version number of each new block data is generated according to the checkpoint round, and a check value is added to each block data stored on the local storage node, and the check value is calculated according to the block data and its version number;

[0031] The system also includes a fault recovery module, which is used to select block data according to the following conditions when restoring the computing node using the checkpoint data:

[0032] Condition 1: The version number of the block data selected on all computing nodes is consistent;

[0033] Condition 2: The check value of the block data stored on the local storage node is consistent with the check copy value of the block data. The check copy value of the block data is calculated based on the copy data block of the block data and its version number, and the method for calculating the check copy value is the same as the method for calculating the check value. The copy data block of the block data is the reconstructed data obtained by reconstructing the block data based on the block data stored on other storage nodes.

[0034] In general, compared with the prior art, the above technical solutions conceived by the present invention mainly have the following beneficial effects:

[0035] 1. The present invention analyzes the checkpoint operation and the computing resources of the computing nodes and finds that the serialization, encoding and distribution of the checkpoint can be operated independently, and are only linked to each other through data transmission, and the computing nodes are usually equipped with different computing resources. Based on the independence of the operation point tasks and the flexibility of computing resource allocation, the present invention proposes a pipeline checkpoint operation method, which splits the entire checkpoint process into three independent tasks: serialization, encoding and distribution. Each task corresponds to a computing resource, and these tasks are processed in parallel during the training idle time period. In this way, the present invention can make full use of system resources during the checkpoint operation process, while reducing interference with the training process, and complete the checkpoint operation as quickly as possible during the training idle time period, so that the checkpoint can be effectively saved without affecting the training efficiency.

[0036] 2. Furthermore, the present invention also introduces a priority flow control mechanism to implement network resource allocation for training data and checkpoint data, which can ensure that the training data stream takes precedence over the checkpoint data stream in network transmission, thereby minimizing network congestion and training delays caused by checkpoint operations.

[0037] 3. Furthermore, the present invention also uses erasure coding technology to encode the serialized data. Compared with the traditional checkpoint data replication method, erasure coding technology greatly reduces the storage and network bandwidth occupancy while ensuring system reliability.

[0038] 4. Furthermore, the present invention also introduces a version number mechanism based on the erasure code technology, which can ensure that the version numbers of the block data selected during fault recovery are consistent, and can effectively deal with the problem of possible data inconsistency during fault recovery caused by dynamic changes in memory data.

[0039] 5. Furthermore, the present invention also introduces a check value mechanism based on the erasure code technology, which can ensure the accuracy of fault recovery using checkpoint data and realize an efficient erasure code fault tolerance mechanism based on memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is an overall framework diagram of performing checkpoint operations during distributed pipeline parallel training in one embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of resource idle time in pipeline parallel training in one embodiment;

[0042] Figure 3 is a schematic diagram of a system traffic scheduling framework in one embodiment of the present invention;

[0043] Figure 4 is the impact of checkpoint operations on throughput in distributed training;

[0044] Figure 5 It is a comparison chart of the fault recovery time between the traditional checkpoint and the present invention;

[0045] Figure 6 This is a comparison of the end-to-end training time of different checkpoint mechanisms under specific interference;

[0046] Figure 7 This is a comparison chart of the impact of using different checkpoint mechanisms on model training accuracy under specific interference. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] Example 1

[0049] like Figure 1 FIG. 1 is an overall framework diagram of performing checkpoint operations in a neural network distributed pipeline parallel training process in one embodiment of the present invention. The pipeline parallel training technology divides a large model into multiple consecutive stages, each stage is processed by a different computing node (such as a server), and these computing nodes perform forward propagation and backward propagation of the model in parallel in a pipeline manner. Figure 2 The figure shows a schematic diagram of resource idle time in pipeline parallel training in one embodiment. When data is transferred from one stage to the next, some devices may need to wait for the data to arrive before they can continue to work, causing these devices to be idle. For example, if the calculation of a certain stage is completed quickly, and the next stage is not ready to receive data, then the device of the previous stage will be idle, forming a "bubble". In addition, if the amount of calculation in each stage is unbalanced, some devices may not be able to continue to perform new work while waiting for other devices to complete their tasks, further exacerbating this idle time phenomenon. Therefore, in pipeline parallel training, there will be "idle bubbles" in the computing nodes. These bubbles appear when the nodes are waiting for other nodes to complete their tasks. They are training idle time periods during the training process.

[0050] Based on the above analysis of the pipeline parallel training process, the present invention proposes a checkpoint operation method. At each computing node, by effectively utilizing the training idle time period analyzed above to perform checkpoint operations, the checkpoints can be effectively saved without affecting the normal training process.

[0051] The checkpoint operation method is a pipeline checkpoint operation method, comprising:

[0052] All computing nodes start the checkpoint operation synchronously, and execute the checkpoint operation of the corresponding computing node during the training idle time period of the computing node; wherein the training idle time period of the computing node is the time period during which the computing node waits for other computing nodes to feedback training data during the forward propagation and backward propagation of the neural network; the checkpoint operation includes three tasks: serialization, encoding and distribution, wherein the serialization is to gradually convert the current state data of the corresponding computing node into a storable byte stream, the encoding is to encode the serialized data, and the distribution is to distribute the encoded data to the storage node for storage, each task of the checkpoint operation is handled by a different processor, and different tasks are executed in parallel in a pipeline manner.

[0053] It should be noted that after the checkpoint operation of each computing node is started synchronously, each computing node will perform the checkpoint operation only during its training idle time period.

[0054] Specifically, the checkpoint operation emphasizes dividing the process into multiple tasks and performing pipelined checkpoint operations on the one hand, and emphasizing the effective use of idle time periods for training on the other hand. These two aspects are combined to achieve effective preservation of checkpoints without affecting the normal training process. The following is a detailed description of these two aspects.

[0055] In the process of large-scale deep neural network training, checkpoint operation (i.e. saving the current model state, optimizer state and other necessary system states) is indispensable for recovery when the system fails. Therefore, checkpoint operation is the core link to ensure that the system can recover after failure. However, traditional checkpoint operation is usually a serial process, and all related tasks need to be executed sequentially. This method is not only time-consuming, but also other tasks of the system are often forced to pause during the execution of the checkpoint operation, resulting in a significant decrease in training efficiency. The core idea of ​​the pipelined checkpoint operation proposed in the present invention is to split the entire process of the checkpoint into multiple independent subtasks, and enable these tasks to be processed in parallel during the idle time period of training. In this way, the present invention can make full use of system resources during the checkpoint operation, while reducing interference with the training process, and completing the checkpoint operation as quickly as possible during the idle time period of training.

[0056] By analyzing the checkpoint operation and the computing resources of the computing nodes, it is found that the serialization, encoding and distribution of the checkpoint can be run independently, and they are only linked to each other through data transmission. In addition, the computing nodes are usually equipped with multi-core CPUs, GPUs and high-speed storage devices. Based on the independence of the operation point tasks and the flexibility of computing resource allocation, the present invention divides the checkpoint operation into the following key tasks:

[0057] (1) Serialization: Convert the current state data of the system (such as model parameters, optimizer status, random seeds, etc.) into a storable byte stream.

[0058] (2) Encoding: Encode the serialized data.

[0059] (3) Distribution: Distribute the encoded data to multiple nodes or storage devices to ensure that the data can be recovered when a node fails.

[0060] Moreover, the present invention distributes the above tasks to different computing resources through a task scheduling mechanism, each task corresponds to a computing resource, serialization operations can be performed on a CPU core, while encoding and data distribution can be performed on different CPU cores or GPUs, and each task can be executed separately, ensuring that each subtask can be executed in parallel as much as possible, so that more checkpoint operations can be completed as much as possible during the training idle time period, thereby improving the overall efficiency. When the serialization task completes the processing of part of the data, this part of the data is immediately passed to the next subtask for encoding without waiting for the entire serialization process to be completed. The encoded data blocks can be distributed immediately, forming a "pipeline" mode in which data flows between each subtask. Through this "generation and processing" pipeline processing, the present invention can complete the checkpoint operation in a shorter time and reduce the waiting time between tasks.

[0061] Experimental results show that compared with traditional synchronous checkpoint operations, the pipeline checkpoint operation method proposed in the present invention can significantly reduce the time overhead of checkpoint operations by embedding checkpoint operations into idle training time periods, decomposing checkpoint tasks, and executing tasks in a pipelined parallel manner. It supports a higher checkpoint frequency and reduces interruptions to the training process, providing strong support for achieving efficient and stable checkpoint operations in large-scale deep neural network training, and providing new ideas for optimizing the use of system resources.

[0062] Since different parts of the model are processed in parallel on multiple computing nodes in distributed deep learning training, these nodes need to exchange data (such as activation values ​​and gradients) through the network. At the same time, in order to ensure the fault tolerance of the training process, checkpoint data also needs to be transmitted between different nodes through the network. For example, the checkpoint data on a server will be partially distributed to other servers for storage. When the server fails, the checkpoint data on the server can be constructed through the checkpoint data stored on other servers, thereby improving the fault tolerance of the system. However, the network bandwidth is limited. When checkpoint data and training data need to be transmitted at the same time, it is easy to cause network congestion, resulting in delays in the training process. Traditional checkpoint operations may compete with training data for network resources, thereby affecting training efficiency.

[0063] In order to solve the above problems, in some specific embodiments, a priority flow control mechanism is also introduced to realize the network resource allocation for training data and checkpoint data. The priority flow control mechanism includes: making the priority of training data higher than the priority of checkpoint data. When there are training data and checkpoint data to be transmitted at the same time, the training data is transmitted first.

[0064] Specifically, the introduced priority flow control mechanism can effectively reduce the competition for network bandwidth caused by checkpoint operations during deep neural network (DNN) training. By introducing the priority flow control mechanism in the kernel network stack of the computing node operating system, it can ensure that the training data stream takes precedence over the checkpoint data stream in network transmission, thereby minimizing network congestion and training delays caused by checkpoint operations.

[0065] By implementing priority flow control in the kernel network stack of the operating system, the network competition problem between checkpoint operation and training data flow is solved. The priority flow control mechanism ensures that the training data flow has a higher priority in network transmission than the checkpoint data flow, thus avoiding the interference of checkpoint operation on the training process.

[0066] Specifically, Figure 3 FIG. 1 is a schematic diagram of a system traffic scheduling framework in an embodiment of the present invention. The implementation of the priority traffic control mechanism includes the following key steps:

[0067] (1) Traffic classification and labeling: First, in the application layer, all packets to be sent are classified and labeled. Training packets and checkpoint packets are assigned different priority labels. Among them, training packets are assigned higher priorities, while checkpoint packets are assigned lower priorities. These priority labels can be automatically generated based on user-defined rules or according to the type of packet (for example, using the TOS field in the IP packet).

[0068] (2) Priority queues in the kernel network stack: In the kernel network stack of the operating system, PipeCheck configures multiple queues for packets of different priorities. The network stack assigns packets to the corresponding queues based on their priority tags. High-priority training packets are placed in the high-priority queue, while low-priority checkpoint packets are placed in the low-priority queue.

[0069] (3) Queue scheduling and packet transmission: The network stack processes packets in queue priority order and transmits them to the ring buffer of the network interface controller (NIC). High-priority packets take precedence over low-priority packets during transmission, which ensures that training data can be sent to the network as soon as possible, while checkpoint data is sent only when there is no training data to be transmitted.

[0070] (4) Hardware acceleration and optimization: In some cases, the scheduling operation of priority flow control may introduce additional CPU overhead. To reduce this overhead, PipeCheck can take advantage of the hardware acceleration function of certain network interface cards (NICs) to offload the priority scheduling task to the NIC hardware layer for processing. This approach not only reduces the burden on the CPU, but also reduces the latency of packet transmission. For more complex network configurations, such as using InfiniBand adapters, PipeCheck can also directly hand over the scheduling task to the NIC hardware through the transport layer API (such as rdma_set_option).

[0071] In order to further improve the fault tolerance of the checkpoint system, in one embodiment, the serialized data is encoded, including: encoding the serialized data using erasure coding technology. Erasure coding is a technology widely used in data storage and transmission, which is used to restore the original data through redundant information when the data is lost or damaged. One of the most commonly used erasure codes is Reed-Solomon code, which can distribute data blocks and redundant blocks among multiple servers, so that complete data can still be restored when a certain number of servers fail. In this embodiment, erasure coding technology is applied to the encoding and decoding process of checkpoint data to improve the fault tolerance of the system and reduce the demand for storage space. Compared with traditional checkpoint data replication methods, erasure coding technology greatly reduces the storage and network bandwidth usage while ensuring system reliability.

[0072] The following is a detailed introduction to the specific applications and advantages of erasure coding technology in PipeCheck:

[0073] When performing a checkpoint operation, PipeCheck first serializes the system state (including model parameters, optimizer state, etc.) into a set of data blocks. These data blocks are then encoded using erasure coding techniques such as Reed-Solomon codes to generate corresponding redundant blocks. The encoded data blocks and redundant blocks are distributed to multiple servers to improve the fault tolerance of the data. For example, in a typical scenario, if checkpoint data needs to be saved on 16 servers and up to 4 servers are allowed to fail, PipeCheck only needs to store redundant data equivalent to 1.33 times the size of the original data, instead of 5 times the storage space required in traditional replication methods. When the system detects that some servers have failed, PipeCheck recovers the lost data using the decoding algorithm of the erasure code by collecting the data blocks and redundant blocks on the surviving servers. This process ensures that even in the case of multiple node failures, the system can recover from the most recent checkpoint and continue the training process.

[0074] Since traditional erasure coding mechanisms are usually applied to relatively stable storage spaces such as disks, and the checkpoint data in the present invention is usually stored in memory space, the memory data is dynamically changing, which can easily cause inconsistent or erroneous data read during fault recovery. It is difficult to effectively detect and correct transient errors or illegal modifications of memory data using traditional erasure coding mechanisms.

[0075] In order to solve the above problems, in some specific embodiments, the traditional erasure coding mechanism is improved as follows:

[0076] The serialized data is encoded using erasure coding technology to obtain multiple block data, which include data blocks and redundant blocks. A version number is attached to each block data and then distributedly stored on local storage nodes and other storage nodes. When a new round of checkpoint data is stored, a version number for each new block data is generated according to the checkpoint round. When the checkpoint data is used to restore the computing node, the block data is selected based on the version number to ensure that the version numbers of the block data selected on all computing nodes are consistent.

[0077] In this way, by introducing a version number mechanism to ensure that the version numbers of the block data selected during fault recovery are consistent, the problem of possible data inconsistency during fault recovery caused by dynamic changes in memory data can be effectively addressed.

[0078] Furthermore, in some embodiments, on the basis of introducing the version number mechanism, a check value is also added to each block data stored on the local storage node; when a new round of checkpoint data is stored, a version number of each new block data is generated according to the checkpoint round, and a check value is added to each block data stored on the local storage node, and the check value is calculated based on the corresponding block data and its version number; when using the checkpoint data to restore the computing node, the block data is selected according to the following conditions: Condition 1, the version numbers of the block data selected on all computing nodes are consistent; Condition 2, the check value of the block data stored on the local storage node is consistent with the check copy value of the block data, the check copy value of the block data is calculated based on the copy data block of the block data and its version number, and the method of calculating the check copy value is the same as the method of calculating the check value, and the copy data block of the block data is the reconstructed data obtained by reconstructing the block data based on the block data stored on other storage nodes.

[0079] In condition two, if the data is tampered with or has a transient error, etc., the reconstructed data will be different from the original data. Since the method of calculating the check copy value is the same as the method of calculating the check value, if the reconstructed data is different from the original data, the calculated result will also be different. Therefore, the data can be further verified to be correct based on the check value.

[0080] The following is an example description of the algorithm process:

[0081] (1) Data writing process: The checked data is divided into data blocks D1, D2, ..., D n and redundant blocks P1, P2, ..., P n , add a version number V to each data block and redundant block i and check value H i , the check value can be a hash value, write the data into the memory, and record the version number update operation;

[0082] (2) Data update process: Generate a new version number V new , and recalculate the updated check value H new ,The updates are synchronized to relevant nodes or memory modules through consistency broadcast.

[0083] (3) Fault recovery process: According to the version number V i Find the latest consistent data block set D′1, D′2, …, D′ n , use the checksum value to verify the correctness of the data, and finally use the erasure code decoding algorithm to reconstruct the lost data blocks or redundant blocks and perform fault recovery.

[0084] (4) Fault detection and verification: Regularly compare the verification value H iPerform consistency checks on data blocks and redundant blocks in memory. If a data block error is detected, the fault recovery process is triggered.

[0085] In this way, the version number mechanism can ensure that the selected version number is consistent, and the check value mechanism can ensure that the read data is correct. By combining the version number mechanism and the check mechanism, the accuracy of failure recovery using checkpoint data can be guaranteed, and an efficient memory-based erasure code fault-tolerant mechanism can be implemented, which can be suitable for high-performance computing, distributed storage systems and memory-intensive application scenarios.

[0086] Example 2

[0087] The present invention also relates to a pipeline checkpoint operating system for performing checkpoint operations in a neural network distributed pipeline parallel training process jointly executed by multiple computing nodes. The system is arranged on a computing node and comprises:

[0088] A synchronization module, used to start the checkpoint operation synchronously with other computing nodes;

[0089] A serialization module is used to perform a checkpoint serialization task during the idle training period of the computing node, wherein the serialization is to gradually convert the current state data of the corresponding computing node into a storable byte stream;

[0090] An encoding module is used to perform the encoding task of the checkpoint during the idle training time period of the computing node, wherein the encoding is to encode the serialized data;

[0091] A distribution module is used to perform a checkpoint distribution task during a training idle period of a computing node, wherein the distribution is to distribute the encoded data to a storage node for storage;

[0092] Among them, the serialization module, encoding module and distribution module are located on different computing resources, and different tasks are executed in parallel through pipeline mode.

[0093] In one embodiment, the system further includes a priority flow control module for making the priority of training data higher than the priority of checkpoint data, and when there are both training data and checkpoint data to be transmitted, the training data is transmitted first.

[0094] In one embodiment, the encoding module is specifically used to: use erasure coding technology to encode the serialized data to obtain multiple block data, the block data includes data blocks and redundant blocks, and add a version number to each block data, and add a check value to each block data stored on the local storage node; whenever a new round of checkpoint data is stored, a version number of each new block data is generated according to the checkpoint round, and a check value is added to each block data stored on the local storage node, and the check value is calculated based on the block data and its version number.

[0095] Furthermore, the system also includes a fault recovery module, which is used to select block data according to the following conditions when restoring the computing node using the checkpoint data:

[0096] Condition 1: The version number of the block data selected on all computing nodes is consistent;

[0097] Condition 2: The check value of the block data stored on the local storage node is consistent with the check copy value of the block data. The check copy value of the block data is calculated based on the copy data block of the block data and its version number, and the method for calculating the check copy value is the same as the method for calculating the check value. The copy data block of the block data is the reconstructed data obtained by reconstructing the block data based on the block data stored on other storage nodes.

[0098] It can be understood that the system also includes a monitoring module and a storage module:

[0099] The monitoring module is used to monitor the training progress and identify the idle time periods for training;

[0100] The storage module is used to store checkpoint data.

[0101] The above modules work together to ensure that checkpoint operations are completed efficiently without interfering with the main training process, and training progress is quickly restored when a system failure occurs.

[0102] The specific process is as follows:

[0103] During the pipeline parallel training process, the monitoring module monitors the status of the computing nodes in real time and automatically identifies the idle training time periods (idle bubbles) during training. When a node is waiting for other nodes to complete calculations or data transmission, the monitoring module marks this period as an idle bubble and immediately triggers the serialization module, encoding module, and distribution module to process the checkpoint tasks in parallel in a pipeline manner. For example, the data serialization task can be executed on one CPU core, while the encoding task can be performed in parallel on another core or GPU. The pipelined processing method ensures seamless connection between tasks, thereby greatly shortening the total time of the checkpoint operation.

[0104] During the pipeline processing, the encoding module uses erasure coding technology to encode serialized data and generate redundant data blocks. These data blocks are passed to the distributed storage module and distributed to multiple servers or storage nodes. The application of erasure coding technology greatly reduces storage requirements and improves the fault tolerance of data, so that even if some nodes fail, the system can restore the original data through the remaining data blocks. In addition, through data version control and verification mechanisms, the consistency and correctness of the data blocks called during fault recovery are ensured, ensuring that the system can accurately identify and restore the latest checkpoint data when a failure occurs.

[0105] In order to prevent the checkpoint data transmission from interfering with the main training process, the system introduces a priority flow control mechanism. This mechanism is integrated into the kernel network stack of the operating system. By classifying and scheduling data packets, it ensures that the training data has a higher priority than the checkpoint data in network transmission. Specifically, the network stack will process high-priority training data packets first, and the checkpoint data packets will be transmitted when network resources are idle.

[0106] When the system detects a failure, the fault recovery module quickly initiates the recovery operation. First, the system checks the data blocks stored on all servers to ensure their consistency. If inconsistencies are found, the system decodes and recovers the lost data blocks through erasure coding technology. After the recovery is complete, the system continues training from the most recent checkpoint, minimizing the impact of the failure on the training progress. Throughout the recovery process, data version control and checksum mechanisms ensure data integrity and avoid recovery failures due to the loss or damage of some data blocks.

[0107] like Figure 4 The figure shows the impact of checkpoint operations on throughput in distributed training, where the horizontal axis represents the checkpoint cycle. The longer the checkpoint cycle, that is, the longer the interval between the previous checkpoint operation and the next checkpoint operation, the more idle training time periods that the computing nodes can use. The computing nodes can directly use the idle training time periods to complete the checkpoint operations, and the interference with training is less.

[0108] like Figure 5 The figure shows a comparison of the fault recovery time of the traditional checkpoint (Strawman) and the present invention (PipeCheck). Experiments show that when processing deep neural network models of different sizes (such as BERT-LARGE, GPT-2 and Llama-2), the PipeCheck system can effectively reduce the time occupied by the checkpoint operation, thereby significantly reducing the interference with the training process.

[0109] like Figure 6The figure shows a comparison of the end-to-end training duration of different checkpoint mechanisms under specific interference. It can be seen from the figure that when a failure occurs during the deep neural network training process using the solution of the present invention (PipeCheck), the system can quickly restore the training progress. Compared with the traditional checkpoint (Strawman) method, PipeCheck shortens the recovery time to 6% to 30% of the traditional method. This means that even in a complex large-scale distributed training environment, the present invention can significantly reduce training interruptions caused by failures and ensure the smooth progress of the training process.

[0110] like Figure 7 The figure shows a comparison of the effects of using different checkpoint mechanisms on the accuracy of model training under specific interference. The present invention improves the convergence speed of the overall training by reducing the delay in the training process. Figure 7 It is shown that in the training of BERT-LARGE and GPT-2 models, the training convergence speed of the PipeCheck system is 1.8 to 2.5 times faster than that of the traditional method. This result proves that the present invention not only performs well in reducing the checkpoint operation overhead and recovery time, but also effectively improves the final effect of training.

[0111] It can be understood that the pipeline checkpoint operating system introduced above can be used to implement the pipeline checkpoint operation method introduced in Example 1, wherein the relevant functional modules are used for the corresponding steps in the method. For details, please refer to the above introduction and will not be repeated here.

[0112] The technical features of the above-mentioned embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. It should be noted that "in one embodiment", "for example", "for example", etc. of the present invention are intended to illustrate the present invention, rather than to limit the present invention.

[0113] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A pipeline checkpointing method for performing checkpointing in a neural network distributed pipeline parallel training process jointly executed by multiple computing nodes, characterized in that: The method comprises: All computing nodes start the checkpoint operation synchronously, and perform the checkpoint operation of the corresponding computing nodes during the training idle time period of the computing nodes; The training idle time period of a computing node is the time period during which the computing node waits for other computing nodes to feed back training data during the forward propagation and backward propagation of the neural network; The checkpoint operation includes three tasks: serialization, encoding and distribution. The serialization is to gradually convert the current state data of the corresponding computing node into a storable byte stream, the encoding is to encode the serialized data, and the distribution is to distribute the encoded data to the storage node for storage. Each task of the checkpoint operation is handled by different computing resources, and different tasks are executed in parallel in a pipeline manner.

2. The pipeline checkpoint operation method according to claim 1, characterized in that: Each task of the checkpoint operation is handled by different computing resources, including: the CPU is responsible for serialization tasks, and two different GPUs are responsible for encoding tasks and distribution tasks respectively.

3. The pipeline checkpoint operation method according to claim 1, characterized in that: The method introduces a priority flow control mechanism to implement network resource allocation for training data and checkpoint data. The priority flow control mechanism includes: making the priority of training data higher than the priority of checkpoint data, and when there are both training data and checkpoint data to be transmitted, the training data is transmitted first.

4. The pipeline checkpoint operation method according to claim 3, characterized in that: The method of implementing priority transmission of training data in the priority flow control mechanism includes: allocating training data to a high priority queue, allocating checkpoint data to a low priority queue, and preferentially transmitting data in the high priority queue. When all data in the high priority queue has been transmitted, the data in the low priority queue is transmitted.

5. The pipeline checkpoint operation method according to claim 1, characterized in that: The serialized data is encoded, including: encoding the serialized data using erasure coding technology.

6. The pipeline checkpoint operation method according to claim 5, characterized in that: The serialized data is encoded using erasure coding technology to obtain multiple block data, wherein the block data includes data blocks and redundant blocks, and a version number is added to each block data and then distributedly stored on a local storage node and other storage nodes; When storing a new round of checkpoint data, a version number for each new block data is generated according to the checkpoint round; When using the checkpoint data to restore the computing node, the block data is selected based on the version number to ensure that the version numbers of the block data selected on all computing nodes are consistent.

7. The pipeline checkpoint operation method according to claim 6, characterized in that: A checksum is also added to each block of data stored on the local storage node; Whenever a new round of checkpoint data is stored, a version number of each new block data is generated according to the checkpoint round, and a check value is added to each block data stored on the local storage node. The check value is calculated based on the block data and its version number. Whenever a compute node is restored using checkpoint data, block data is selected according to the following conditions: Condition 1: The version number of the block data selected on all computing nodes is consistent; Condition 2: The check value of the block data stored on the local storage node is consistent with the check copy value of the block data. The check copy value of the block data is calculated based on the copy data block of the block data and its version number, and the method for calculating the check copy value is the same as the method for calculating the check value. The copy data block of the block data is the reconstructed data obtained by reconstructing the block data based on the block data stored on other storage nodes.

8. A pipeline checkpoint operating system for performing checkpoint operations in a neural network distributed pipeline parallel training process jointly executed by multiple computing nodes, characterized in that: The system comprises: A synchronization module, used to start the checkpoint operation synchronously with other computing nodes; A serialization module is used to perform a checkpoint serialization task during the idle training period of the computing node, wherein the serialization is to gradually convert the current state data of the corresponding computing node into a storable byte stream; An encoding module is used to perform the encoding task of the checkpoint during the idle training time period of the computing node, wherein the encoding is to encode the serialized data; A distribution module is used to perform a checkpoint distribution task during a training idle period of a computing node, wherein the distribution is to distribute the encoded data to a storage node for storage; Among them, the serialization module, encoding module and distribution module are located on different computing resources, and different tasks are executed in parallel through pipeline mode.

9. The pipeline checkpoint operating system according to claim 8, characterized in that: The system also includes a priority flow control module, which is used to make the priority of training data higher than the priority of checkpoint data. When there are both training data and checkpoint data to be transmitted, the training data is transmitted first.

10. The pipeline checkpoint operating system according to claim 8, characterized in that: The encoding module is used to: use erasure coding technology to encode the serialized data to obtain multiple block data, the block data includes data blocks and redundant blocks, and add a version number to each block data, and add a check value to each block data stored on the local storage node; Whenever a new round of checkpoint data is stored, a version number of each new block data is generated according to the checkpoint round, and a check value is added to each block data stored on the local storage node. The check value is calculated based on the block data and its version number. The system also includes a fault recovery module, which is used to select block data according to the following conditions when restoring the computing node using the checkpoint data: Condition 1: The version number of the block data selected on all computing nodes is consistent; Condition 2: The check value of the block data stored on the local storage node is consistent with the check copy value of the block data. The check copy value of the block data is calculated based on the copy data block of the block data and its version number, and the method for calculating the check copy value is the same as the method for calculating the check value. The copy data block of the block data is the reconstructed data obtained by reconstructing the block data based on the block data stored on other storage nodes.

Images