A model training checkpoint file storage method and device and related equipment
By employing a multi-level caching asynchronous persistence strategy during model training, checkpoint files are efficiently stored, solving the problem of checkpoint file loss, improving training efficiency and data integrity, and ensuring the continuity of model training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU SHANGHANG INFORMATION TECH CO LTD
- Filing Date
- 2026-03-13
- Publication Date
- 2026-07-03
AI Technical Summary
During model training, frequent hardware failures or power outages can lead to the loss of checkpoint files, affecting training efficiency. Existing technologies struggle to efficiently store checkpoint files, resulting in significant computational losses.
A multi-level caching asynchronous persistence strategy is adopted, from node to local unified memory to distributed storage system. Checkpoint data is copied to local unified memory asynchronously, solid-state drives with non-volatile memory host controller interface specification protocol are used, and checkpoint files are synchronously written to the distributed storage system to ensure data integrity.
It shortens the storage time of checkpoint data, avoids node idleness, improves training efficiency, ensures the integrity and recoverability of checkpoint data, and reduces the impact of training interruption.
Smart Images
Figure CN121833632B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of cloud computing technology, and in particular to a method, apparatus and related equipment for storing checkpoint files during model training. Background Technology
[0002] Checkpoint files play a crucial role in model training. They are primarily used to save the model's state at a specific point in time, including its weights, biases, and optimizer state. By periodically saving checkpoint files, the optimal model can be selected for evaluation and testing after training. By comparing the performance of different checkpoints, the best model configuration can be found. Furthermore, since model training is time-consuming, interruptions due to hardware failures, power outages, or other reasons are inevitable. In such cases, the periodically saved checkpoint files can be used to resume training from the most recent checkpoint, avoiding starting from scratch.
[0003] However, to prevent the loss of checkpoint files due to failures, it is necessary to frequently pause the training task. For large models with hundreds of billions or trillions of parameters, a complete training process usually requires tens of thousands of devices and millions of iterations of optimization. Frequent pauses in the training task will result in a significant loss of computing power and reduced training efficiency. Therefore, how to store checkpoint files during model training has been a long-standing concern. Summary of the Invention
[0004] In view of this, this application provides a method, apparatus and related equipment for storing checkpoint files during model training, so as to facilitate the storage of checkpoint files during model training.
[0005] To achieve the above objectives, the following solution is proposed:
[0006] A method for storing checkpoint files during model training includes:
[0007] Based on the number of nodes and parallel strategy input by the user, generate checkpoint configuration information corresponding to the training process on each node;
[0008] Based on the configuration information of the checkpoints, corresponding checkpoint index information and shard number k are allocated according to the identifier of each training process. This allows each training process on a node to send its own checkpoint data to the training processes on other nodes based on the corresponding checkpoint index information and shard number k, and to receive checkpoint data from k-1 other training processes on other nodes. This ensures that each training process on a node contains its own checkpoint data and checkpoint data from k-1 other training processes on other nodes.
[0009] When the number of training iterations reaches the preset number of iterations, a checkpoint data saving instruction is sent to each node so that each node can asynchronously copy its own checkpoint data and the checkpoint data of k-1 other training processes to the local unified memory. The local unified memory consists of system memory and a solid-state drive based on the non-volatile memory host controller interface specification protocol.
[0010] Each node copies its own checkpoint file to a solid-state drive based on the non-volatile memory host controller interface specification protocol and writes it to the distributed storage system synchronously, so that the distributed storage system stores complete checkpoint data.
[0011] Optional, also includes:
[0012] Monitor the status of each node, and when a fault is detected, count the number of fault training processes.
[0013] If the number of faulty training processes is less than k, a checkpoint data saving instruction is sent to the non-faulty nodes so that each non-faulty node can asynchronously copy its own checkpoint data and the checkpoint data of k-1 other training processes to the local unified memory.
[0014] Select a backup node from the non-faulty nodes, and control the backup node to copy its own checkpoint file and the checkpoint file corresponding to the faulty node to the solid-state drive based on the non-volatile memory host controller interface specification protocol. Write them to the distributed storage system in a synchronous manner, and control other non-faulty nodes to copy their own checkpoint files to the solid-state drive based on the non-volatile memory host controller interface specification protocol. Write them to the distributed storage system in a synchronous manner, so that the distributed storage system stores the complete checkpoint data at the breakpoint.
[0015] Optional, also includes:
[0016] After troubleshooting, retrieve the complete checkpoint data at the breakpoint from the distributed storage system so that the model can continue training from the breakpoint.
[0017] Optional, also includes:
[0018] If the number of fault training processes is greater than or equal to k, then control each node to clear the checkpoint files copied to the solid-state drives based on the non-volatile memory host controller interface specification protocol, and retrieve the latest complete checkpoint files from the distributed storage system to continue training.
[0019] Optionally, the training process on each node sends its own checkpoint data to the training processes on other nodes based on the corresponding checkpoint index information and the number of shards k, and receives checkpoint data from the training processes on k-1 other nodes, including:
[0020] Based on the corresponding checkpoint index information and the number of shards k, the training process on each node uses the high-speed interconnection switching chip within each node to send its own checkpoint data to the training processes on other nodes, and receives checkpoint data from the training processes on k-1 other nodes.
[0021] A checkpoint file storage device for model training, comprising:
[0022] The configuration information generation module is used to generate checkpoint configuration information for the training process on each node based on the number of nodes and parallel strategy input by the user.
[0023] The data processing module is used to allocate corresponding checkpoint index information and shard number k according to the configuration information of the checkpoints and the identifier of each training process, so that the training process on each node can send its own checkpoint data to the training processes on other nodes based on the corresponding checkpoint index information and shard number k, and receive the checkpoint data of the training processes on k-1 other nodes, so that the training process on each node contains its own checkpoint data and the checkpoint data of the training processes on k-1 other nodes.
[0024] The local data storage module is used to send a checkpoint data saving instruction to each node when the number of training iterations reaches the preset number of iterations, so that each node can copy its own checkpoint data and the checkpoint data of k-1 other training processes to the local unified memory in an asynchronous manner. The local unified memory consists of system memory and a solid-state drive based on the non-volatile memory host controller interface specification protocol.
[0025] The distributed data storage module controls each node to write its own checkpoint files, which are copied to solid-state drives based on the non-volatile memory host controller interface specification protocol, into the distributed storage system in a synchronous manner, so that the distributed storage system stores complete checkpoint data.
[0026] Optional, also includes:
[0027] The fault handling module monitors the status of each node. When a fault is detected, it counts the number of faulty training processes. If the number of faulty training processes is less than k, it sends a checkpoint data saving instruction to the non-faulty nodes. This allows each non-faulty node to asynchronously copy its own checkpoint data and the checkpoint data of k-1 other training processes to its local unified memory. The module then selects a backup node from the non-faulty nodes and controls it to synchronously write its own checkpoint file (copied to a solid-state drive based on the non-volatile memory host controller interface specification protocol) and the checkpoint file corresponding to the faulty node into the distributed storage system. It also controls other non-faulty nodes to synchronously write their own checkpoint files (copied to solid-state drives based on the non-volatile memory host controller interface specification protocol) into the distributed storage system, ensuring that the distributed storage system stores complete checkpoint data.
[0028] Optionally, after the fault handling module performs the process of monitoring the status of each node and counting the number of fault training processes when a fault status is detected, it also includes:
[0029] If the number of fault training processes is greater than or equal to k, then control each node to clear the checkpoint files copied to the solid-state drives based on the non-volatile memory host controller interface specification protocol, and retrieve the latest complete checkpoint files from the distributed storage system to continue training.
[0030] A checkpoint file storage device for model training includes: a memory and a processor;
[0031] The memory is used to store programs;
[0032] The processor is configured to execute the program to implement the various steps of the checkpoint file storage method in model training as described in any of the preceding embodiments.
[0033] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the checkpoint file storage method for model training as described in any of the preceding claims.
[0034] As can be seen from the above technical solutions, the method for storing checkpoint files during model training provided in this application includes: generating checkpoint configuration information corresponding to the training process on each node based on the number of nodes and parallel strategy input by the user; allocating corresponding checkpoint index information and shard number k based on the checkpoint configuration information and the identifier of each training process, so that the training process on each node can send its own checkpoint data to the training processes on other nodes based on the corresponding checkpoint index information and shard number k, and receive checkpoint data from k-1 other training processes on other nodes, so that the training process on each node contains its own checkpoint data and k shard number k. -1 checkpoint data of the training process on other nodes; when the training iteration count reaches the preset iteration count, a checkpoint data saving instruction is sent to each node, so that each node can asynchronously copy its own checkpoint data and the checkpoint data of k-1 other training processes to local unified memory, which consists of system memory and solid-state drives based on the non-volatile memory host controller interface specification protocol; control each node to write its own checkpoint file copied to the solid-state drive based on the non-volatile memory host controller interface specification protocol to the distributed storage system in a synchronous manner, so that the distributed storage system stores complete checkpoint data. This application adopts a multi-level caching asynchronous persistence strategy from node to local unified memory to distributed storage system. During normal training, each node only persists its own checkpoint file from local unified memory to distributed storage system, so that the distributed storage system stores complete checkpoint data, which shortens the checkpoint data storage time to a certain extent and avoids node idleness. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0036] Figure 1 A flowchart illustrating a method for storing checkpoint files during model training, provided in an embodiment of this application;
[0037] Figure 2 This application provides a schematic diagram of a checkpoint file storage architecture for model training.
[0038] Figure 3 A schematic diagram of a checkpoint file storage device for model training provided in this application embodiment;
[0039] Figure 4This is a hardware structure block diagram of a checkpoint file storage device for model training provided in an embodiment of this application. Detailed Implementation
[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0041] Figure 1 A flowchart illustrating a method for storing checkpoint files during model training, provided in an embodiment of this application, is included. This method may include:
[0042] Step S100: Based on the number of nodes and parallel strategy input by the user, generate checkpoint configuration information corresponding to the training process on each node.
[0043] Specifically, during the distributed startup of a large model, based on the number of input nodes and the parallel strategy, such as 3D parallelism or 5D parallelism, checkpoint configuration information corresponding to the training process on each node is generated to clarify the affiliation of each training process in the parallel architecture.
[0044] For example, when the input parameters are tensor parallelism tp=4, pipeline parallelism pp=2, data parallelism dp=2, and total number of training processes=16, a three-layer parallel distributed training architecture is generated. The distributed training process grouping information is as follows, where the numbers represent the ID identifier of the training process.
[0045] Parallel pipeline grouping: [[0, 8], [1, 9], [2, 10], [3, 11], [4, 12], [5, 13], [6, 14], [7, 15]]
[0046] Tensor parallel grouping: [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15]]
[0047] Parallel data grouping: [[0, 4], [1, 5], [2, 6], [3, 7], [8, 12], [9, 13], [10, 14], [11, 15]]
[0048] The above configuration information provides the foundation for subsequent checkpoint shard allocation and communication. When launching a large model in a distributed manner, the 16 training processes (ID 0-15) are regrouped according to three dimensions: pipeline parallelism (pp), tensor parallelism (tp), and data parallelism (dp). Checkpoint configuration information is generated, which essentially determines which training processes belong to the same group under this parallelism dimension. This grouping rule is based on the parallelism strategy of tp=4, pp=2, and dp=2. Since the splitting rules of tp, pp, and dp are fixed, the saved training process information can accurately restore the role of each process, which pipeline stage, which data copy, and which tensor shard it belongs to. This ensures that each training process clearly understands its role in the parallel architecture and its associated training processes, thereby correctly completing tensor splitting, pipeline scheduling, and data distribution.
[0049] Step S101: Based on the checkpoint configuration information, allocate the corresponding checkpoint index information and the number of shards k according to the identifier of each training process.
[0050] Specifically, the checkpoint index information is the coordinate of the checkpoint location in the training process. It is generated based on the ID of the training process's rank and the rules of tp, pp, and dp. The core is to tell the training process the index position of the corresponding checkpoint file in the global checkpoint list. For example, rank0 corresponds to index 0, and rank8 corresponds to index 8. The checkpoint belongs to which parallel dimension group. For example, the index information of rank0 will be marked pp=0, dp=0, tp=0, corresponding to pp group [0,8], tp group [0,1,2,3], and dp group [0,4]. The checkpoint needs to be merged with which training process's checkpoints. For example, the index information of rank0 will point to rank1 / 2 / 3 (same as tp group), rank8 (same as pp group), and rank4 (same as tp group).
[0051] The number of slices k is a value strongly related to the parallel strategy. The core idea is how many slices the model part corresponding to the current training process has been split into, and which slice it is. This allows each training process to know which file to read when loading or saving checkpoint files through the checkpoint index information; it also knows that the corresponding file is only a part of the model and its position in the global hierarchy through the number of slices k and its own slice number; and it can be merged with the slices of other training processes to form a complete model during training recovery.
[0052] By assigning corresponding checkpoint index information and shard number k to each training process, the training process on each node sends its own checkpoint data to the training processes on other nodes based on the corresponding checkpoint index information and shard number k, and receives checkpoint data from k-1 other training processes on other nodes. This ensures that each training process on a node contains its own checkpoint data and checkpoint data from k-1 other training processes on other nodes. The sending and receiving of checkpoint data by each node can be achieved using the high-speed interconnect switching chip within each node.
[0053] For training process i, the checkpoint data of training processes ik, i-(k-1), ..., i-1 (a total of k-1 training processes) are sent to training process i via point-to-point communication, so that training process i can back up the checkpoint data of k-1 other training processes, forming a multi-shard storage structure in which each training process contains its own checkpoint data and the checkpoint data backups of k-1 other training processes.
[0054] The process of N training processes simultaneously sending and receiving k fragments forms an N×k two-dimensional matrix. The All-to-All communication operator is used to optimize this communication process, which significantly reduces the interference of redundant checkpoint fragment transmission on the training process and reduces additional computing power loss.
[0055] Step S102: When the number of training iterations reaches the preset number of iterations, send a checkpoint data saving instruction to each node.
[0056] Specifically, the preset number of iterations can be set according to actual needs. During training, after each preset number of iterations, the model slices, optimizer status, and parallel configuration corresponding to each training process will be saved to a file. The saved file usually contains the training process number identifier. When training is resumed, each training process will load its corresponding slice file to ensure the consistency of the parallel architecture.
[0057] Each node asynchronously copies its own checkpoint data and the checkpoint data of k-1 other training processes to local unified memory. Local unified memory consists of system memory and solid-state drives (NVMe-SSDs) based on the Non-volatile Memory Host Controller Interface Specification (NVMe-SSD) protocol. During training, checkpoint files are saved frequently. The low latency and high throughput of NVMe-SSDs can efficiently complete the reading and writing of checkpoint files, reducing training interruption time.
[0058] Step S103: Control each node to copy its own checkpoint file to the solid-state drive based on the non-volatile memory host controller interface specification protocol, and write it to the distributed storage system in a synchronous manner, so that the distributed storage system stores complete checkpoint data.
[0059] For details, please refer to Figure 2 As shown, Figure 2 This is a schematic diagram of a checkpoint file storage architecture provided in an embodiment of this application. When the training process is proceeding normally, each node only persists its own checkpoint file. For example, node 1 persists its own checkpoint file 01.ckpt, node 2 persists its own checkpoint file 02.ckpt, and node N persists its own checkpoint file N.ckpt. A multi-level cache asynchronous persistence strategy is adopted, which goes from the node GPU to the local unified memory and then to the distributed storage system, to shorten the storage time of checkpoint data and avoid node idleness.
[0060] After a preset number of iterations, the checkpoint file is saved. The node copies the checkpoint file from the video memory to the local unified memory in an asynchronous point-to-point manner. The local unified memory consists of system memory and NVMe-SSD solid-state drives based on the Non-Volatile Memory Host Controller Interface Specification protocol. The system memory has a bandwidth of 20GB / s and features high throughput, while the NVMe-SSD features persistence and data loss prevention. Together, they construct the local unified memory, forming a multi-level cache space with both high bandwidth and disk persistence, compensating for the bandwidth difference between the node and traditional storage, and can shorten the asynchronous copy time to a certain extent.
[0061] After the checkpoint files are persisted to the local NVMe SSD, the data is written to the distributed storage system synchronously to ensure global data accessibility. The checkpoint file of a single training process is part of the complete model, and the checkpoint files of all training processes are merged to form the complete CKPT file. After all training processes have completed the persistence of checkpoint files, the backup fragment files on each training process are cleaned up to release GPU memory space and prepare for the next round of iteration training.
[0062] As can be seen from the above technical solution, the method for storing checkpoint files in model training provided in this application includes: generating checkpoint configuration information corresponding to the training process on each node according to the number of nodes and parallel strategy input by the user; based on the checkpoint configuration information, allocating corresponding checkpoint index information and shard number k according to the identifier of each training process, so that the training process on each node can send its own checkpoint data to the training processes on other nodes based on the corresponding checkpoint index information and shard number k, and receive checkpoint data from k-1 other training processes on other nodes, so that the training process on each node contains its own checkpoint data and k shard number k. The system asynchronously copies its own checkpoint data and the checkpoint data of k-1 other training processes to local unified memory. This local unified memory consists of system memory and a solid-state drive (SSD) based on the Non-Volatile Memory Host Controller Interface (NVMC) protocol. Each node then synchronously writes its own checkpoint file, copied to the SSD, to the distributed storage system, ensuring complete checkpoint data is stored in the distributed storage system. This application employs a multi-level caching asynchronous persistence strategy from nodes to local unified memory to the distributed storage system. During normal training, each node only persists its own checkpoint file from local unified memory to the distributed storage system, ensuring complete checkpoint data is stored in the distributed storage system. This shortens the checkpoint data retention time to some extent and prevents nodes from becoming idle.
[0063] In some embodiments of this application, since model training is time-consuming, training interruptions are inevitable due to hardware failures, power outages, or other reasons. Therefore, the method for storing checkpoint files during model training may further include:
[0064] S11. Monitor the status of each node. When a fault is detected, count the number of fault training processes.
[0065] Specifically, during the training process, the node status is monitored in real time. After a failure occurs, the number of failed training processes is immediately counted, and it is determined whether the checkpoint file can be recovered. If the number of failed training processes is less than k, then S12 is executed.
[0066] S12. Send a checkpoint data save command to non-faulty nodes.
[0067] Specifically, if the number of faulty training processes is less than k, it is a recoverable situation. Each non-faulty node asynchronously copies its own checkpoint data and the checkpoint data of k-1 other training processes to the local unified memory.
[0068] S13. Select a backup node from the non-faulty nodes, and control the backup node to copy its own checkpoint file and the checkpoint file corresponding to the faulty node to the solid-state drive based on the non-volatile memory host controller interface specification protocol, and write them to the distributed storage system in a synchronous manner. Also control other non-faulty nodes to copy their own checkpoint files to the solid-state drive based on the non-volatile memory host controller interface specification protocol, and write them to the distributed storage system in a synchronous manner.
[0069] Specifically, the above steps enable the distributed storage system to store complete checkpoint data at the breakpoint.
[0070] Furthermore, in some embodiments of this application, after troubleshooting, based on the complete checkpoint data at the breakpoint stored in the distributed storage system in the above embodiments, the complete checkpoint data at the breakpoint can be retrieved from the distributed storage system so that the model can continue training from the breakpoint, ensuring the continuity of the training process.
[0071] In some embodiments of this application, if the number of faulty training processes is greater than or equal to k, it indicates that the checkpoint file is unrecoverable. In this case, each node can be controlled to clear the checkpoint file copied to the solid-state drive based on the non-volatile memory host controller interface specification protocol, and the latest complete checkpoint file can be retrieved from the distributed storage system to continue training.
[0072] The following describes a checkpoint file storage device for model training provided in an embodiment of this application. The checkpoint file storage device for model training described below can be referred to in correspondence with the checkpoint file storage method for model training described above.
[0073] Figure 3 This application provides a schematic diagram of a checkpoint file storage device structure for model training, as shown in the embodiments of the present application. Figure 3 As shown, the device may include:
[0074] The configuration information generation module 10 is used to generate checkpoint configuration information for the training process on each node based on the number of nodes and parallel strategy input by the user.
[0075] The data processing module 20 is used to allocate corresponding checkpoint index information and shard number k according to the identifier of each training process based on the checkpoint configuration information. This allows each training process on a node to send its own checkpoint data to the training processes on other nodes based on the corresponding checkpoint index information and shard number k, and to receive checkpoint data from k-1 other training processes on other nodes. This ensures that each training process on a node contains its own checkpoint data and checkpoint data from k-1 other training processes on other nodes.
[0076] The local data storage module 30 is used to send a checkpoint data saving instruction to each node when the number of training iterations reaches the preset number of iterations, so that each node can copy its own checkpoint data and the checkpoint data of k-1 other training processes to the local unified memory in an asynchronous manner. The local unified memory consists of system memory and solid-state hard disk based on the non-volatile memory host controller interface specification protocol.
[0077] The distributed data storage module 40 is used to control each node to write its own checkpoint file, which is copied to a solid-state drive based on the non-volatile memory host controller interface specification protocol, into the distributed storage system in a synchronous manner, so that the distributed storage system stores complete checkpoint data.
[0078] As can be seen from the above technical solutions, the checkpoint file storage device for model training provided in this application includes: a configuration information generation module 10, used to generate checkpoint configuration information corresponding to the training process on each node according to the number of nodes and parallel strategy input by the user; and a data processing module 20, used to allocate corresponding checkpoint index information and shard number k according to the identifier of each training process based on the checkpoint configuration information, so that the training process on each node can send its own checkpoint data to the training processes on other nodes based on the corresponding checkpoint index information and shard number k, and receive checkpoint data from k-1 other training processes, so that the training process on each node contains its own checkpoint data and k-1 other checkpoint data. The system includes: checkpoint data for the training process on one other node; a local data storage module 30, used to send a checkpoint data saving instruction to each node when the training iteration count reaches a preset number of iterations, so that each node can asynchronously copy its own checkpoint data and the checkpoint data of k-1 other training processes to a local unified memory, which consists of system memory and a solid-state drive based on the Non-volatile Memory Host Controller Interface Specification (NTHMI) protocol; and a distributed data storage module 40, used to control each node to synchronously write its own checkpoint file copied to the NHTMI to the distributed storage system, so that the distributed storage system stores complete checkpoint data. This application adopts a multi-level caching asynchronous persistence strategy from node to local unified memory to the distributed storage system. During normal training, each node only persists its own checkpoint file from local unified memory to the distributed storage system, ensuring that the distributed storage system stores complete checkpoint data, thus shortening the checkpoint data retention time to some extent and avoiding node idleness.
[0079] Optionally, the checkpoint file storage device during model training may also include:
[0080] The fault handling module monitors the status of each node. When a fault is detected, it counts the number of faulty training processes. If the number of faulty training processes is less than k, it sends a checkpoint data saving instruction to the non-faulty nodes. This allows each non-faulty node to asynchronously copy its own checkpoint data and the checkpoint data of k-1 other training processes to its local unified memory. The module then selects a backup node from the non-faulty nodes and controls it to synchronously write its own checkpoint file (copied to a solid-state drive based on the non-volatile memory host controller interface specification protocol) and the checkpoint file corresponding to the faulty node into the distributed storage system. It also controls other non-faulty nodes to synchronously write their own checkpoint files (copied to solid-state drives based on the non-volatile memory host controller interface specification protocol) into the distributed storage system, ensuring that the distributed storage system stores complete checkpoint data.
[0081] Optionally, after the fault handling module performs the process of monitoring the status of each node and counting the number of fault training processes when a fault status is detected, it may also include:
[0082] If the number of fault training processes is greater than or equal to k, then control each node to clear the checkpoint files copied to the solid-state drives based on the non-volatile memory host controller interface specification protocol, and retrieve the latest complete checkpoint files from the distributed storage system to continue training.
[0083] This application also provides a checkpoint file storage device for model training. Figure 4 This diagram illustrates the hardware structure of a checkpoint file storage device used in model training. (Refer to...) Figure 4 The hardware structure of a checkpoint file storage device in model training may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4.
[0084] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;
[0085] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0086] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;
[0087] The memory stores a program, which the processor can call. The program is used to implement the various processing steps in the aforementioned method for storing checkpoint files during model training.
[0088] This application embodiment also provides a storage medium that can store a program suitable for processor execution, the program being used to implement various processing flows in the aforementioned method for storing checkpoint files in model training.
[0089] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0090] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined with each other, and the same or similar parts can be referred to each other.
[0091] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for storing checkpoint files during model training, characterized in that, include: Based on the number of nodes and parallel strategy input by the user, generate checkpoint configuration information corresponding to the training process on each node; Based on the configuration information of the checkpoints, corresponding checkpoint index information and shard number k are allocated according to the identifier of each training process. This allows each training process on a node to send its own checkpoint data to the training processes on other nodes based on the corresponding checkpoint index information and shard number k, and to receive checkpoint data from k-1 other training processes on other nodes. This ensures that each training process on a node contains its own checkpoint data and checkpoint data from k-1 other training processes on other nodes. When the number of training iterations reaches the preset number of iterations, a checkpoint data saving instruction is sent to each node so that each node can asynchronously copy its own checkpoint data and the checkpoint data of k-1 other training processes to the local unified memory. The local unified memory consists of system memory and a solid-state drive based on the non-volatile memory host controller interface specification protocol. Each node copies its own checkpoint file to a solid-state drive based on the non-volatile memory host controller interface specification protocol and writes it to the distributed storage system synchronously, so that the distributed storage system stores complete checkpoint data.
2. The method according to claim 1, characterized in that, Also includes: Monitor the status of each node, and when a fault is detected, count the number of fault training processes. If the number of faulty training processes is less than k, a checkpoint data saving instruction is sent to the non-faulty nodes so that each non-faulty node can asynchronously copy its own checkpoint data and the checkpoint data of k-1 other training processes to the local unified memory. Select a backup node from the non-faulty nodes, and control the backup node to copy its own checkpoint file and the checkpoint file corresponding to the faulty node to the solid-state drive based on the non-volatile memory host controller interface specification protocol. Write them to the distributed storage system in a synchronous manner, and control other non-faulty nodes to copy their own checkpoint files to the solid-state drive based on the non-volatile memory host controller interface specification protocol. Write them to the distributed storage system in a synchronous manner, so that the distributed storage system stores the complete checkpoint data at the breakpoint.
3. The method according to claim 2, characterized in that, Also includes: After troubleshooting, retrieve the complete checkpoint data at the breakpoint from the distributed storage system so that the model can continue training from the breakpoint.
4. The method according to claim 2, characterized in that, Also includes: If the number of fault training processes is greater than or equal to k, then control each node to clear the checkpoint files copied to the solid-state drives based on the non-volatile memory host controller interface specification protocol, and retrieve the latest complete checkpoint files from the distributed storage system to continue training.
5. The method according to claim 1, characterized in that, Each training process on a node sends its own checkpoint data to the training processes on other nodes based on the corresponding checkpoint index information and the number of shards k, and receives checkpoint data from k-1 other training processes on other nodes, including: Based on the corresponding checkpoint index information and the number of shards k, the training process on each node uses the high-speed interconnection switching chip within each node to send its own checkpoint data to the training processes on other nodes, and receives checkpoint data from the training processes on k-1 other nodes.
6. A checkpoint file storage device for model training, characterized in that, include: The configuration information generation module is used to generate checkpoint configuration information for the training process on each node based on the number of nodes and parallel strategy input by the user. The data processing module is used to allocate corresponding checkpoint index information and shard number k according to the configuration information of the checkpoints and the identifier of each training process, so that the training process on each node can send its own checkpoint data to the training processes on other nodes based on the corresponding checkpoint index information and shard number k, and receive the checkpoint data of the training processes on k-1 other nodes, so that the training process on each node contains its own checkpoint data and the checkpoint data of the training processes on k-1 other nodes. The local data storage module is used to send a checkpoint data saving instruction to each node when the number of training iterations reaches the preset number of iterations, so that each node can copy its own checkpoint data and the checkpoint data of k-1 other training processes to the local unified memory in an asynchronous manner. The local unified memory consists of system memory and a solid-state drive based on the non-volatile memory host controller interface specification protocol. The distributed data storage module controls each node to write its own checkpoint files, which are copied to solid-state drives based on the non-volatile memory host controller interface specification protocol, into the distributed storage system in a synchronous manner, so that the distributed storage system stores complete checkpoint data.
7. The apparatus according to claim 6, characterized in that, Also includes: The fault handling module is used to monitor the status of each node. When a fault is detected, the number of fault training processes is counted. If the number of faulty training processes is less than k, a checkpoint data saving instruction is sent to the non-faulty nodes. This allows each non-faulty node to asynchronously copy its own checkpoint data and the checkpoint data of k-1 other training processes to its local unified memory. A backup node is selected from the non-faulty nodes, and this backup node synchronously writes its own checkpoint file (copied to a solid-state drive based on the non-volatile memory host controller interface specification protocol) and the checkpoint file corresponding to the faulty node into the distributed storage system. Simultaneously, other non-faulty nodes also synchronously write their own checkpoint files (copied to solid-state drives based on the non-volatile memory host controller interface specification protocol) into the distributed storage system, ensuring that the distributed storage system stores complete checkpoint data.
8. The apparatus according to claim 7, characterized in that, After monitoring the status of each node and counting the number of fault training processes when a fault is detected, the fault handling module also includes: If the number of fault training processes is greater than or equal to k, then control each node to clear the checkpoint files copied to the solid-state drives based on the non-volatile memory host controller interface specification protocol, and retrieve the latest complete checkpoint files from the distributed storage system to continue training.
9. A checkpoint file storage device for model training, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the checkpoint file storage method in model training as described in any one of claims 1-5.
10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the checkpoint file storage method in model training as described in any one of claims 1-5.