Checkpoint loading method and device in distributed training, storage medium and program product

Abstract

The invention relates to a check point loading method and device in distributed training, a storage medium and a program product. The method comprises the following steps: segmenting a check point file in the shared storage device into multiple pieces of block data; for any training node in a training node cluster corresponding to the target distributed training task, loading block data corresponding to the training node from a shared storage device, and storing the block data corresponding to the training node into a local storage device of the training node; and exchanging block data among different training nodes of the training node cluster by adopting a point-to-point technology until each training node of the training node cluster respectively obtains each piece of block data corresponding to the check point file. According to the method and the device, the loading speed of the check point can be obviously increased, and the loading bottleneck problem caused by storage bandwidth limitation is effectively avoided.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a checkpoint loading method in distributed training, a checkpoint loading device in distributed training, a non-volatile computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of artificial intelligence technologies, large model training has become a key link in promoting technological progress. Large model training usually requires the assistance of hundreds or thousands of acceleration cards, and the training cycle may last for weeks or even months. However, during such a long training process, it is common for training to be interrupted due to faults. To ensure the continuity and reliability of training, the training system must have fault tolerance, that is, the training program can resume the state of the model and optimizer before the fault after the fault occurs, so as to continue training.

[0003] To achieve this goal, existing training programs generally adopt a periodic checkpoint scheme to persistently store the training state in a storage device regularly. When recovering from a fault, the system will load the checkpoint from the storage device to restore the training state. However, the checkpoint loading process of the distributed training scheme is time-consuming. Summary of the Invention

[0004] In view of this, the present disclosure provides a technical solution for checkpoint loading in distributed training.

[0005] According to one aspect of the present disclosure, there is provided a checkpoint loading method in distributed training, including:

[0006] Dividing a checkpoint file in a shared storage device into multiple pieces of chunk data;

[0007] For any training node in a training node cluster corresponding to a target distributed training task, loading the chunk data corresponding to the training node from the shared storage device, and storing the chunk data corresponding to the training node in a local storage device of the training node;

[0008] Exchanging chunk data between different training nodes in the training node cluster by using a peer-to-peer technology until each training node in the training node cluster obtains each piece of chunk data corresponding to the checkpoint file.

[0009] In a possible implementation manner, a peer-to-peer server and a peer-to-peer client are respectively set on each training node;

[0010] The exchanging chunk data between different training nodes in the training node cluster by using a peer-to-peer technology includes:

[0011] For the first training node in the training node cluster, a chunk data reading request is sent to the peer-to-peer server of the second training node in the training node cluster through the peer-to-peer client of the first training node; for the second training node, in response to the chunk data reading request, the peer-to-peer server of the second training node returns the chunk data corresponding to the second training node to the peer-to-peer client of the first training node; wherein, the first training node and the second training node are any two different training nodes in the training node cluster.

[0012] In a possible implementation, chunk data is exchanged between different training nodes through a Remote Direct Memory Access (RDMA) network.

[0013] In a possible implementation, the Remote Direct Memory Access (RDMA) network includes a wireless bandwidth network and / or an Ethernet-fused Remote Direct Memory Access (RDMA) network.

[0014] In a possible implementation, the splitting of the checkpoint file in the shared storage device into multiple chunks of data includes:

[0015] The checkpoint file in the shared storage device is split into multiple chunks of data according to the number of training subtasks in the training node cluster, wherein the number of chunks of data is equal to the number of training subtasks.

[0016] In a possible implementation, for any training node in the training node cluster corresponding to a target distributed training task, loading the chunk data corresponding to the training node from the shared storage device and storing the chunk data corresponding to the training node in the local storage device of the training node includes:

[0017] For any training subtask in any training node in the training node cluster corresponding to a target distributed training task, loading the chunk data corresponding to the training subtask from the shared storage device and storing the chunk data corresponding to the training subtask in the local storage device of the training node.

[0018] In a possible implementation, the local storage device of the training node includes memory and external storage;

[0019] The storing of the chunk data corresponding to the training node in the local storage device of the training node includes:

[0020] In response to the remaining capacity of the memory of the training node being greater than or equal to the data volume of the chunk data corresponding to the training node, storing the chunk data corresponding to the training node in the memory of the training node;

[0021] Or,

[0022] In response to the remaining capacity of the memory of the training node being less than the amount of chunk data corresponding to the training node, store the chunk data corresponding to the training node in the external memory of the training node.

[0023] In a possible implementation, the external memory includes a storage device based on the Non-Volatile Memory Express (NVMe) protocol.

[0024] In a possible implementation, the method further includes:

[0025] For any training node in the training node cluster, in response to the training node obtaining each piece of chunk data corresponding to the checkpoint file, load the complete checkpoint file from the local storage device of the training node and resume training.

[0026] In a possible implementation, the method is integrated into a distributed training framework.

[0027] According to another aspect of the present disclosure, there is provided a checkpoint loading device in distributed training, including:

[0028] A splitting module, configured to split a checkpoint file in a shared storage device into multiple pieces of chunk data;

[0029] A loading module, configured to, for any training node in a training node cluster corresponding to a target distributed training task, load the chunk data corresponding to the training node from the shared storage device and store the chunk data corresponding to the training node in the local storage device of the training node;

[0030] A swapping module, configured to swap chunk data between different training nodes in the training node cluster by using a peer-to-peer technology until each training node in the training node cluster respectively obtains each piece of chunk data corresponding to the checkpoint file.

[0031] In a possible implementation, a peer-to-peer server and a peer-to-peer client are respectively set on each training node;

[0032] The swapping module is configured to:

[0033] For the first training node in the training node cluster, send a chunk data reading request to the peer-to-peer server of the second training node in the training node cluster through the peer-to-peer client of the first training node; for the second training node, in response to the chunk data reading request through the peer-to-peer server of the second training node, return the chunk data corresponding to the second training node to the peer-to-peer client of the first training node; wherein, the first training node and the second training node are any two different training nodes in the training node cluster.

[0034] In a possible implementation, chunk data is exchanged between different training nodes through a Remote Direct Memory Access (RDMA) network.

[0035] In a possible implementation, the Remote Direct Memory Access (RDMA) network includes a wireless bandwidth network and / or an Ethernet-fused Remote Direct Memory Access (RDMA) network.

[0036] In a possible implementation, the splitting module is configured to:

[0037] Split the checkpoint file in the shared storage device into multiple chunks of data according to the number of training subtasks in the training node cluster, wherein the number of chunks of data is equal to the number of training subtasks.

[0038] In a possible implementation, the loading module is configured to:

[0039] For any training subtask in any training node in the training node cluster corresponding to a target distributed training task, load the chunk data corresponding to the training subtask from the shared storage device and store the chunk data corresponding to the training subtask in the local storage device of the training node.

[0040] In a possible implementation, the local storage device of the training node includes memory and external storage;

[0041] The loading module is configured to:

[0042] In response to the remaining capacity of the memory of the training node being greater than or equal to the data volume of the chunk data corresponding to the training node, store the chunk data corresponding to the training node in the memory of the training node;

[0043] Or,

[0044] In response to the remaining capacity of the memory of the training node being less than the data volume of the chunk data corresponding to the training node, store the chunk data corresponding to the training node in the external storage of the training node.

[0045] In a possible implementation, the external memory includes a storage device based on the Non-Volatile Memory Express (NVMe) protocol.

[0046] In a possible implementation, the apparatus further includes:

[0047] A recovery training module, configured to, for any training node in the training node cluster, in response to the training node obtaining the respective chunk data corresponding to the checkpoint file, load the complete checkpoint file from the local storage device of the training node, and resume training.

[0048] In a possible implementation, the apparatus is integrated into a distributed training framework.

[0049] According to another aspect of the present disclosure, there is provided a checkpoint loading apparatus in distributed training, including a memory, a processor, and a computer program stored on the memory, where the processor executes the computer program to implement the steps of the above method.

[0050] According to another aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0051] According to another aspect of the present disclosure, there is provided a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0052] In the embodiments of the present disclosure, by splitting the checkpoint file in the shared storage device into multiple chunks of data, for any training node in the training node cluster corresponding to the target distributed training task, loading the chunk data corresponding to the training node from the shared storage device, storing the chunk data corresponding to the training node in the local storage device of the training node, and exchanging chunk data between different training nodes in the training node cluster using a peer-to-peer technology until each training node in the training node cluster respectively obtains the respective chunk data corresponding to the checkpoint file. Thus, by splitting the checkpoint file and using the high-speed network and peer-to-peer technology between training nodes for chunk data exchange, the loading speed of the checkpoint can be significantly accelerated, effectively avoiding the loading bottleneck problem caused by storage bandwidth limitations. In addition, this method can be seamlessly integrated into the existing distributed training framework to achieve fast recovery without the user's awareness, improving the overall training efficiency.

[0053] According to the following detailed description of exemplary embodiments with reference to the accompanying drawings, other features and aspects of the present disclosure will become apparent. Description of the Drawings

[0054] The accompanying drawings, which are included in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and are used to explain the principles of the present disclosure together with the specification.

[0055] Figure 1 A flowchart showing the checkpoint loading method in distributed training provided by an embodiment of the present disclosure.

[0056] Figure 2 A schematic diagram showing the division of a checkpoint file according to the number of training nodes in the checkpoint loading method in distributed training provided by an embodiment of the present disclosure.

[0057] Figure 3 A schematic diagram showing the division of a checkpoint file according to the number of training subtasks in the checkpoint loading method in distributed training provided by an embodiment of the present disclosure.

[0058] Figure 4 A schematic diagram showing an application scenario of the checkpoint loading method in distributed training provided by an embodiment of the present disclosure.

[0059] Figure 5 A block diagram showing the checkpoint loading device in distributed training provided by an embodiment of the present disclosure.

[0060] Figure 6 A block diagram of a checkpoint loading device 1900 in distributed training shown according to an exemplary embodiment. Detailed implementation manners

[0061] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0062] As used herein, the terms "comprising", "including", "having", or variations thereof are open-ended and include one or more stated features, wholes, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, wholes, elements, steps, components, functions, or groups thereof.

[0063] When an element is referred to as being "connected", "coupled", "responsive" or variations thereof to another element, it can be directly connected, coupled, or responsive to the other element, or there can be intervening elements.

[0064] Although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Thus, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0065] As used herein, the term "exemplary" means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or better than other embodiments.

[0066] In addition, for a better illustration of the present disclosure, numerous specific details are given in the following detailed description. Those skilled in the art should understand that the present disclosure can be implemented without some specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0067] As described above, there are obvious bottlenecks in the distributed training scheme in the related art when loading checkpoints. Specifically, all training nodes will load the same checkpoint file from the storage device simultaneously. For large models, the volume of the checkpoint file is usually large, which makes the loading process time-consuming.

[0068] Taking the training in kcal as an example, assuming the storage bandwidth is 30 GB / s, the time required to load a 2 TB checkpoint file is: 2 TB×1000 / (30 GB / s) = 18 hours. However, in actual applications, due to the competition for storage bandwidth by other services, the actual loading time may far exceed 18 hours. If it is required to complete the loading within 30 minutes, the required storage bandwidth will reach: 2 TB×1000 / (30 min×60 s / min) = 1111 GB / s. This means that at least 370 open-source Ceph storage nodes, or 12 commercial storage DDN (Data Distribution Network) controllers (single control 90 GB / s) are required to meet the demand.

[0069] It can be seen that the bandwidth that the backend storage in the related art can provide is relatively limited, which severely restricts the rapid loading of large model checkpoints. This storage bandwidth bottleneck not only leads to an overly long training recovery time but may also affect the efficiency and reliability of the entire training system. Therefore, how to break through the storage bandwidth bottleneck and achieve the rapid loading of large model checkpoints has become an urgent technical problem to be solved in the current field of distributed training.

[0070] To solve the technical problems similar to those described above, embodiments of the present disclosure provide a checkpoint loading method in distributed training. By splitting the checkpoint file in the shared storage device into multiple chunks of data, for any training node in the training node cluster corresponding to the target distributed training task, load the chunk of data corresponding to the training node from the shared storage device, and store the chunk of data corresponding to the training node in the local storage device of the training node. And between different training nodes in the training node cluster, use peer-to-peer technology to exchange chunk data until each training node in the training node cluster respectively obtains each chunk of data corresponding to the checkpoint file. Thus, by splitting the checkpoint file and using the high-speed network and peer-to-peer (P2P) technology between training nodes for chunk data exchange, the loading speed of the checkpoint can be significantly accelerated, and the loading bottleneck problem caused by storage bandwidth limitations can be effectively avoided. In addition, this method can be seamlessly integrated into the existing distributed training framework to achieve fast recovery without the user's awareness and improve the overall training efficiency.

[0071] The following will describe in detail the checkpoint loading method in distributed training provided by the embodiments of the present disclosure with reference to the accompanying drawings.

[0072] Figure 1 The flowchart showing the checkpoint loading method in distributed training provided by the embodiments of the present disclosure is shown. In a possible implementation manner, the execution subject of the checkpoint loading method in distributed training may be a checkpoint loading device in distributed training. For example, the checkpoint loading method in distributed training may be executed by a terminal device, a server, or other electronic devices. Among them, the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, or a wearable device, etc. In some possible implementation manners, the checkpoint loading method in distributed training may be implemented by a processor calling computer-readable instructions stored in a memory. As Figure 1 shown, the checkpoint loading method in distributed training includes steps S11 to S13.

[0073] In step S11, split the checkpoint file in the shared storage device into multiple chunks of data.

[0074] In step S12, for any training node in the training node cluster corresponding to the target distributed training task, load the chunk of data corresponding to the training node from the shared storage device, and store the chunk of data corresponding to the training node in the local storage device of the training node.

[0075] In step S13, between different training nodes of the training node cluster, the point-to-point technology is adopted to exchange the chunk data until each training node of the training node cluster respectively obtains each chunk data corresponding to the checkpoint file.

[0076] In the embodiments of the present disclosure, the target distributed training task may refer to a specific training task that needs to be executed or is being executed in a distributed training environment. In distributed training, usually a large training task is decomposed into multiple training subtasks, and then these training subtasks are assigned to different training nodes (which can also be called computing nodes, physical nodes or worker nodes) for parallel processing.

[0077] The training node cluster may refer to a collaborative working group composed of multiple training nodes in a distributed training environment. These training nodes can be connected through a high-speed network and jointly undertake the computing load of the distributed training task. Each training node can be equipped with computing resources (such as CPU (Central Processing Unit, central processor), GPU (Graphics Processing Unit, graphics processor) or TPU (Tensor Processing Unit, tensor processor)) and storage resources, used to execute specific training subtasks and perform data interaction and collaborative work within the cluster. In this way, the training node cluster can efficiently process large-scale training tasks, improve training efficiency, and at the same time have better fault tolerance and resource scalability to meet complex model training requirements.

[0078] In a possible implementation manner, any training node may include at least one GPU and may run at least one training subtask. For example, each training node may respectively include 8 GPUs, and one training subtask may be run on each GPU respectively. Thus, 8 training subtasks may be run on each training node.

[0079] In the embodiments of the present disclosure, the shared storage device may refer to a device or storage system that provides unified storage services for multiple training nodes in a distributed system. The shared storage device allows different training nodes to access and share the data in the same storage space through the network, ensuring data consistency and availability. In a distributed training scenario, the shared storage device can be used to store data related to the training task, such as model parameters, training datasets, and checkpoint files. In this way, each training node can work collaboratively, read or write data from the shared storage device, thereby realizing centralized management and efficient utilization of data, and at the same time facilitating data backup and recovery in case of training node failure or task recovery.

[0080] In a distributed training system, checkpoint files are used to save the state of the model (including model parameters, optimizer state, etc.) so that training can be resumed from them in case of a failure. Checkpoint files are usually large, especially for large-scale deep learning models, and their size can reach the order of several terabytes. If the entire checkpoint file is directly loaded from a shared storage device, due to the limitation of storage bandwidth, the loading time will be very long, resulting in a slow training recovery process. In the embodiments of the present disclosure, the checkpoint file can be logically divided into multiple smaller chunk data, and each chunk data can be assigned a unique identifier (ID). The number of chunk data can be determined according to factors such as the number of training nodes or the number of training subtasks, which is not limited herein. The sizes of the divided chunk data can be the same or different. For example, the size of the chunk data can be adjusted according to factors such as the performance of the storage device in the training node and the network bandwidth.

[0081] In a possible implementation, the splitting of the checkpoint file in the shared storage device into multiple chunk data includes: splitting the checkpoint file in the shared storage device into multiple chunk data according to the number of training nodes in the training node cluster, where the number of the chunk data is equal to the number of the training nodes.

[0082] In this implementation, the number of chunk data can be determined according to the number of training nodes in the training node cluster. For example, if there are 100 training nodes, then the checkpoint file can be split into 100 chunk data. Each chunk data can be assigned to a specific training node. This means that each training node is only responsible for loading its own chunk data instead of the entire checkpoint file. Each chunk data can be assigned a unique identifier (ID), and this ID can correspond to the rank value of the training node so that the training node can identify and load the correct chunk data.

[0083] Figure 2 FIG. shows a schematic diagram of splitting a checkpoint file according to the number of training nodes in the checkpoint loading method in distributed training provided by the embodiments of the present disclosure. In Figure 2 In the example shown, the training node cluster corresponding to the target distributed training task includes 3 training nodes (training node 1, training node 2, and training node 3). Therefore, the checkpoint file can be split into 3 chunk data (chunk 0, chunk 1, chunk 2). Training node 1 can load chunk 0 from the shared storage device and store chunk 0 in the local storage device of training node 1; training node 2 can load chunk 1 from the shared storage device and store chunk 1 in the local storage device of training node 2; training node 3 can load chunk 2 from the shared storage device and store chunk 2 in the local storage device of training node 3.

[0084] Since each training node only loads its own chunk of data, the training nodes can load data from the shared storage device in parallel, rather than all training nodes loading the entire checkpoint file simultaneously, thus reducing the competition for network bandwidth. After the loading is completed, the training nodes can exchange chunk data through peer-to-peer (P2P) technology so that each training node can finally obtain the complete checkpoint file.

[0085] By adopting this implementation method, the loading process of the checkpoint file can be more efficient because each training node only processes a part of the data instead of the entire checkpoint file. This method can significantly reduce the loading time of the checkpoint file, improve the speed of training recovery, and thus optimize the overall performance of distributed training.

[0086] In another possible implementation method, the splitting of the checkpoint file in the shared storage device into multiple chunks of data includes: splitting the checkpoint file in the shared storage device into multiple chunks of data according to the number of training subtasks in the training node cluster, where the number of the chunk data is equal to the number of the training subtasks.

[0087] In a distributed training system, a training task can be composed of multiple training subtasks, and each training subtask can be executed on different training nodes. To improve the efficiency of checkpoint loading, in this implementation method, the checkpoint file in the shared storage device can be split according to the number of training subtasks. For example, if there are 200 training subtasks, then the checkpoint file can be split into 200 chunks.

[0088] Figure 3 A schematic diagram showing the splitting of the checkpoint file according to the number of training subtasks in the checkpoint loading method in distributed training provided by the embodiments of the present disclosure is shown. In Figure 3 In the shown example, the training node cluster corresponding to the target distributed training task includes 3 training nodes, and 2 training subtasks are respectively run on each training node. Therefore, the checkpoint file can be split into 6 chunks of data. And each training subtask can respectively read the corresponding chunk of data. The training subtasks can exchange chunk data through an InfiniBand (IB) network.

[0089] In this implementation method, by splitting the checkpoint file into multiple chunks of data and allowing each training subtask to only load its own chunk of data, the loading speed of the checkpoint can be significantly improved.

[0090] As an example of this implementation manner, for any training node in the training node cluster corresponding to the target distributed training task, loading the chunk data corresponding to the training node from the shared storage device and storing the chunk data corresponding to the training node in the local storage device of the training node includes: for any training subtask in any training node in the training node cluster corresponding to the target distributed training task, loading the chunk data corresponding to the training subtask from the shared storage device and storing the chunk data corresponding to the training subtask in the local storage device of the training node.

[0091] In this example, each chunk of data can be assigned to a specific training subtask. This means that each training subtask is only responsible for loading its own chunk of data, rather than the entire checkpoint file. Each chunk of data can be assigned a unique identifier (ID), and this ID can correspond to the rank value of the training subtask, so that the training subtask can identify and load the correct chunk of data.

[0092] Since each training subtask only loads its own chunk of data, data can be loaded from the shared storage device in parallel, rather than all training subtasks loading the entire checkpoint file simultaneously, thus reducing competition for network bandwidth. After the loading is completed, the training subtasks can exchange chunk data through peer-to-peer (P2P) technology so that each training subtask can finally obtain the complete checkpoint file.

[0093] In this example, the loading process of the checkpoint file can be more efficient because each training subtask only processes a part of the data, rather than the entire checkpoint file. This method can significantly reduce the loading time and improve the speed of training recovery, thereby optimizing the overall performance of distributed training.

[0094] In a possible implementation manner, the local storage device of the training node includes memory and external storage; storing the chunk data corresponding to the training node in the local storage device of the training node includes: in response to the remaining capacity of the memory of the training node being greater than or equal to the data volume of the chunk data corresponding to the training node, storing the chunk data corresponding to the training node in the memory of the training node; or, in response to the remaining capacity of the memory of the training node being less than the data volume of the chunk data corresponding to the training node, storing the chunk data corresponding to the training node in the external storage of the training node.

[0095] In this implementation, each training node is equipped with a local storage device, including memory and external storage. Among them, the memory has a very fast read / write speed but limited capacity; while the external storage usually has a larger capacity but slower read / write speed. To optimize performance and resource utilization, in this implementation, the storage of chunk data can be dynamically determined to be stored in memory or external storage according to the remaining capacity of the memory.

[0096] Specifically, the remaining capacity of the memory of the current training node can be first checked to determine whether it can accommodate the chunk data corresponding to the training node. If the remaining capacity of the memory is greater than or equal to the data volume of the chunk data (i.e., the memory is sufficient to store the chunk data), the chunk data can be stored in the memory. In this way, the high-speed read / write characteristics of the memory can be utilized to accelerate subsequent training tasks. If the remaining capacity of the memory is less than the data volume of the chunk data (i.e., the memory is not sufficient to store the chunk data), the chunk data can be stored in the external storage. Although the read / write speed of the external storage is slower, its larger capacity can ensure that the data can be completely stored.

[0097] The purpose of this dynamic storage strategy is to make the most of the high-speed characteristics of the memory to improve the training efficiency as much as possible under limited memory resources, while avoiding the situation where data cannot be stored due to insufficient memory. By flexibly selecting the storage location, a balance can be achieved between performance and resource utilization, thereby optimizing the overall efficiency of distributed training.

[0098] As an example of this implementation, the external storage includes a storage device based on the Non-Volatile Memory Express (NVMe) protocol.

[0099] In this example, each training node can be equipped with one or more storage devices based on the NVMe protocol (such as solid-state drives). The storage device based on the NVMe protocol can be used to store checkpoint files and other temporary data during the training process. When loading the checkpoint file, the high bandwidth and low latency characteristics of the storage device based on the NVMe protocol can significantly reduce the data loading time, thereby accelerating the recovery speed of the training task. Therefore, by equipping each training node with a storage device based on the NVMe protocol, fast data read / write operations can be achieved, thereby improving the overall performance and reliability of the distributed training system.

[0100] In a possible implementation, a peer-to-peer server and a peer-to-peer client are respectively set on each training node; exchanging chunk data between different training nodes in the training node cluster by using peer-to-peer technology includes: for a first training node in the training node cluster, sending a chunk data reading request to the peer-to-peer server of a second training node in the training node cluster through the peer-to-peer client of the first training node; for the second training node, in response to the chunk data reading request through the peer-to-peer server of the second training node, returning the chunk data corresponding to the second training node to the peer-to-peer client of the first training node; where the first training node and the second training node are any two different training nodes in the training node cluster.

[0101] In this implementation, a peer-to-peer server and a peer-to-peer client can be respectively set on each training node. Among them, the peer-to-peer server of each training node can be used to manage local data storage and provide a chunk data reading service for other training nodes; the peer-to-peer client of each training node can be used to read chunk data from other training nodes.

[0102] Suppose there are two training nodes: a first training node and a second training node. The P2P client of the first training node needs to obtain specific chunk data from the second training node. The P2P client of the first training node sends a chunk data reading request to the P2P server of the second training node. After receiving the request, the P2P server of the second training node finds the corresponding chunk data from the local storage device. The P2P server of the second training node sends the chunk data back to the P2P client of the first training node.

[0103] Through P2P technology, chunk data can be directly exchanged between training nodes, reducing the dependence on the central storage device, thereby reducing the load on the central storage. This direct data exchange method can make more full use of network bandwidth and improve the efficiency of data transmission. In addition, P2P data exchange can also reduce the latency of data transmission because the data is directly transmitted between training nodes instead of being relayed through a central node. Therefore, this method of exchanging chunk data between training nodes by using peer-to-peer technology can improve the speed and efficiency of checkpoint loading in distributed training, while reducing the demand for central storage resources and improving the scalability and robustness of the entire system.

[0104] In another possible implementation, a peer-to-peer server and a peer-to-peer client for each training subtask are respectively set on each training node; exchanging chunk data between different training nodes in the training node cluster by using peer-to-peer technology includes: for a first training subtask in a first training node in the training node cluster, sending a chunk data reading request to a peer-to-peer server of a second training subtask in a second training node in the training node cluster through the peer-to-peer client of the first training subtask; and the peer-to-peer server of the second training subtask returns the chunk data corresponding to the second training subtask to the peer-to-peer client of the first training subtask in response to the chunk data reading request; wherein, the first training node and the second training node are any two different training nodes in the training node cluster.

[0105] In this implementation, on each training node, a P2P server and a P2P client are respectively set for each training subtask. The P2P server can be responsible for managing the chunk data of this training subtask and providing data reading services for other training subtasks. The P2P client can be responsible for requesting the required chunk data from other training subtasks.

[0106] As Figure 2 shown, training subtask 2 and training subtask 3 are running on training node 1, training subtask 0 and training subtask 1 are running on training node 2, and training subtask 4 and training subtask 5 are running on training node 3. A peer-to-peer server and a peer-to-peer client corresponding to training subtask 2, and a peer-to-peer server and a peer-to-peer client corresponding to training subtask 3 can be set on training node 1; a peer-to-peer server and a peer-to-peer client corresponding to training subtask 0, and a peer-to-peer server and a peer-to-peer client corresponding to training subtask 1 can be set on training node 2; a peer-to-peer server and a peer-to-peer client corresponding to training subtask 4, and a peer-to-peer server and a peer-to-peer client corresponding to training subtask 5 can be set on training node 3.

[0107] Suppose there are two different training nodes: a first training node and a second training node. Multiple training subtasks are running on each training node. The first training subtask on the first training node needs to obtain specific chunk data from the second training subtask on the second training node. The P2P client of the first training subtask sends a chunk data reading request to the P2P server of the second training subtask on the second training node. After receiving the request, the P2P server of the second training subtask finds the corresponding chunk data from the local storage device. The P2P server of the second training subtask sends the chunk data back to the P2P client of the first training subtask.

[0108] Through P2P technology, chunked data can be directly exchanged between training subtasks, reducing the dependence on the central storage device, thereby reducing the load on the central storage. This direct data exchange method can make more full use of the network bandwidth and improve the efficiency of data transmission. P2P data exchange can also reduce the latency of data transmission because the chunked data is directly transmitted between nodes instead of being relayed through the central node. Therefore, this method of exchanging chunked data between training subtasks using the peer-to-peer technology can improve the speed and efficiency of checkpoint loading in distributed training, while reducing the demand for central storage resources and improving the scalability and robustness of the entire system. This method is particularly suitable for large-scale distributed training environments where training tasks are decomposed into multiple subtasks and these subtasks are distributed on different nodes. In this way, each subtask can directly communicate with other subtasks, thereby optimizing the data loading and recovery process.

[0109] In a possible implementation, chunked data is exchanged between different training nodes through a Remote Direct Memory Access (RDMA) network.

[0110] RDMA is a technology that allows devices to directly access system memory without the continuous intervention of the CPU. This means that data transmission can be carried out without CPU intervention, thereby improving the efficiency and speed of data transmission.

[0111] In a distributed training system, each training node has its local memory. When data needs to be exchanged between training nodes, RDMA can directly read data from the memory of one training node and write it into the memory of another training node. This direct memory-to-memory data transmission bypasses the CPU and reduces the CPU load.

[0112] As Figure 2 shown, chunked data can be exchanged between different training nodes through the RMDA network.

[0113] By using RDMA technology, different training nodes in a distributed training system can exchange chunked data more quickly and efficiently, which is crucial for improving the performance and scalability of large-scale distributed training.

[0114] As an example of this implementation, the remote direct memory access network includes an InfiniBand (IB) network and / or a Remote Direct Memory Access over Converged Ethernet (RoCE) network.

[0115] In a distributed training system, the performance of the data transmission network is crucial for the overall system efficiency. Remote Direct Memory Access (RDMA) networks allow direct data exchange between nodes without the intervention of the CPU, which can significantly improve the data transmission speed and system performance. In this implementation, RDMA can be achieved through different types of network technologies, including wireless bandwidth networks and Ethernet-fused remote direct memory access networks.

[0116] In one example, different training nodes can exchange chunked data through a wireless bandwidth network.

[0117] In another example, different training nodes can exchange chunked data through an Ethernet-fused remote direct memory access network.

[0118] In this example, the training nodes in the distributed training system can perform high-speed and low-latency RDMA data transmission through InfiniBand or RoCE networks. For example, one training node can directly transfer chunked data into the memory of another training node through an InfiniBand network, or achieve the same operation over Ethernet through RoCE. This direct memory-to-memory data transmission method not only improves the data transmission efficiency but also reduces the burden on the CPU, enabling the CPU to focus on other computing tasks.

[0119] By implementing RDMA using InfiniBand or RoCE networks, the distributed training system can quickly and efficiently exchange chunked data between different training nodes, thus optimizing the performance and scalability of the entire system. The choice of this network technology can be determined according to specific system requirements, cost budgets, and existing infrastructure.

[0120] In one possible implementation, the method further includes: for any training node in the training node cluster, in response to the training node obtaining each chunk of data corresponding to the checkpoint file, loading the complete checkpoint file from the local storage device of the training node and resuming training.

[0121] In this implementation, for any training node, after the training node has collected all the chunked data belonging to the checkpoint file, these chunked data can be combined or concatenated into a complete checkpoint file. This step may include sorting the chunked data to ensure they are combined in the correct order.

[0122] After the combination is completed, the training node can load this complete checkpoint file from its local storage device. The loading process may include moving the data from a temporary storage area to the memory area required for model training.

[0123] After loading the checkpoint file, the training node can use the information in the checkpoint file to restore the state of the model and the state of the optimizer. This means that the parameters of the model and the training progress are reset to the state when the checkpoint file was saved. Once the restoration is complete, the training node can continue with the training task that was interrupted previously or perform the next training operation as needed.

[0124] The checkpoint loading method in distributed training provided by the embodiments of the present disclosure can be applied to technical fields such as shared storage, checkpoints, distributed training, peer-to-peer (P2P), etc., and is not limited herein.

[0125] The following uses a specific application scenario to illustrate the checkpoint loading method in distributed training provided by the embodiments of the present disclosure.

[0126] Figure 4 A schematic diagram showing the application scenario of the checkpoint loading method in distributed training provided by the embodiments of the present disclosure. As Figure 4 shown, the system architecture can consist of multiple layers, including a training task layer, a distributed training framework layer, a P2P service layer, an RDMA module layer, and a tool layer, which work together to achieve efficient checkpoint loading and training recovery.

[0127] In this architecture, the training task layer can include multiple training subtasks, and each training subtask can be executed on different distributed training frameworks, such as DeepSpeed, ColossalAI, PyTorch, etc. These frameworks integrate P2P clients internally, so that when users use these frameworks for training, they do not need to perceive the underlying P2P service layer. The P2P service layer can be implemented by a Python module and is responsible for exchanging chunked data and managing local data storage among training nodes through RDMA. The underlying RDMA module can be implemented in C++ to ensure high performance in data exchange.

[0128] In addition, the system also provides a checkpoint tool that allows users to manually trigger the loading of checkpoints. This checkpoint tool can interact with the Python module and utilize the P2P service layer and the RDMA module to achieve fast checkpoint loading.

[0129] Through this architecture design, not only the efficiency of distributed training is improved, but also flexibility and ease of use are provided, making large-scale model training more efficient and reliable.

[0130] Suppose in a large-scale distributed training scenario, 1000 GPU cards are used for training, the storage bandwidth is 30 GB / s, and the checkpoint file size is 2 TB.

[0131] In the related art, each training node separately loads the entire checkpoint file from the shared storage device, and the required time is: 2TB×1000 / (30GB / s) = 18 hours. In fact, due to the competition for storage bandwidth by other services, the actual loading time may be much longer than 18 hours. This long loading process seriously affects the efficiency of training recovery.

[0132] Using the method provided by the embodiments of the present disclosure, the checkpoint file is divided into multiple block data. Each training node only loads the block data belonging to itself and exchanges the block data among the training nodes through the peer-to-peer technology. Among them, the time required to load the checkpoint file from the shared storage device is: 2TB / (30GB / s) = 66 seconds. Assuming that the write bandwidth of the local NVMe disk is 16GB / s, the time for exchanging the block data through the IB network is 2TB / (16GB / s) = 125 seconds. Assuming that the read bandwidth of the local NVMe disk is 20GB / s, the time for Pytorch to load the checkpoint from the local NVMe disk is: 2TB / (20GB / s) = 100 seconds. Therefore, the total time is: 66 seconds + 125 seconds + 100 seconds = 291 seconds.

[0133] Compared with 18 hours in the related art, the embodiments of the present disclosure only need less than 5 minutes to complete the loading of the checkpoint file, significantly improving the efficiency of training recovery.

[0134] It can be understood that the above-mentioned method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not elaborate. Those skilled in the art can understand that in the above-mentioned method of the specific implementation manner, the specific execution order of each step should be determined according to its function and possible internal logic.

[0135] In addition, the present disclosure also provides a checkpoint loading device, a non-volatile computer-readable storage medium, and a computer program product in distributed training. The above can all be used to implement any one of the checkpoint loading methods provided by the present disclosure. The corresponding technical solutions and technical effects can be seen in the corresponding records of the method part and will not be elaborated here.

[0136] Figure 5 The block diagram of the checkpoint loading device in distributed training provided by the embodiments of the present disclosure is shown. As Figure 5 shown, the checkpoint loading device in distributed training includes:

[0137] A loading module 51, configured to, for any training node in the training node cluster corresponding to the target distributed training task, load the block data corresponding to the training node from the shared storage device and store the block data corresponding to the training node in the local storage device of the training node;

[0138] The exchange module 52 is used to exchange chunk data between different training nodes in the training node cluster by using the peer-to-peer technology until each training node in the training node cluster obtains each chunk data corresponding to the checkpoint file respectively.

[0139] In a possible implementation, a peer-to-peer server and a peer-to-peer client are respectively set on each training node;

[0140] The exchange module 52 is used for:

[0141] For the first training node in the training node cluster, send a chunk data reading request to the peer-to-peer server of the second training node in the training node cluster through the peer-to-peer client of the first training node; for the second training node, in response to the chunk data reading request through the peer-to-peer server of the second training node, return the chunk data corresponding to the second training node to the peer-to-peer client of the first training node; wherein, the first training node and the second training node are any two different training nodes in the training node cluster.

[0142] In a possible implementation, chunk data is exchanged between different training nodes through a Remote Direct Memory Access (RDMA) network.

[0143] In a possible implementation, the Remote Direct Memory Access (RDMA) network includes a wireless bandwidth network and / or a Remote Direct Memory Access (RDMA) network integrated with an Ethernet network.

[0144] In a possible implementation, the splitting module is used for:

[0145] According to the number of training subtasks in the training node cluster, split the checkpoint file in the shared storage device into multiple chunks of data, wherein the number of chunks of data is equal to the number of training subtasks.

[0146] In a possible implementation, the loading module 51 is used for:

[0147] For any training subtask in any training node in the training node cluster corresponding to the target distributed training task, load the chunk data corresponding to the training subtask from the shared storage device and store the chunk data corresponding to the training subtask into the local storage device of the training node.

[0148] In a possible implementation, the local storage device of the training node includes a memory and an external storage;

[0149] The loading module 51 is used for:

[0150] In response to the remaining capacity of the memory of the training node being greater than or equal to the data volume of the chunk data corresponding to the training node, store the chunk data corresponding to the training node in the memory of the training node;

[0151] Or,

[0152] In response to the remaining capacity of the memory of the training node being less than the data volume of the chunk data corresponding to the training node, store the chunk data corresponding to the training node in the external memory of the training node.

[0153] In a possible implementation, the external memory includes a storage device based on the Non-Volatile Memory Express (NVMe) protocol.

[0154] In a possible implementation, the device further includes:

[0155] A resume training module, configured to, for any training node in the training node cluster, in response to the training node obtaining the respective chunk data corresponding to the checkpoint file, load the complete checkpoint file from the local storage device of the training node, and resume training.

[0156] In a possible implementation, the device is integrated into a distributed training framework.

[0157] In some embodiments, the functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the methods described in the method embodiments above. The specific implementation and technical effects can be referred to the descriptions of the method embodiments above. For the sake of brevity, they will not be repeated here.

[0158] The embodiments of the present disclosure further provide a checkpoint loading device in distributed training, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the above method.

[0159] The embodiments of the present disclosure further provide a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0160] The embodiments of the present disclosure further provide a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0161] Figure 6 is a block diagram of a checkpoint loading device 1900 in distributed training shown according to an exemplary embodiment. For example, the device 1900 can be provided as a server or a terminal device. Refer to Figure 6, Device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0162] Device 1900 may also include a power component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0163] In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, such as the memory 1932 including computer program instructions, and the computer program instructions can be executed by the processing component 1922 of the device 1900 to complete the above method.

[0164] A computer-readable storage medium can be a tangible device that can hold and store programs / instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding devices, such as punched cards or raised structures in grooves storing instructions thereon, and any suitable combination of the above. The computer-readable storage medium used herein is not construed as an instantaneous signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through an optical fiber cable), or electrical signals transmitted through wires.

[0165] The computer programs (or computer-readable program instructions) described herein can be downloaded to various computing / processing devices from a computer-readable storage medium or downloaded to an external computer or an external storage device via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0166] The computer programs (or computer program instructions) for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0167] Aspects of the present disclosure are described herein with reference to the flowchart and / or block diagram of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart and / or block diagram, and the combinations of blocks in the flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0168] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when the instructions are executed by the processor of the computer or other programmable data processing apparatus, an apparatus is created that implements the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises a manufacture, the instructions comprising aspects of implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0169] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, such that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0170] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, and the module, segment of a program, or portion of an instruction comprises one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0171] The computer program product may be implemented specifically by means of hardware, software, or a combination thereof. In an alternative embodiment, the computer program product is embodied as a computer storage medium, and in another alternative embodiment, the computer program product is embodied as a software product, such as a Software Development Kit (SDK), etc.

[0172] The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, and their similarities or likenesses may be referred to each other. For the sake of brevity, they are not elaborated herein.

[0173] If the technical solutions of the embodiments of the present disclosure involve personal information, the products applying the technical solutions of the embodiments of the present disclosure have clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solutions of the embodiments of the present disclosure involve sensitive personal information, the products applying the technical solutions of the embodiments of the present disclosure have obtained the individual's separate consent before processing the sensitive personal information, and at the same time meet the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information. The personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0174] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A checkpoint loading method in distributed training, characterized in that, Including: Dividing the checkpoint file in the shared storage device into multiple chunks of data; For any training node in the training node cluster corresponding to the target distributed training task, loading the chunk of data corresponding to the training node from the shared storage device and storing the chunk of data corresponding to the training node in the local storage device of the training node; Among different training nodes in the training node cluster, exchanging chunk of data using peer-to-peer technology until each training node in the training node cluster respectively obtains each chunk of data corresponding to the checkpoint file.

2. The method according to claim 1, characterized in that, A peer-to-peer server and a peer-to-peer client are respectively set on each training node; The exchanging chunk of data using peer-to-peer technology among different training nodes in the training node cluster includes: For the first training node in the training node cluster, sending a chunk data reading request to the peer-to-peer server of the second training node in the training node cluster through the peer-to-peer client of the first training node; for the second training node, in response to the chunk data reading request through the peer-to-peer server of the second training node, returning the chunk of data corresponding to the second training node to the peer-to-peer client of the first training node; wherein, the first training node and the second training node are any two different training nodes in the training node cluster.

3. The method according to claim 1 or 2, wherein Chunk of data is exchanged among different training nodes through a Remote Direct Memory Access (RDMA) network.

4. The method according to claim 3, wherein The Remote Direct Memory Access network includes a wireless bandwidth network and / or an RDMA network integrated with an Ethernet network.

5. The method according to claim 1, wherein The dividing the checkpoint file in the shared storage device into multiple chunks of data includes: Dividing the checkpoint file in the shared storage device into multiple chunks of data according to the number of training subtasks in the training node cluster, wherein the number of chunks of data is equal to the number of training subtasks.

6. The method according to claim 5, wherein The for any training node in the training node cluster corresponding to the target distributed training task, loading the chunk of data corresponding to the training node from the shared storage device and storing the chunk of data corresponding to the training node in the local storage device of the training node includes: For any training subtask in any training node in the training node cluster corresponding to the target distributed training task, loading the chunk of data corresponding to the training subtask from the shared storage device and storing the chunk of data corresponding to the training subtask in the local storage device of the training node.

7. The method according to claim 1, wherein The local storage device of the training node includes memory and external storage; The storing the chunk of data corresponding to the training node in the local storage device of the training node includes: In response to the remaining capacity of the memory of the training node being greater than or equal to the data volume of the chunk of data corresponding to the training node, storing the chunk of data corresponding to the training node in the memory of the training node; Or, In response to the remaining capacity of the memory of the training node being less than the data volume of the chunk of data corresponding to the training node, storing the chunk of data corresponding to the training node in the external storage of the training node.

8. The method according to claim 7, characterized in that, The external memory includes a storage device based on the Non-Volatile Memory Express (NVMe) protocol.

9. The method according to claim 1, characterized in that, The method further includes: For any training node in the training node cluster, in response to the training node obtaining the respective chunk data corresponding to the checkpoint file, loading the complete checkpoint file from the local storage device of the training node and resuming training.

10. The method according to claim 1, characterized in that, The method is integrated into a distributed training framework.

11. A checkpoint loading device in distributed training, characterized in that, It includes: A splitting module, configured to split the checkpoint file in the shared storage device into multiple chunks of data; A loading module, configured to, for any training node in the training node cluster corresponding to a target distributed training task, load the chunk data corresponding to the training node from the shared storage device and store the chunk data corresponding to the training node in the local storage device of the training node; A swapping module, configured to swap the chunk data between different training nodes in the training node cluster by using a peer-to-peer technology until each training node in the training node cluster respectively obtains the respective chunk data corresponding to the checkpoint file.

12. A checkpoint loading device in distributed training, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 10.

13. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program, or a non-volatile computer-readable storage medium carrying the computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images