Checkpoint read-write method and device in distributed training, storage medium and program product

Abstract

The invention relates to a check point reading and writing method and device in distributed training, a storage medium and a program product. The method comprises the steps that in response to check point file generation and the number of check point files in a memory of a training node cluster corresponding to a target distributed training task does not reach N, an idle target memory is determined in the memory of the training node cluster, and the check point files are written into the target memory, the memory of the training node cluster comprises the memory of each training node in the training node cluster; and in response to a fault of any training node in the training node cluster, reading the target check point file from the memory of the training node cluster according to the metadata of the target check point file corresponding to the training node, and recovering training based on the target check point file. According to the method and the device, the check point file can be quickly read from the memory of the training node for recovery when the training node fails.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a checkpoint reading and writing method in distributed training, a checkpoint reading and writing device in distributed training, a non-volatile computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of artificial intelligence technologies, large model training has become one of the core tasks in the field of deep learning. In practical applications, large model training usually requires the use of hundreds or thousands of acceleration cards (such as GPUs (Graphics Processing Units)), and the training cycle ranges from several weeks to several months. However, during such large-scale training, the occurrence of faults is inevitable, which may lead to training interruption and thus affect the integrity and efficiency of the training task.

[0003] To achieve training fault tolerance, the training system must meet two key requirements: First, it should be able to quickly resume the training process and continue training after a fault occurs; second, the training program needs to be able to restore to the model and optimizer states before the fault to ensure the continuity and accuracy of training.

[0004] To meet these requirements, the training program usually adopts a periodic checkpoint scheme to persist the training state into the storage system. However, the conventional checkpoint scheme has the following problems:

[0005] Time-consuming problem: The generation and saving of conventional checkpoints are closely related to the size of the model and the storage I / O (Input / Output) performance, often taking from several minutes to dozens of minutes. This not only increases the overall training time but may also cause frequent interruptions during training.

[0006] Balance problem between frequency and efficiency: There is a dilemma in setting the frequency of checkpoints. If the setting is too high, it will significantly reduce the available training time; if the setting is too low, the low-frequency checkpoint interval will result in a large number of wasted iteration steps after a fault, further reducing the training efficiency. Summary of the Invention

[0007] In view of this, the present disclosure provides a technical solution for checkpoint reading and writing in distributed training.

[0008] According to one aspect of the present disclosure, there is provided a checkpoint reading and writing method in distributed training, including:

[0009] In response to the generation of a checkpoint file, and when the number of copies of the checkpoint file in the memory of the training node cluster corresponding to the target distributed training task has not reached N, determine free target memory in the memory of the training node cluster, write the checkpoint file into the target memory, and record the metadata of the checkpoint file, where N is an integer greater than or equal to 2, and the memory of the training node cluster includes the memories of each training node in the training node cluster;

[0010] In response to any training node in the training node cluster failing, read the target checkpoint file from the memory of the training node cluster according to the metadata of the target checkpoint file corresponding to the training node, and resume training based on the target checkpoint file.

[0011] In one possible implementation, the method further includes:

[0012] In response to the generation of the checkpoint file, and when the number of copies of the checkpoint file in the memory of the training node cluster has reached N, record the metadata of the checkpoint file.

[0013] In one possible implementation, the step of in response to any training node in the training node cluster failing, reading the target checkpoint file from the memory of the training node cluster according to the metadata of the target checkpoint file corresponding to the training node includes:

[0014] In response to any training node in the training node cluster failing, and when the metadata of the target checkpoint file corresponding to the training node indicates that the checkpoint file exists in the memory of the training node, read the target checkpoint file from the memory of the training node;

[0015] Or,

[0016] In response to any training node in the training node cluster failing, and when the metadata of the target checkpoint file corresponding to the training node indicates that the checkpoint file does not exist in the memory of the training node, determine the target training node storing the target checkpoint file according to the metadata of the target checkpoint file, and read the target checkpoint file from the memory of the target training node.

[0017] In one possible implementation, the step of reading the target checkpoint file from the memory of the target training node includes:

[0018] Read the target checkpoint file from the memory of the target training node through a Remote Direct Memory Access (RDMA) network.

[0019] In a possible implementation, determining free target memory in the memory of the training node cluster, writing the checkpoint file into the target memory, and recording metadata of the checkpoint file, includes:

[0020] Determining free target memory in the memory of the training node cluster through a background task, writing the checkpoint file into the target memory, and recording metadata of the checkpoint file.

[0021] In a possible implementation, the metadata includes:

[0022] Identification information of the training node where the checkpoint file is located, and the name of the checkpoint file.

[0023] In a possible implementation, the writing the checkpoint file into the target memory includes:

[0024] Generating an erasure code file corresponding to the checkpoint file through erasure code encoding;

[0025] Writing the erasure code file into the target memory.

[0026] In a possible implementation, determining free target memory in the memory of the training node cluster, and writing the checkpoint file into the target memory, includes:

[0027] Dividing the checkpoint file into multiple block data;

[0028] Determining multiple free target memories in the memory of the training node cluster;

[0029] Writing the multiple block data into the multiple target memories respectively.

[0030] In a possible implementation, after writing the checkpoint file into the target memory, the method further includes:

[0031] In response to the checkpoint file not existing in the shared storage device of the training node cluster, writing the checkpoint file into the shared storage device.

[0032] According to another aspect of the present disclosure, there is provided a checkpoint reading and writing device in distributed training, including:

[0033] A write and record module, which is used to, in response to the generation of a checkpoint file and when the number of copies of the checkpoint file in the memory of the training node cluster corresponding to the target distributed training task has not reached N, determine free target memory in the memory of the training node cluster, write the checkpoint file into the target memory, and record the metadata of the checkpoint file, where N is an integer greater than or equal to 2, and the memory of the training node cluster includes the memories of each training node in the training node cluster;

[0034] A read and recovery module, which is used to, in response to any training node in the training node cluster failing, read the target checkpoint file from the memory of the training node cluster according to the metadata of the target checkpoint file corresponding to the training node, and resume training based on the target checkpoint file.

[0035] In a possible implementation manner, the write and record module is further used to:

[0036] In response to the generation of the checkpoint file and when the number of copies of the checkpoint file in the memory of the training node cluster has reached N, record the metadata of the checkpoint file.

[0037] In a possible implementation manner, the read and recovery module is used to:

[0038] In response to any training node in the training node cluster failing and when the metadata of the target checkpoint file corresponding to the training node indicates that the checkpoint file exists in the memory of the training node, read the target checkpoint file from the memory of the training node;

[0039] Or,

[0040] In response to any training node in the training node cluster failing and when the metadata of the target checkpoint file corresponding to the training node indicates that the checkpoint file does not exist in the memory of the training node, determine the target training node storing the target checkpoint file according to the metadata of the target checkpoint file, and read the target checkpoint file from the memory of the target training node.

[0041] In a possible implementation manner, the read and recovery module is used to:

[0042] Read the target checkpoint file from the memory of the target training node through a remote direct memory access network.

[0043] In a possible implementation manner, the write and record module is used to:

[0044] Determine free target memory in the memory of the training node cluster through a background task, write the checkpoint file to the target memory, and record the metadata of the checkpoint file.

[0045] In a possible implementation, the metadata includes:

[0046] The identification information of the training node where the checkpoint file is located and the name of the checkpoint file.

[0047] In a possible implementation, the writing and recording module is used to:

[0048] Generate an erasure code file corresponding to the checkpoint file through erasure code encoding;

[0049] Write the erasure code file to the target memory.

[0050] In a possible implementation, the writing and recording module is used to:

[0051] Split the checkpoint file into multiple block data;

[0052] Determine multiple free target memories in the memory of the training node cluster;

[0053] Write the multiple block data to the multiple target memories respectively.

[0054] In a possible implementation, the writing and recording module is used to:

[0055] In response to the absence of the checkpoint file in the shared storage device of the training node cluster, write the checkpoint file to the shared storage device.

[0056] According to another aspect of the present disclosure, there is provided a checkpoint reading and writing device in distributed training, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the above method.

[0057] According to another aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0058] According to another aspect of the present disclosure, there is provided a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0059] In the embodiments of the present disclosure, in response to the generation of a checkpoint file and when the number of copies of the checkpoint file in the memory of the training node cluster corresponding to the target distributed training task has not reached N, free target memory is determined in the memory of the training node cluster, the checkpoint file is written into the target memory, and metadata of the checkpoint file is recorded, where N is an integer greater than or equal to 2, and the memory of the training node cluster includes the memories of the individual training nodes in the training node cluster. In response to a failure of any training node in the training node cluster, according to the metadata of the target checkpoint file corresponding to the training node, the target checkpoint file is read from the memory of the training node cluster, and training is resumed based on the target checkpoint file. Thus, by using the distributed memory technology to implement checkpoint reading and writing in distributed training, it is possible to quickly read the checkpoint file from the memory of the training node for recovery when a training node fails, significantly reduce the training interruption time caused by the failure, improve the fault tolerance and efficiency of distributed training, and at the same time make full use of the idle memory resources of the training node cluster to avoid the problem of failed checkpoint file saving due to insufficient memory of a single training node.

[0060] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings, which are included in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and are used to explain the principles of the present disclosure.

[0062] Figure 1 A schematic diagram showing the generation time of a checkpoint and the occurrence time of a failure.

[0063] Figure 2 A flowchart showing the checkpoint reading and writing method in distributed training provided by the embodiments of the present disclosure.

[0064] Figure 3 A physical architecture diagram of a distributed training system provided by the embodiments of the present disclosure.

[0065] Figure 4 A logical architecture diagram of a distributed training system provided by the embodiments of the present disclosure.

[0066] Figure 5 An interaction architecture between a client and a server of a distributed memory service provided by the embodiments of the present disclosure.

[0067] Figure 6 A schematic diagram showing a write process in the checkpoint reading and writing method in distributed training provided by the embodiments of the present disclosure.

[0068] Figure 7 Another schematic diagram showing the writing process in the checkpoint reading and writing method in distributed training provided by an embodiment of the present disclosure.

[0069] Figure 8 A schematic diagram showing the reading process in the checkpoint reading and writing method in distributed training provided by an embodiment of the present disclosure.

[0070] Figure 9 A block diagram showing the checkpoint reading and writing device in distributed training provided by an embodiment of the present disclosure.

[0071] Figure 10 It is a block diagram of a checkpoint reading and writing device 1900 in distributed training shown according to an exemplary embodiment. Detailed implementation manners

[0072] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0073] As used herein, the terms "comprising", "including", "having", or variations thereof are open-ended and include one or more stated features, wholes, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, wholes, elements, steps, components, functions, or groups thereof.

[0074] When an element is referred to as being "connected", "coupled", "responsive", or variations thereof to another element, it can be directly connected, coupled, or responsive to the other element, or there can be intervening elements.

[0075] Although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Thus, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0076] The term "exemplary" as used herein means "serving as an example, embodiment, or illustration". Any embodiment illustrated herein as "exemplary" should not necessarily be construed as being superior to or better than other embodiments.

[0077] In addition, to better illustrate the present disclosure, numerous specific details are provided in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can also be implemented without some specific details. In some instances, methods, means, elements, and circuits well-known to those skilled in the art are not described in detail to highlight the gist of the present disclosure.

[0078] Figure 1 A schematic diagram showing the generation time of a checkpoint and the occurrence time of a fault. In Figure 1 the example shown, the training program fails before generating checkpoint 4 and needs to resume training from checkpoint 3. If the time from checkpoint 3 to the fault point is long, a large amount of GPU time will be wasted, seriously affecting the continuity and efficiency of training.

[0079] Therefore, a low-overhead checkpoint scheme is particularly important. It can not only significantly reduce the training pause time but also support high-frequency checkpoints, thereby reducing the number of wasted iteration steps during fault tolerance. To achieve fast checkpoint saving and recovery, related technologies usually directly save the checkpoint to local memory and asynchronously save the checkpoint to shared storage in the background to avoid blocking GPU training.

[0080] However, the solutions in related technologies have the following problems:

[0081] Data loss problem after node failure: Saving the checkpoint to memory during training improves the write speed, but the data in memory will be lost after node failure, resulting in new joined nodes being unable to directly read the checkpoint from memory during fault recovery.

[0082] Uneven memory load problem: Each node independently saves the checkpoint to local memory, resulting in different memory loads and different memory occupancies on different nodes. If some nodes have insufficient memory, they will be unable to save the checkpoint.

[0083] Cluster resource waste problem: Related technologies cannot make full use of the idle memory of the entire cluster, resulting in waste of storage resources.

[0084] To solve the technical problems similar to those described above, an embodiment of the present disclosure provides a checkpoint reading and writing method in distributed training. By responding to the generation of a checkpoint file and when the number of copies of the checkpoint file in the memory of the training node cluster corresponding to the target distributed training task does not reach N, determining free target memory in the memory of the training node cluster, writing the checkpoint file into the target memory, and recording the metadata of the checkpoint file, where N is an integer greater than or equal to 2, and the memory of the training node cluster includes the memories of each training node in the training node cluster. In response to any training node in the training node cluster failing, according to the metadata of the target checkpoint file corresponding to the training node, reading the target checkpoint file from the memory of the training node cluster, and resuming training based on the target checkpoint file. Thus, by using distributed memory technology to implement checkpoint reading and writing in distributed training, it is possible to quickly read the checkpoint file from the memory of the training node for recovery when a training node fails, significantly reducing the training interruption time caused by the failure, improving the fault tolerance and efficiency of distributed training, and at the same time making full use of the idle memory resources of the training node cluster to avoid the problem of failed checkpoint file saving due to insufficient memory of a single training node.

[0085] The following will describe in detail the checkpoint reading and writing method in distributed training provided by the embodiments of the present disclosure with reference to the accompanying drawings.

[0086] Figure 2 The flowchart showing the checkpoint reading and writing method in distributed training provided by the embodiments of the present disclosure is shown. In a possible implementation, the execution subject of the checkpoint reading and writing method in distributed training may be a checkpoint reading and writing device in distributed training. For example, the checkpoint reading and writing method in distributed training may be executed by a terminal device, a server, or other electronic devices. Among them, the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, or a wearable device, etc. In some possible implementations, the checkpoint reading and writing method in distributed training may be implemented by a processor calling computer-readable instructions stored in a memory. As Figure 2 shown, the checkpoint reading and writing method in distributed training includes steps S21 to S22.

[0087] In step S21, in response to the generation of the checkpoint file and when the number of copies of the checkpoint file in the memory of the training node cluster corresponding to the target distributed training task has not reached N, determine free target memory in the memory of the training node cluster, write the checkpoint file into the target memory, and record the metadata of the checkpoint file, where N is an integer greater than or equal to 2, and the memory of the training node cluster includes the memories of the respective training nodes in the training node cluster.

[0088] In step S22, in response to a failure of any one of the training nodes in the training node cluster, read the target checkpoint file from the memory of the training node cluster according to the metadata of the target checkpoint file corresponding to the training node, and resume training based on the target checkpoint file.

[0089] In the embodiments of the present disclosure, the target distributed training task may refer to a specific training task that needs to be executed or is being executed currently in a distributed training environment. In distributed training, usually a large training task is decomposed into multiple training subtasks, and then these training subtasks are assigned to different training nodes (which may also be referred to as computing nodes, physical nodes, or worker nodes) for parallel processing.

[0090] The training node cluster may refer to a collaborative working group composed of multiple training nodes in a distributed training environment. These training nodes can be connected through a high-speed network and jointly undertake the computing load of the distributed training task. Each training node may be equipped with computing resources (such as a CPU (Central Processing Unit, central processor), a GPU (Graphics Processing Unit, graphics processor), or a TPU (Tensor Processing Unit, tensor processor)) and storage resources, used to execute specific training subtasks and perform data interaction and collaborative work within the cluster. In this way, the training node cluster can efficiently process large-scale training tasks, improve training efficiency, and at the same time have better fault tolerance and resource scalability to meet complex model training requirements.

[0091] In a possible implementation, any one of the training nodes may include at least one GPU and may run at least one training subtask. For example, each training node may respectively include 8 GPUs, and one training subtask may be run on each GPU respectively. Thus, 8 training subtasks may be run on each training node.

[0092] In a possible implementation, a background service can be deployed on each training node respectively. When a target training task is scheduled to be executed on a certain training node, the background service on this training node can automatically add this training node to the training node cluster corresponding to the target training task to provide the service for reading checkpoint files.

[0093] In distributed training, in order to implement a fault tolerance mechanism, the training program will regularly generate checkpoint files. The checkpoint files contain key information such as the model state and optimizer state during the training process, enabling the training to be resumed from the nearest checkpoint in case of a failure instead of starting from scratch.

[0094] In the embodiments of the present disclosure, in order to improve the reading speed, fault tolerance ability, and data reliability of checkpoint files, the same checkpoint file can be stored in the memory of N training nodes so that in case a certain training node fails, the data can be restored from the memory of other training nodes, where N can represent a predetermined redundancy number, and N is an integer greater than or equal to 2.

[0095] In the embodiments of the present disclosure, after the checkpoint file is generated, it can be checked whether N copies of the checkpoint file already exist in the memory of the training node cluster.

[0096] If the number of checkpoint files has not reached N, free target memory can be searched in the memory of the entire training node cluster. For example, in the scenario of Fully Sharded Data Parallel (FSDP), the generated checkpoint files usually do not have redundancy.

[0097] Among them, the memory of the training node cluster can refer to the set of memory resources of the training nodes in the training node cluster. Through the distributed memory management mechanism, it is possible to dynamically identify which training nodes' memory is free, thus providing available space for the storage of checkpoint files.

[0098] In the embodiments of the present disclosure, after determining the target memory, the checkpoint file can be written into the target memory.

[0099] In a possible implementation, determining free target memory in the memory of the training node cluster and writing the checkpoint file into the target memory includes: splitting the checkpoint file into multiple block data; determining multiple free target memories in the memory of the training node cluster; and writing the multiple block data into the multiple target memories respectively.

[0100] In this implementation, in distributed training, to efficiently utilize the cluster memory and achieve redundant storage of data, the checkpoint file can be split into multiple smaller chunk data. Subsequently, multiple idle target memories can be dynamically found in the memory of the entire training node cluster, and these multiple target memories are the memories of different training nodes. Then, the split chunk data can be written into these idle target memories respectively. In this way, the checkpoint file is scattered and stored in the memories of multiple training nodes, realizing the distributed storage and backup of data.

[0101] In this implementation, by splitting and scattering the storage of the checkpoint file into the memories of multiple training nodes, the idle memory resources of the cluster are fully utilized, avoiding the storage bottleneck caused by insufficient memory in a single training node and improving the overall utilization rate of memory resources. In addition, this chunk storage method can also reduce the storage pressure on a single training node, reduce memory access latency, improve data read and write efficiency, and further optimize the performance of distributed training.

[0102] In another possible implementation, only one idle target memory can be determined in the memory of the training node cluster, and the checkpoint file is written into the target memory. In this implementation, the checkpoint file may not be split, and the complete checkpoint file is written into the same target memory.

[0103] In a possible implementation, determining the idle target memory in the memory of the training node cluster, writing the checkpoint file into the target memory, and recording the metadata of the checkpoint file includes: determining the idle target memory in the memory of the training node cluster through a background task, writing the checkpoint file into the target memory, and recording the metadata of the checkpoint file.

[0104] In this implementation, in a distributed training system, to reduce the interference to the main training task and improve the overall efficiency, the writing and management operations of the checkpoint file can be processed through a background task. Specifically, the background task can scan the memory of the entire training node cluster to find idle target memory areas. This process will not block the main training task, thus ensuring the continuity of the training task. After finding suitable idle memory, the background task can write the checkpoint file into the target memory. Since this operation is executed in the background, the main training task can continue to run without being delayed due to waiting for the storage operation to complete. After completing the writing of the checkpoint file, the background task can record the metadata related to the checkpoint file.

[0105] By performing the write and management operations of the checkpoint file through a background task, the impact on the main training task can be significantly reduced. The main training task usually has high performance requirements, and any blocking operation may lead to a decrease in training efficiency. The introduction of the background task allows the system to asynchronously complete the persistence and redundant backup of checkpoint-related data without affecting the main task. This design not only improves the overall efficiency of the system but also enhances the fault tolerance and resource utilization of the system, enabling the distributed training system to fully utilize the cluster resources while ensuring high availability.

[0106] In one possible implementation, the method further includes: in response to the generation of the checkpoint file and the number of copies of the checkpoint file in the memory of the training node cluster reaching N, recording the metadata of the checkpoint file.

[0107] In this implementation, if the number of copies of the checkpoint file in the memory of the training node cluster has reached N, there is no need to generate a new copy of the checkpoint file, but directly record the metadata of the checkpoint file.

[0108] For example, in the 3D parallel (Data Parallelism, Model Parallelism, Pipeline Parallelism) scenario of distributed training, redundant storage of checkpoint files has been achieved among Data Parallelism (DP) groups through the data distribution mechanism during training. This means that when generating the checkpoint file, the complete checkpoint file has been saved on the training nodes of each data parallel group, thus ensuring the high availability and fault tolerance of the data. Therefore, when importing the checkpoint file into the distributed memory, there is no need to generate additional copy data again. The system only needs to inform the distributed memory service of the metadata of the checkpoint file to complete the import operation of the checkpoint file. This optimization method makes full use of the existing data redundancy characteristics in distributed training, avoids unnecessary data replication and storage overhead, improves the efficiency of checkpoint writing, and reduces the impact on the training task.

[0109] If there are already enough redundant copies of the checkpoint file, generating a new copy will waste memory resources and storage bandwidth. By recording the metadata, the system can quickly locate and access the checkpoint file when needed without regenerating or replicating the data.

[0110] In the embodiments of the present disclosure, metadata can be descriptive information about the checkpoint file, used to manage and locate the checkpoint file.

[0111] In a possible implementation, the metadata includes: the identification information of the training node where the checkpoint file is located, and the name of the checkpoint file.

[0112] In this implementation, the identification information of the training node can be used to uniquely determine on which training node the checkpoint file is stored, while the name of the checkpoint file provides an identification name for the checkpoint file. These information are crucial for managing and locating the checkpoint file, enabling the system to quickly find and access the required checkpoint file, whether during the normal training process or during fault recovery. By maintaining detailed metadata, the distributed memory service can effectively organize and index the checkpoint data in the cluster memory, ensuring data consistency and accessibility.

[0113] In another possible implementation, the metadata includes: the identification information of the training node where the checkpoint file is located, the starting address of the checkpoint file, and the size of the checkpoint file.

[0114] In a possible implementation, writing the checkpoint file into the target memory includes: generating an erasure code file corresponding to the checkpoint file through erasure code (EC) encoding; writing the erasure code file into the target memory.

[0115] Among them, erasure code encoding is an encoding technology for data redundancy and fault tolerance. Erasure code encoding realizes redundant storage of data by splitting the original data into multiple data blocks and generating a certain number of parity blocks. When some data blocks are lost or damaged, the original data can be recovered through the remaining data blocks and parity blocks. This technology can be used in distributed training to optimize the storage and fault tolerance ability of checkpoint files.

[0116] As an example of this implementation, the checkpoint file can be split into k data blocks, each data block containing a part of the original data. m parity blocks can be generated based on these k data blocks through an erasure code algorithm (such as Reed - Solomon encoding). The number m of parity blocks can be adjusted according to the fault tolerance requirements and storage efficiency. These k data blocks and m parity blocks can be combined into an erasure code file. This erasure code file contains sufficient redundant information for recovering the original data when some data is lost.

[0117] In this implementation, through erasure code encoding, redundant storage of checkpoint files can be achieved with less storage space. Compared with simple data replicas, erasure codes can significantly save memory space, especially in large-scale distributed training. Additionally, even if some data is lost, as long as the number of remaining data blocks and parity blocks meets the recovery conditions, the original checkpoint file can be recovered through the erasure code algorithm. This greatly improves the fault tolerance and reliability of the system.

[0118] As an example of this implementation, an erasure code file corresponding to the checkpoint file can be generated in the background through erasure code encoding. In a distributed training system, to reduce interference with the training task and improve efficiency, the system can perform the erasure code encoding operation asynchronously in the background. Specifically, when the checkpoint file is generated, the system does not immediately block the training task to generate redundant data, but instead executes this process in the background. This method not only ensures data redundancy and fault tolerance but also avoids delays in the training task caused by synchronous execution of the encoding operation, enabling the training task to continue running efficiently while completing the construction of data redundancy in the background, taking both performance and reliability into account.

[0119] In a possible implementation, after writing the checkpoint file to the target memory, the method further includes: in response to the absence of the checkpoint file in the shared storage device of the training node cluster, writing the checkpoint file to the shared storage device.

[0120] The shared storage device can refer to a device or storage system that provides unified storage services for multiple training nodes in a distributed system. The shared storage device allows different training nodes to access and share data in the same storage space through the network, ensuring data consistency and availability. In a distributed training scenario, the shared storage device can be used to store data related to training tasks, such as model parameters, training datasets, and checkpoint files. In this way, each training node can work together to read or write data from the shared storage device, thereby achieving centralized management and efficient utilization of data, and also facilitating data backup and recovery in case of training node failures or task recovery.

[0121] In an embodiment of the present disclosure, when a failure of a certain training node is detected (such as node crash, network interruption, memory corruption, etc.), the failure recovery mechanism will be activated. At this time, it is necessary to quickly restore the training state of the training node to ensure that the entire distributed training task can continue rather than starting from scratch. In an embodiment of the present disclosure, the checkpoint file related to the training node (i.e., the target checkpoint file) can be located according to the pre-recorded metadata. The metadata includes the storage location of the checkpoint file (such as distributed in the memory of which training node or nodes), the file name, etc. The training node can read the target checkpoint file from the memory of the training node cluster according to the information in the metadata of the target checkpoint file, and can restore the training state based on the target checkpoint file. For example, it can include restoring the weights of the model, the state of the optimizer, and other training-related context information. In this way, the training task can continue from the most recent checkpoint when the failure occurs rather than starting from scratch, thereby greatly reducing the impact of the failure on the training efficiency.

[0122] In a possible implementation manner, the step of, in response to a failure of any training node in the training node cluster, reading the target checkpoint file from the memory of the training node cluster according to the metadata of the target checkpoint file corresponding to the training node, includes: in response to a failure of any training node in the training node cluster, and the metadata of the target checkpoint file corresponding to the training node indicates that the checkpoint file exists in the memory of the training node, reading the target checkpoint file from the memory of the training node; or, in response to a failure of any training node in the training node cluster, and the metadata of the target checkpoint file corresponding to the training node indicates that the checkpoint file does not exist in the memory of the training node, determining the target training node storing the target checkpoint file according to the metadata of the target checkpoint file, and reading the target checkpoint file from the memory of the target training node.

[0123] In this implementation manner, it is possible to preferentially attempt to read the target checkpoint file from the local memory of the failed node. In this case, the target checkpoint file has been stored in the local memory of the failed node, so it can be directly and quickly loaded without cross-node access, thereby minimizing the recovery time and resource overhead. Among them, the advantage of local memory loading lies in low latency and high efficiency, especially in scenarios with frequent failure recovery, which can significantly improve the fault tolerance performance of the system.

[0124] If the target checkpoint file cannot be read from the local memory of the faulty node, the other target training nodes storing the checkpoint file can be determined according to the metadata of the target checkpoint file, and the target checkpoint file can be read from the other target training nodes. Thus, the target checkpoint file can be quickly obtained using the high-speed network between the training nodes without reading from the backend storage.

[0125] In a possible implementation, reading the target checkpoint file from the memory of the target training node includes: reading the target checkpoint file from the memory of the target training node through a Remote Direct Memory Access (RDMA) network.

[0126] RDMA is a technology that allows devices to directly access system memory without the continuous intervention of the CPU. This means that data transfer can be performed without CPU intervention, thereby improving the efficiency and speed of data transfer. In a distributed training system, each training node has its local memory. When data needs to be transferred between training nodes, RDMA can directly read data from the memory of one training node and write it into the memory of another training node. This direct memory-to-memory data transfer bypasses the CPU and reduces the CPU load.

[0127] In this implementation, reading the target checkpoint file from the memory of the target training node through the RDMA network can obtain the target checkpoint file more quickly and efficiently, thereby improving the fault recovery efficiency.

[0128] In a possible implementation, the method further includes: in response to any training node in the training node cluster failing, the metadata of the target checkpoint file corresponding to the training node indicates that the checkpoint file does not exist in the memory of the training node, and the checkpoint file also does not exist in the memories of the other training nodes in the training node cluster, reading the target checkpoint file from a shared storage device.

[0129] The checkpoint reading and writing method in distributed training provided by the embodiments of the present disclosure can be applied to technical fields such as checkpoints, distributed training, and distributed memory, which are not limited herein.

[0130] The following uses a specific application scenario to illustrate the checkpoint reading and writing method in distributed training provided by the embodiments of the present disclosure.

[0131] Figure 3 The physical architecture diagram of the distributed training system provided by the embodiments of the present disclosure is shown. In Figure 3 the example shown, the training node cluster includes 3 training nodes (i.e., Figure 3The physical nodes (a, b, c) therein, and a training container and a distributed memory service container are respectively deployed on each training node.

[0132] Among them, the training container can be responsible for executing specific training tasks. 8 training tasks (such as training task_1, training task_2,..., training task_8) are respectively run in each training container. Each training task corresponds to a flash checkpoint module and a distributed memory client respectively. Among them, the flash checkpoint module can be used to quickly generate checkpoint files. The distributed memory client can be responsible for communicating with the distributed memory service and managing the storage and recovery of checkpoints. There is also an asynchronous checkpoint process in each training container, which can be responsible for generating copy data of checkpoints in the background and writing the copy data into the distributed memory (i.e., shared memory). This process does not block the execution of training tasks, thereby reducing the impact on training performance.

[0133] The memory of each training node can be respectively used as shared memory. The shared memory can be used to quickly share data between training nodes, including checkpoint files. Through the shared memory, the training container can quickly access and write checkpoint data.

[0134] The distributed memory service can be responsible for managing the saving and recovery of checkpoint files in the cluster memory, including the metadata service and the data service. Among them, the metadata service can provide the organization form and addressing information of data in the distributed memory, and can include the cluster node information file and the metadata corresponding to each checkpoint file. Among them, the cluster node information file can be used to record the information of each training node in the training node cluster. For example, the cluster node information file can contain the address (such as IP address) of the training node and the ID of the corresponding training node, which is used to uniquely identify each training node in the distributed training system. The data service can provide read and write access services for checkpoint files and is responsible for the organization and reading and writing of data locally.

[0135] The shared storage can be used to long-term save checkpoint data as the final data backup. When a certain training node fails, it can first try to load the checkpoint file from the local memory. If the checkpoint file does not exist in the local memory, then according to the metadata, load the checkpoint file from the memory of other training nodes. If the checkpoint file also does not exist in the memory of other training nodes, then the checkpoint file can be loaded from the shared storage device.

[0136] Figure 4 Shows the logical architecture diagram of the distributed training system provided by the embodiments of the present disclosure. In Figure 4In the example shown, distributed training frameworks such as Megatron and DeepSpeed can be adopted to support the training of large-scale models. The distributed training framework can be integrated with the flash checkpoint function to achieve efficient checkpoint management. Among them, the flash checkpoint module can be responsible for generating and managing checkpoints. The flash checkpoint module can interact with the distributed memory client to achieve the saving and recovery of checkpoints.

[0137] The distributed memory client can include functions such as Import, Put, and Get. The Import function can be used to import metadata into the distributed memory management system. The Put function can be used to save the checkpoint file to the remote memory (i.e., the memory of other training nodes). This can be achieved by the interaction between the distributed memory client and the distributed memory service. The Get function can be used to copy data from the remote memory to the local memory through RDMA technology for quick recovery of checkpoints when needed.

[0138] The distributed memory service can include a data service and a metadata service. Among them, the data service can be responsible for managing the checkpoint files stored in the shared memory of the training nodes (such as the shared memory of training node 1 (Node1 / shm) and the shared memory of training node 2 (Node2 / shm)), and for reading and writing the checkpoint files. The metadata service can provide the organization form and addressing information of the data in the distributed memory.

[0139] Figure 5 Shows the interaction architecture between the client and the server of the distributed memory service provided by the embodiments of the present disclosure. In Figure 5 it, the distributed memory client is the entry point for users to interact with the distributed memory service. The distributed memory client can be responsible for initiating requests for saving and recovering checkpoints, and communicating with the backend metadata service and data service. The metadata service can be responsible for managing and storing metadata related to checkpoints. The data service can be responsible for actual data reading and writing operations.

[0140] Figure 6 Shows a schematic diagram of the write process in the checkpoint reading and writing method in distributed training provided by the embodiments of the present disclosure. In Figure 6In the illustrated example, the training task process in the training node generates a checkpoint file during execution, and the number of copies of the checkpoint file in the memory of the training node cluster has not reached N. Therefore, the checkpoint file can be written to the local memory of the training node, and through the asynchronous checkpoint process, the checkpoint file can be written to the memory of other training nodes. Since the checkpoint file does not exist in the shared storage device of the training node cluster, the checkpoint file can also be written to the shared storage device.

[0141] Figure 7 Another schematic diagram showing the write process in the checkpoint reading and writing method in distributed training provided by an embodiment of the present disclosure. In Figure 7 In the illustrated example, the training task process in the training node generates a checkpoint file during execution, and the number of copies of the checkpoint file in the memory of the training node cluster has reached N. Therefore, only the metadata of the checkpoint file can be recorded, and no copy of the checkpoint file is generated.

[0142] Figure 8 A schematic diagram showing the read process in the checkpoint reading and writing method in distributed training provided by an embodiment of the present disclosure. As Figure 8 shown, when a certain training node fails, it is possible to first try to load the checkpoint file from the local memory. If the checkpoint file does not exist in the local memory, the checkpoint file can be loaded from the memory of other training nodes according to the metadata. If the checkpoint file also does not exist in the memory of other training nodes, the checkpoint file can be loaded from the shared storage device.

[0143] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not elaborate further. Those skilled in the art can understand that in the above methods of the specific implementation manner, the specific execution order of each step should be determined according to its function and possible internal logic.

[0144] In addition, the present disclosure also provides a checkpoint reading and writing device, a non-volatile computer-readable storage medium, and a computer program product in distributed training. The above can all be used to implement any one of the checkpoint reading and writing methods provided by the present disclosure. The corresponding technical solutions and technical effects can be seen in the corresponding records in the method part and will not be elaborated further.

[0145] Figure 9 A block diagram showing the checkpoint reading and writing device in distributed training provided by an embodiment of the present disclosure. As Figure 9 shown, the checkpoint reading and writing device in distributed training includes:

[0146] A write and record module 91, configured to, in response to the generation of a checkpoint file and when the number of copies of the checkpoint file in the memory of the training node cluster corresponding to the target distributed training task has not reached N, determine free target memory in the memory of the training node cluster, write the checkpoint file into the target memory, and record metadata of the checkpoint file, where N is an integer greater than or equal to 2, and the memory of the training node cluster includes the memories of the respective training nodes in the training node cluster;

[0147] A read and recovery module 92, configured to, in response to any training node in the training node cluster failing, read the target checkpoint file from the memory of the training node cluster according to the metadata of the target checkpoint file corresponding to the training node, and resume training based on the target checkpoint file.

[0148] In a possible implementation manner, the write and record module 91 is further configured to:

[0149] In response to the generation of the checkpoint file and when the number of copies of the checkpoint file in the memory of the training node cluster has reached N, record the metadata of the checkpoint file.

[0150] In a possible implementation manner, the read and recovery module 92 is configured to:

[0151] In response to any training node in the training node cluster failing and when the metadata of the target checkpoint file corresponding to the training node indicates that the checkpoint file exists in the memory of the training node, read the target checkpoint file from the memory of the training node;

[0152] Or,

[0153] In response to any training node in the training node cluster failing and when the metadata of the target checkpoint file corresponding to the training node indicates that the checkpoint file does not exist in the memory of the training node, determine a target training node storing the target checkpoint file according to the metadata of the target checkpoint file, and read the target checkpoint file from the memory of the target training node.

[0154] In a possible implementation manner, the read and recovery module 92 is configured to:

[0155] Read the target checkpoint file from the memory of the target training node through a remote direct memory access network.

[0156] In a possible implementation manner, the write and record module 91 is configured to:

[0157] Determine free target memory in the memory of the training node cluster through a background task, write the checkpoint file into the target memory, and record metadata of the checkpoint file.

[0158] In a possible implementation, the metadata includes:

[0159] The identification information of the training node where the checkpoint file is located, and the name of the checkpoint file.

[0160] In a possible implementation, the writing and recording module 91 is used for:

[0161] Generate an erasure code file corresponding to the checkpoint file through erasure code encoding;

[0162] Write the erasure code file into the target memory.

[0163] In a possible implementation, the writing and recording module 91 is used for:

[0164] Split the checkpoint file into multiple block data;

[0165] Determine multiple free target memories in the memory of the training node cluster;

[0166] Write the multiple block data into the multiple target memories respectively.

[0167] In a possible implementation, the writing and recording module 91 is used for:

[0168] In response to the absence of the checkpoint file in the shared storage device of the training node cluster, write the checkpoint file into the shared storage device.

[0169] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementation and technical effects can refer to the descriptions of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0170] The embodiments of the present disclosure also provide a checkpoint reading and writing device in distributed training, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the above method.

[0171] The embodiments of the present disclosure also provide a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0172] Embodiments of the present disclosure also provide a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0173] Figure 10 FIG. 4 is a block diagram of a checkpoint read / write device 1900 in distributed training according to an exemplary embodiment. For example, the device 1900 may be provided as a server or a terminal device. Referring to Figure 10 , the device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0174] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0175] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as the memory 1932 including computer program instructions. The above computer program instructions may be executed by the processing component 1922 of the device 1900 to complete the above method.

[0176] A computer-readable storage medium can be a tangible device that can retain and store programs / instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device, such as a punched card or raised structures in grooves storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not construed as being an instantaneous signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0177] The computer programs (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0178] A computer program (or computer program instructions) for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via an Internet service provider through the Internet). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present disclosure.

[0179] Aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer - readable program instructions.

[0180] These computer - readable program instructions can be provided to a processor of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data - processing apparatus, create a means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, which causes a computer, a programmable data - processing apparatus, and / or other devices to operate in a particular manner. Thus, the computer - readable medium storing the instructions comprises a manufacture, which includes instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0181] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other devices implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0182] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the boxes may occur in a different order than noted in the figures. For example, two consecutive boxes may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and combinations of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0183] The computer program product may be implemented specifically by hardware, software, or a combination thereof. In an alternative embodiment, the computer program product is specifically embodied as a computer storage medium. In another alternative embodiment, the computer program product is specifically embodied as a software product, such as a Software Development Kit (SDK), etc.

[0184] The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments. The similarities or likenesses among them can be referred to each other. For the sake of brevity, they will not be elaborated herein.

[0185] If the technical solutions of the embodiments of the present disclosure involve personal information, the products applying the technical solutions of the embodiments of the present disclosure have clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solutions of the embodiments of the present disclosure involve sensitive personal information, the products applying the technical solutions of the embodiments of the present disclosure have obtained the individual's separate consent before processing the sensitive personal information, and at the same time meet the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information. The personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0186] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A checkpoint reading and writing method in distributed training, characterized in that, Including: In response to the generation of a checkpoint file, and when the number of copies of the checkpoint file in the memory of the training node cluster corresponding to the target distributed training task has not reached N, determine free target memory in the memory of the training node cluster, write the checkpoint file into the target memory, and record metadata of the checkpoint file, where N is an integer greater than or equal to 2, and the memory of the training node cluster includes the memories of each training node in the training node cluster; In response to any training node in the training node cluster failing, read the target checkpoint file from the memory of the training node cluster according to the metadata of the target checkpoint file corresponding to the training node, and resume training based on the target checkpoint file.

2. The method according to claim 1, characterized in that The method further includes: In response to the generation of the checkpoint file, and when the number of copies of the checkpoint file in the memory of the training node cluster has reached N, record the metadata of the checkpoint file.

3. The method according to claim 1 or 2, characterized in that, The step of, in response to any training node in the training node cluster failing, reading the target checkpoint file from the memory of the training node cluster according to the metadata of the target checkpoint file corresponding to the training node, includes: In response to any training node in the training node cluster failing, and when the metadata of the target checkpoint file corresponding to the training node indicates that the checkpoint file exists in the memory of the training node, read the target checkpoint file from the memory of the training node; Or, In response to any training node in the training node cluster failing, and when the metadata of the target checkpoint file corresponding to the training node indicates that the checkpoint file does not exist in the memory of the training node, determine the target training node storing the target checkpoint file according to the metadata of the target checkpoint file, and read the target checkpoint file from the memory of the target training node.

4. The method according to claim 3, wherein The step of reading the target checkpoint file from the memory of the target training node includes: Reading the target checkpoint file from the memory of the target training node through a Remote Direct Memory Access (RDMA) network.

5. The method according to claim 1 or 2, characterized in that, The step of determining free target memory in the memory of the training node cluster, writing the checkpoint file into the target memory, and recording the metadata of the checkpoint file includes: Determining free target memory in the memory of the training node cluster through a background task, writing the checkpoint file into the target memory, and recording the metadata of the checkpoint file.

6. The method according to claim 1 or 2, characterized in that, The metadata includes: The identification information of the training node where the checkpoint file is located, and the name of the checkpoint file.

7. The method according to claim 1 or 2, characterized in that, The step of writing the checkpoint file into the target memory includes: Generating an erasure code file corresponding to the checkpoint file through erasure code encoding; Writing the erasure code file into the target memory.

8. The method according to claim 1 or 2, characterized in that, The step of determining free target memory in the memory of the training node cluster and writing the checkpoint file into the target memory includes: Dividing the checkpoint file into multiple block data; Determining multiple free target memories in the memory of the training node cluster; Write the multiple chunks of data into the multiple target memories respectively.

9. The method according to claim 1 or 2, characterized in that, After writing the checkpoint file into the target memory, the method further includes: In response to the absence of the checkpoint file in the shared storage device of the training node cluster, write the checkpoint file into the shared storage device.

10. A checkpoint reading and writing device in distributed training, characterized in that, It includes: A writing and recording module, configured to, in response to the generation of a checkpoint file and when the number of copies of the checkpoint file in the memory of the training node cluster corresponding to the target distributed training task does not reach N, determine free target memories in the memory of the training node cluster, write the checkpoint file into the target memories, and record the metadata of the checkpoint file, where N is an integer greater than or equal to 2, and the memory of the training node cluster includes the memories of each training node in the training node cluster; A reading and recovery module, configured to, in response to a failure of any training node in the training node cluster, read the target checkpoint file from the memory of the training node cluster according to the metadata of the target checkpoint file corresponding to the training node, and resume training based on the target checkpoint file.

11. A checkpoint reading and writing device in distributed training, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

12. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images