Large model training fault recovery method and device based on distributed memory management

Abstract

The present invention discloses a large model training fault recovery method based on distributed memory management. When it is determined that the current training round needs to perform a checkpoint preservation service, the model state at the current moment is obtained, and the obtained model state is serialized into a storable format to generate checkpoint data; then the checkpoint data is transmitted from the device-side memory in the node to the host-side memory; and after the transmission is completed, the training task is continued, and asynchronous checkpoint preservation is performed at the same time; the asynchronous checkpoint preservation includes two types of data distributed memory backup and data persistence to disk; when the training task fails, the task is recovered through the checkpoint data. The present invention gives priority to reading backup data from the memory during fault recovery, which can significantly reduce the delay caused by the disk I / O bottleneck, thereby reducing the training suspension or efficiency reduction caused by the operation of saving or loading checkpoint data, maintaining the efficient use of GPU or other computing resources, and improving the continuity and stability of large-scale model training.

Description

Technical Field

[0001] The present invention relates to the field of model training, and in particular to a large model training fault recovery method and device based on distributed memory management. Background Art

[0002] Large language models (LLMs) have gradually become an important branch in the field of artificial intelligence (AI), showing great potential in many aspects, such as machine translation, text summarization, and conversational agents. Training large language models is a difficult task that requires huge computing resources and massive training data. Model size and training data size are the two most critical factors that determine the capabilities of large models. In order to achieve state-of-the-art model capabilities, many AI projects are committed to training large models with hundreds of billions or trillions of parameters on datasets with hundreds of billions or even trillions of annotations.

[0003] With the rapid development of artificial intelligence technology, general-purpose computing on graphics processing units (GPGPU) and other dedicated parallel computing processors have also made rapid progress, and a series of hardware devices for large model training have emerged. The progress of these basic conditions has effectively improved the feasibility and efficiency of large language model training. However, even with such hardware support, large model training may still be interrupted due to long-term computing requirements, hardware failures, energy limitations and other factors, resulting in the loss of previous training results and the need to start training from scratch, which greatly reduces the training efficiency. In addition, the number of parameters of large models is huge, and the demand for memory is extremely high, often exceeding the memory limit of a single computing device. This requires a mechanism that can effectively manage and use model parameters under limited memory resources to support the training of large models. In response to the above problems, checkpoint technology came into being. Checkpoint technology allows the state of the model to be saved at any time during the training process, including but not limited to model parameters, optimizer status, training rounds and other information. When training is interrupted or training tasks need to be migrated, the model state can be restored by loading the previously saved checkpoint, so as to continue training, avoiding duplication of work and waste of resources. At the same time, checkpoint technology also facilitates operations such as continuous model training and model fine-tuning.

[0004] At present, the mainstream checkpoint data storage solutions in the large model training process are mainly divided into two types: synchronous storage and asynchronous storage:

[0005] (1) Checkpoint data synchronization storage solution: In large model training, the checkpoint data synchronization storage solution is a common solution to ensure data integrity and consistency. This solution requires that at each key point in model training, that is, when a checkpoint is generated, the model's state information, including weights, optimizer status, and intermediate calculation results, must be completely written to persistent storage, such as a hard disk drive (HDD) or solid state drive (SSD). This process requires all computing nodes involved in the training to reach a consensus, that is, each node must successfully write data to the above non-volatile storage system, and all nodes must be able to access the data before continuing the next step of training. During the data storage process, the large model training task is in a suspended state. In addition, synchronous storage has high requirements for the load of the network and storage system, and may require higher-performance hardware support. Therefore, the synchronous storage solution is suitable for scenarios that have extremely high requirements for data consistency and relatively loose requirements for training time, such as training tasks with fewer breakpoints during training and can tolerate delays caused by synchronization. Because large model training tasks have high requirements for timeliness and computing card utilization, the checkpoint data synchronization storage solution cannot meet the requirements of large model training tasks.

[0006] (2) Checkpoint data asynchronous storage solution: The checkpoint data asynchronous storage solution is another data storage strategy commonly used in large model training. It allows the training process to continue without pausing when the checkpoint data is written to the persistent storage system, that is, the training can continue while the data is being written, without waiting for the data to be fully synchronized. However, although the asynchronous storage solution saves task waiting time when saving checkpoint data, when a training task fails and the checkpoint data is used for recovery, it still needs to be read from the persistent storage system, and the task recovery speed is limited by the disk I / O speed. In addition, in order to ensure correct recovery in the event of a failure, a more complex fault tolerance and data recovery mechanism needs to be designed, which may increase the complexity and maintenance cost of the system. The asynchronous storage solution is more suitable for training tasks that have high requirements for training speed and high checkpoint generation frequency.

[0007] In the above two storage solutions, checkpoint data is stored in non-volatile storage, such as hard disk drives (HDD) or solid state drives (SSD). As model parameters continue to grow, the checkpoint data also continues to increase. In practical applications, the amount of checkpoint data can range from tens of GB to several TB. When a failure occurs during model training and checkpoint data recovery tasks are used, checkpoint data needs to be read from non-volatile storage. Excessive data reading time will lead to a waste of computing resources. How to recover large model training tasks in a shorter time after a failure has posed a severe challenge to model training fault recovery technology. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention proposes a large model training fault recovery method and device based on distributed memory management. The specific technical solution is as follows:

[0009] A large model training fault recovery method based on distributed memory management includes the following steps:

[0010] Step S1: During the cluster large model training process, determine whether the current training round needs to perform a checkpoint preservation service; if the current training round does not need to perform a checkpoint preservation service, continue the training task; if the current training round needs to perform a checkpoint preservation service, execute steps S2 to S5 to trigger the checkpoint data preservation process and complete the data backup;

[0011] Step S2: Obtain the current model state, including model weight, optimizer state, and training metadata;

[0012] Step S3: Serialize the acquired model state into a storable format to generate checkpoint data;

[0013] Step S4: transferring the checkpoint data from the device-side memory in the node to the host-side memory;

[0014] Step S5: Determine whether the model state serialization data is completely transferred to the host memory. If the transfer is complete, continue the training task and save the checkpoint data generated by the data integrity check algorithm, and perform asynchronous checkpoint saving at the same time; asynchronous checkpoint saving includes data distributed memory backup and data persistence to disk;

[0015] Step S6: When a training task fails, the task is restored through the checkpoint data.

[0016] Furthermore, the distributed backup of data in asynchronous checkpoint storage specifically includes:

[0017] (1) Filter nodes whose remaining memory is greater than m times the maximum checkpoint file size, and then sort them according to the affinity AFFI. The node ranked first is the optimal backup node Bk. The first n nodes are selected to form the optimal backup node combination Bk_Group, where n is the number of computing cards in a single node;

[0018] (2) The checkpoint data in the memory of this node is transferred to the memory of the optimal backup node Bk or the optimal backup node combination Bk_Group;

[0019] (3) After the checkpoint data transmission is completed, the memory backup completion flag FIN_M is set to TRUE.

[0020] Furthermore, persisting the data in the asynchronous checkpoint storage to the disk specifically includes: transferring the checkpoint data in the host memory to the disk of the local node, and setting the disk backup completion flag FIN_D of the local node to TRUE after the transfer is completed.

[0021] Furthermore, the calculation formula of affinity AFFI is:

[0022] AFFI=w1•B+ w2•(1 / L)+ w3•H- w4•PLR- w5•T- w6•(S / F)

[0023] Among them, w1, w2, w3, w4, w5, and w6 are weights assigned according to actual conditions, B is the bandwidth score, L is the latency, H is the backup history between nodes, PLR is the packet loss rate, T is the number of node failures, S is the maximum checkpoint file size in this node, and F is the free memory size of the target node.

[0024] Furthermore, when the checkpoint data in the memory of this node is transmitted to the memory of the optimal backup node Bk or the optimal backup node combination Bk_Group respectively, if the number of remaining nodes in the cluster is less than the number of computing cards n in the node, then all n copies of the checkpoint data are transmitted to the memory of the node Bk with the highest affinity in the cluster through RDMA, and the network address of the node Bk and the memory address of the checkpoint data are recorded at the same time; if the number of remaining nodes in the cluster is not less than the number of computing cards n in the node, then n copies of the data are transmitted to the memory of the optimal backup node combination Bk_Group respectively through RDMA, and the network address of each node in Bk_Group and the memory address of the checkpoint data are recorded at the same time.

[0025] Furthermore, the step S6 specifically includes the following sub-steps:

[0026] S6_1: If the checkpoint data in the memory of this node exists, use the checkpoint data in the memory of this node to recover the task;

[0027] S6_2: If the node is restarted or the task of the node is scheduled to another node, the checkpoint data in the memory of the backup node Bk or the optimal backup node combination Bk_Group is used to recover the task;

[0028] S6_3: If the checkpoint data in the memory of the backup node Bk or a node in the optimal backup node combination Bk_Group is lost or damaged, the checkpoint data in the disk is used to recover the task.

[0029] Further, after the task recovery in step S6 is performed, the method further includes:

[0030] Step S7: Manage the saved checkpoint data according to the task status.

[0031] Furthermore, step S7 specifically includes the following sub-steps:

[0032] S7_1: If the task has resumed normal training and completed the next round of checkpoint data saving, delete the checkpoint data saved in the memory of the previous round of Bk nodes or nodes in Bk_Group;

[0033] S7_2: If the task has resumed normal training but has not completed the next round of checkpoint data saving, the checkpoint data saved in the memory of the previous round of Bk nodes or nodes in Bk_Group is retained;

[0034] S7_3: If the training task ends normally, delete the checkpoint data saved in the memory of the previous round of Bk nodes or nodes in Bk_Group.

[0035] A large model training fault recovery device based on distributed memory management, the device includes one or more processors, and is used to implement a large model training fault recovery method based on distributed memory management.

[0036] An electronic device, comprising:

[0037] one or more processors;

[0038] A storage device is used to store one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements a large model training fault recovery method based on distributed memory management.

[0039] The beneficial effects of the present invention are:

[0040] The large model training fault recovery method based on distributed memory management of the present invention preferentially reads backup data from the memory, which can significantly reduce the delay caused by the disk I / O bottleneck, thereby reducing the training suspension or efficiency decline caused by saving or loading checkpoint data operations, maintaining the efficient use of GPU or other computing resources, and improving the continuity and stability of large-scale model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of a common checkpoint data structure involved in the present invention.

[0042] Figure 2 It is a data backup flow chart of a large model training fault recovery method based on distributed memory management in an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of a computing cluster storing checkpoint data in distributed memory based on the method of an embodiment of the present invention.

[0044] Figure 4 It is a fault recovery flow chart of a large model training fault recovery method based on distributed memory management according to an embodiment of the present invention.

[0045] Figure 5 It is a structural diagram of a large model training fault recovery device based on distributed memory management according to another embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0047] In the following description, reference is made to “one specific embodiment”, which describes a subset of all possible embodiments, but it can be understood that “one specific embodiment” describes the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0048] Unless otherwise defined, all technical science and technology used herein have the same meaning as commonly understood by those skilled in the art of the present invention. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0049] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following explanations.

[0050] (1) Large models refer to machine learning models with large-scale parameters and complex computational structures. These models are usually built with deep neural networks and have billions or even hundreds of billions of parameters. Large models learn complex patterns and features by training massive amounts of data, have stronger generalization capabilities, and can make accurate predictions on unseen data.

[0051] (2) Epoch: In the training process of machine learning, it refers to the process of inputting the entire training data set into the neural network for a complete training. This process is usually repeated multiple times so that the model can better learn the features and patterns in the data set. After each epoch, the model will adjust and optimize parameters based on the current data batch processing results. When all training samples have been used by the model for training once, an epoch is completed. The number of epochs is an important hyperparameter in the training process, which directly affects the training effect and final performance of the model.

[0052] (3) Batch refers to a group of samples used to train the model in one iteration. This means that the model only processes a small portion of the data at a time, rather than processing the entire dataset at once. For large datasets, loading all the data for training at once may exceed the capacity of the memory or video memory. Using batches can effectively manage and utilize limited computing resources. The batch size is another important hyperparameter that affects the stability and convergence speed of model training.

[0053] (4) Checkpoints: Checkpoints are used to save the model state during model training to cope with possible interruptions during training. These checkpoints usually include model parameters (such as weights and biases), optimizer status, and other relevant training information. By saving these checkpoints, the current state of the model can be saved periodically during training so that training can be resumed when needed or used for model evaluation and inference. Figure 1 As shown in FIG. 1 , a schematic diagram of a common checkpoint data structure in the field is shown.

[0054] (5) Disk I / O, or disk input / output, refers to the process of data exchange between a computer system and a disk storage device, involving data reading and writing operations. This process includes the computer initiating a read or write request, and the disk drive responding to these requests by moving the head to the specified position to read data or write data to the disk.

[0055] (6) Host: usually refers to a complete computing system that can independently execute programs and manage other devices. In a wider network environment, a host can be any computer system connected to the network, including personal computers, servers, smartphones, etc. A host usually has core components such as a central processing unit (CPU), memory (RAM), hard disk drive (HDD) or solid-state drive (SSD), and an operating system (OS) that can run applications and process data. In a distributed system or client-server model, a host may refer to a server that provides services to clients, such as file storage, data processing, or website hosting.

[0056] (7) Device: refers to any hardware component that can be recognized and controlled by a computer system. This includes input devices (such as keyboards, mice), output devices (such as monitors, printers), storage devices (such as hard disks, USB drives), network communication devices (such as routers, modems), and more specialized devices such as graphics processing units (GPUs), sound cards, etc. In a computer system, a device can be any hardware that requires a driver to communicate and interact with the host. With the development of Internet of Things (IoT) technology, the concept of device has also expanded to various smart devices, such as smart home devices, sensors, etc., which can connect to the Internet and exchange data. In computer architecture, the relationship between host and device is usually a master-slave relationship, with the host acting as the central control unit and the device responding to the host's instructions to operate. The host is responsible for coordinating the operation of the device, processing data, and feeding back the results to the user or other systems when necessary. The device focuses on performing specific tasks, such as data input, output, or processing. This division of labor and cooperation ensures the efficient operation and flexibility of the computer system.

[0057] (8) RDMA: Remote Direct Memory Access is a high-performance network communication technology that allows one computer system in a network to directly access the memory of another computer system without the intervention of the central processing unit (CPU) of the target system. This direct memory access capability reduces the latency and overhead of data transmission by bypassing the traditional operating system network stack and CPU processing, thereby improving data transmission efficiency.

[0058] Based on the above architecture, an embodiment of the present invention proposes a large model training fault recovery method and device based on distributed memory management, which can maintain efficient utilization of GPU or other computing resources and improve the continuity and stability of large-scale model training.

[0059] like Figure 2 and Figure 4As shown, the large model training fault recovery method based on distributed memory management in an embodiment of the present invention includes two parts: data backup and fault recovery, and specifically includes the following steps:

[0060] Step S1: During the cluster large model training process, determine whether the current training round requires a checkpoint save service; if the current training round does not require a checkpoint save service, continue the training task; if the current training round requires a checkpoint save service, execute steps S2 to S5 to trigger the checkpoint data save process and complete the data backup.

[0061] In this embodiment, when the large model training task meets the condition that "the current training epoch round is a multiple of 10", checkpoint data is saved.

[0062] Step S2: Get the current model status, including model weights, optimizer status, and training metadata (such as epoch number, batch number, best validation metric, etc.);

[0063] Step S3: Serialize the acquired model state into a storable format to generate checkpoint data. The storable formats include but are not limited to PyTorch's .pt format, TensorFlow's .ckpt format, etc. In this embodiment, the checkpoint data format is PyTorch's .pt format.

[0064] Step S4: Transfer the checkpoint data from the device memory in the node to the host memory. The device here includes general-purpose graphics processors and other dedicated parallel computing processors.

[0065] Step S5: Determine whether the model state serialization data is completely transferred to the host memory. If the transfer is complete, continue the training task and save the checkpoint data generated by the hash algorithm (one of the data integrity verification algorithms), and perform asynchronous checkpoint saving at the same time; asynchronous checkpoint saving includes data distributed memory backup and data persistence to disk;

[0066] Among them, data distributed backup includes:

[0067] (1) Nodes whose remaining memory is greater than m times the maximum checkpoint file size are selected, and then sorted according to the affinity AFFI. The node ranked first is the optimal backup node Bk. The first n nodes are selected to form the optimal backup node combination Bk_Group, where n is the number of single-node computing cards; in this embodiment, m=1.5 and n=4.

[0068] AFFI=w1•B+ w2•(1 / L)+ w3•H- w4•PLR- w5•T- w6•(S / F)

[0069] Among them, w1, w2, w3, w4, w5, and w6 are weights allocated according to actual conditions. In this embodiment, w1=25%, w2=20%, w3=20%, w4=10%, w5=10%, and w6=15%. B is the bandwidth score, L is the latency, H is the backup history between nodes, PLR is the packet loss rate, T is the number of node failures, S is the maximum checkpoint file size in the current node, and F is the free memory size of the target node.

[0070] (2) The checkpoint data in the memory of this node is transferred to the memory of the optimal backup node Bk or the optimal backup node combination Bk_Group.

[0071] Preferably, if the number of remaining nodes in the cluster is less than the number of computing cards n in the node, then all n copies of checkpoint data are transmitted to the memory of the node Bk with the highest affinity in the cluster via RDMA, and the network address of node Bk and the memory address of the checkpoint data are recorded; if the number of remaining nodes in the cluster is not less than the number of computing cards n in the node, then the n copies of data are transmitted to the memory of the optimal backup node combination Bk_Group via RDMA, and the network address of each node in Bk_Group and the memory address of the checkpoint data are recorded.

[0072] (3) After the checkpoint data transmission is completed, the memory backup completion flag FIN_M is set to TRUE.

[0073] The specific steps for persisting data to disk are:

[0074] The checkpoint data in the host memory is transferred to the local node disk. After the transfer is completed, the local node disk backup completion flag FIN_D is set to TRUE. The specific data backup process is as follows: Figure 2 shown.

[0075] like Figure 3As shown in the figure, the computing cluster has 5 nodes A~E. The first line of each node represents the checkpoint data generated by the computing card of this node stored in the memory, and the second line is the checkpoint data transmitted to the memory of this node from other nodes. For example, the checkpoint data generated by 4 computing cards in node A will be divided into 4 blocks, A_1~A_4, and stored in the host memory of this node. Then, these 4 copies of data will be transmitted to the host memory of other nodes in the cluster for storage.

[0076] Step S6: When a training task fails, the task is restored through the checkpoint data, such as Figure 3 As shown, the specific steps include:

[0077] S6_1: If the checkpoint data exists in the memory of this node, use the checkpoint data in the memory of this node to recover the task.

[0078] S6_2: If the node is restarted or the task of the node is scheduled to another node, the checkpoint data in the memory of the backup node Bk or the optimal backup node combination Bk_Group is used to recover the task.

[0079] S6_3: If the checkpoint data in the memory of the backup node Bk or a node in the optimal backup node combination Bk_Group is lost or damaged, use the checkpoint data in the disk to recover the task.

[0080] Step S7: After the training task is restored, the checkpoint data saved in the previous steps is managed according to the task status, specifically including:

[0081] S7_1: If the task has resumed normal training and completed the next round of checkpoint data saving, delete the checkpoint data saved in the memory of the previous round of Bk nodes or nodes in Bk_Group.

[0082] S7_2: If the task has resumed normal training but has not completed the next round of checkpoint data saving, retain the checkpoint data saved in the memory of the previous round of Bk nodes or nodes in Bk_Group.

[0083] S7_3: If the training task ends normally, delete the checkpoint data saved in the memory of the previous round of Bk nodes or nodes in Bk_Group.

[0084] The present invention realizes rapid recovery of faults in the training process of large models in computing clusters based on distributed memory management, ensures that large model training tasks can be carried out continuously and reliably, avoids long-term interruption of training due to faults, thereby wasting GPU or other computing resources, and helps to improve the continuity and stability of large-scale model training.

[0085] Corresponding to the aforementioned embodiment of a large model training fault recovery method based on distributed memory management, the present invention also provides an embodiment of a large model training fault recovery device based on distributed memory management.

[0086] See also Figure 5 The large model training fault recovery device based on distributed memory management provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a large model training fault recovery method based on distributed memory management in the above embodiment.

[0087] An embodiment of a large model training fault recovery device based on distributed memory management of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From a hardware perspective, if Figure 5 As shown, it is a hardware structure diagram of any device with data processing capability where the large model training fault recovery device based on distributed memory management of the present invention is located, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0088] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0089] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0090] An embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a large model training fault recovery method based on distributed memory management in the above embodiment is implemented.

[0091] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A large model training fault recovery method based on distributed memory management, characterized in that: The following steps are involved: Step S1: During the cluster large model training process, determine whether the current training round requires a checkpoint save service; if the current training round does not require a checkpoint save service, continue the training task; If the current training round requires a checkpoint save service, execute steps S2 to S5 to trigger the checkpoint data save process and complete the data backup; Step S2: Obtain the current model status, including model weight, optimizer status, and training metadata; Step S3: Serialize the acquired model state into a storable format to generate checkpoint data; Step S4: transferring the checkpoint data from the device-side memory in the node to the host-side memory; Step S5: Determine whether the model state serialization data is completely transferred to the host memory. If the transfer is complete, continue the training task and save the checkpoint data generated by the data integrity check algorithm, and perform asynchronous checkpoint saving at the same time; Asynchronous checkpoint storage includes two types: data distributed memory backup and data persistence to disk. The data distributed memory backup specifically includes: (1) Filter nodes whose remaining memory is greater than m times the maximum checkpoint file size, and then sort them according to the affinity AFFI. The node ranked first is the optimal backup node Bk. The first n nodes are selected to form the optimal backup node combination Bk_Group, where n is the number of computing cards in a single node; (2) The checkpoint data in the memory of this node is transferred to the memory of the optimal backup node Bk or the optimal backup node combination Bk_Group; (3) After the checkpoint data transmission is completed, the memory backup completion flag FIN_M is set to TRUE; Step S6: When a training task fails, the task is restored through the checkpoint data.

2. The large model training fault recovery method based on distributed memory management according to claim 1 is characterized in that: Persisting the data in the asynchronous checkpoint storage to the disk specifically includes: transferring the checkpoint data in the host memory to the disk of the local node, and setting the disk backup completion flag FIN_D of the local node to TRUE after the transfer is completed.

3. The large model training fault recovery method based on distributed memory management according to claim 1, characterized in that: The calculation formula of affinity AFFI is: AFFI=w1•B+ w2•(1 / L)+ w3•H- w4•PLR- w5•T- w6•(S / F) Among them, w1, w2, w3, w4, w5, and w6 are weights assigned according to actual conditions, B is the bandwidth score, L is the latency, H is the backup history between nodes, PLR is the packet loss rate, T is the number of node failures, and S is the maximum checkpoint file size in this node. F is the free memory size of the target node.

4. The large model training fault recovery method based on distributed memory management according to claim 3 is characterized in that: When the checkpoint data in the memory of this node is transferred to the memory of the optimal backup node Bk or the optimal backup node combination Bk_Group, if the number of remaining nodes in the cluster is less than the number of computing cards n in the node, then all n copies of the checkpoint data are transferred to the memory of the node Bk with the highest affinity in the cluster through RDMA, and the network address of the node Bk and the memory address of the checkpoint data are recorded; if the number of remaining nodes in the cluster is not less than the number of computing cards n in the node, then n copies of the data are transferred to the memory of the optimal backup node combination Bk_Group through RDMA, and the network address of each node in Bk_Group and the memory address of the checkpoint data are recorded.

5. The large model training fault recovery method based on distributed memory management according to claim 1 is characterized in that: The step S6 specifically includes the following sub-steps: S6_1: If the checkpoint data in the memory of this node exists, use the checkpoint data in the memory of this node to recover the task; S6_2: If the node is restarted or the task of the node is scheduled to another node, the checkpoint data in the memory of the backup node Bk or the optimal backup node combination Bk_Group is used to recover the task; S6_3: If the checkpoint data in the memory of the backup node Bk or a node in the optimal backup node combination Bk_Group is lost or damaged, the checkpoint data in the disk is used to recover the task.

6. The large model training fault recovery method based on distributed memory management according to claim 1 is characterized in that: After the task recovery in step S6 is completed, the following steps are also included: Step S7: Manage the saved checkpoint data according to the task status.

7. The large model training fault recovery method based on distributed memory management according to claim 6 is characterized in that: Step S7 specifically includes the following sub-steps: S7_1: If the task has resumed normal training and completed the next round of checkpoint data saving, delete the checkpoint data saved in the memory of the previous round of Bk nodes or nodes in Bk_Group; S7_2: If the task has resumed normal training but has not completed the next round of checkpoint data saving, the checkpoint data saved in the memory of the previous round of Bk nodes or nodes in Bk_Group is retained; S7_3: If the training task ends normally, delete the checkpoint data saved in the memory of the previous round of Bk nodes or nodes in Bk_Group.

8. A large model training fault recovery device based on distributed memory management, characterized in that: The device includes one or more processors for implementing the large model training fault recovery method based on distributed memory management as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the electronic device, enables the electronic device to implement the large model training fault recovery method based on distributed memory management as described in any one of claims 1 to 7.

Images