Large model training system and method, computing device and storage medium

Abstract

The invention provides a large model training system and method, computing equipment and a storage medium, and belongs to the technical field of artificial intelligence. In the method, a plurality of computing cards in a large model training system are grouped by utilizing the characteristic that a plurality of computing cards executing the same training task generate the same checkpoint, a plurality of backups of the same checkpoint are formed in a memory of a computing node where one group of computing cards are located, and in response to a training recovery instruction, the computing cards in different computing card groups communicate with each other. The method comprises the following steps of: obtaining a full-amount checkpoint, and carrying out training recovery based on the full-amount checkpoint. Due to the fact that part of the checkpoint is stored in the computing node where each computing card is located, the data size is small, when training recovery is carried out, the efficiency of loading the checkpoint by the computing card is high, the training recovery efficiency is high, and waste of computing resources can be reduced.

Description

[0001] This application claims priority to Chinese patent application No. 202410389583.5 filed on April 1, 2024, entitled “A method, device and other equipment for data processing”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a large-scale model training system, method, computing device, and storage medium. Background Art

[0003] In the training of large artificial intelligence (AI) models, large models are usually trained through distributed clusters. The distributed cluster includes multiple computing nodes, and each computing node includes at least one computing card. During the distributed training process, parameters such as model weights, optimizer states, gradients, and the number of training cycles are generated. The distributed cluster saves the generated parameters as checkpoints at a fixed frequency, so that when a computing node in the distributed cluster fails, the distributed cluster can load the most recently saved checkpoint and resume training based on the checkpoint, thereby avoiding training from the beginning, saving failure losses, and improving training efficiency.

[0004] In related technologies, each computing node in a distributed cluster stores the checkpoint generated by the computing card on the node in the memory of the node, and forms a copy on each of the other m computing nodes according to a certain multi-copy strategy (m is greater than or equal to 1). If the user suspends training and then restarts training, the computing card on each computing node loads the checkpoint from the memory of the computing node where it is located and performs training recovery based on the checkpoint; if any computing node fails and the checkpoint in the memory of the computing node is lost, the replacement computing node can load the checkpoint from the computing node that saves the checkpoint copy to the memory of the replacement computing node, and the computing card on each computing node then loads the checkpoint from the memory of the computing node where it is located and performs training recovery based on the checkpoint.

[0005] However, in the above method, all m computing nodes store the full checkpoint, which has a large amount of data. When recovering training, the full checkpoint needs to be transmitted across nodes, which leads to low training recovery efficiency and waste of computing resources. Summary of the Invention

[0006] Embodiments of the present application provide a large model training system, method, computing device and storage medium, which can improve the loading efficiency of checkpoint, improve the training recovery efficiency, and reduce the waste of computing resources. The technical solution is as follows.

[0007] In a first aspect, a large model training system is provided, which includes a plurality of computing cards located on a plurality of computing nodes, the computing nodes including memory, the plurality of computing cards including a first computing card group and a second computing card group, the plurality of computing card groups being used to execute the same training task, in a first iteration of the training task, the computing cards in the first computing card group and the second computing card group respectively generate the same checkpoint, the checkpoint including at least one of a weight parameter, an optimizer state, a gradient and a number of training periods of the first iteration of the training task, the checkpoint including a first checkpoint and a second checkpoint, a plurality of copies of the first checkpoint being stored on a plurality of computing nodes on which the plurality of computing cards in the first computing card group are located, a plurality of copies of the second checkpoint being stored on a plurality of computing nodes on which the plurality of computing cards in the second computing card group are located;

[0008] The computing cards in the first computing card group are configured to: in response to a training recovery instruction, load the first checkpoint from the memory of the computing node on which the computing card in the first computing card group is located, send the first checkpoint to the computing cards in the second computing card group, receive the second checkpoint sent by the computing cards in the second computing card group, the training recovery instruction indicating that training recovery is performed according to the first iteration; and based on the first checkpoint and the second checkpoint, recover the training task.

[0009] In the above system, the plurality of computing cards executing the same training task are grouped based on the feature that the same checkpoint is generated, a plurality of backups of the same checkpoint are formed in the memory of the computing node on which the computing cards in a group are located, in response to a training recovery instruction, the computing cards in different computing card groups communicate the checkpoint to obtain the full checkpoint, and then the training recovery is performed based on the full checkpoint. Since each computing card is located on a computing node that only saves part of the checkpoint, the data volume is small, and when training recovery is performed, the loading efficiency of the checkpoint is high, the training recovery efficiency is high, and the waste of computing resources can be reduced.

[0010] Optionally, the first computing card group includes a first computing card and a second computing card, and the second computing card group includes a third computing card and a fourth computing card;

[0011] The first computing card is used to store the first checkpoint in the memory of the computing node where the first computing card is located;

[0012] The second computing card is used to store the first checkpoint in the memory of the computing node where the second computing card is located;

[0013] The third computing card is used to store the second checkpoint in the memory of the computing node where the third computing card is located;

[0014] The fourth computing card is used to store the second checkpoint in the memory of the computing node where the fourth computing card is located.

[0015] Optionally, the first checkpoint and the second checkpoint constitute the checkpoint.

[0016] Optionally, the first computing card and the second computing card are located in different computing nodes, and the third computing card and the fourth computing card are located in different computing nodes.

[0017] Optionally, the large model training system further includes a control node, which is used to:

[0018] Based on the topological structure information and the preset backup number of the multiple computing cards, the first computing card group and the second computing card group among the multiple computing cards are determined, the topological structure information is used to indicate the parallel relationship between the multiple computing cards, and the number of computing cards in the first computing card group and the second computing card group is equal to the preset backup number.

[0019] Optionally, the computing cards in the first computing card group and the second computing card group are further used for:

[0020] If the first iteration satisfies the first frequency, metadata of the checkpoint is sent to the control node through the respective computing nodes.

[0021] The control node is further configured to determine the first checkpoint and the second checkpoint according to the metadata of the checkpoint.

[0022] Optionally, a difference between the data volume of the first checkpoint, the data volume of the second checkpoint, and an average data volume is smaller than a target value, where the average data volume is determined based on the data volume of the checkpoint and the preset number of backups.

[0023] In the above system, checkpoints are divided into multiple copies based on the average data volume. This ensures that the data volume of the multiple checkpoints is uniform, thereby evenly distributing the checkpoint storage tasks to different computing card groups. This helps to unify the time taken by different computing card groups to store the checkpoints they are responsible for.

[0024] Optionally, the control node is further configured to:

[0025] Sending a first save instruction to the first computing card, where the first save instruction is used to instruct the first computing card to store the first checkpoint in the memory of the computing node where the card is located;

[0026] Sending a second save instruction to the second computing card, where the second save instruction is used to instruct the second computing card to store the first checkpoint in the memory of the computing node where the card is located;

[0027] Sending a third save instruction to the third computing card, where the third save instruction is used to instruct the third computing card to store the second checkpoint in the memory of the computing node where the card is located;

[0028] A fourth save instruction is sent to the fourth computing card, where the fourth save instruction is used to instruct the fourth computing card to store the second checkpoint in the memory of the computing node where the card is located.

[0029] Optionally, the control node is further configured to:

[0030] If the computing nodes where the computing cards in the first computing card group and the second computing card group are located have insufficient memory, sending a data deletion request for a second iteration to the computing nodes where the computing cards in the first computing card group and the second computing card group are located, respectively, where the second iteration is the iteration whose generated checkpoint is stored in the computing node memory and whose iteration order is the first number of iterations ahead;

[0031] The computing nodes where the computing cards in the first computing card group and the second computing card group are located are further used to: respectively receive the data deletion request and delete the checkpoint generated in the second iteration from the memory.

[0032] Optionally, the large model training system further includes a storage device;

[0033] The control node is also used to:

[0034] If the first iteration meets the data persistence condition, a data persistence request for the first iteration is sent to the computing nodes where the computing cards in the first computing card group and the second computing card group are located, respectively, where the data persistence condition indicates that the first iteration meets the second frequency, the iteration order of the first iteration is a preset order, or the checkpoint generated by the first iteration is currently persistent;

[0035] The computing nodes where the computing cards in the first computing card group and the second computing card group are located are further used to: respectively receive the data persistence request, and store the first checkpoint and the second checkpoint from the memory to the storage device.

[0036] Optionally, the memory of the computing node where the first computing card is located includes a first buffer zone and a second buffer zone;

[0037] The first computing card is configured to: store the first checkpoint in the first buffer, and store a third checkpoint in the second buffer, where the third checkpoint is a partial checkpoint generated by computing cards in the first computing card group and the second computing card group in a third iteration, where the third iteration is an iteration subsequent to the first iteration;

[0038] The computing node where the first computing card is located is used to: store the first checkpoint from the first buffer to the storage device while the first computing card stores the third checkpoint to the second buffer.

[0039] In the above system, two buffers in the memory are used to receive data written by the computing card. When the computing card writes data to one of the buffers, the computing node where the computing card is located can persist the data in the other buffer to the storage device. The two buffers continuously exchange positions as training progresses, which can avoid the checkpoint persistence process blocking the checkpoint saving process.

[0040] Optionally, the memory of the computing node where the first computing card is located further includes a third buffer zone;

[0041] The computing node where the first computing card is located is used for:

[0042] If, when the first computing card stores the fourth checkpoint in the first buffer, there is a first checkpoint in the first buffer that has not been stored in the storage device, the first checkpoint in the first buffer is written from the first buffer to the third buffer. The fourth checkpoint is a partial checkpoint generated by the computing cards in the first computing card group and the second computing card group in a fourth iteration, which is an iteration after the third iteration.

[0043] While storing the fourth checkpoint in the first buffer, the first checkpoint in the third buffer is stored in the storage device.

[0044] In the above system, checkpoints are stored in the first buffer and the second buffer of the memory. The persistence of the checkpoint in the first buffer has not been completed. When the computing card needs to write the latest checkpoint to the memory, the computing node writes the checkpoint that has not been persisted in the first buffer into a temporary buffer, and then can write the latest checkpoint to the first buffer. At the same time as the computing card writes the latest checkpoint to the first buffer, the computing node writes the checkpoint that has not been persisted from the temporary buffer to the storage device. Since the efficiency of data copying between buffers is high, the blocking of the checkpoint persistence process on the checkpoint saving process can be reduced.

[0045] Optionally, the computing cards in the first computing card group are further used to:

[0046] In response to the training resume instruction, load the first checkpoint from the memory of the computing node, and send the first checkpoint to the computing cards in the second computing card group through the communication link between the multiple computing cards;

[0047] The second checkpoint sent by the computing card in the second computing card group is received through the communication link between the multiple computing cards.

[0048] In the above system, parameter-plane communication between computing cards is used for checkpoint completion. Compared with the traditional method of obtaining full checkpoints through cross-node communication, this system utilizes high-speed inter-card bandwidth to obtain checkpoints, which can significantly improve the efficiency of resuming training.

[0049] Optionally, the large model training system further includes a first computing node, the first computing node being used to replace a failed computing node; the first computing node being used to:

[0050] In the event that the computing node where the second computing card is located fails, in response to the training recovery instruction, obtaining the first checkpoint from the memory of the computing node where the first computing card is located;

[0051] The first checkpoint is stored in the memory of the first computing node.

[0052] In a second aspect, a large model training method is provided, which is applied to a large model training system. The large model training system includes multiple computing cards, the multiple computing cards are located on multiple computing nodes, the computing nodes include memory, the multiple computing cards include a first computing card group and a second computing card group, the multiple computing card groups are used to perform the same training task, in a first iteration of the training task, the computing cards in the first computing card group and the second computing card group respectively generate the same checkpoint, the checkpoint includes at least one of the weight parameters, optimizer state and gradient of the first iteration of the training task, the checkpoint includes a first checkpoint and a second checkpoint, multiple copies of the first checkpoint are stored on the multiple computing nodes where the multiple computing cards in the first computing card group are located, and multiple copies of the second checkpoint are stored on the multiple computing nodes where the multiple computing cards in the second computing card group are located;

[0053] The method includes:

[0054] The computing card in the first computing card group responds to the training resume instruction by loading the first checkpoint from the memory of the computing node where the computing card is located, sending the first checkpoint to the computing card in the second computing card group, and receiving the second checkpoint sent by the computing card in the second computing card group, wherein the training resume instruction instructs to resume training according to the first iteration;

[0055] The computing cards in the first computing card group recover the training task based on the first checkpoint and the second checkpoint.

[0056] Optionally, the first computing card group includes a first computing card and a second computing card, and the second computing card group includes a third computing card and a fourth computing card;

[0057] The method further includes:

[0058] The first computing card stores the first checkpoint into the memory of the computing node where the first computing card is located.

[0059] The second computing card stores the first checkpoint into the memory of the computing node where the second computing card is located.

[0060] The third computing card stores the second checkpoint into the memory of the computing node where the third computing card is located.

[0061] The fourth computing card stores the second checkpoint into the memory of the computing node where the fourth computing card is located.

[0062] Optionally, the first checkpoint and the second checkpoint constitute the checkpoint.

[0063] Optionally, the first computing card and the second computing card are located in different computing nodes, and the third computing card and the fourth computing card are located in different computing nodes.

[0064] Optionally, the large model training system further comprises a control node, and the method further comprises:

[0065] The control node determines the first computing card group and the second computing card group in the plurality of computing cards based on the topology information of the plurality of computing cards and the pre-device quantity, the topology information being used to indicate the parallel relationship between the plurality of computing cards, and the number of computing cards in the first computing card group and the second computing card group being equal to the pre-device quantity.

[0066] Optionally, the method further comprises:

[0067] If the first iteration satisfies a first frequency, the computing cards in the first computing card group and the second computing card group respectively send the metadata of the checkpoint to the control node through the computing nodes where the computing cards are located.

[0068] The control node determines the first checkpoint and the second checkpoint according to the metadata of the checkpoint.

[0069] Optionally, the difference between the data amount of the first checkpoint and the data amount of the second checkpoint and an average data amount is less than a target value, and the average data amount is determined based on the data amount of the checkpoint and the pre-device quantity.

[0070] Optionally, the method further comprises:

[0071] The control node sends a first save instruction to the first computing card, where the first save instruction is used to instruct the first computing card to store the first checkpoint in the memory of the computing node where the card is located;

[0072] The control node sends a second save instruction to the second computing card, where the second save instruction is used to instruct the second computing card to store the first checkpoint in the memory of the computing node where the card is located;

[0073] The control node sends a third save instruction to the third computing card, where the third save instruction is used to instruct the third computing card to store the second checkpoint in the memory of the computing node where the card is located;

[0074] The control node sends a fourth save instruction to the fourth computing card, where the fourth save instruction is used to instruct the fourth computing card to store the second checkpoint in the memory of the computing node where the card is located.

[0075] Optionally, the method further includes:

[0076] If the computing nodes where the computing cards in the first computing card group and the second computing card group are located have insufficient memory, the control node sends a data deletion request for a second iteration to the computing nodes where the computing cards in the first computing card group and the second computing card group are located, respectively. The second iteration is the iteration whose generated checkpoint is stored in the computing node memory and whose iteration order is the first number of iterations ahead;

[0077] The computing nodes where the computing cards in the first computing card group and the second computing card group are located respectively receive the data deletion request and delete the checkpoint generated in the first iteration from the memory.

[0078] Optionally, the large model training system further includes a storage device;

[0079] The method further includes:

[0080] If the first iteration meets the data persistence condition, the control node sends a data persistence request for the first iteration to the computing nodes where the computing cards in the first computing card group and the second computing card group are located, respectively, where the data persistence condition indicates that the first iteration meets the second frequency, the iteration order of the first iteration is a preset order, or the checkpoint generated by the first iteration is currently persistent;

[0081] The computing nodes where the computing cards in the first computing card group and the second computing card group are located respectively receive the data persistence request, and store the first checkpoint and the second checkpoint from the memory to the storage device.

[0082] Optionally, the memory of the computing node where the first computing card is located includes a first buffer zone and a second buffer zone;

[0083] The computing node where the computing card in the first computing card group is located receives the data persistence request and stores the first checkpoint from the memory to the storage device, including:

[0084] The first computing card stores the first checkpoint in the first buffer and stores a third checkpoint in the second buffer, where the third checkpoint is a partial checkpoint generated by computing cards in the first computing card group and the second computing card group in a third iteration, where the third iteration is an iteration after the first iteration;

[0085] The computing node where the first computing card is located stores the first checkpoint from the first buffer to the storage device while the first computing card stores the third checkpoint to the second buffer.

[0086] Optionally, the memory of the computing node where the first computing card is located further includes a third buffer zone;

[0087] Storing the first checkpoint from the first buffer to the storage device includes:

[0088] If, when the first computing card stores the fourth checkpoint in the first buffer, there is a first checkpoint in the first buffer that has not been stored in the storage device, the computing node where the first computing card is located writes the first checkpoint in the first buffer from the first buffer to the third buffer;

[0089] While storing the fourth checkpoint in the first buffer, the computing node where the first computing card is located stores the first checkpoint in the third buffer in the storage device.

[0090] Optionally, the computing card in the first computing card group, in response to the training resume instruction, loads the first checkpoint from the memory of the computing node where it is located, sends the first checkpoint to the computing card in the second computing card group, and receives the second checkpoint sent by the computing card in the second computing card group, including:

[0091] In response to the training resume instruction, a computing card in the first computing card group loads the first checkpoint from a memory of the computing node where the computing card is located, and sends the first checkpoint to a computing card in the second computing card group through a communication link between the plurality of computing cards;

[0092] The computing card in the first computing card group receives the second checkpoint sent by the computing card in the second computing card group through the communication link between the multiple computing cards.

[0093] Optionally, the large model training system further includes a first computing node, the first computing node being used to replace a failed computing node;

[0094] The method further includes:

[0095] When the computing node where the second computing card is located fails, the first computing node obtains the first checkpoint from the memory of the computing node where the first computing card is located in response to the training recovery instruction;

[0096] The first computing node stores the first checkpoint in the memory of the first computing node.

[0097] In a third aspect, a large model training device is provided, which includes at least one functional module, and the at least one functional module is used to execute the large model training method provided by the aforementioned second aspect or any possible implementation method of the second aspect.

[0098] In a fourth aspect, a computing card is provided, which includes a computing core and a memory. The computing card is used to execute the large model training method provided by the aforementioned second aspect or any possible implementation of the second aspect, the memory is used to store computing data, and the computing core is used to perform computing operations on the computing data stored in the memory.

[0099] In a fifth aspect, a computing device is provided, wherein the computing node includes a processor, a computing card and a memory, and the processor is used to execute the large model training method provided in the aforementioned second aspect or any possible implementation of the second aspect.

[0100] In the sixth aspect, a computing device cluster is provided, which includes at least one computing node, each of which includes a processor, a computing card and a memory, and the processor of the at least one computing node is used to execute instructions so that the computing device cluster executes the large model training method provided by the aforementioned second aspect or any possible implementation of the second aspect.

[0101] In the seventh aspect, a computer program product comprising instructions is provided, which, when executed by a computing device cluster, enables the computing device cluster to execute the large model training method provided in the aforementioned second aspect or any possible implementation of the second aspect.

[0102] In an eighth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the large model training method provided in the aforementioned second aspect or any possible implementation of the second aspect.

[0103] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 This is a schematic diagram of the structure of a large model training system provided by this application;

[0105] Figure 2 This is a flowchart of the checkpoint saving process in a large model training method provided in an embodiment of the present application;

[0106] Figure 3 This is a schematic diagram of the checkpoint division process in a large model training method provided in an embodiment of the present application;

[0107] Figure 4 This is a flowchart of a checkpoint saving process in a large model training method provided in an embodiment of the present application;

[0108] Figure 5 This is a flowchart of a checkpoint saving process during a large model training process provided by an embodiment of the present application;

[0109] Figure 6 This is a flowchart of the checkpoint persistence process in a large model training method provided in an embodiment of the present application;

[0110] Figure 7 This is a schematic diagram of the checkpoint persistence process in a large model training method provided in an embodiment of the present application;

[0111] Figure 8 This is an example diagram of the checkpoint persistence process in a large model training method provided in an embodiment of the present application;

[0112] Figure 9 This is a flowchart of the checkpoint loading process in a large model training method provided in an embodiment of the present application;

[0113] Figure 10 This is an example diagram of the checkpoint loading process in a large model training method provided in an embodiment of the present application;

[0114] Figure 11This is a flowchart of the checkpoint loading process in a large model training method provided in an embodiment of the present application;

[0115] Figure 12 This is a flowchart of a checkpoint loading process in a large model training method provided in an embodiment of the present application;

[0116] Figure 13 This is an example diagram of the checkpoint loading process in a large model training method provided in an embodiment of the present application;

[0117] Figure 14 This is a schematic diagram of the structure of a large model training device provided in an embodiment of the present application;

[0118] Figure 15 is a structural diagram of a computing device provided in an embodiment of the present application;

[0119] Figure 16 is a schematic diagram of a computing device cluster provided in an embodiment of the present application;

[0120] Figure 17 This is a schematic diagram of a possible implementation of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0121] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0122] In order to make the technical solution provided by this application clearer, before describing the technical solution provided by this application in detail, the relevant terms are first explained.

[0123] Data parallelism (DP): Data parallelism is the most common distributed training strategy. The key idea is to partition large-scale data into multiple subsets, each processed by a compute card. The results from these cards are then integrated through parameter passing and model synchronization. Specifically, each compute card obtains a portion of the entire batch of data and uses this data to determine the gradient for the first iteration of the large model training task. The gradients determined by each compute card are then averaged to obtain the average gradient, which is then used to update the model weights. Data parallelism comes in many forms. In the most basic form, such as distributed data parallelism (DDP) in PyTorch, each compute card stores a copy of all model weights, optimizer state, gradients, and the number of training cycles. In other words, the model on each compute card is exactly the same. In the first stage of the zero redundancy optimizer (ZeRO-1), a form of data parallelism, multiple compute cards participating in data parallelism each store a portion of the model weights. During the first iteration of forward training, each compute card obtains the model weights stored by other compute cards through parameter passing, and then determines the optimizer state for the portion of the data being processed.

[0124] Pipeline parallelism (PP): also known as model parallelism, in which a large model is divided into multiple parts, each part is placed on a different computing card, and then different computing cards perform training on these parts separately. This form of parallelism is usually used when a single computing card cannot accommodate the entire model. Pipeline parallelism can be regarded as vertical model parallelism. For example, if there is a single large model with 12 layers, the different layers of the large model will be placed on 3 different computing cards. When the latter computing card processes the previous batch of data, the previous computing card can process the next batch of data. In other words, the latter computing card does not need to wait for the previous computing card to complete the calculation. By overlapping the calculations of different batches of data, all computing cards are working at the same time, which improves the throughput of model training.

[0125] Tensor parallelism (TP): In tensor parallelism, the model weights of a large model are horizontally sliced, and each compute card only processes a portion of the model weights. Tensor parallelism is usually regarded as horizontal model parallelism. Specifically, each compute card processes the same batch of data, determines the activation value based on the model weights possessed by the compute card, and the compute cards exchange the data they need with each other, and then determine the gradient based on the model weights possessed by each compute card. Tensor parallelism involves different sharding methods. The most commonly used one is 1D sharding, which is to divide the model weights according to a certain dimension (slicing horizontally or vertically).

[0126] 3D parallel training: Simultaneously apply data parallelism, pipeline parallelism, and tensor parallelism to large model training.

[0127] 3D parallel training with ZeRO-1, applying ZeRO-1's data parallelism, pipeline parallelism, and tensor parallelism to large model training simultaneously.

[0128] Checkpoint (ckpt): Parameters such as model weights, optimizer state, gradients, and number of training cycles generated in one iteration of a large model training task.

[0129] The above introduces the relevant terms involved in this application, and the following introduces the implementation environment of the embodiments of this application.

[0130] Figure 1 This is a structural diagram of a large model training system provided by this application, such as Figure 1 As shown, the large model training system includes multiple computing nodes 101, control nodes 102 and storage devices 103, wherein each computing node 101 includes at least one computing card, a processor and memory, each computing card includes high-bandwidth memory (HBM), and a software development kit (SDK) is deployed on each computing card; a worker agent is deployed on each processor. A control module (controller) is deployed on the control node 102. The multiple computing nodes 101, control nodes 102 and storage devices 103 are connected to each other by wired or wireless means.

[0131] The computing node 101 can be a standalone physical server, a server cluster composed of multiple physical servers, a cloud server providing cloud services, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (CDN), and basic cloud computing services such as big data and artificial intelligence platform. The processor on the computing node 101 is, for example, a central processing unit (CPU), and the memory on the computing node 101 is, for example, a dynamic random access memory (DRAM). The processor of the computing node 101 is used to control the communication of the control node 102, write the checkpoint into the memory of the computing node 101, store the checkpoint in the memory of the computing node 101 into the storage device 103, and load the checkpoint in the storage device 103 to the computing card on the computing node 101. In some embodiments, the above functions of the processor of the computing node 101 are implemented by a worker agent.

[0132] The computing card on the computing node 101 is, for example, a CPU, a graphics processing unit (GPU), a network processor unit (NPU), a switching module processor unit (SMPU), a microprocessor, and further, for example, one or more integrated circuits for implementing the schemes of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Figure 1Only the example of a computing card being a GPU is used as an example. The embodiments of the present application do not limit the type of computing card. In some embodiments, the computing card is referred to as an accelerator. Exemplarily, the computing card is used to perform computational acceleration, and the SDK on the computing card is used to provide an interactive interface for the training process running on the computing card. The computing card includes an HBM, which is used to cache the model weights and optimizer states required by the computing card when performing training tasks, as well as to cache data generated during the training process.

[0133] Among them, the control node 102 can be a server, a cluster controller or a host in a computing device cluster. Among them, the large model training system may include one or more control nodes 102, and the embodiment of the present application does not limit the number of control nodes 102. In some embodiments, the control node 102 is acted by one or more computing nodes among multiple computing nodes 101. Exemplarily, the control node 102 is used to determine the checkpoint to be saved by the computing card, to determine the checkpoint to be loaded by the computing card, to determine the checkpoint to be persisted by the computing node 101, and to determine the timing of checkpoint saving, loading and persistence. In some embodiments, the above functions of the control node 102 are implemented by a control module (controller), wherein the controller includes a storage module (saving module), a loading module (loading module) and a persistence module (presistence module). It should be noted that, Figure 1 The division of the controller into multiple functional modules shown in FIG is merely exemplary, and the embodiments of the present application do not limit the division method of the controller.

[0134] The storage device 103 may be a solid state disk (SSD) or a hard disk drive (HDD), or a remote persistence storage device such as an object storage service (OBS), an elastic volume service (EVS), or a scalable file service (SFS). Figure 1 The storage device 103 is only taken as an example as a remote persistent storage device, and the embodiment of the present application does not limit the type of the storage device 103.

[0135] In some embodiments, the wireless network or wired network described above uses standard communication technologies and / or protocols. The network is typically a Transmission Control Protocol / Internet Protocol (TCP / IP) network in a data center network and an RDMA network, such as an RDMA over converged Ethernet (RoCE) network or an InfiniBand (IB) network, without limitation. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the above-mentioned data communication technologies.

[0136] An embodiment of the present application provides a large-model training method, which is applied to a large-model training system. In this method, the multiple computing cards in the large-model training system are grouped, utilizing the characteristic that multiple computing cards performing the same training task generate the same checkpoint. Multiple backups of the same checkpoint are formed in the memory of the computing node where a group of computing cards are located. In response to a training recovery instruction, computing cards in different computing card groups perform checkpoint communication to obtain a full set of checkpoints, and then perform training recovery based on the full set of checkpoints. Since the computing node where each computing card is located only stores a portion of the checkpoints and the amount of data is small, the computing card is more efficient in loading checkpoints when performing training recovery, and the training recovery efficiency is higher, which can reduce the waste of computing resources.

[0137] The above method can be applied to scenarios where data parallel mode is used to train large models in parallel, where scenarios where data parallel mode is used to train large models in parallel include data parallel scenarios, 3D parallel scenarios, or 3D parallel with ZeRO-1 scenarios, etc. The embodiments of this application do not limit the training scenarios where data parallel mode is used.

[0138] The following further explains the process of the aforementioned large model training method. This method includes checkpoint saving, checkpoint persistence, and checkpoint loading. The checkpoint saving process refers to the process by which the compute card stores the checkpoint to be saved from the HBM to the memory of the compute node where it resides; the checkpoint persistence process refers to the process by which the compute node stores the checkpoint to be persisted from the memory to a storage device; and the checkpoint loading process refers to the process by which the compute card loads the checkpoint from the compute node's memory to the HBM and completes the full checkpoint. Each of these three processes will be described below.

[0139] First, the checkpoint saving process is introduced. Figure 2 This is a flowchart of the checkpoint saving process in a large model training method provided in an embodiment of the present application, such as Figure 2 As shown, applied to the above-mentioned large model training system, the method includes the following steps 201 to 208.

[0140] Step 201: The control node determines a first computing card group and a second computing card group among the multiple computing cards based on the topological structure information and the preset backup number of the multiple computing cards in the large model training system. The first computing card group includes a first computing card and a second computing card, and the second computing card group includes a third computing card and a fourth computing card. The computing cards in the first computing card group and the second computing card group are used to perform the same training task. The topological structure information is used to indicate the parallel relationship between the multiple computing cards. The number of computing cards in the first computing card group and the second computing card group is equal to the preset backup number.

[0141] The parallel relationship between multiple computing cards includes at least one of data parallelism, pipeline parallelism, and tensor parallelism. For example, a large model includes three layers, where the training task of the first layer is performed by computing card AD, the training task of the second layer is performed by computing card EH, and the training task of the third layer is performed by computing card IL. Then, computing cards AD, EH, and IL are in a pipeline parallel relationship; the model weights of the first layer include T1 and T2, computing cards A and B perform the first part of the training task of the first layer based on T1, and computing cards C and D perform the second part of the training task of the first layer based on T2. ​​Then, computing cards A and B and computing cards C and D are in a tensor parallel relationship; the input data of computing card A is data 1, and the input data of computing card B is data 2. Then, computing cards A and B are in a data parallel relationship.

[0142] In which, the control node stores and maintains the topological structure information of multiple computing cards in the large model system. In some embodiments, in the initialization process of the large model training system, multiple computing cards in the large model training system respectively send the topological parameters of the computing card to the control node through the computing nodes where they are located. The topological parameters include the parallel mode in which the computing card is located and the number of the computing card in the parallel mode; the control node receives the topological parameters sent by the multiple computing cards and obtains the topological structure information of the multiple computing cards. In which, the AI ​​training framework of the large model training system generates the topological structure information of multiple computing cards, and configures the topological parameters to the multiple computing cards based on the topological structure information. The AI ​​training framework includes MindSpore, TensorFlow, Pytorch, etc., and the embodiments of the present application do not limit the AI ​​framework. In other embodiments, in the initialization process of the large model training system, the AI ​​training framework sends the topological structure information of multiple computing cards to the control node. The embodiments of the present application do not limit the way in which the control node obtains the topological structure information of multiple computing cards.

[0143] Among them, the preset backup number is set according to actual needs and is not limited in the embodiments of the present application. The computing cards in the first computing card group and the second computing card group perform the same training task, which means that in the same iteration process, the computing cards in the first computing card group and the second computing card group perform forward calculations based on the same model weights, that is, the parallel relationship between the computing cards in the first computing card group and the computing cards in the second computing card group is a data parallel relationship.

[0144] Wherein, the process of the control node determining the first computing card group and the second computing card group in the plurality of computing card groups includes: the control node determines the plurality of computing cards in the plurality of computing card groups in a data parallel relationship based on the topological structure information of the plurality of computing cards; the control node divides the plurality of computing cards in the data parallel relationship into a plurality of computing card groups based on the preset backup number, the plurality of computing card groups including the first computing card group and the second computing card group, the number of computing cards in the first computing card group and the second computing card group being equal to the preset backup number. It should be noted that, in order to simplify the description of this embodiment, in the embodiment of the present application, only two computing card groups and a preset backup number of 2 are used as an example for explanation, but it does not mean that the plurality of computing cards in the data parallel relationship can only be divided into two computing card groups, nor does it mean that each computing card group can only include 2 computing cards. The number of computing card groups divided is determined based on the number of computing cards in the data parallel relationship and the preset backup number, and the number of computing cards included in each computing card group is equal to the preset backup number, wherein the number of computing cards in the data parallel relationship is also the degree of data parallelism.

[0145] Wherein, the control node determines the computing cards included in each computing card group based on the positional relationship of the multiple computing nodes where the multiple computing cards are located. In some embodiments, the multiple computing cards included in a computing card group are located on different computing nodes. In other embodiments, the multiple computing cards included in a computing card group are located on different computing nodes and the multiple computing cards in the same computing card group are evenly distributed on computing nodes in different spaces. For example, the computing cards in a data parallel relationship are computing cards AD, which are respectively located on computing nodes C1-C4, wherein C1 and C2 are located in one rack, and C3 and C4 are located in another rack. Then the control node divides computing cards A and computing card C into one computing card group, and divides computing cards B and computing card D into another computing card group. In the above embodiment, when grouping computing cards, the spatial relationship between the computing nodes where the computing cards are located is taken into consideration, so that multiple computing cards in the same computing card group can be evenly distributed on computing nodes in different spaces, which is conducive to evenly distributing different copies of the same checkpoint on computing nodes in different spaces. Since the probability of multiple computing nodes in different spaces failing at the same time is low, the probability of multiple computing nodes where multiple computing cards in the same computing card group are located failing at the same time is low, and thus the probability of multiple copies of the same checkpoint being lost at the same time is small, which can improve the reliability of the copies.

[0146] It should be noted that the above step 201 is an initialization step. In some embodiments, the initialization step is performed once during the initialization process of the large model training system, and there is no need to perform the initialization step before each iteration.

[0147] It should be noted that the above step 201 is described using the example of the control node determining the first computing card group and the second computing card group. In some embodiments, the first computing node among the multiple computing nodes determines the first computing card group and the second computing card group, that is, the first computing node acts as the control node. In other embodiments, during the initialization process of the large model training system, the computing card group to which each computing card belongs is preset, that is, the first computing card group and the second computing card group are preset. The control node stores and maintains the grouping information of the multiple computing cards. Therefore, it is not necessary to determine the first computing card group and the second computing card group by executing the above step 201.

[0148] Step 202: The computing cards in the first computing card group and the second computing card group respectively execute the first iteration of the training task and generate the same checkpoint.

[0149] The first iteration is any iteration of the training task. The first and second computing cards in the first computing card group and the third and fourth computing cards in the second computing card group execute the first iteration of the training task based on the same model weights, and each generates the same checkpoint. The computing cards in the first and second computing card groups each store the generated checkpoint in the HBM of the computing card.

[0150] Step 203: If the first iteration satisfies the first frequency, the computing cards in the first computing card group and the second computing card group respectively send the metadata of the checkpoint to the control node through the computing nodes where they are located.

[0151] The first frequency is the frequency at which the large model training system saves checkpoints. The first frequency can be set according to actual needs, and the embodiment of the present application does not limit the value of the first frequency.

[0152] Among them, the metadata of the checkpoint includes the names of multiple parameters generated in the first iteration. In some embodiments, the metadata of the checkpoint also includes the data volume of multiple parameters. Taking the first computing card as an example, the process of the first computing card sending the metadata of the checkpoint to the control node includes: if the target frequency meets the first frequency, the SDK on the first computing card sends a checkpoint save request to the worker agent on the computing node where the first computing card is located. The checkpoint save request carries the metadata of the checkpoint generated by the first computing card in the first iteration; the work agent on the computing node where the first computing card is located forwards the checkpoint save request to the control node. It should be noted that the process of the second computing card, the third computing card and the fourth computing card sending the metadata of the checkpoint to the control node is the same as that of the first computing card and will not be repeated.

[0153] In some embodiments, taking the first computing card as an example, the control node receives a checkpoint save request from the first computing card, creates a checkpoint save record for the first computing card in the first iteration, and sets the save record to the created state. In some embodiments, the control node is configured with distributed coordination service software (Zookeeper, ZK), and the control node uses ZK to store and maintain the checkpoint save record for the first computing card in the first iteration, but this embodiment of the application is not limited to this. The same principles apply to other computing cards as the first computing card and will not be further described.

[0154] Step 204: The control node determines the first checkpoint and the second checkpoint in the checkpoint according to the metadata of the checkpoint.

[0155] Among them, when the multiple computing cards in data parallelism only include the first computing card group and the second computing card group, the first checkpoint and the second checkpoint constitute the checkpoint. When the multiple computing cards in data parallelism also include computing card groups other than the first computing card group and the second computing card group, the first checkpoint and the second checkpoint only constitute part of the checkpoint. That is, if the multiple computing cards in data parallelism include several computing card groups, the checkpoint is divided into several parts, one part for each computing card group.

[0156] In which, the control node determines the metadata of the first checkpoint and the metadata of the second checkpoint based on the metadata of the checkpoint, the metadata of the first checkpoint indicates the first checkpoint, and the metadata of the second checkpoint indicates the second checkpoint. In some embodiments, the process of the control node determining the first checkpoint and the second checkpoint in the checkpoint includes: the control node determines the average data volume based on the data volume of each parameter in the checkpoint and the number of computing card groups; the control node divides the checkpoint into multiple parts based on the average data volume, and the difference between the data volume of each checkpoint and the average data volume is less than a preset value. In the above embodiment, dividing the checkpoint into multiple parts based on the average data volume can ensure that the data volume of the multiple checkpoints is uniform, so that the storage tasks of the checkpoints can be evenly distributed to different computing card groups, which is conducive to unifying the time consumed by different computing card groups to store the checkpoints they are responsible for.

[0157] In some embodiments, the process of the control node dividing the checkpoint into multiple parts based on the average data volume includes: the control node sorts the parameters in the checkpoint according to the size of the data volume, and traverses the parameters in the checkpoint in order from small to large data volume. Each time a parameter is traversed, the parameter is divided into a parameter group. If the data volume of the parameter group reaches the average data volume, the next traversed parameter is divided into the next parameter group, and so on, to obtain multiple parameter groups. A parameter group is also a checkpoint. This division process is to divide the checkpoint using a greedy algorithm. In the above embodiment, a greedy algorithm is used to divide the checkpoint into multiple parts. The data volume of the parameters in the same checkpoint is similar, that is, the size of the parameters in the same checkpoint is relatively uniform, and it is more efficient to store them in the memory of the same computing node later. The following is through Figure 3 Let’s take an example to illustrate the above process. Figure 3 This is a schematic diagram of the checkpoint division process in a large model training method provided in an embodiment of the present application, such as Figure 3 As shown in the upper-middle figure, this checkpoint includes multiple parameters. The control node sorts the parameters in ascending order of data volume. The control node traverses the parameters in sequence, assigning parameters 1 through 3 to the first group of checkpoints. After parameter 3 is assigned to the first group of checkpoints, the data volume of the first group of checkpoints reaches the average data volume. The control node then traverses parameter 4 and assigns it to the second group of checkpoints, and so on.

[0158] It should be noted that the above steps 203 and 204 are described by taking the computing cards in the first computing card group and the second computing card group as an example of a normal data parallel relationship. In some embodiments, the computing cards in the first computing card group and the second computing card group are in a data parallel relationship in the form of ZeRO-1, that is, the computing cards in the first computing card group and the second computing card group respectively store 1 / 4 of the model weights of the same training task. When any computing card performs the forward calculation of the first iteration, it needs to obtain the model weights stored in other computing cards to obtain the complete model weights of the training task; under this data parallel relationship , the computing cards in the same computing card group respectively obtain the model weights stored on other computing cards in the same computing card group, determine the optimizer state of the computing card based on the input data of the computing card in the first iteration and the model weights stored by the computing card, generate the optimizer state of other computing cards in the same computing card group in the first iteration based on the input data of the computing card in the first iteration and the obtained model weights, and store the obtained model weights and the generated optimizer state in the memory of the computing node where they are located, thereby forming multiple copies of the same model weights and multiple copies of the same optimizer state in the computing card group. In the above embodiment, under the data parallel relationship of the ZeRO-1 form, the computing card will obtain the model weights during the forward calculation of the first iteration and store them in the memory of the computing node where it is located, instead of discarding the model weights after the forward calculation is completed. This can form multiple copies of the same model weights in the same computing card group, and the forward calculation and copy storage overlap in time. Therefore, it can avoid the interference of copy storage on the training process, improving the efficiency of copy storage and model training.

[0159] In some embodiments, taking the first computing card as an example, the control node adds metadata of the first checkpoint to the checkpoint save record of the first computing card in the first iteration. The other computing cards are similar to the first computing card and will not be described in detail.

[0160] It should be noted that the embodiment of this application only takes the first computing card group and the second computing card group as an example to illustrate, and the checkpoint is divided into the first checkpoint and the second checkpoint. The case of more than two computing card groups is the same as the case of two computing card groups, and will not be repeated.

[0161] It should be noted that the above steps 203 and 204 are described using the example of the control node determining the checkpoint to be saved for each computing card. In some embodiments, the control node is the first computing node among multiple computing nodes, and the first computing node determines the checkpoint to be saved for each computing card. In other embodiments, the checkpoint to be saved for each computing card is determined by each computing card itself. For example, in some embodiments, each computing card stores and maintains topological structure information of multiple computing cards, and each computing card determines the checkpoint to be saved for the computing card based on the topological structure information and the generated checkpoint. The embodiments of this application do not limit who performs the checkpoint division.

[0162] Step 205: The control node sends a first save instruction to the first computing card. The first computing card receives the first save instruction and stores the first checkpoint in the memory of the computing node where the first computing card is located.

[0163] Among them, the first save instruction is used to instruct the first computing card to store the first checkpoint in the memory of the computing node where the first computing card is located. The first save instruction carries the metadata of the first checkpoint. The first computing card receives the first save instruction, and based on the metadata of the first checkpoint carried by the first save instruction, stores the first checkpoint of the checkpoint in the HBM of the first computing card in the memory of the computing node where it is located; the first computing card sends a save end message for the first checkpoint to the control node through the computing node where it is located, and clears other data in the checkpoint in the HBM. The second computing card is similar to the first computing card and will not be repeated here.

[0164] In some embodiments, the first computing card writes the first checkpoint in batches to the memory of the computing node where it is located through the SDK. In other embodiments, the first computing card writes the first checkpoint to the memory of the computing node where it is located at one time through the SDK. The embodiments of the present application are not limited to this.

[0165] In some embodiments, a first memory address in the memory of the compute node where the first computing card resides is used to store checkpoints saved by the computing card on the compute node. Specifically, a memory space is allocated from the compute node's memory to store checkpoints. In some embodiments, this first memory address may be referred to as the compute node's memory pool, which is not limited in this embodiment of the present application.

[0166] In some embodiments, the control node receives a save completion message from the first computing card for the first checkpoint, and changes the state of the first computing card's checkpoint save record for the first iteration from a created state to a saved state. In other embodiments, the control node stores and maintains a record of the number of checkpoints currently saved by the first computing card.

[0167] It should be noted that the above step 205 is an implementation method for the first computing card to store the first checkpoint in the memory of the computing node where the first computing card is located. In some embodiments, this step is also implemented based on other methods, and the embodiments of the present application are not limited to this.

[0168] Step 206: The control node sends a second save instruction to the second computing card. The second computing card receives the second save instruction and stores the first checkpoint in the memory of the computing node where the second computing card is located.

[0169] Step 207: The control node sends a third save instruction to the third computing card. The third computing card receives the third save instruction and stores the second checkpoint in the memory of the computing node where the third computing card is located.

[0170] Step 208: The control node sends a fourth save instruction to the fourth computing card. The fourth computing card receives the fourth save instruction and stores the second checkpoint in the memory of the computing node where the fourth computing card is located.

[0171] The above steps 206 to 208 are similar to the above step 205 and are not described in detail.

[0172] It should be noted that the embodiment of the present application does not limit the execution order of the above steps 205 to 208.

[0173] Below through Figure 3 and Figure 4 The process shown in steps 205 to 208 is described with an example. Figure 3 As shown in the figure below, the first computing card group includes two computing cards, located on computing nodes 1 and 2 respectively. The first set of checkpoints is stored in the memory of computing node 1, and the first set of checkpoints is stored in the memory of computing node 2, forming two copies of the first set of checkpoints. The second computing card group includes two computing cards, located on computing nodes 3 and 4 respectively. The second set of checkpoints is stored in the memory of computing node 3, and the second set of checkpoints is stored in the memory of computing node 4, forming two copies of the second set of checkpoints. The same applies to other computing card groups and is not further described. Figure 4 This is a flow chart of the checkpoint saving process in a large model training method provided in an embodiment of the present application. Figure 4 As shown, there is a data parallel relationship between multiple computing nodes. The computing card in each computing node saves the generated multiple parameters in the memory of the computing node, thus forming multiple copies of the same checkpoint. Figure 4 The example in which the computing card is a GPU is used for illustration, and the embodiment of the present application does not limit the type of computing card.

[0174] In the above steps 205 to 208, the characteristic that multiple computing cards performing the same training task generate the same checkpoint is utilized to group the multiple computing cards in the large model training system. Each group of computing cards saves a portion of the full checkpoint to the memory of the computing node where it is located. Among them, multiple computing cards belonging to the same group save the same part of the full checkpoint, thereby forming multiple backups of the same checkpoint in the memory of the computing node where a group of computing cards are located. Therefore, there is no need to save the full checkpoint across multiple computing nodes, which avoids the full copy of the checkpoint and the interference of the checkpoint saving process on the training process, thereby improving the checkpoint saving efficiency and the model training efficiency.

[0175] In some embodiments, if the computing nodes where the computing cards in the first computing card group and the second computing card group are located are insufficient in memory, the control node sends a data deletion request for the second iteration to the computing nodes where the computing cards in the first computing card group and the second computing card group are located, respectively. The second iteration is the first iteration in which the generated checkpoint is stored in the computing node memory and the iteration order is ranked first; the computing nodes where the computing cards in the first computing card group and the second computing card group are located receive the data deletion request respectively, and delete the checkpoint generated in the second iteration from the memory. Taking the first computing card as an example, the first computing card determines the available capacity of the computing node memory where the first computing card is located through the SDK, and the first computing card reports the available capacity to the control node. The same is true for other computing cards, which will not be repeated. Insufficient memory of the computing node means that the memory capacity of the computing node is less than a preset threshold, or the available capacity of the computing node memory is less than the average amount of data to be saved for each computing card checkpoint. The value of the first number can be set according to actual needs, and the embodiments of the present application do not limit this. In some embodiments, the first number is less than the total number of iterations of checkpoints stored in the memory of the computing node, thereby ensuring that after the checkpoints of the first number of iterations are deleted from the memory, there are checkpoints generated by iterations with a closer iteration order in the memory, thereby ensuring that when training is restored, the checkpoints generated by iterations with a closer iteration order can be loaded from the memory, thereby reducing the waste of computing resources. The following example illustrates this embodiment. The computing nodes where the computing cards in the first computing card group and the second computing card group are located store checkpoints generated by the 100th, 200th, 300th, 400th and 500th iterations. The first number is 2, then the second iteration includes the 100th and 200th iterations. If the computing nodes where the computing cards in the first computing card group and the second computing card group are located have insufficient memory, then the checkpoints generated by the 100th and 200th iterations are deleted from the memory of the computing node.

[0176] Below through Figure 5 The process shown in the above steps 201 to 208 is described with an example. Figure 5 This is a flowchart of a checkpoint saving process during a large model training process provided by an embodiment of the present application. Figure 5 In the example, mindspore, modelcheckpoint, and serialization_py are modules in the AI ​​training framework MindSpore, and mempoolclicent is the memory pool module on the computing node. Figure 5As shown in the figure, during training, the AI ​​training framework MindSpore saves checkpoints at a user-specified frequency. When the AI ​​training framework issues a save_checkpoint request, the SDK notifies the control node through the worker agent on the compute node to start the save task. The SDK then calls the memory pool API to create and open a namespace. After receiving the save_checkpoint request, the controller on the control node verifies the input parameters, constructs a checkpoint object for this single card, and saves it. The SDK then calculates the actual checkpoint content for each card and saves it to the memory pool based on the redundancy strategy. This redundancy strategy in the SDK is based on data parallelism and does not require cross-network processing. Each card directly writes the checkpoint content based on the redundancy strategy to DRAM, achieving multiple replicas. After calculating and saving the redundancy strategy, the SDK continuously loops and directly saves the required checkpoints to the memory pool. After saving, the SDK notifies the control node of the save result through the worker agent. Upon receiving the checkpoint save completion request, the control node refreshes the checkpoint object status and the overall checkpoint save status for this iteration. When the checkpoints in the memory pool are almost full, the control node determines whether the checkpoints need to be deleted. The worker agent calls the delete namespace interface of the memory pool based on the namespace information of the checkpoints that may need to be removed, removes the checkpoints that need to be removed, and finally returns the removal results to the control node.

[0177] In the above method, the characteristic that multiple computing cards performing the same training task generate the same checkpoint is utilized to group the multiple computing cards in the large model training system. Multiple backups of the same checkpoint are formed in the memory of the computing node where a group of computing cards are located. In response to the training recovery instruction, the computing cards in different computing card groups communicate with each other on the checkpoint to obtain the full checkpoint, and then perform training recovery based on the full checkpoint. Since the computing node where each computing card is located only stores part of the checkpoint and the data volume is small, the computing card is more efficient in loading checkpoints and the training recovery efficiency is higher during training recovery, which can reduce the waste of computing resources. Furthermore, when grouping the computing cards, the spatial relationship between the nodes where the computing cards are located is taken into consideration, and multiple computing cards in the same computing card group are evenly distributed on computing nodes in different spaces, which is conducive to evenly distributing different copies of a checkpoint on computing nodes in different spaces. Since the probability of multiple computing nodes in different spaces failing at the same time is low, the probability of multiple computing nodes where multiple computing cards in the same computing card group fail at the same time is low, and thus the probability of multiple copies being lost at the same time is low, which can improve the reliability of the copies. In addition, based on the average data volume, the checkpoint is divided into multiple copies, which can ensure that the data volume of the multiple checkpoints is uniform, so that the checkpoint storage tasks can be evenly distributed to different computing card groups, which is conducive to unifying the time spent by different computing card groups to store the checkpoints they are responsible for.

[0178] The following describes the checkpoint persistence process. Figure 6 This is a flowchart of the checkpoint persistence process in a large model training method provided in an embodiment of the present application, such as Figure 6 As shown, it is applied to the above-mentioned large model training system, which includes multiple computing cards, and the multiple computing cards are located on multiple computing nodes. The method includes the following steps 601 to 604.

[0179] Step 601: If the first iteration meets the persistence condition, the control node sends a data persistence request for the first iteration to the computing nodes where the computing cards in the first computing card group and the second computing card group are located, respectively. The data persistence condition indicates that the first iteration meets the second frequency, the iteration order of the first iteration is a preset order, or the checkpoint generated by the first iteration is currently persistent.

[0180] The second frequency is the frequency at which the large model training system persists checkpoints. The second frequency can be set according to actual needs. For example, checkpoint persistence is performed every 200 iterations, or checkpoint persistence is performed every 10 times the checkpoint is saved to memory. The embodiment of the present application does not limit the value of the first frequency. The iteration order of the first iteration is a preset order, that is, the iteration order of the first iteration is a specified order.

[0181] Among them, the checkpoint generated by the first iteration is currently persistent, which means that after the computing node completes the most recent checkpoint persistence operation to the storage device, the computing card is executing the first iteration, that is, the first iteration is the most recent iteration, and the computing node persists the checkpoint generated by the first iteration. This persistence condition can be understood as the computing node trying to persist the checkpoint generated by the most recent iteration. Figure 7 Let's take an example to illustrate the persistence condition. Figure 7 This is a schematic diagram of the checkpoint persistence process in a large model training method provided in an embodiment of the present application, such as Figure 7 As shown, the compute node's memory has saved checkpoints for three iterations, V1, V2, and V3. The compute node is writing V1 from memory to the storage device (OBS). The AI ​​process on the compute card is writing the checkpoint of V4 to memory. In other words, V4 is the most recent iteration. After the compute node finishes writing V1 to OBS, it writes V4 from memory to OBS.

[0182] It should be noted that the persistence condition shown in step 601 is only exemplary. Based on step 601, those skilled in the art can easily think of other persistence conditions. The embodiment of the present application does not limit the persistence conditions.

[0183] In some embodiments, the data persistence request carries the identifier of the first iteration to indicate that the checkpoint generated by the first iteration is to be persisted. In some embodiments, the control node sends a first data persistence request to the first computing card, and the first data persistence request carries the metadata of the first checkpoint; the control node sends a second data persistence request to the second computing card, and the second data persistence request carries the metadata of the first checkpoint; the control node sends a third data persistence request to the third computing card, and the third data persistence request carries the metadata of the second checkpoint; the control node sends a fourth data persistence request to the fourth computing card, and the fourth data persistence request carries the metadata of the second checkpoint. The embodiments of the present application do not limit data persistence requests.

[0184] In the above step 601, when the persistence condition is met in the first iteration, the control node sends a data persistence request to the computing node where the computing card is located to persist the checkpoint generated by the first iteration, that is, the checkpoint is selectively persisted. Since persisting the checkpoint in the memory to the storage device will to a certain extent hinder the checkpoint from being written from the HBM on the computing card to the memory, selective checkpoint persistence can improve the reliability of checkpoint storage while reducing the pressure on the storage device and reducing the obstruction of writing the checkpoint to the memory.

[0185] Step 602: The computing nodes where the computing cards in the first computing card group and the second computing card group are located respectively receive the data persistence request.

[0186] Step 603: The computing nodes where the first computing card and the second computing card are located respectively store the first checkpoint from the memory to the storage device.

[0187] Taking the first computing card as an example, the memory of the computing node where the first computing card resides includes a first buffer and a second buffer. The first computing card stores a first checkpoint in the first buffer and a third checkpoint in the second buffer. The third checkpoint is a partial checkpoint generated by computing cards in the first and second computing card groups in the third iteration, which is an iteration after the first iteration. The process by which the computing node where the first computing card resides stores the first checkpoint from memory to a storage device includes: the computing node where the first computing card resides stores the first checkpoint from the first buffer to the storage device simultaneously with the first computing card storing the third checkpoint in the second buffer. The same process applies to the second computing card and will not be further described. In the above process, two buffers in memory are used to receive data written by the computing card to memory. When a computing card writes data to one buffer, the computing node where the computing card resides can persist data in the other buffer to the storage device. The two buffers continuously exchange positions as training progresses, preventing the checkpoint persistence process from blocking the checkpoint saving process.

[0188] In some embodiments, taking the first computing card as an example, the memory of the computing node where the first computing card is located also includes a third buffer. If, when the first computing card stores the fourth checkpoint to the first buffer, there is a first checkpoint in the first buffer that has not been stored to the storage device, the computing node where the first computing card is located writes the first checkpoint in the first buffer from the first buffer to the third buffer; while the first computing card stores the fourth checkpoint to the first buffer, it stores the first checkpoint in the third buffer to the storage device. The fourth checkpoint is a partial checkpoint generated by the computing cards in the first computing card group and the second computing card group in the fourth iteration, and the fourth iteration is the iteration after the third iteration. After the persistence of the first checkpoint is completed, the computing node where the first computing card is located releases the third buffer. The second computing card is similar to the first computing card and will not be described in detail. In the above embodiment, checkpoints are stored in the first buffer and the second buffer of the memory, the persistence of the checkpoint in the first buffer has not been completed, and when the computing card needs to write the latest checkpoint to the memory, the computing node writes the checkpoint that has not been persisted in the first buffer to a temporary buffer, and then can write the latest checkpoint to the first buffer. At the same time as the computing card writes the latest checkpoint to the first buffer, the computing node writes the checkpoint that has not been persisted from the temporary buffer to the storage device. Since the efficiency of data copying between buffers is high, the blocking of the checkpoint persistence process on the checkpoint saving process can be reduced.

[0189] Below through Figure 8 The following example illustrates the above checkpoint persistence process. Figure 8 This is an example diagram of the checkpoint persistence process in a large model training method provided in an embodiment of the present application, such as Figure 8As shown, the memory of the computing node includes bufferA (equivalent to the first buffer), bufferB (equivalent to the second buffer) and bufferC (equivalent to the third buffer), the GPU writes ckpti-1 (equivalent to the first checkpoint) into bufferA and ckpti (equivalent to the third checkpoint) into bufferB; while ckpti is written into bufferB, ckpti-1 is written from bufferA into OBS; the GPU writes ckpti+1 (equivalent to the fourth checkpoint) into bufferA, while the first checkpoint in bufferA is not yet completed, the computing node writes the first checkpoint in bufferA which is not yet completed into bufferC to free bufferA for ckpti+1; the GPU writes ckpti+1 into bufferA, while the computing node writes the first checkpoint in bufferC into OBS.

[0190] In some embodiments, taking the first computing card as an example, after the computing node where the first computing card is located completes the persistence of the first checkpoint, the computing node returns a persistence end message for the first checkpoint to the control node, and the control node receives the persistence end message and modifies the state of the first computing card in the checkpoint saving record of the first iteration from having been saved to memory to having been persisted. The second computing card is the same as the first computing card, and thus will not be described again.

[0191] In step 604, the computing node where the third computing card and the fourth computing card are located respectively stores the second checkpoint from the memory to the storage device.

[0192] The step 604 is the same as the step 603 described above, and thus will not be described again.

[0193] It should be noted that the embodiments of the present application do not limit the execution order of the step 603 and the step 604.

[0194] In the above method, multiple compute cards in a large model training system are grouped, leveraging the fact that multiple compute cards executing the same training task generate the same checkpoint. Multiple copies of the same checkpoint are then formed in the memory of the compute nodes containing a group of compute cards. In response to a training recovery instruction, the compute cards in different groups communicate to obtain the full checkpoint, and then perform training recovery based on the full checkpoint. Because each compute card's compute node stores a partial checkpoint, the data volume is relatively small. This allows for efficient checkpoint loading and recovery during training recovery, reducing the waste of computing resources. Furthermore, checkpoint persistence is selectively performed. Because persisting in-memory checkpoints to storage devices can hinder the writing of checkpoints from the HBM on the compute card to memory, selective checkpoint persistence improves checkpoint storage reliability while reducing pressure on the storage device and blocking checkpoint writes. Furthermore, the checkpoints persisted by each compute node are partial checkpoints determined based on the topological structure between the compute cards, rather than full checkpoints, improving the efficiency of checkpoint persistence.

[0195] The following describes the checkpoint loading process. This process can be applied to checkpoint loading scenarios that don't involve compute node failures, such as when a user suspends and then restarts training of a large model. It can also be applied to checkpoint loading scenarios that involve compute node failures, such as when one or more compute nodes in a large model training system fail. The following describes the checkpoint loading process using these two scenarios as examples.

[0196] First, the checkpoint loading process is introduced, taking the checkpoint loading scenario without computing node failure as an example. Figure 9 This is a flowchart of the checkpoint loading process in a large model training method provided in an embodiment of the present application, such as Figure 9 As shown, it is applied to the above-mentioned large model training system, which includes multiple computing cards, and the multiple computing cards are located on multiple computing nodes. The method includes the following steps 901 to 905.

[0197] In step 901, the computing cards in the first computing card group and the second computing card group respectively send checkpoint query requests to the control node in response to a training recovery instruction, the training recovery instruction indicating to perform training recovery according to a first iteration, and the checkpoint query request being used to query checkpoint saving information of the first iteration.

[0198] The training recovery instruction is used to indicate to perform training recovery according to a specified iteration, and the first iteration is taken as an example in step 901, and embodiments of the present application do not limit the specified iteration. In some embodiments, by default, training recovery is performed according to the latest iteration in which checkpoint is saved, without specifying which iteration to perform training recovery.

[0199] The checkpoint query request carries a query condition, the query condition including a user identifier (taskID), a computing card identifier (rankID), and a training task name (taskName), and optionally, the query condition further includes an iteration order (globalStepNum) to specify which iteration to perform training recovery.

[0200] In step 902, the control node receives the checkpoint query requests sent by the computing cards, respectively determines saving information of checkpoint required by each computing card, sends a first data loading instruction to the computing cards in the first computing card group, and sends a second data loading instruction to the computing cards in the second computing card group.

[0201] The checkpoint required by the computing card includes part of the checkpoint saved by the computing card and part of the checkpoint saved by the computing cards in the computing card group that does not include the computing card, that is, the full amount of checkpoint. The saving information of the checkpoint required by the computing card includes which computing cards save each part of the checkpoint required by the computing card and storage locations of each part of the checkpoint. The storage location of the checkpoint is used to indicate whether the checkpoint is located in the memory of the computing node or in the storage device.

[0202] In some embodiments, using the first computing card as an example, a control node receives a checkpoint query request from the computing card and queries the ZK on the control node to determine whether a first checkpoint required by the first computing card is located in the memory of the computing node where the first computing card resides, and whether a second checkpoint required by the first computing card is located in the memory of the computing node where a computing card in the second computing card group resides. The control node then sends a first data loading instruction to the first computing card, which carries the saved checkpoint information required by the first computing card. The same procedure applies to other computing cards and is not further described.

[0203] Step 903: In response to the first data loading instruction, a computing card in the first computing card group loads the first checkpoint of the first iteration from the memory of the computing node where the computing card in the first computing card group is located, sends the first checkpoint to the computing card in the second computing card group through the communication link between the multiple computing cards, and receives the second checkpoint sent by the computing card in the second computing card group.

[0204] Taking the first computing card as an example, the process of the first computing card loading the first checkpoint from the memory of the computing node where it is located includes: if the checkpoint saving information carried by the first data loading instruction indicates that the first checkpoint is saved in the memory of the computing node where the first computing card is located, then the first computing card loads the first checkpoint from the memory of the computing node to the HBM of the first computing card through the communication network between the computing node memory and HBM; if the checkpoint saving information carried by the first data loading instruction indicates that the first checkpoint is saved in the storage device, then the first computing card loads the first checkpoint to the memory of the computing node where the first computing card is located through the computing node where the first computing card is located, and the first computing card loads the first checkpoint from the memory of the computing node to the HBM of the first computing card through the communication link between the computing node memory and HBM. The second computing card is similar to the first computing card and will not be repeated.

[0205] Among them, the computing cards in the first computing card group and the computing cards in the second computing card group perform checkpoint communication in an all-reduce manner so that each computing card obtains the required full checkpoint. In some embodiments, the communication process can be implemented by an all-reduce operator. In some embodiments, when performing checkpoint communication in an all-reduce manner, the destination computing card of the communicated checkpoint is determined according to a strategy of giving priority to adjacent computing nodes. For example, if the computing node where the first computing card is located is closer to the computing node where the third computing card is located, and farther away from the computing node where the fourth computing card is located, then the first computing card sends the first checkpoint to the third computing card and receives the second checkpoint sent by the third computing card. Since the data transmission efficiency between computing cards that are closer is higher, it is beneficial to improve the checkpoint loading efficiency, thereby improving the training recovery efficiency.

[0206] Below through Figure 10 The process shown in the above step 903 is described with an example. Figure 10 This is an example diagram of the checkpoint loading process in a large model training method provided in an embodiment of the present application, such as Figure 10 As shown, in step 01, a computing card in computing card group 3 loads checkpoint 1 from the memory of the computing node where it is located to the HBM of the computing card; in step 02, the computing card sends the checkpoint 1 to the computing cards in other computing card groups through the communication link between the computing cards.

[0207] The above-mentioned step 903 is an implementation method in which, in response to the training resumption instruction, the computing card in the first computing card group loads the first checkpoint from the memory of the computing node where it is located, sends the first checkpoint to the computing card in the second computing card group, and receives the second checkpoint sent by the computing card in the second computing card group. In some embodiments, this step is also implemented based on other methods, which is not limited in the embodiments of the present application.

[0208] Step 904: The computing cards in the second computing card group respond to the second data loading instruction, load the second checkpoint of the first iteration from the memory of the computing node where the computing cards in the second computing card group are located, and send the second checkpoint to the computing cards in the first computing card group through the communication link between the multiple computing cards.

[0209] Step 904 is similar to step 903 and will not be described in detail.

[0210] Step 905: The computing cards in the first computing card group and the computing cards in the second computing card group respectively restore the training task based on the first checkpoint and the second checkpoint.

[0211] In some embodiments, using the first computing card as an example, the first computing card stores the checkpoints obtained from the computing cards in the second computing card group in the memory of the computing node where the first computing card is located, so that the memory of the computing node stores the full set of checkpoints. The same applies to the other computing cards. Each computing card reads the full set of checkpoints from the memory of the computing node where it is located and executes the training task based on the full set of checkpoints, thereby achieving recovery of the training task.

[0212] It should be noted that after the training task is resumed, the large model training system will continue as before. Figure 2 and Figure 6 The process shown here performs checkpoint preservation and persistence, which will not be described in detail.

[0213] In the above method, multiple computing cards in a large model training system are grouped, taking advantage of the fact that multiple computing cards performing the same training task generate the same checkpoint. Multiple copies of the same checkpoint are then formed in the memory of the computing node where a group of computing cards reside. In response to a training recovery instruction, computing cards in different computing card groups communicate to obtain the full checkpoint, and then perform training recovery based on the full checkpoint. Because each computing card's computing node stores a partial checkpoint and the data volume is relatively small, the computing card can load the checkpoint from memory more efficiently during training recovery, resulting in higher training recovery efficiency and reduced waste of computing resources. Furthermore, based on the topological structure information between multiple computing cards, computing cards storing copies of the same checkpoint are identified. The computing node where the computing card resides first reads the checkpoint from local memory, and then uses the topological structure information and parameter-plane communication between computing cards to complete the checkpoint. Compared to the traditional method of obtaining the full checkpoint through cross-node communication, this method utilizes high-speed inter-card bandwidth to obtain checkpoints, increasing the efficiency of training recovery by more than five times.

[0214] The following describes the checkpoint loading process, taking the checkpoint loading scenario involving a compute node failure as an example. Figure 11 This is a flowchart of the checkpoint loading process in a large model training method provided in an embodiment of the present application, such as Figure 11As shown, it is applied to the above-mentioned large model training system, which includes multiple computing cards, and the multiple computing cards are located on multiple computing nodes. The method includes the following steps 1101 to 1107.

[0215] Step 1101: When the computing node where the second computing card is located fails, in response to a training recovery instruction, the first computing node obtains a first checkpoint from the memory of the computing node where the first computing card is located. The first computing node is used to replace the computing node where the second computing card is located.

[0216] It should be noted that the above step 1101 is described by taking the failure of the computing node where the second computing card is located as an example. In some embodiments, there are two or more computing node failures. The embodiment of the present application only takes the failure of one computing node as an example for description. The situation of multiple computing node failures is the same as the situation of one computing node failure. When a failure occurs, as long as the computing nodes where all computing cards in the same computing card group are located do not fail at the same time, the checkpoint in the memory can be restored by memory broadcast. Specifically, since the embodiment of the present application takes into account the spatial distance between different computing nodes when dividing the computing card group, the computing nodes where multiple computing cards in a computing card group are located are close to each other and are far away. When a computing node fails, the pull preference is selected according to the distance, and the checkpoint copy is pulled from the computing node closest to the group first. When the node closest to the group cannot be obtained, it is pulled from the computing node farther away.

[0217] Step 1102: The first computing node stores the first checkpoint in the memory of the first computing node.

[0218] Step 1103: The computing cards in the first computing card group and the second computing card group respectively send checkpoint query requests to the control node in response to the training recovery instruction. The training recovery instruction instructs to resume training according to the first iteration, and the checkpoint query request is used to query the checkpoint storage information of the first iteration.

[0219] Step 1104: The control node receives the checkpoint query request sent by each computing card, determines the checkpoint storage information required by each computing card, sends a first data loading instruction to the computing card in the first computing card group, and sends a second data loading instruction to the computing card in the second computing card group.

[0220] Step 1105: The computing cards in the first computing card group respond to the first data loading instruction, load the first checkpoint of the first iteration from the memory of the computing node where the computing cards in the first computing card group are located, send the first checkpoint to the computing cards in the second computing card group through the communication link between the multiple computing cards, and receive the second checkpoint sent by the computing cards in the second computing card group.

[0221] Step 1106: The computing cards in the second computing card group respond to the second data loading instruction, load the second checkpoint of the first iteration from the memory of the computing node where the computing cards in the second computing card group are located, and send the second checkpoint to the computing cards in the first computing card group through the communication link between multiple computing cards.

[0222] Step 1107: The computing cards in the first computing card group and the computing cards in the second computing card group restore the training task based on the first checkpoint and the second checkpoint, respectively.

[0223] The above steps 1103 to 1107 are similar to the above steps 901 to 905. The difference is that in steps 1103 to 1107, the computing node where the second computing card in steps 901 and 905 is located is replaced by the first computing node, and the second computing card in steps 901 and 905 is replaced by the computing card on the first computing node. The similarities are not repeated here.

[0224] The process shown in steps 1101 to 1107 can be found in Figure 12 and Figure 13 . Figure 12 This is a flowchart of the checkpoint loading process in a large model training method provided in an embodiment of the present application. Figure 13 This is an example diagram of the checkpoint loading process in a large model training method provided in an embodiment of the present application.

[0225] In the above method, the multiple computing cards in the large model training system are grouped, taking advantage of the fact that multiple computing cards performing the same training task generate the same checkpoint. Multiple backups of the same checkpoint are formed in the memory of the computing node where one computing card is located. In response to a training recovery instruction, computing cards in different computing card groups communicate to obtain the full checkpoint, and then perform training recovery based on the full checkpoint. Because the computing node where each computing card is located stores a portion of the checkpoint, the data volume is relatively small. During training recovery, the computing card can load checkpoints more efficiently, which improves training recovery efficiency and reduces the waste of computing resources. Furthermore, when a failure occurs, as long as the computing nodes where all computing cards in the same computing card group are located do not fail simultaneously, the checkpoints in memory can be recovered through memory broadcast. Specifically, since the embodiment of the present application takes into account the spatial distance between different computing nodes when dividing the computing card groups, the computing nodes where multiple computing cards are located in a computing card group are either close or far away. When a computing node fails, the pulling preference is selected according to the distance, and the checkpoint copy is pulled from the computing node closest to the group first. When the nearest node cannot be obtained, it is pulled from a farther computing node. This can improve communication efficiency and thus improve the efficiency of recovery training.

[0226] It should be noted that the above Figure 2 、 Figure 6 、 Figure 9 and Figure 11 The embodiments shown are all described by taking 4 computing cards to perform the same training task as an example. In some embodiments, the number of computing cards performing the same training task can be greater. The embodiments of the present application do not limit the number of computing cards performing the same training task. In addition, the above Figure 2 、 Figure 6 、 Figure 9 and Figure 11 The embodiments shown are all described by taking two computing card groups and each computing card group including two computing cards as an example. In one embodiment, the large model training system includes more than two computing card groups, and each computing card group includes two or more computing cards. The embodiment of the present application does not limit the number and scale of computing card groups, as long as the number of computing card groups is greater than or equal to 2 and the number of computing cards in the computing card group is greater than or equal to 2; in addition, Figure 2 、 Figure 6 、 Figure 9 and Figure 11The embodiments shown are all explained by taking the example that the first computing card in the first computing card group stores the first checkpoint, the second computing card in the second computing card group stores the second checkpoint, and the first checkpoint and the second checkpoint constitute a complete checkpoint. In some embodiments, the large model training system includes other computing card groups in addition to the first computing card group and the second computing card group. Then, the checkpoints stored by each computing card group constitute a complete checkpoint. In summary, the embodiments of the present application can be expanded from a scenario with two computing card groups, each of which includes two computing cards, to a scenario with more than two computing card groups, each of which includes two or more computing cards. The checkpoint saving method, checkpoint persistence method, and checkpoint loading method in this scenario are the same as those in the previous embodiment. Figure 2 、 Figure 6 、 Figure 9 and Figure 11 The same applies to the embodiments shown, and no further details will be given.

[0227] Figure 14 2. It is a structural schematic diagram of a large model training device provided by an embodiment of the present application, wherein the device is applied to a large model training system, wherein the large model training system includes multiple computing cards, wherein the multiple computing cards are located on multiple computing nodes, wherein the computing nodes include memory, and the multiple computing cards include a first computing card group and a second computing card group. The multiple computing card groups are used to perform the same training task. In the first iteration of the training task, the computing cards in the first computing card group and the second computing card group respectively generate the same checkpoint, which includes at least one of the weight parameters, optimizer state, and gradient of the first iteration of the training task. The checkpoint includes a first checkpoint and a second checkpoint. Multiple copies of the first checkpoint are stored on the multiple computing nodes where the multiple computing cards in the first computing card group are located, and multiple copies of the second checkpoint are stored on the multiple computing nodes where the multiple computing cards in the second computing card group are located.

[0228] The device includes a loading module 1401 and a recovery module 1402 .

[0229] The loading module 1401 is configured to, in response to a training resume instruction, load the first checkpoint from the memory of the computing node where the computing card in the first computing card group is located, send the first checkpoint to the computing card in the second computing card group, and receive the second checkpoint sent by the computing card in the second computing card group. The training resume instruction instructs to resume training according to the first iteration.

[0230] The recovery module 1402 is configured to recover the training task using the computing cards in the first computing card group based on the first checkpoint and the second checkpoint.

[0231] Optionally, the first computing card group includes a first computing card and a second computing card, and the second computing card group includes a third computing card and a fourth computing card;

[0232] The device also includes a storage module for:

[0233] Storing the first checkpoint in the memory of the computing node where the first computing card is located through the first computing card;

[0234] Storing the first checkpoint in the memory of the computing node where the second computing card is located through the second computing card;

[0235] Storing the second checkpoint in the memory of the computing node where the third computing card is located through the third computing card;

[0236] The second checkpoint is stored in the memory of the computing node where the fourth computing card is located through the fourth computing card.

[0237] Optionally, the first checkpoint and the second checkpoint constitute the checkpoint.

[0238] Optionally, the first computing card and the second computing card are located in different computing nodes, and the third computing card and the fourth computing card are located in different computing nodes.

[0239] Optionally, the large model training system further includes a control node, and the device further includes:

[0240] A first determination module is used to determine the first computing card group and the second computing card group among the multiple computing cards based on topological structure information and a preset backup number of the multiple computing cards, the topological structure information is used to indicate the parallel relationship between the multiple computing cards, and the number of computing cards in the first computing card group and the second computing card group is equal to the preset backup number.

[0241] Optionally, the device further comprises:

[0242] a sending module, configured to send the checkpoint metadata to the control node through computing nodes where computing cards in the first computing card group and the second computing card group are located, respectively, if the first iteration meets the first frequency;

[0243] The second determining module is configured to determine the first checkpoint and the second checkpoint according to the metadata of the checkpoint.

[0244] Optionally, a difference between the data volume of the first checkpoint, the data volume of the second checkpoint, and an average data volume is smaller than a target value, where the average data volume is determined based on the data volume of the checkpoint and the preset number of backups.

[0245] Optionally, the sending module is further configured to:

[0246] Sending a first save instruction to the first computing card, where the first save instruction is used to instruct the first computing card to store the first checkpoint in the memory of the computing node where the card is located;

[0247] Sending a second save instruction to the second computing card, where the second save instruction is used to instruct the second computing card to store the first checkpoint in the memory of the computing node where the card is located;

[0248] Sending a third save instruction to the third computing card, where the third save instruction is used to instruct the third computing card to store the second checkpoint in the memory of the computing node where the card is located;

[0249] A fourth save instruction is sent to the fourth computing card, where the fourth save instruction is used to instruct the fourth computing card to store the second checkpoint in the memory of the computing node where the card is located.

[0250] Optionally, the device further includes a deletion module, configured to:

[0251] If the computing nodes where the computing cards in the first computing card group and the second computing card group are located have insufficient memory, sending a data deletion request for a second iteration to the computing nodes where the computing cards in the first computing card group and the second computing card group are located, respectively, where the second iteration is the iteration whose generated checkpoint is stored in the computing node memory and whose iteration order is the first number of iterations ahead;

[0252] The checkpoint generated in the first iteration is deleted from the memory through the computing nodes where the computing cards in the first computing card group and the second computing card group are located.

[0253] Optionally, the large model training system further includes a storage device;

[0254] The device also includes a persistence module, including:

[0255] a sending unit, configured to send a data persistence request for the first iteration to the computing nodes where the computing cards in the first computing card group and the second computing card group are located, respectively, if the first iteration satisfies a data persistence condition, where the data persistence condition indicates that the first iteration satisfies a second frequency, an iteration order of the first iteration is a preset order, or a checkpoint generated by the first iteration is currently persistable;

[0256] The persistence unit is used to store the first checkpoint and the second checkpoint from the memory to the storage device through the computing nodes where the computing cards in the first computing card group and the second computing card group are located.

[0257] Optionally, the memory of the computing node where the first computing card is located includes a first buffer zone and a second buffer zone;

[0258] The persistence unit includes:

[0259] A first storage subunit is configured to store the first checkpoint in the first buffer and a third checkpoint in the second buffer via the first computing card, wherein the third checkpoint is a partial checkpoint generated by computing cards in the first computing card group and the second computing card group in a third iteration, where the third iteration is an iteration subsequent to the first iteration.

[0260] The second storage subunit is configured to store the first checkpoint from the first buffer to the storage device while the first computing card stores the third checkpoint to the second buffer through the computing node where the first computing card is located.

[0261] Optionally, the memory of the computing node where the first computing card is located further includes a third buffer zone;

[0262] The second storage subunit is used for:

[0263] If, when the first computing card stores the fourth checkpoint in the first buffer, there is a first checkpoint in the first buffer that has not been stored in the storage device, the first checkpoint in the first buffer is written from the first buffer to the third buffer through the computing node where the first computing card is located;

[0264] While the fourth checkpoint is stored in the first buffer, the first checkpoint in the third buffer is stored in the storage device through the computing node where the first computing card is located.

[0265] Optionally, the loading module 1401 includes:

[0266] a loading unit configured to, in response to the training resume instruction, load the first checkpoint from a memory of a computing node in the first computing card group through a computing card in the first computing card group, and send the first checkpoint to a computing card in the second computing card group through a communication link between the plurality of computing cards;

[0267] The receiving unit is configured to receive the second checkpoint sent by the computing card in the second computing card group through the computing card in the first computing card group and the communication link between the plurality of computing cards.

[0268] Optionally, the large model training system further includes a first computing node, the first computing node being used to replace a failed computing node;

[0269] The device also includes:

[0270] an acquisition module, configured to acquire, through the first computing node, the first checkpoint from a memory of the computing node where the second computing card is located in response to the training recovery instruction when the computing node where the second computing card is located fails;

[0271] The first computing node stores the first checkpoint in the memory of the first computing node.

[0272] The loading module 1401 and the recovery module 1402 can be implemented by software or hardware. For example, the implementation of the loading module 1401 will be described below using the loading module 1401 as an example. Similarly, the implementation of the recovery module 1402 can refer to the implementation of the loading module 1401.

[0273] As an example of a software functional unit, the loading module 1401 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the loading module 1401 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0274] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0275] As an example of a hardware functional unit, the loading module 1401 may include at least one computing device, such as a server. Alternatively, the loading module 1401 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0276] The multiple computing devices included in load module 1401 can be distributed in the same region or in different regions. The multiple computing devices included in load module 1401 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in load module 1401 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0277] It should be noted that, in other embodiments, the loading module 1401 can be used to execute any step in the large model training method, and the recovery module 1402 can be used to execute any step in the large model training method. The steps that the loading module 1401 and the recovery module 1402 are responsible for implementing can be specified as needed, and the loading module 1401 and the recovery module 1402 respectively implement different steps in the large model training method to achieve the following. Figure 14 The full functionality of the large model training device is shown.

[0278] The present application also provides a computing device 1500 . Figure 15 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application, such as Figure 15 As shown, computing device 1500 includes a bus 1501, a processor 1502, a memory 1503, and a communication interface 1504. Processor 1502, memory 1503, and communication interface 1504 communicate with each other via bus 1501. Computing device 1500 can be a computing device or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 1500.

[0279] The bus 1501 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 15 The bus 1501 may include a path for transmitting information between various components of the computing device 1500 (eg, memory 1503, processor 1502, communication interface 1504).

[0280] The processor 1502 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0281] The memory 1503 may include a volatile memory, such as a random access memory (RAM). The memory 1503 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0282] Memory 1503 stores executable program code, which processor 1502 executes to implement the functions of loading module 1401 and recovery module 1402, respectively, thereby implementing the steps of the large model training method. That is, memory 1503 stores instructions for executing the large model training method.

[0283] The communication interface 1504 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1500 and other devices or a communication network.

[0284] An embodiment of the present application also provides a computing device cluster. Figure 16 is a schematic diagram of a computing device cluster provided in an embodiment of the present application, such as Figure 16 As shown, the computing device cluster includes at least one computing device 1500. The memory 1503 in one or more computing devices 1500 in the computing device cluster may store the same instructions for executing the large model training method.

[0285] In some possible implementations, the memory 1503 of one or more computing devices 1500 in the computing device cluster may also store some instructions for executing the large model training method. In other words, the combination of one or more computing devices 1500 can jointly execute the instructions for executing the large model training method.

[0286] It should be noted that the memory 1503 in different computing devices 1500 in the computing device cluster can store different instructions, each used to perform part of the functions of the large model training apparatus. In other words, the instructions stored in the memory 1503 in different computing devices 1500 can implement the functions of one or more modules in the aforementioned loading module 1401 and recovery module 1402.

[0287] It should be understood that Figure 16 The functionality of the computing device 1500 shown in FIG. 15 may also be performed by multiple computing devices 1500 .

[0288] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 17 A possible implementation is shown. Figure 17 This is a schematic diagram of a possible implementation of a computing device cluster provided in an embodiment of the present application, such as Figure 17 As shown, computing device 1500A and computing device 1500B are connected via a network. Specifically, each computing device is connected to the network via a communication interface in each computing device. In this possible implementation, memory 1503 in computing device 1500A stores instructions for executing the functions of loading module 1401. Figure 17 In the example, the memory 1503 in the computing device 1500A stores instructions for the function of loading the module 1401. Meanwhile, the memory 1503 in the computing device 1500B stores instructions for the function of executing the restoration module 1402. Figure 17In the example, the memory 1503 in the computing device 1500B stores instructions for executing the functions of the recovery module 1402.

[0289] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similarly referred to as Figure 16 or Figure 17 The difference is that the memory 1503 in one or more computing devices 1500 in the computing device cluster may store the same instructions for executing the large model training method.

[0290] In some possible implementations, the memory 1503 of one or more computing devices 1500 in the computing device cluster may also store some instructions for executing the large model training method. In other words, the combination of one or more computing devices 1500 can jointly execute the instructions for executing the large model training method.

[0291] It should be noted that the memory 1503 in different computing devices 1500 in the computing device cluster can store different instructions, each used to perform part of the functions of the large model training apparatus. In other words, the instructions stored in the memory 1503 in different computing devices 1500 can implement the functions of one or more modules in the aforementioned loading module 1401 and recovery module 1402.

[0292] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute a large model training method provided by any possible implementation method in the aforementioned method embodiment.

[0293] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the large model training method.

[0294] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the topology information involved in this application is obtained with full authorization.

[0295] Those skilled in the art will appreciate that the various method steps and units described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0296] Those skilled in the art will clearly understand that, for the sake of convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0297] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0298] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0299] In addition, each unit in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software units.

[0300] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the present application, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computing device (which can be a personal computer, a server, or a computing device) to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0301] The terms "first", "second", and the like in the present application are used to distinguish between items or similar items with substantially the same function and action. It should be understood that there is no logical or chronological dependency between "first", "second", and "nth", and the quantity and execution order are not limited. It should also be understood that although the following description uses the terms first, second, and the like to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of various examples, a first computing card can be referred to as a second computing card, and similarly, a second computing card can be referred to as a first computing card. The first computing card and the second computing card can both be node computing cards, and in some cases, can be separate and different computing cards.

[0302] In the present application, the term "at least one" means one or more, and the term "multiple" in the present application means two or more. The terms "system" and "network" are often used interchangeably in the present application.

[0303] It should also be understood that the term "if" can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting". Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]", depending on the context.

[0304] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0305] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer program instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0306] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital video disc (DVD), or a semiconductor medium (e.g., a solid-state drive)), etc.

[0307] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0308] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A large model training system, characterized in that: The large model training system includes multiple computing cards, which are located on multiple computing nodes. The computing nodes include memory. The multiple computing cards include a first computing card group and a second computing card group. The multiple computing card groups are used to perform the same training task. In a first iteration of the training task, the computing cards in the first computing card group and the second computing card group respectively generate the same checkpoint. The checkpoint includes at least one of the weight parameters, optimizer state, gradient and number of training cycles of the first iteration of the training task. The checkpoint includes a first checkpoint and a second checkpoint. Multiple copies of the first checkpoint are stored on the multiple computing nodes where the multiple computing cards in the first computing card group are located, and multiple copies of the second checkpoint are stored on the multiple computing nodes where the multiple computing cards in the second computing card group are located. The computing cards in the first computing card group are used for: In response to a training resume instruction, loading the first checkpoint from the memory of the computing node where the computing card in the first computing card group is located, sending the first checkpoint to the computing card in the second computing card group, and receiving the second checkpoint sent by the computing card in the second computing card group, wherein the training resume instruction instructs to resume training according to the first iteration; Restore the training task based on the first checkpoint and the second checkpoint.

2. The system according to claim 1, wherein: The first computing card group includes a first computing card and a second computing card, and the second computing card group includes a third computing card and a fourth computing card; The first computing card is used to store the first checkpoint in the memory of the computing node where the first computing card is located; The second computing card is used to store the first checkpoint in the memory of the computing node where the second computing card is located; The third computing card is used to store the second checkpoint in the memory of the computing node where the third computing card is located; The fourth computing card is used to store the second checkpoint in the memory of the computing node where the fourth computing card is located.

3. The system according to claim 1, wherein: The first checkpoint and the second checkpoint constitute the checkpoint.

4. The system according to any one of claims 1 to 3, characterized in that The large model training system further includes a control node, which is used to: Based on the topological structure information and the preset backup number of the multiple computing cards, the first computing card group and the second computing card group among the multiple computing cards are determined, the topological structure information is used to indicate the parallel relationship between the multiple computing cards, and the number of computing cards in the first computing card group and the second computing card group is equal to the preset backup number.

5. The system according to claim 4, characterized in that The computing cards in the first computing card group and the second computing card group are further used for: If the first iteration satisfies the first frequency, sending the metadata of the checkpoint to the control node through the computing nodes where the checkpoints are respectively located; The control node is further configured to determine the first checkpoint and the second checkpoint according to the metadata of the checkpoint.

6. The system according to claim 4 or 5, characterized in that The control node is further configured to: If the computing nodes where the computing cards in the first computing card group and the second computing card group are located have insufficient memory, sending a data deletion request for a second iteration to the computing nodes where the computing cards in the first computing card group and the second computing card group are located, respectively, where the second iteration is the iteration whose generated checkpoint is stored in the computing node memory and whose iteration order is the first number of iterations before; The computing nodes where the computing cards in the first computing card group and the second computing card group are located are used to: respectively receive the data deletion request and delete the checkpoint generated in the first iteration from the memory.

7. The system according to any one of claims 4 to 6, characterized in that The large model training system also includes a storage device; The control node is further configured to: If the first iteration meets the data persistence condition, a data persistence request for the first iteration is sent to the computing nodes where the computing cards in the first computing card group and the second computing card group are located, respectively, where the data persistence condition indicates that the first iteration meets the second frequency, the iteration order of the first iteration is a preset order, or the checkpoint generated by the first iteration is currently persistable; The computing nodes where the computing cards in the first computing card group and the second computing card group are located are further used to: respectively receive the data persistence request and store the first checkpoint and the second checkpoint from the memory to the storage device.

8. The system according to claim 7, characterized in that The memory of the computing node where the first computing card is located includes a first buffer zone and a second buffer zone; The first computing card is configured to: store the first checkpoint in the first buffer, and store a third checkpoint in the second buffer, where the third checkpoint is a partial checkpoint generated by computing cards in the first computing card group and the second computing card group in a third iteration, where the third iteration is an iteration subsequent to the first iteration; The computing node where the first computing card is located is used to: store the first checkpoint from the first buffer to the storage device at the same time as the first computing card stores the third checkpoint to the second buffer.

9. The system according to claim 8, characterized in that The memory of the computing node where the first computing card is located further includes a third buffer zone; The computing node where the first computing card is located is used for: If, when the first computing card stores the fourth checkpoint in the first buffer, there is a first checkpoint in the first buffer that has not been stored in the storage device, writing the first checkpoint in the first buffer from the first buffer to the third buffer; While the first computing card stores the fourth checkpoint in the first buffer, the first checkpoint in the third buffer is stored in the storage device.

10. The system according to any one of claims 1 to 9, characterized in that The computing cards in the first computing card group are used for: In response to the training resume instruction, load the first checkpoint from the memory of the computing node where the first checkpoint is located, and send the first checkpoint to the computing cards in the second computing card group through the communication link between the plurality of computing cards; The second checkpoint sent by the computing card in the second computing card group is received through the communication link between the multiple computing cards.

11. The system according to claim 10, wherein: The large model training system further includes a first computing node, wherein the first computing node is used to replace a failed computing node; The first computing node is used for: In the event that the computing node where the second computing card is located fails, in response to the training recovery instruction, obtaining the first checkpoint from the memory of the computing node where the first computing card is located; The first checkpoint is stored in the memory of the first computing node.

12. A large model training method, characterized in that: Applied to a large model training system, the large model training system includes multiple computing cards, the multiple computing cards are located on multiple computing nodes, the computing nodes include memory, the multiple computing cards include a first computing card group and a second computing card group, the multiple computing card groups are used to perform the same training task, in the first iteration of the training task, the computing cards in the first computing card group and the second computing card group respectively generate the same checkpoint, the checkpoint includes at least one of the weight parameters, optimizer state and gradient of the first iteration of the training task, the checkpoint includes a first checkpoint and a second checkpoint, multiple copies of the first checkpoint are stored on the multiple computing nodes where the multiple computing cards in the first computing card group are located, and multiple copies of the second checkpoint are stored on the multiple computing nodes where the multiple computing cards in the second computing card group are located; The method comprises: The computing cards in the first computing card group, in response to the training resume instruction, load the first checkpoint from the memory of the computing node where the computing card is located, send the first checkpoint to the computing cards in the second computing card group, and receive the second checkpoint sent by the computing cards in the second computing card group, wherein the training resume instruction instructs to resume training according to the first iteration; The computing cards in the first computing card group restore the training task based on the first checkpoint and the second checkpoint.

13. The method according to claim 12, characterized in that The first computing card group includes a first computing card and a second computing card, and the second computing card group includes a third computing card and a fourth computing card; The method further comprises: The first computing card stores the first checkpoint in a memory of a computing node where the first computing card is located; The second computing card stores the first checkpoint in the memory of the computing node where the second computing card is located; The third computing card stores the second checkpoint in the memory of the computing node where the third computing card is located; The fourth computing card stores the second checkpoint in the memory of the computing node where the fourth computing card is located.

14. The method according to any one of claims 12 or 13, characterized in that The large model training system further includes a control node, and the method further includes: The control node determines the first computing card group and the second computing card group among the multiple computing cards based on the topological structure information and the preset backup number of the multiple computing cards, the topological structure information is used to indicate the parallel relationship between the multiple computing cards, and the number of computing cards in the first computing card group and the second computing card group is equal to the preset backup number.

15. The method according to claim 14, characterized in that The method further comprises: If the first iteration satisfies the first frequency, the computing cards in the first computing card group and the second computing card group respectively send the metadata of the checkpoint to the control node through the computing nodes where they are located; The control node determines the first checkpoint and the second checkpoint according to the metadata of the checkpoint.

16. The method according to claim 15, characterized in that The large model training system also includes a storage device; The method further comprises: If the first iteration satisfies a data persistence condition, the control node sends a data persistence request for the first iteration to the computing nodes where the computing cards in the first computing card group and the second computing card group are located, respectively, where the data persistence condition indicates that the first iteration satisfies a second frequency, the iteration order of the first iteration is a preset order, or the checkpoint generated by the first iteration is currently persistable; The computing nodes where the computing cards in the first computing card group and the second computing card group are located respectively receive the data persistence request and store the first checkpoint and the second checkpoint from the memory to the storage device.

17. The method according to claim 16, characterized in that The memory of the computing node where the first computing card is located includes a first buffer zone and a second buffer zone; The computing node where the computing card in the first computing card group is located receives the data persistence request and stores the first checkpoint from the memory to the storage device, including: The first computing card stores the first checkpoint in the first buffer. While storing the third checkpoint in the second buffer, the computing node where the first computing card is located stores the first checkpoint from the first buffer to the storage device. The third checkpoint is a partial checkpoint generated by the computing cards in the first computing card group and the second computing card group in a third iteration. The third iteration is an iteration after the first iteration.

18. The method according to claim 17, characterized in that The memory of the computing node where the first computing card is located further includes a third buffer zone; Storing the first checkpoint from the first buffer to the storage device includes: If, when the first computing card stores the fourth checkpoint in the first buffer, there is a first checkpoint in the first buffer that has not been stored in the storage device, the computing node where the first computing card is located writes the first checkpoint in the first buffer from the first buffer to the third buffer; While storing the fourth checkpoint in the first buffer, the computing node where the first computing card is located stores the first checkpoint in the third buffer in the storage device.

19. The method according to any one of claims 12 to 18, characterized in that The computing card in the first computing card group, in response to the training resume instruction, loads the first checkpoint from the memory of the computing node where the computing card is located, sends the first checkpoint to the computing card in the second computing card group, and receives the second checkpoint sent by the computing card in the second computing card group, including: In response to the training resume instruction, the computing cards in the first computing card group load the first checkpoint from the memory of the computing node where they are located, and send the first checkpoint to the computing cards in the second computing card group through the communication link between the multiple computing cards; The computing cards in the first computing card group receive the second checkpoint sent by the computing cards in the second computing card group through the communication link between the multiple computing cards.

20. The method according to claim 19, characterized in that The large model training system further includes a first computing node, wherein the first computing node is used to replace a failed computing node; The method further comprises: When the computing node where the second computing card is located fails, the first computing node obtains the first checkpoint from the memory of the computing node where the first computing card is located in response to the training recovery instruction; The first computing node stores the first checkpoint in the memory of the first computing node.

21. A computing card, characterized in that: The computing card includes a computing core and a memory. The computing card is used to execute the operations performed by the computing card in the large model training method described in any one of claims 12 to 20. The memory is used to store computing data, and the computing core is used to perform computing operations on the computing data stored in the memory.

22. A computing device, characterized in that The computing node includes a processor, a computing card and a memory, and the processor is used to execute the large model training method according to any one of claims 12 to 20.

23. A computing device cluster, characterized in that: The computing device cluster includes at least one computing node, each of the computing nodes includes a processor, a computing card and a memory, and the processor of the at least one computing node is used to execute instructions so that the computing device cluster performs the large model training method as described in any one of claims 12 to 20.

24. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the large model training method as described in any one of claims 12 to 20.

25. A computer-readable storage medium, characterized in that The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the large model training method according to any one of claims 12 to 20.

Images