Large-model training system, apparatus and method, and computing device cluster

Abstract

The present application belongs to the technical field of cloud computing. Provided are a large-model training system, apparatus and method, and a computing device cluster. In the method, a plurality of computing cards of a large-model training system are grouped according to the fact that checkpoints generated by means of the plurality of computing cards have two states, i.e., being identical and being different, under data parallelism and optimizer parallelism, and checkpoints in shared memories of computing nodes where the plurality of computing cards are located have identical items and different items; and in response to a training resumption instruction, a first computing card comprised in a first computing card group loads identical items from a storage device, and broadcasts same to other computing cards in the first computing card group for training resumption. Since only one computing card is selected from the first computing card group to load a checkpoint from the storage device, the occurrence of a bandwidth bottleneck that is caused by all computing cards in the group simultaneously loading checkpoints from the storage device is avoided, such that the training resumption efficiency can be improved.

Description

Large model training systems, devices, methods, and clusters of computing equipment

[0001] This application claims priority to Chinese Patent Application No. 202411045418.4, filed on July 31, 2024, entitled “Large Model Training System, Apparatus, Method and Computing Equipment Cluster”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of cloud computing technology, and in particular to a large model training system, apparatus, method, and computing device cluster. Background Technology

[0003] In training large-scale artificial intelligence (AI) models, a distributed cluster is typically used. This distributed cluster consists of multiple computing nodes, each containing at least one computing card. During distributed training, parameters such as model weights, optimizer states, gradients, and the number of training epochs are generated. The distributed cluster saves these parameters as checkpoints at a fixed frequency. This allows the distributed cluster to load the most recently saved checkpoint and resume training based on it in the event of a computing node failure, thus avoiding retraining from scratch, saving on failure losses, and improving training efficiency.

[0004] In related technologies, each computing node in a distributed cluster stores the checkpoint generated by its computing card in its own memory and creates a copy on each of the other m computing nodes (m is greater than or equal to 1) according to a certain multi-copy strategy. If a user stops training and then restarts training, the computing card on each computing node loads the checkpoint from the memory of its own computing node and resumes training based on the checkpoint. If any computing node fails and the checkpoint in its memory is lost, the replacement computing node can load the checkpoint from the computing node that stores the checkpoint copy into the memory of the replacement computing node. If the computing node that stores the checkpoint copy also fails and no copy of the checkpoint exists, the replacement computing node needs to read the checkpoint from a remote persistent storage device and resume training based on the checkpoint.

[0005] However, in the above method, when m computing nodes simultaneously read checkpoints from persistent remote storage devices, a bandwidth bottleneck occurs, which prolongs the checkpoint reading time, affects the training recovery time, and thus leads to low training recovery efficiency. Summary of the Invention

[0006] This application provides a large model training system, apparatus, method, and computing device cluster, which can improve checkpoint loading efficiency, improve training recovery efficiency, and avoid bandwidth bottlenecks. The technical solution is as follows.

[0007] In a first aspect, a large model training system is provided. The system includes a storage device and at least one computing card, which is disposed on at least one computing node. The at least one computing card includes a first computing card group for executing a first training task. The first training task includes a first sub-training task and a second sub-training task. The first computing card group includes a first computing card and a second computing card. The first computing card is used to execute the first sub-training task and generate a first checkpoint, which includes model parameters and optimizer state for the target iteration of the first sub-training task. The second computing card is used to execute the second sub-training task and generate a second checkpoint, which includes model parameters and optimizer state for the target iteration of the second sub-training task. The first computing card is further used to, in response to a first training recovery instruction, load identical items from the first checkpoint and the second checkpoint from the storage device, recover the first sub-training task based on these identical items, and send the identical items to the second computing card. The first training recovery instruction is used to instruct training recovery to proceed according to the target iteration. The second computing card is further used to receive the identical items and recover the second sub-training task based on these identical items.

[0008] In the aforementioned system, the checkpoints generated by multiple computing cards under data parallelism and optimizer parallelism can have two states: identical and different. The multiple computing cards in the large model training system are grouped, and the checkpoints in the shared memory of the computing nodes where the cards reside contain both identical and different entries. When a failure occurs and training needs to be resumed, if multiple computing cards in the first computing card group generate checkpoints with identical entries, only one computing card in the first group is selected to load this identical entry from the storage device. This entry is then broadcast to the other computing cards in the first computing card group for training resumption. This avoids all computing cards in the group simultaneously loading checkpoints from the storage device, which could lead to bandwidth bottlenecks and affect training resumption time. This reduces the bandwidth required to load checkpoints from the storage device during training resumption, resulting in higher checkpoint loading efficiency and overall training resumption efficiency.

[0009] In one possible implementation, at least one computing card further includes a third computing card and a fourth computing card belonging to the second computing card group. The third computing card is used to execute the second training task and generate a third checkpoint; the fourth computing card is used to execute the third training task and generate a fourth checkpoint, wherein the model parameters and optimizer states included in the fourth checkpoint differ from those included in the third checkpoint; the third computing card is also used to, in response to a second training recovery instruction, load the different items of the third checkpoint from a storage device and recover the second training task based on the different items of the third checkpoint; the fourth computing card is also used to, in response to a second training recovery instruction, load the different items of the fourth checkpoint from a storage device and recover the third training task based on the different items of the fourth checkpoint.

[0010] In one possible implementation, each compute node includes multiple compute cards that share memory. The first compute card is further configured to load the same item from the shared memory of the compute node where the first compute card resides before loading it from a storage device; the second compute card is further configured to load the same item from the shared memory of the compute node where the second compute card resides before loading it from a storage device, and if the same item is not stored in the shared memory of the compute node where the first compute card resides, and then send the same item to the first compute card.

[0011] In one possible implementation, the compute node on which the fourth compute card resides stores copies of the distinct items of the third checkpoint. The third compute card is further configured to load the distinct items of the third checkpoint from the shared memory of the compute node on which it resides before loading them from the storage device; the fourth compute card is further configured to load copies of the distinct items of the third checkpoint from the shared memory of the compute node on which it resides, before loading them from the storage device, and in the absence of copies of the distinct items of the third checkpoint stored in the shared memory of the compute node on which it resides, and to send the copies of the distinct items of the third checkpoint to the third compute card.

[0012] In one possible implementation, the identical items include a first identical item fragment and a second identical item fragment. The first computing card is further configured to, in response to a first training recovery instruction, load the first identical item fragment from a storage device and send the first identical item fragment to a second computing card; the second computing card is further configured to, in response to the first training recovery instruction, load the second identical item fragment from a storage device and send the second identical item fragment to the first computing card; the first computing card is further configured to recover a first sub-training task based on the first identical item fragment and the second identical item fragment; the second computing card is further configured to recover a second sub-training task based on the first identical item fragment and the second identical item fragment.

[0013] In the above system, by equally distributing identical items within the computing card group, and having each computing card in the group load a fragment of the identical item from the storage device, the loading task of identical items can be evenly distributed to each computing card in the group. This ensures that each computing card in the group loads the fragment of the identical item from the storage device in a balanced manner to obtain the complete identical item. This is beneficial for increasing the concurrency of reading identical items and avoids the bottleneck of a single computing card being overloaded.

[0014] In one possible implementation, each computing card includes video memory, and the first computing card is further configured to write the checkpoint generated by the first computing card from the video memory of the first computing card to the shared memory of the computing node where the first computing card is located in the original data format, wherein the checkpoint in the original data format is not serialized.

[0015] In the aforementioned system, removing the serialization operation from the stage of saving the checkpoint to the shared memory of the computing node where the computing card resides can hide the time consumed by saving the checkpoint and avoid long-term training blockage. Furthermore, during training recovery, the first computing card can load the same items in the original data format from the shared memory of the computing node where it resides, saving the time consumed by the deserialization operation and improving the training recovery speed.

[0016] In one possible implementation, the first computing card and the second computing card are used to determine whether the computing node where the first computing card is located writes the same item from the shared memory of the computing node where the first computing card is located to the storage device.

[0017] In one possible implementation, each computing card includes card status information indicating the health status of the computing card. The first computing card is further configured to send its card status information to a second computing card; the second computing card is further configured to receive the card status information from the first computing card; the computing node where the second computing card resides is configured to, in the event of a failure of the first computing card, or in the event that the same item is not stored in the storage device, write the same item from the shared memory of the computing node where the second computing card resides to the storage device, wherein the failure of the first computing card is determined based on the card status information of the first computing card.

[0018] In the above system, the first compute card persistently writes the identical item from the shared memory of the compute node where it resides to the storage device. Other compute cards in the first compute card group, such as the second compute card, check whether the identical item has been successfully written to the storage device. If the identical item is not stored in the storage device, a new compute card other than the first compute card is elected from the first compute card group to persistently write the identical item to the storage device. Thus, through this fault-tolerant checking mechanism, the possibility of checkpoint loss due to failure to persistently write to the storage device can be avoided, ensuring the reliability of the checkpoint persistent writing to the storage device.

[0019] In one possible implementation, the computing node where the first computing card is located is used to write the first identical item fragment from the shared memory of the computing node where the first computing card is located to the storage device; the computing node where the second computing card is located is also used to write the second identical item fragment from the shared memory of the computing node where the first computing card is located to the storage device.

[0020] In the above system, by evenly distributing identical items within the computing card group, the storage tasks of identical items can be evenly distributed to each computing card in the group, ensuring that each computing card in the group writes identical items to the storage device in a balanced manner. This helps to increase the concurrency of writing identical items and avoids the bottleneck of a single computing card being overloaded.

[0021] In one possible implementation, the computing node where the third computing card is located is used to write the different items of the third checkpoint from the shared memory of the computing node where the third computing card is located to the storage device; the computing node where the fourth computing card is located is used to write the different items of the fourth checkpoint from the shared memory of the computing node where the fourth computing card is located to the storage device.

[0022] In one possible implementation, the first computing card is further configured to send its card status information to the second computing card; the second computing card is further configured to receive the card status information of the first computing card; the computing node where the second computing card is located is further configured to, in the event of a failure of the first computing card, or in the event that the first identical item fragment is not stored in the storage device, write the first identical item fragment from the shared memory of the computing node where the second computing card is located to the storage device; the third computing card is further configured to send its card status information to the fourth computing card; the fourth computing card is further configured to receive the card status information of the third computing card; the computing node where the fourth computing card is located is further configured to, in the event of a failure of the third computing card, or in the event that the different items of the third checkpoint are not stored in the storage device, write a copy of the different items of the third checkpoint from the shared memory of the computing node where the fourth computing card is located to the storage device.

[0023] In one possible implementation, the storage device further includes a first-layer storage device and a second-layer storage device. The computing node where the first computing card is located is also used to write the checkpoint generated by the first computing card from the shared memory of the computing node where the first computing card is located to the first-layer storage device in its original data format. The computing node where the first computing card is located is also used to serialize the checkpoint generated by the first computing card and write the serialized checkpoint from the shared memory of the computing node where the first computing card is located to the first-layer storage device. The first-layer storage device is used to write the serialized checkpoint from the first-layer storage device to the second-layer storage device.

[0024] In the above system, the computing node where the first computing card is located stores the checkpoint in the original data format and the serialized checkpoint in the first-level storage device. The checkpoint in the original data format is not serialized. After the computing node fails, the checkpoint in the original data format is loaded from the first-level storage device first. The loading efficiency of the checkpoint is relatively high, which can save the deserialization process and improve the training recovery efficiency.

[0025] Secondly, a large model training apparatus is provided, applied to a large model training system. The large model training system includes a storage device and at least one computing card. The at least one computing card is disposed on at least one computing node. The at least one computing card includes a first computing card group for executing a first training task. The first training task includes a first sub-training task and a second sub-training task. The first computing card group includes a first computing card and a second computing card. The apparatus includes: an allocation module for allocating the first sub-training task to the first computing card. The first sub-training task is used to instruct the first computing card to execute the first sub-training task and generate a first checkpoint. The first checkpoint includes the target iteration of the first sub-training task. The system includes: a model parameter and optimizer state; an allocation module that allocates a second sub-training task to a second computing card, the second sub-training task instructing the second computing card to execute the first sub-training task and generate a second checkpoint, the second checkpoint including the model parameters and optimizer state of the target iteration of the second sub-training task; a determination module that, in response to a first training recovery instruction, determines the common items of the first checkpoint and the second checkpoint, the first training recovery instruction instructing training recovery according to the target iteration; and a sending module that sends the information recording the common items to the first computing card, so that the first computing card loads the common items from the storage device based on the information and sends the common items to the second computing card.

[0026] Thirdly, a large model training method is provided, which is applied to a large model training system. The large model training system includes a storage device and at least one computing card. The at least one computing card is disposed on at least one computing node. The at least one computing card includes a first computing card group for executing a first training task. The first training task includes a first sub-training task and a second sub-training task. The first computing card group includes a first computing card and a second computing card. The method includes: the first computing card executing the first sub-training task and generating a first checkpoint, the first checkpoint including model parameters and optimizer state for the target iteration of the first sub-training task; the second computing card executing the second sub-training task and generating a second checkpoint, the second checkpoint including model parameters and optimizer state for the target iteration of the second sub-training task; the first computing card, in response to a first training recovery instruction, loading identical items of the first checkpoint and the second checkpoint from the storage device, recovering the first sub-training task based on the identical items, and sending the identical items to the second computing card. The first training recovery instruction is used to instruct training recovery to be performed according to the target iteration; the second computing card receiving the identical items and recovering the second sub-training task based on the identical items.

[0027] In one possible implementation, at least one computing card further includes a third computing card and a fourth computing card belonging to the second computing card group. The method further includes: the third computing card performing a second training task and generating a third checkpoint; the fourth computing card performing the third training task and generating a fourth checkpoint, wherein the model parameters and optimizer states included in the fourth checkpoint differ from those included in the third checkpoint; the third computing card, in response to a second training recovery instruction, loading the different items of the third checkpoint from a storage device and recovering the second training task based on the different items of the third checkpoint; and the fourth computing card, in response to the second training recovery instruction, loading the different items of the fourth checkpoint from a storage device and recovering the third training task based on the different items of the fourth checkpoint.

[0028] In one possible implementation, each computing node includes multiple computing cards sharing memory. Before loading the same item from a storage device, the process includes: a first computing card loading the same item from the shared memory of the computing node where the first computing card resides; and a second computing card loading the same item from the shared memory of the computing node where the second computing card resides if the same item is not stored in the shared memory of the computing node where the first computing card resides, and sending the same item to the first computing card.

[0029] In one possible implementation, the compute node on which the fourth compute card resides stores copies of the different items of the third checkpoint. Before loading the different items of the third checkpoint from the storage device, the method includes: the third compute card is further configured to load the different items of the third checkpoint from the shared memory of the compute node on which the third compute card resides; the fourth compute card is further configured to, if the shared memory of the compute node on which the third compute card resides does not store the different items of the third checkpoint, load copies of the different items of the third checkpoint from the shared memory of the compute node on which the fourth compute card resides, and send copies of the different items of the third checkpoint to the third compute card.

[0030] In one possible implementation, the identical items include a first identical item fragment and a second identical item fragment. The method further includes: a first computing card, in response to a first training recovery command, loading the first identical item fragment from a storage device and sending the first identical item fragment to a second computing card; the second computing card, in response to the first training recovery command, loading the second identical item fragment from a storage device and sending the second identical item fragment to the first computing card; the first computing card recovering a first sub-training task based on the first identical item fragment and the second identical item fragment; and the second computing card recovering a second sub-training task based on the first identical item fragment and the second identical item fragment.

[0031] In one possible implementation, each computing card includes video memory, and the method further includes: the first computing card writes the checkpoint generated by the first computing card from the video memory of the first computing card to the shared memory of the computing node where the first computing card is located in the original data format, wherein the checkpoint in the original data format is not serialized.

[0032] In one possible implementation, after the first computing card writes the checkpoint generated by the first computing card from the video memory of the first computing card to the shared memory of the computing node where the first computing card is located in the original data format, the process includes: the first computing card and the second computing card determining that the computing node where the first computing card is located writes the same item from the shared memory of the computing node where the first computing card is located to the storage device.

[0033] In one possible implementation, each computing card includes card status information to indicate the health status of the computing card. The method further includes: a first computing card sending its card status information to a second computing card; the second computing card receiving the card status information from the first computing card; and the computing node where the second computing card is located writing the same item from the shared memory of the computing node where the second computing card is located to the storage device in the event of a failure of the first computing card or, in the event that the storage device does not store the same item, wherein the failure of the first computing card is determined based on the card status information of the first computing card.

[0034] In one possible implementation, the method further includes: the computing node where the first computing card is located writing the first identical item fragment from the shared memory of the computing node where the first computing card is located to the storage device; and the computing node where the second computing card is located writing the second identical item fragment from the shared memory of the computing node where the first computing card is located to the storage device.

[0035] In one possible implementation, the method further includes: the computing node where the third computing card is located writing the different items of the third checkpoint from the shared memory of the computing node where the third computing card is located to the storage device; and the computing node where the fourth computing card is located writing the different items of the fourth checkpoint from the shared memory of the computing node where the fourth computing card is located to the storage device.

[0036] In one possible implementation, the method further includes: a first computing card sending its card status information to a second computing card; the second computing card receiving the card status information from the first computing card; the computing node where the second computing card is located writing the first identical item fragment from the shared memory of the computing node where the second computing card is located to the storage device in the event of a failure of the first computing card, or in the event that the first identical item fragment is not stored in the storage device; a third computing card sending its card status information to a fourth computing card; the fourth computing card receiving the card status information from the third computing card; and the computing node where the fourth computing card is located writing a copy of the different item of the third checkpoint from the shared memory of the computing node where the fourth computing card is located to the storage device in the event of a failure of the third computing card, or in the event that the different item of the third checkpoint is not stored in the storage device.

[0037] In one possible implementation, the storage device further includes a first-layer storage device and a second-layer storage device. After writing the checkpoint generated by the first computing card from the video memory of the first computing card to the shared memory of the computing node where the first computing card is located in its original data format, the process includes: the computing node where the first computing card is located writing the checkpoint generated by the first computing card from the shared memory of the computing node where the first computing card is located to the first-layer storage device in its original data format; the computing node where the first computing card is located serializing the checkpoint generated by the first computing card and writing the serialized checkpoint from the shared memory of the computing node where the first computing card is located to the first-layer storage device; and the first-layer storage device being used to write the serialized checkpoint from the first-layer storage device to the second-layer storage device.

[0038] Fourthly, a computing card is provided, which includes a computing core and a memory. The computing card is used to execute a large model training method as provided in the third aspect or any possible implementation thereof. The memory is used to store computational data, and the computing core is used to perform computational operations on the computational data stored in the memory.

[0039] Fifthly, a computing device is provided, the computing node including a processor, a computing card and memory, the processor being used to execute a large model training method as provided in the third aspect or any possible implementation thereof.

[0040] In a sixth aspect, a computing device cluster is provided, the computing device cluster including at least one computing node, each computing node including a processor, a computing card and memory, the processor of the at least one computing node being used to execute instructions such that the computing device cluster performs a large model training method as provided in the third aspect or any possible implementation thereof.

[0041] In a seventh aspect, a computer program product containing instructions is provided that, when executed by a cluster of computing devices, causes the cluster of computing devices to perform a large model training method as provided in the third aspect or any possible implementation thereof.

[0042] Eighthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a cluster of computing devices, execute a large model training method as provided in the third aspect or any possible implementation thereof.

[0043] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0044] Figure 1 is a schematic diagram of the structure of a large model training system provided in this application;

[0045] Figure 2 is a schematic diagram of the distribution of checkpoints in a large model training method provided in an embodiment of this application;

[0046] Figure 3 is a schematic diagram of checkpoint copy formation in a large model training method provided in an embodiment of this application;

[0047] Figure 4 is a flowchart of the checkpoint saving process in a large model training method provided in an embodiment of this application;

[0048] Figure 5 is a schematic diagram of saving the checkpoint to shared memory in a large model training method provided in an embodiment of this application;

[0049] Figure 6 is a schematic diagram of saving the checkpoint to a storage device in a large model training method provided in an embodiment of this application;

[0050] Figure 7 is a schematic diagram of saving the checkpoint to a storage device in a large model training method provided in an embodiment of this application;

[0051] Figure 8 is a flowchart of the checkpoint fault-tolerant checking process in a large model training method provided in an embodiment of this application;

[0052] Figure 9 is a flowchart of the checkpoint loading process in a large model training method provided in an embodiment of this application;

[0053] Figure 10 is a schematic diagram of the checkpoint loading process in a large model training method provided in an embodiment of this application;

[0054] Figure 11 is a schematic diagram of the checkpoint loading process in a large model training method provided in an embodiment of this application;

[0055] Figure 12 is a schematic diagram of the checkpoint loading process in a large model training method provided in an embodiment of this application;

[0056] Figure 13 is a flowchart of the checkpoint loading process in a large model training method provided in an embodiment of this application;

[0057] Figure 14 is a flowchart of the checkpoint verification process in a large model training method provided in an embodiment of this application;

[0058] Figure 15 is a schematic diagram of a large model training device provided in an embodiment of this application;

[0059] Figure 16 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0060] Figure 17 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0061] Figure 18 is a schematic diagram of a network connection between computing devices provided in an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0063] To make the technical solutions provided in this application clearer, the relevant terms will be explained before describing the technical solutions in detail.

[0064] Deep learning is a type of machine learning technique based on deep neural network algorithms. Its main characteristic is the use of multiple nonlinear transformations to process and analyze data. It is primarily applied in perception and decision-making scenarios within the field of artificial intelligence, such as image and speech recognition, natural language translation, and computer games.

[0065] Data parallelism (DP): Data parallelism is the most common distributed training strategy. The main idea is to divide a large dataset into multiple subsets, each processed by a separate computing card. The results from each card are then integrated through parameter passing and model synchronization. Specifically, each computing card acquires a portion of the entire batch of data and determines the gradient for the target iteration of the large model training task based on this portion. The gradients determined by each computing card are then averaged to obtain the average gradient, which is used to update the model weights. Data parallelism has various forms. In the most basic form, such as distributed data parallelism (DDP) in PyTorch, each computing card stores a copy of all model weights, optimizer states, gradients, and the number of training epochs; that is, the model on each computing card is identical.

[0066] Optimizer Parallel: During data-parallel training, the parameter update part of the model involves redundant computation across different GPUs. Optimizer parallelism effectively reduces memory consumption and improves network performance on large-scale networks by distributing the optimizer's computation across GPUs in the data-parallel dimension. Currently, the most popular method for optimizer parallelism is the ZeRO redundancy optimizer (ZeRO), which includes three levels of partitioning: optimizer partitioning, optimizer and gradient partitioning, and optimizer, gradient, and parameter partitioning. For example, in the first stage of the zero-redundancy optimizer (ZeRO redundancy optimizer-1, ZeRO-1), the optimizer state is partitioned. Each of the multiple GPUs participating in data parallelism stores a portion of the optimizer state. During the forward training of the target iteration, each GPU obtains the optimizer states stored by other GPUs through parameter passing, thereby determining the optimizer state for the data portion being processed.

[0067] Pipeline parallelism (PP), also known as model parallelism, involves dividing a large model into multiple parts, each placed on a different compute card, and then having each card perform training on those parts separately. This form of parallelism is typically used when a single compute card cannot handle the entire model. Pipeline parallelism can be viewed as vertical model parallelism. For example, if there is a single large model with 12 layers, the different layers of this large model would be placed on three different compute cards. While a later compute card is processing a previous batch of data, the previous compute card can process the next batch of data. That is, the later compute card does not need to wait for the previous compute card to finish its computation. By overlapping the computation of different batches of data, all compute cards are working simultaneously, which can improve the throughput of model training.

[0068] Tensor parallelism (TP): In tensor parallelism, the model weights of a large model are horizontally sliced, with each computation card processing only a portion of the model weights. Tensor parallelism is often considered horizontal model parallelism. Specifically, each computation card processes the same batch of data, determines activation values ​​based on the model weights it possesses, exchanges the data they need with each other, and then determines the gradient based on the model weights held by each computation card. Tensor parallelism involves different sharding methods, with 1D sharding being the most common, which divides the model weights along a single dimension (horizontally or vertically).

[0069] 3D Parallel Training: Simultaneously applying data parallelism, pipeline parallelism, and tensor parallelism to the training of large models.

[0070] 3D Parallel Training and ZeRO-1: This approach applies ZeRO-1-style data parallelism, pipeline parallelism, and tensor parallelism simultaneously to the training of large models.

[0071] Serialization: Converts an object into a byte stream for easy storage in memory, files, or databases; deserialization is the reverse process, restoring the object from the byte stream.

[0072] Process-level failures refer to situations where a process encounters an error or exception, causing it to fail to execute normally or exhibit abnormal behavior. After a failure occurs, the system will attempt to restart the process; this does not involve restarting the container, and the host memory is retained.

[0073] Job-level failures: Hardware failures cause individual faulty nodes to be isolated, which may lead to an attempt to restart the entire system or require rescheduling of training nodes, and loss of all or part of the host memory.

[0074] Checkpoint (ckpt): Parameters generated during one iteration of a large model training task, including model parameters, optimizer state (OS), and the number of training epochs. Model parameters include weights and biases during training; optimizer state, for gradient descent optimizers, includes state parameters such as momentum and learning rate.

[0075] The relevant terminology used in this application has been introduced above. The implementation environment of the embodiments of this application is described below.

[0076] Figure 1 is a schematic diagram of a large model training system provided in this application. As shown in Figure 1, the large model training system includes a computing device cluster 101, a first-layer storage device 102, and a second-layer storage device 103. Each computing node includes at least one computing card, a processor, and memory. Each computing card includes high-bandwidth memory (HBM), and each computing card is equipped with a software development kit (SDK). The computing nodes in the computing device cluster 101, the first-layer storage device 102, and the second-layer storage device 103 are connected via wired or wireless means.

[0077] The training task of the large model can run on containers, physical machines, or virtual machines. Compute nodes can be independent physical servers, server clusters consisting of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The processor on the compute node is, for example, a central processing unit (CPU), and the memory on the compute node is, for example, dynamic random access memory (DRAM). When compute cards 3 and 4 are located on the same compute node, a contiguous memory space can be allocated in the memory of that compute node, and processors on the same compute node can access this memory space, i.e., shared memory. For example, compute card 3 on the compute node is used to write the checkpoint into the shared memory of the compute node, and compute card 4 on the compute node can also access this checkpoint. For example, when there are data-parallel processors on the same compute node, the compute card can store only one checkpoint in shared memory. Other data-parallel processors on the compute node can access this checkpoint for fault recovery, avoiding redundant checkpoint storage. For example, the compute card of the compute node is used to write the checkpoint into the compute node's memory, store the checkpoint in the compute node's memory into the second-level storage device 103, and load the checkpoint from the second-level storage device 103 onto the compute card on the compute node.

[0078] The computing card on the computing node is, for example, a CPU, a graphics processing unit (GPU), a neural processor unit (NPU), a switching module processor unit (SMPU), a microprocessor, or one or more integrated circuits for implementing the scheme of this application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Figure 1 is only an example using a GPU as the computing card; this application embodiment does not limit the type of computing card. In some embodiments, the computing card is referred to as an accelerator. The computing cards communicate with each other using a high-performance parametric plane network. Exemplarily, the computing card is used for computational acceleration, and the SDK on the computing card provides an interactive interface for the training process running on the computing card. The computing card includes HBM, which is used to cache model weights and optimizer states required by the computing card when performing training tasks, as well as to cache data generated during training.

[0079] In this computing device cluster 101, multiple computing nodes are used to execute large model training tasks. One or more of these computing nodes can be used to run a large model training device. As shown in Figure 1, exemplarily, the large model training device runs on CPU1. This device is used to determine the parallel relationship between multiple computing cards executing training tasks, allocate model training tasks to multiple computing cards according to the parallel relationship, determine the same and different items included in the checkpoints generated by multiple computing cards based on the parallel relationship, and send the information recording the same and different items to the multiple computing cards. The large model training device may include an allocation module, a determination module, and a sending module. The program code for the allocation module, determination module, and sending module can be stored in the DRAM of the computing node where CPU1 is located. It should be noted that the above division of the large model training device into multiple functional modules is only exemplary, and this application embodiment does not limit the division method of the large model training device.

[0080] The first-tier storage device 102 can be composed of one or more storage media, such as a solid-state disk (SSD), a hard disk drive (HDD), or a scalable file service (SFS), etc., used to save checkpoints before persisting them to cloud storage. Loading checkpoints from the first-tier storage device 102 is faster than loading checkpoints from the second-tier storage device 103. For example, the first-tier storage device 102 can be an SFS, which can store checkpoints as files and organize and present them in a hierarchical structure of files and folders. It should be noted that this embodiment does not limit the type of the first-tier storage device 102.

[0081] The second-layer storage device 103 can be a solid-state drive (SSD) or a hard disk drive (HDD), or, for example, a remote persistent storage device such as an object storage service (OBS) or an elastic volume service (EVS). In some embodiments, the first-layer storage device 102 can be an elastic file service (SFS), and the second-layer storage device 103 can be an object storage service (OBS). The elastic file service (SFS) can work in conjunction with the object storage service (OBS) to achieve mutual recognition of file semantics and object semantics, and supports importing from the elastic file service (SFS) to the object storage service (OBS), as well as exporting from the object storage service (OBS) to the elastic file service (SFS). It should be noted that Figure 1 is only used as an example of the second-layer storage device 103 being a remote persistent storage device, and the embodiments of this application do not limit the type of the second-layer storage device 103.

[0082] In some embodiments, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The networks are typically Transmission Control Protocol / Internet Protocol (TCP / IP) networks and RDMA networks in data center networks, such as RDMA over Converged Ethernet (RoCE) networks, InfiniBand (IB) networks, etc., and are not limited thereto. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0083] This application provides a large model training method applied to a large model training system. In this method, checkpoints generated by multiple computing cards performing training tasks using techniques such as data parallelism and optimizer parallelism contain identical items in their model parameters and optimizer states. The multiple computing cards in the large model training system are grouped, and within each computing card group, multiple copies of the identical items are formed in the memory of the computing node where each card resides, and these identical items are stored in a storage device. When all identical items are lost in the memory of the computing node where the computing cards in the group reside, in response to a training recovery command, one of the computing cards in the group loads the identical items from the storage device and broadcasts them to the other computing cards in the group. The computing cards in the group then perform training recovery based on the checkpoints composed of these identical items. During training recovery, only one compute card in the compute card group is selected to perform the operation of loading the checkpoint from the storage device, and then broadcast to the other compute cards in the group. This reduces the bandwidth required to load the checkpoint from the storage device during training recovery, thereby speeding up the training recovery process. It also avoids all compute cards in the group loading the same items included in the checkpoint from the storage device at the same time, which would otherwise cause storage bandwidth to become a bottleneck.

[0084] The above method can be applied to scenarios where large models are trained in parallel using data parallel mode. These scenarios include data parallel scenarios, scenarios that use a combination of data parallelism and optimizer parallelism, etc. This application does not limit the training scenarios that use data parallelism.

[0085] Large model training tasks utilize parallel techniques such as data parallelism and optimizer parallelism to partition model parameters and datasets on computing cards for accelerated training. Based on this, checkpoints generated by computing cards in this large model training system can have two states: identical or different model parameters and optimizer states. It should be noted that during large model training, the model parameters and optimizer states included in the checkpoints occupy a large amount of storage space, while other parameter data is relatively small. Therefore, this application embodiment only groups the model parameters and optimizer states included in the checkpoints generated by computing cards in this large model training system. This application embodiment groups checkpoints generated by multiple computing cards based on the existence of two states: identical or different model parameters and optimizer states. The checkpoint situation between computing cards under different data parallelism and optimizer parallelism is shown in Table 1, where DP is the data parallelism dimension, ZeRO is the optimizer parallelism dimension, and ZeRO Stage is the stage of the zero-redundancy optimizer. For example, ZeRO Stage = 1 is the first stage of the zero-redundancy optimizer, i.e., ZeRO-1.

[0086] Table 1. Checkpointing results under different data parallelism and optimizer parallelism.

[0087] Based on whether the model parameters and optimizer states included in the checkpoints generated by the computing cards under different data parallelism and optimizer parallelism conditions are the same, the computing cards are divided into a first computing card group and a second computing card group. When multiple computing cards in the first computing card group execute training tasks based on data parallelism or optimizer parallelism, there are common items among the checkpoints generated by the multiple computing cards in the first computing card group. These common items can be model parameters and optimizer states, or model parameters, including both completely identical and partially identical scenarios as shown in Table 1. Specifically, when the first training task under data parallelism is assigned to multiple computing cards in the first computing card group, each computing card in the first computing card group executes a sub-training task of the first training task. When each computing card executes each sub-training task, the subset of data used for training by each computing card is different, but the model parameters and optimizer states are completely identical; therefore, the common items are the model parameters and optimizer states. Specifically, the first training task under data parallelism and ZeRO-1 or data parallelism and ZeRO-2 is assigned to multiple computing cards in the first computing card group. Each computing card in the first computing card group executes a sub-training task of the first training task. When each computing card executes each sub-training task, the subset of data used for training by each computing card is different, the model parameters are the same, and the optimizer state is different. The same item is the model parameters.

[0088] There are differences among the checkpoints generated by the multiple computing cards in the second computing card group. These differences can be model parameters and optimizer states, or simply optimizer states, including two scenarios as shown in Table 1: completely different and partially the same. Specifically, in non-data parallel or data parallel and ZeRO-3 scenarios, different training tasks are assigned to the multiple computing cards in the second computing card group. Since the training tasks, input datasets, model parameters, and optimizer states of the multiple computing cards in the second computing card group are different, the differences are in the model parameters and optimizer states. Specifically, when the first training task under data parallel and ZeRO-1 or data parallel and ZeRO-2 scenarios is assigned to the multiple computing cards in the second computing card group, each computing card in the second computing card group executes a sub-training task of the first training task. When executing each sub-training task, each computing card uses a different subset of data for training, has the same model parameters, and a different optimizer state; therefore, the differences are in the optimizer state.

[0089] In one example, the checkpoints generated by the four compute cards have both identical and different items, where DP=4, ZeRO=4, and ZeRO_stage=1, as shown in Figure 2. It can be seen that the identical items are model parameters, and the different items are optimizer states. That is, the model parameters are copies of each other across the four compute cards, and the optimizer states are divided into four optimizer state fragments and distributed across the four compute cards. Therefore, when saving checkpoints, the four compute cards with identical model parameters can be grouped into one compute card group. It should be noted that Figure 2 uses a GPU as an example; this embodiment does not limit the type of compute card.

[0090] In this embodiment, for the different items included in the checkpoint generated between computing cards, in order to ensure the reliability of checkpoint storage, the different items are copied to the memory of other computing nodes through the communication link between computing nodes to form a copy of the different items. Since the probability of different computing nodes failing at the same time is low, the probability of the different items and their copies being lost at the same time is low, which can improve the reliability of the copies.

[0091] In one example, checkpoints generated by four compute cards contain both identical and different items, where DP = 4, ZeRO = 4, and ZeRO_stage = 1, as shown in Figure 3. It can be seen that the identical items are model parameters, and the different items are optimizer states. That is, the model parameters are copies of each other across the four compute cards, and the optimizer state is divided into four optimizer state fragments and distributed across the four compute cards. To ensure the reliability of checkpoint storage, the optimizer state fragments stored on each of the four compute cards are copied to another compute card to form a copy of that optimizer state fragment. For example, optimizer state fragment 1 on compute card 1 is transmitted to the shared memory of the compute node where compute card 2 is located via the communication link between compute nodes to form a copy of optimizer state fragment 1. Compute card 1 and compute card 2 can be located on different compute nodes. Optimizer state fragment 2 on compute card 2 is transmitted to the shared memory of the compute node where compute card 3 is located to form a copy of optimizer state fragment 2, and so on, storing four copies of optimizer state fragments on the four compute cards in a circular manner. The correspondence between the computing card and each shard and shard copy is recorded on each computing card. For example, the shared memory of the computing node where computing card 1 is located stores copies of optimizer state shard 1 and optimizer state shard 4, and computing card 1 is responsible for storing optimizer state shard 1. It should be noted that Figure 3 is an example using a GPU as the computing card, and this embodiment does not limit the type of computing card.

[0092] The large model training device determines the common and different items in the checkpoints generated by the multiple computing cards in the large model training system based on the parallel relationship between them, and sends the information recording the common and different items to the multiple computing cards in the large model training system. The large model training device is also used to distribute training tasks to the multiple computing cards in the large model training system according to the parallel relationship between them. The large model training device can be an AI training framework, such as MindSpore, TensorFlow, and PyTorch; this application does not limit the AI ​​training framework. In other embodiments, the AI ​​training framework generates topology information of the multiple computing cards and configures topology parameters on the multiple computing cards based on this topology information. This topology information is used to indicate the parallel relationship between the multiple computing cards and the number of computing cards in the parallel relationship. This application does not limit the method of generating the topology information of the multiple computing cards. It should be noted that the above steps are an initialization process. In some embodiments, this initialization step is performed once during the initialization process of the large model training system, instead of being performed before each iteration.

[0093] The following section further elaborates on the process of the large-scale model training method described above. This method includes a checkpoint saving process, a checkpoint fault tolerance check process, a checkpoint loading process, and a checkpoint verification process. The checkpoint saving process refers to the process by which the computing card synchronously stores the checkpoint to be saved from its video memory to the memory of the computing node, and asynchronously persists the checkpoint to be persisted from memory to the storage device. The checkpoint fault tolerance check process performs fault tolerance checks on the checkpoint saved to the storage device. The checkpoint loading process refers to the process by which the computing card loads the checkpoint from the storage device to its video memory. The checkpoint verification process verifies both the saved checkpoint and the loaded checkpoint. These four processes will be described in detail below.

[0094] First, the checkpoint saving process is described. Figure 4 is a flowchart of the checkpoint saving process in a large model training method provided in an embodiment of this application. As shown in Figure 4, the method is applied to the above-mentioned large model training system, which includes a storage device and at least one computing card. The at least one computing card is set on at least one computing node. The method includes the following steps 401 to 404.

[0095] Step 401: The first computing card writes the checkpoint generated by the first computing card from the video memory of the first computing card to the shared memory of the computing node where the first computing card is located in the original data format.

[0096] It should be noted that the computing card generates checkpoints during the training process. The computing card needs to wait for the generated checkpoints to be written from the computing card's video memory to the shared memory of the computing node where the computing card is located before it can continue training. In other words, saving the checkpoints to the shared memory of the computing node where the computing card is located is a synchronous operation, which will block model training.

[0097] In this embodiment, the checkpoint in the original data format is not serialized. The first computing card writes the unserialized checkpoint from the graphics memory of the computing card to the shared memory of the computing node where the first computing card is located, which can reduce the overhead of saving checkpoints and reduce the time of blocking training.

[0098] In one example, taking the first computing card as an example, as shown in Figure 5, the first computing card generates a checkpoint during the execution of the training task. When saving the checkpoint, the original data format is written from the video memory of the first computing card to the shared memory of the computing node where the first computing card is located, and the serialization is removed from the stage of saving the checkpoint.

[0099] Specifically, if this is the first time a checkpoint is saved, the size of the checkpoint needs to be calculated based on the training task. Based on this calculated checkpoint size, a memory space, i.e., shared memory, is initialized in the memory of the computing node where the computing card resides. In some embodiments, this shared memory reserves a portion of the memory space for users to save custom parameters. If this is not the first time a checkpoint is saved, the shared memory allocated during the first checkpoint saving is overwritten, and this shared memory will not be released until the container exits.

[0100] Specifically, when saving a checkpoint, the first computing card first parses the checkpoint's metadata. This metadata can include the shape and data type of parameters such as model parameters and optimizer states in the checkpoint. For example, the model parameters and optimizer states in the checkpoint can be tensors, where the shape can be the matrix dimension of the tensor, and the data type can indicate whether the tensor is integer, floating-point, etc. The first computing card generates checkpoint metadata based on the parsed metadata and saves the checkpoint metadata to the shared memory of the computing node where the first computing card resides. It also persistently stores the checkpoint metadata in JSON format on a storage device. This checkpoint metadata can be loaded from the shared memory for training recovery.

[0101] Furthermore, the first computing card directly copies the model parameters, optimizer states, and other parameters stored in its video memory to the shared memory of the computing node where the first computing card resides, using the raw data format of the checkpoint. This raw data format checkpoint is not serialized, thus eliminating the need for serialization. If the shared memory of the computing node where the first computing card resides is insufficient, the shared memory is recalculated and allocated according to the size of the checkpoint to store it. It should be noted that the same process applies to other computing cards as with the first computing card, and will not be elaborated further.

[0102] Thus, removing the serialization operation from the stage of saving the checkpoint to the shared memory of the computing node where the computing card is located can hide the time spent saving the checkpoint and avoid long-term blocking during training.

[0103] Step 402: The computing node where the first computing card is located writes the same item from the shared memory of the computing node where the first computing card is located to the storage device.

[0104] In this embodiment, for the same item in checkpoints generated by multiple computing cards, the computing node where the first computing card is located is selected from the first computing card group, and the same item is persistently written from the shared memory of the computing node where the first computing card is located to the storage device. This can reduce the waste of input / output I / O for persistent writing of checkpoints to the storage device, and reduce the requirements for write bandwidth and storage capacity.

[0105] In terms of the strategy for determining the first computing card, in order to ensure load balancing among computing nodes, the first computing card in multiple first computing card groups used to persistently write the same item from the shared memory of the computing node to the storage device can be set on different computing nodes.

[0106] In some embodiments, the large model training system can provide users with an extreme load balancing option. When this option is enabled, identical items generated by each compute card can be evenly distributed within a compute card group. Each compute card in the group then persistently stores a portion of the equally distributed identical item fragment to a storage device. For example, in a first compute card group, the compute node containing the first compute card writes the first identical item fragment from its shared memory to the storage device, and the compute node containing the second compute card writes the second identical item fragment from its shared memory to the storage device. The identical item includes both the first and second identical item fragments. By evenly distributing identical items within the compute card group, the storage task of identical items can be evenly allocated to each compute card in the group, ensuring that each compute card in the group evenly persists the identical item fragments to the storage device. This increases the concurrency of identical item writes and prevents a single compute card from becoming overloaded and a bottleneck.

[0107] In one example, assuming the first compute card group consists of four compute cards that produce identical checkpoints, as shown in Figure 6(a), multiple compute cards producing identical checkpoints under data parallelism have identical model parameters and optimizer states. That is, the identical items are the checkpoints produced by these multiple compute cards. For example, if the checkpoint with sequence number 0 in the shared memory of compute nodes 601 to 604 is identical, then one of the four compute cards that produced checkpoint 0 is selected as the first compute card, or Leader. For instance, the compute card that produced checkpoint 0 in compute node 601 is selected to write checkpoint 0 from the shared memory of compute node 601 to the storage device. If the checkpoint with sequence number 1 in compute nodes 601 to 604 is identical, then the compute card that produced checkpoint 1 in compute node 602 is selected from the four compute cards that produced checkpoint 1 to write checkpoint 1 from the shared memory of compute node 602 to the storage device. Similarly, only one compute card is selected from the compute cards that produced the same checkpoint to write one copy of the checkpoint to the storage device. The computing cards selected for persistently writing checkpoints to storage devices can be set on different computing nodes. For example, computing cards used to store checkpoints with sequence number 0, 1, 2, and 3 can be set on different computing nodes.

[0108] In one example, assuming the first compute card group comprises four compute cards that generate identical checkpoints, as shown in Figure 6(b), multiple compute cards generating identical checkpoints under data parallelism have identical model parameters and optimizer states; that is, the identical items are the checkpoints generated by these multiple compute cards. For example, checkpoint 0 in the shared memory of compute nodes 611 to 614 is identical. The compute card generating checkpoint 0 is assigned to a compute card group. Taking checkpoint 0 as an example, after enabling the extreme load balancing option, checkpoint 0 can be divided into four parts within this compute card group, where the number of shards is calculated based on the number of identical checkpoints included in the compute card group. The computing card in computing node 611 within the computing card group writes 1 / 4 of checkpoint0 to the storage device, the computing card in computing node 612 writes 1 / 4 of checkpoint0, and so on. Each computing card in computing nodes 611 to 614 writes 1 / 4 of checkpoint0 to the storage device, that is, writes a complete checkpoint0 to the storage device.

[0109] In one example, assuming the first compute card group comprises four compute cards that generate identical checkpoints (as shown in Figure 7(a)), the checkpoints generated by multiple compute cards in data parallelism and optimizer parallelism are identical, meaning the model parameters are the same, but the optimizer states differ. The common element is the model parameters in the checkpoints generated by these multiple compute cards. Optimizer parallelism can be either ZeRO-1 or ZeRO-2. For example, if the model parameter with index m0 in the shared memory of compute nodes 701 and 702 is the same, then one compute card is selected from the four compute cards that generated the model parameter m0 and designated as the first compute card, or Leader. For instance, the first compute card in compute node 701 is selected to write the model parameter m0 from the shared memory of compute node 701 to the storage device. The selection of the compute card is still based on load balancing principles, which will not be elaborated further here.

[0110] In one example, assuming the first compute card group comprises four compute cards that generate identical checkpoint portions, as shown in Figure 7(b), the checkpoint portions generated by multiple compute cards in data parallelism and optimizer parallelism are identical, meaning the model parameters are the same, but the optimizer states are different. The common item is the model parameters in the checkpoints generated by these multiple compute cards. Optimizer parallelism can be in the ZeRO-1 or ZeRO-2 stage. After enabling the extreme load balancing option, for the same model parameter m0, the same model parameter m0 can be evenly distributed within the compute card group. For example, the same model parameter can be evenly divided into four parts within the compute card group, with the first compute card of compute node 711 within the compute card group receiving 1 / 4 of the model parameter m0. 00 The model parameters m are written to the storage device, and then 1 / 4 of the model parameters m are transferred by the second computing card of compute node 711. 01 Write to the storage device, and so on. Each compute card in compute nodes 711 and 712 writes 1 / 4 of the model parameters to the storage device, that is, persists a complete set of model parameters m0 to the storage device.

[0111] Step 403: The computing node where the third computing card is located writes the different items of the third checkpoint from the shared memory of the computing node where the third computing card is located to the storage device.

[0112] In this embodiment, for the different items of the checkpoints generated by multiple computing cards, the third computing card in the second computing card group persistently writes the different items of the third checkpoint generated by the third computing card from the shared memory of the computing node where the third computing card is located to the storage device.

[0113] Step 404: The computing node where the fourth computing card is located writes the different items of the fourth checkpoint from the shared memory of the computing node where the fourth computing card is located to the storage device.

[0114] In this embodiment, for the different items of the checkpoints generated by multiple computing cards, the fourth computing card in the second computing card group persistently writes the different items of the fourth checkpoint generated by the fourth computing card from the shared memory of the computing node where the fourth computing card is located to the storage device.

[0115] In one example, taking the second compute card group as an example where the checkpoints generated by the four compute cards are identical, continuing to refer to Figure 7(a), the checkpoints generated by the multiple compute cards in data parallelism and optimizer parallelism are identical, that is, the model parameters are the same, but the optimizer states are different. The difference lies in the optimizer states within the checkpoints generated by these multiple compute cards. Here, optimizer parallelism can be a ZeRO-1 or ZeRO-2 stage. For example, the optimizer states generated by the compute cards in the shared memory of compute nodes 701 and 702 are different. Specifically, the first compute card of compute node 701 will have optimizer state O. 00 The copy is stored in the shared memory of the second compute card of compute node 701, and the optimizer state O generated by the second compute card of compute node 701... 01 The optimizer state shards are stored in the shared memory of the first compute card in compute node 702, and so on, in a circular fashion across the four compute cards in compute nodes 701 and 702. During persistent writes to storage, the compute cards in compute nodes 701 and 702 write their respective optimized state from the shared memory of their respective compute nodes to the storage device. For example, the first compute card in compute node 701 writes the optimizer state O... 00 Write to the storage device; the second compute card in compute node 701 will set the optimizer state to O. 01 The first compute card in compute node 702 writes the optimizer state to the storage device. 02 Write to storage device, the second compute card in compute node 702 will set the optimizer state to O 03 Write to the storage device.

[0116] It should be noted that the embodiments of this application do not limit the execution order of steps 402 to 404 above.

[0117] In some embodiments, the storage device includes a first-level storage device and a second-level storage device. Loading a checkpoint from the first-level storage device is faster than loading a checkpoint from the second-level storage device. During the process of writing a checkpoint from shared memory to the storage device, taking a first computing card as an example, after the first computing card writes the checkpoint generated by the first computing card in its original data format from its video memory to the shared memory of the computing node where the first computing card resides, the computing node where the first computing card resides stores the checkpoint in its original data format and the serialized checkpoint in the first-level storage device, respectively. The checkpoint in its original data format is not serialized. In the event of a computing node failure, the checkpoint in its original data format is loaded first from the first-level storage device, resulting in higher loading efficiency and eliminating the need for deserialization, thus improving training recovery efficiency. Meanwhile, continuing to refer to Figure 5, the serialized checkpoint is saved to the first-level storage device for loading and deserialization. Since persistent writing to the storage device is asynchronous, it does not block model training. Furthermore, after sending the original data format checkpoint and the serialized checkpoint to the first-level storage device, the first-level storage device stores the serialized checkpoint to the second-level storage device.

[0118] In the above method, the checkpoints generated by multiple computing cards under data parallelism and optimizer parallelism can have two states: identical and different. The multiple computing cards in the large model training system are grouped, and the checkpoints in the shared memory of the computing nodes where the multiple computing cards reside contain both identical and different items. For checkpoints in the first computing card group that contain identical items, only one computing card in the first group is selected to persistently write the identical item from the shared memory of its computing node to the storage device. For checkpoints in the second computing card group that contain different items, each computing card in the second group persistently writes the different items of its generated checkpoint from the shared memory of its computing node to the storage device. Having only one copy of the identical checkpoint items stored in the storage device by the computing node where only one computing card resides reduces the pressure on the storage device. This reduces the blocking of persistently writing checkpoints to storage devices and lowers the waste of write bandwidth and storage resources. Furthermore, storing copies of different items of the generated checkpoints on other compute cards improves the reliability of the copies. In addition, based on load balancing, the same items are evenly distributed within the compute card group, which can distribute the storage tasks of the same items to each compute card in the group. This ensures that each compute card in the group writes the same items fragments persistently to the storage device in a balanced manner, which helps to increase the concurrency of writing the same items and avoids the bottleneck of a single compute card being overloaded.

[0119] The checkpoint fault tolerance process is described below. Figure 8 is a flowchart of the checkpoint fault tolerance process in a large model training method provided in an embodiment of this application. As shown in Figure 8, the method is applied to the large model training system described above. The large model training system includes a storage device and at least one computing card. The at least one computing card is located at at least one computing node. The method includes the following steps 801 to 803.

[0120] Step 801: The first computing card sends its card status information to the second computing card.

[0121] In this embodiment, for the first computing card in the first computing card group, the first computing card persistently writes the same item from the shared memory of the computing node where the first computing card is located to the storage device. During the process of the first computing card persistently writing the same item from the shared memory of the computing node where the first computing card is located to the storage device, card status information is continuously sent to other computing cards included in the first computing card group, such as to the second computing card. The card status information is used to indicate the health status of the first computing card.

[0122] It should be noted that the card status information can be heartbeat information. For example, the first computing card periodically sends a custom heartbeat packet or heartbeat frame to the second computing card to determine that the first computing card has not malfunctioned. This card status information can be sent from the first computing card to the second computing card to determine the health status of the first computing card, or it can be sent to all computing cards in the first computing card group except the first computing card to determine the health status of the first computing card. This embodiment does not limit this.

[0123] Step 802: The second computing card receives the card status information from the first computing card.

[0124] In this embodiment, the second computing card receives the card status information of the first computing card to determine whether the first computing card has malfunctioned.

[0125] Step 803: In the event of a failure of the first computing card, or in the event that the storage device does not store the same item, the computing node where the second computing card is located writes the same item from the shared memory of the computing node where the second computing card is located to the storage device.

[0126] In this embodiment, after the second computing card determines that the first computing card has failed based on the card status information sent by the first computing card, it elects another computing card as the leader from the first computing card group. This is done by the second computing card writing the same item from the shared memory of the computing node where it resides to the storage device. Alternatively, the computing cards in the first computing card group other than the first computing card will check whether the same item has been successfully persisted to the storage device after a preset time. If the storage device does not store the same item three times consecutively, another computing card is elected as the leader from the first computing card group. This is done by the second computing card writing the same item from the shared memory of the computing node where it resides to the storage device. It should be noted that the election strategy for electing another computing card as the leader from the first computing card group can be a distributed election algorithm, such as the Bully algorithm, Raft algorithm, or ZRB algorithm. This embodiment does not limit this.

[0127] In this embodiment, before the third and fourth computing cards in the second computing card group write the different items of the third checkpoint and the different items of the fourth checkpoint from the shared memory of the computing node where the third computing card is located and the computing node where the fourth computing card is located to the storage device, a communication connection is established between the third computing card and the fourth computing card. The third computing card transmits the file name and metadata information of the different items of the third checkpoint to the fourth computing card. The shared memory of the fourth computing card stores a copy of the different items of the third checkpoint. During the persistent writing to the storage device, the third computing card continuously sends its card status information to the fourth computing card. After receiving the card status information of the third computing card, the fourth computing card determines that the third computing card is faulty, or checks whether the different items of the third checkpoint are stored in the storage device after a preset time. If the different items of the third checkpoint are not stored in the storage device, the fourth computing card writes a copy of the different items of the third checkpoint from the shared memory of the computing node where the fourth computing card is located to the storage device.

[0128] In this embodiment, the fault-tolerance mechanism for writing the first identical item fragment and the second identical item fragment from the shared memory of the computing node where the first computing card is located and the computing node where the second computing card is located to the storage device with the difference item of the third checkpoint mentioned above will not be described in detail here.

[0129] In the above method, the checkpoints generated by multiple computing cards under data parallelism and optimizer parallelism can have two states: the same and different. The multiple computing cards of the large model training system are grouped, and the checkpoints in the shared memory of the computing nodes where the multiple computing cards are located have the same items and different items. If multiple compute cards in the first compute card group generate checkpoints with identical entries, only one compute card in the first compute card group is selected to persistently write the identical entry from the shared memory of the compute node where the compute card resides to the storage device. The other compute cards in the first compute card group are used to check whether the identical entry has been successfully written to the storage device. If the storage device does not store the identical entry, a compute card other than the first compute card is re-elected in the first compute card group to persistently write the identical entry to the storage device. If multiple compute cards in the second compute card group generate checkpoints with different entries, each compute card in the second compute card group persistently writes the different entries of the generated checkpoint from the shared memory of the compute node where the compute card resides to the storage device. The compute card storing a copy of the different entry is used to check whether the different entry has been successfully written to the storage device. If the storage device does not store the different entry, the compute card storing a copy of the different entry persistently writes the copy of the different entry to the storage device. Therefore, this fault-tolerance check mechanism can avoid the possibility of checkpoint loss due to persistent write failure to the storage device, and ensure the reliability of persistent write of checkpoint to the storage device.

[0130] The following describes the checkpoint loading process using a checkpoint loading scenario involving compute node failure as an example. Figure 9 is a flowchart of the checkpoint loading process in a large model training method provided in an embodiment of this application. As shown in Figure 9, the method is applied to the aforementioned large model training system, which includes a storage device and at least one compute card. The at least one compute card is located on at least one compute node. The method includes the following steps 901 to 906 and steps 1101 to 1106.

[0131] First, the loading process of the same item from multiple checkpoints generated by the computing cards is described, which includes the following steps 901 to 906.

[0132] Step 901: The first computing card executes the first sub-training task and generates the first checkpoint.

[0133] In this embodiment, the large model training device distributes the first training task, which is either data-parallel or data-parallel and optimizer-parallel, to multiple computing cards included in the first computing card group. The first computing card in the first computing card group executes the target iteration of the first sub-training task and generates a first checkpoint. The target iteration is any iteration of the first sub-training task.

[0134] Step 902: The second computing card executes the second sub-training task and generates the second checkpoint.

[0135] In this embodiment, the large model training device assigns the first training task, which is either data parallel or data parallel and optimizer parallel, to multiple computing cards included in the first computing card group. The second computing card in the first computing card group performs the target iteration of the second sub-training task and generates a second checkpoint.

[0136] It should be noted that, as seen from the checkpoint saving process described above, the first checkpoint and the second checkpoint can have two states: the model parameters and the optimizer state can be the same or different. Specifically, in the first computation card group under data parallelism, the first checkpoint generated by the first computation card and the second checkpoint generated by the second computation card share the same items: model parameters and optimizer state. In the first computation card group under both data parallelism and optimizer parallelism, the first checkpoint generated by the first computation card and the second checkpoint generated by the second computation card share the same item: model parameters. Optimizer parallelism can be either the ZeRO-1 or ZeRO-2 stage.

[0137] Step 903: First computing card, load the same item from the shared memory of the computing node where the first computing card is located.

[0138] In this embodiment, it should be noted that step 903 is described using the example of a failure in the computing node where the first computing card resides. In some embodiments, there may be two or more computing node failures. This embodiment only describes a single computing node failure as an example. The situation of multiple computing node failures is similar to that of a single computing node failure. When a failure occurs, if the identical entries of the first checkpoint and the second checkpoint stored in the shared memory of the computing node where the first computing card resides are not lost, the failure is considered a process-level failure. In this case, the first computing card can load the identical entry from the shared memory of the computing node where the first computing card resides.

[0139] Furthermore, the first computing card can load the same items in the original data format from the shared memory of the computing node where the first computing card is located, thereby saving the time consumed by the deserialization operation and improving the training recovery speed.

[0140] Step 904: If the same item is not stored in the shared memory of the computing node where the first computing card is located, the second computing card loads the same item from the shared memory of the computing node where the second computing card is located and sends the same item to the first computing card.

[0141] In this embodiment, when a fault occurs, as long as the computing nodes where all computing cards within the same computing card group reside fail simultaneously, the checkpoint in memory can be recovered via memory broadcast. That is, the computing nodes where the computing cards within the same computing card group reside still have usable checkpoints. Specifically, if the shared memory of the computing node where the first computing card resides does not store the same item, indicating a task-level fault, the second computing card, acting as the originator, can load the same item from its shared memory and send it to the receiving end, such as the first computing card, for the first computing card to recover training for the first sub-training task.

[0142] Optionally, when a computing node in the first computing card group fails, the originating point for broadcasting can be determined based on the distance. The computing node containing the nearest computing card broadcasts the same item to the computing node that has restarted after the failure. If the nearest computing node cannot obtain the item, the computing node that is farther away broadcasts the same item to the computing node that has restarted after the failure.

[0143] It should be noted that the first computing card needs to load the metadata of the identical item from the storage device. This metadata includes the shape, data type, etc., of the identical item. Based on the loaded metadata of the identical item, the first computing card assigns a value to the received identical item using the shape in the metadata and sets the data type to the data type indicated in the metadata, thereby obtaining the identical item that can be used for training recovery.

[0144] In one example, taking compute node 1001 as the replacement compute node after a failure, and compute node 1002 having the same item in its shared memory, as shown in Figure 10(a), after compute node 1001 is replaced by compute card 2 in compute node 1002, the same item can be loaded from shared memory. Compute node 1001 loads the metadata of the same item from the storage device, constructs the format of the same item, and compute node 1002 broadcasts the loaded same item to compute node 1001. Compute node 1001 constructs the same item for training recovery based on the metadata of the same item.

[0145] Step 905: In response to the first training recovery command, the first computing card loads the same items of the first checkpoint and the second checkpoint from the storage device, recovers the first sub-training task based on the same items, and sends the same items to the second computing card.

[0146] In this embodiment, when all computing nodes in the first computing card group fail simultaneously, and the identical items stored in the shared memory of all computing nodes in the computing card group are lost, a computing card in the first computing card group is selected as the originating point, such as the first computing card. The first computing card loads the identical item from the storage device, recovers the first sub-training task based on the identical item, and broadcasts the identical item to the computing cards in the first computing card group, such as sending the identical item to the second computing card.

[0147] Furthermore, the first computing card can preferentially load identical items in the original data format from the first-level storage device to save time spent on deserialization operations and improve training recovery speed. Here, the example of the first computing card loading identical items from the storage device is used; the first computing card can be any computing card in the first computing card group. This example focuses on the first computing card and does not impose any limitations on it.

[0148] Optionally, selecting computing cards that load the same items from storage devices in the first computing card group can be based on the principle of load balancing to avoid computing cards that load checkpoints from storage devices in multiple computing card groups being set on the same computing node, which would result in a large load on that computing node.

[0149] Step 906: The second computing card receives the same item and recovers the second sub-training task based on the same item.

[0150] In this embodiment, the second computing card receives the same item and recovers the second sub-training task based on the same item.

[0151] It should be noted that the second computing card needs to load the metadata of the same item from the storage device. Based on the loaded metadata of the same item, it assigns a value to the received same item using the shape in the metadata and sets the data type to the data type indicated in the metadata, thereby obtaining the same item that can be used for training recovery.

[0152] In one example, compute nodes 1011 and 1012 are used as replacement compute nodes after a failure, as shown in Figure 10(b). Compute card 1 of compute node 1011 loads the same item from the storage device and sends it to compute card 2 in compute node 1012. Compute card 2 in compute node 1012 loads the metadata of the same item from the storage device and constructs a similar item for training recovery based on the received similar item's metadata.

[0153] In one example, assuming the first compute card group comprises four compute cards that produce identical checkpoints, as shown in Figure 11(a), multiple compute cards producing identical checkpoints under data parallelism have identical model parameters and optimizer states; that is, the identical items are the checkpoints produced by these multiple compute cards. For example, if the first compute card in compute nodes 1101 to 1104 all produces the identical checkpoint with sequence number 0, the compute card producing the checkpoint with sequence number 0 is assigned to a compute card group. When compute nodes 1101 through 1104 fail simultaneously, all shared memory stored by the compute nodes is lost. Then, one of the four compute cards that generated checkpoint 0 is selected as the first compute card, or the originator, and the other three compute cards in the group are the receivers. For example, the first compute card in compute node 1101 that generated checkpoint 0 loads checkpoint 0 from the storage device and broadcasts it to the other three compute cards in the group. Similarly, among the four compute cards that generated checkpoint 1, one of the compute cards in compute node 1102 that generated checkpoint 1 loads checkpoint 1 from the storage device and broadcasts it to the other three compute cards in the group. This process continues until compute nodes 1101 through 1104 fail simultaneously. Then, one of the compute cards that generated the same checkpoint loads the checkpoint from the storage device. Among them, the computing cards selected for loading checkpoints from storage devices can be set on different computing nodes. For example, the computing cards used to load checkpoints with sequence number 0, 1, 2, and 3 can be set on different computing nodes.

[0154] In one example, assuming the first compute card group comprises four compute cards that generate the same checkpoint portion, as shown in Figure 11(b). In cases of data parallelism and optimizer parallelism, multiple compute cards generate the same checkpoint portion, where the model parameters are identical but the optimizer states differ; that is, the identical item is the model parameter. For example, compute nodes 1111 and 1112 comprise four compute cards that all generate the same model parameter with the sequence number m0. Compute cards generating the same model parameter m0 are grouped into a compute card group. When compute nodes 1111 and 1112 fail simultaneously, one compute card is selected from the four that generated model parameter m0 and designated as the first compute card, or initiator. For example, the first compute card in compute node 1111 might load model parameter m0 from the storage device and broadcast it to the other three compute cards in the group.

[0155] In some embodiments, during checkpoint loading, the large model training system can provide users with an extreme load balancing option. When this option is enabled, identical items generated by each compute card can be evenly distributed within a compute card group. Each compute card in this group loads a portion of the evenly distributed identical item fragment from the storage device to obtain the complete identical item. For example, in response to a first training recovery command, the first compute card in the first compute card group loads a first identical item fragment from the storage device and sends it to the second compute card; the second compute card, in response to the first training recovery command, loads a second identical item fragment from the storage device and sends it to the first compute card. This identical item includes both the first and second identical item fragments. Thus, the first compute card recovers the first sub-training task based on the first and second identical item fragments, and the second compute card recovers the second sub-training task based on both fragments. By distributing identical items equally within a compute card group, and having each compute card in the group load a fragment of the identical item from the storage device, the loading task of identical items can be evenly distributed to each compute card in the group. This ensures that each compute card in the group loads the fragment of the identical item from the storage device in a balanced manner to obtain the complete identical item. This helps to increase the concurrency of reading identical items and avoids the bottleneck of a single compute card being overloaded.

[0156] In one example, assuming the first compute card group contains two compute cards that produce identical checkpoints, as shown in Figure 12(a), multiple compute cards producing identical checkpoints under data parallelism have identical model parameters and optimizer states; that is, the identical items are the checkpoints produced by these multiple compute cards. For example, the first compute cards in compute nodes 1201 and 1202 both produce identical checkpoints with sequence number 0. The compute cards that produce checkpoints with sequence number 0 are grouped into one compute card group. When both compute nodes 1201 and 1202 fail simultaneously, taking checkpoint 0 (serial number 0) as an example, after enabling the extreme load balancing option, checkpoint 0 can be divided into two parts within the compute card group. The compute card of compute node 1201 in this compute card group loads half of checkpoint 0 from the storage device and broadcasts this half of checkpoint 0 to the compute card of compute node 1202 in the compute card group. The compute card of compute node 1202 in this compute card group loads the other half of checkpoint 0 from the storage device and broadcasts this half of checkpoint 0 to compute node 1201 in the compute card group. In this way, a complete checkpoint 0 can be loaded from the storage device, and through broadcasting, the compute cards included in compute nodes 1201 and 1202 can perform training recovery based on a complete checkpoint 0.

[0157] In one example, assuming the first compute card group includes two compute cards that produce the same checkpoint portion, as shown in Figure 12(b), multiple compute cards in data parallelism and optimizer parallelism produce the same checkpoint portion. The model parameters are the same, but the optimizer states are different; that is, the identical item is the model parameter. For example, compute nodes 1211 and 1212 include four compute cards that all produce the same model parameter with the same sequence number m0. Compute cards producing the same model parameter with sequence number m0 are grouped into a compute card group. After enabling the extreme load balancing option, when compute nodes 1211 and 1212 fail simultaneously, the model parameter can be divided into four equal parts within the group. The first compute card of compute node 1211 in this compute card group loads 1 / 4 of the model parameter m from the storage device. 00 And the 1 / 4 model parameter m 00 Broadcast to the other three compute cards in the compute card group, the second compute card of compute node 1211 loads 1 / 4 of the model parameters m from the storage device. 01 And the 1 / 4 model parameter m 01The data is broadcast to the other three computing cards in the computing card group, and so on. The two computing cards in computing node 1212 each load 1 / 4 of the model parameters m0 from the storage device to obtain a complete set of model parameters m0. The data is then broadcast to enable the computing cards in computing nodes 1211 and 1212 to perform training recovery based on the complete set of model parameters m0.

[0158] Furthermore, the loading process of different items of checkpoints generated by multiple computing cards is described, as shown in Figure 13. The loading process of these different items includes the following steps 1301 to 1306.

[0159] Step 1301: The third computing card performs the second training task and generates the third checkpoint.

[0160] In this embodiment, the checkpoints generated by the multiple computing cards in the second computing card group have different items. The large model training device assigns the second training task to the third computing card and the third training task to the fourth computing card. Specifically, under non-data parallelism, the second and third training tasks are two training tasks under non-data parallelism, and the input datasets of the third and fourth computing cards are different, resulting in different checkpoints. Under data parallelism and optimizer parallelism, the second and third training tasks are two sub-training tasks of one training task, and the input data subsets of the third and fourth computing cards are different, resulting in different items in the model parameters and optimizer states included in the generated checkpoints. It should be noted that under data parallelism and ZeRO-1 and data parallelism and ZeRO-2, the different items in the generated checkpoints are the optimizer states; under non-data parallelism and data parallelism and ZeRO-3, the different items in the generated checkpoints are the model parameters and optimizer states.

[0161] Step 1302: The fourth computing card performs the third training task and generates the fourth checkpoint.

[0162] Step 1303: Third computing card, load different items of the third checkpoint from the shared memory of the computing node where the third computing card is located.

[0163] In this embodiment, when a fault occurs, if the different items of the third checkpoint stored in the shared memory of the computing node where the third computing card is located are not lost, that is, the fault is a process-level fault, the third computing card can load the different items of the third checkpoint from the shared memory of the computing node where the third computing card is located.

[0164] Step 1304: Fourth computing card, if the shared memory of the computing node where the third computing card is located does not store the different items of the third checkpoint, loads a copy of the different items of the third checkpoint from the shared memory of the computing node where the fourth computing card is located, and sends the copy of the different items of the third checkpoint to the third computing card.

[0165] In this embodiment, the shared memory of the fourth computing card stores copies of the different items of the third checkpoint. When a fault occurs, if the shared memory of the computing node where the third computing card resides does not store the different items of the third checkpoint, the fourth computing card loads copies of the different items of the third checkpoint from the shared memory of the computing node where the fourth computing card resides, and sends the copies of the different items of the third checkpoint to the third computing card for training recovery of the second training task.

[0166] Step 1305: In response to the second training recovery command, the third computing card loads the different items of the third checkpoint from the storage device and recovers the second training task based on the different items of the third checkpoint.

[0167] Step 1306: In response to the second training recovery instruction, the fourth computing card loads the different items of the fourth checkpoint from the storage device and recovers the third training task based on the different items of the fourth checkpoint.

[0168] In this embodiment, when all computing nodes in the second computing card group fail simultaneously, and all the different items stored on all computing cards in the group are lost, the third computing card, in response to the second training recovery command, loads the different items of the third checkpoint from the storage device and recovers the second training task based on the different items of the third checkpoint. The fourth computing card, in response to the second training recovery command, loads the different items of the fourth checkpoint from the storage device and recovers the third training task based on the different items of the fourth checkpoint. That is, it is not necessary to select a computing card in the second computing card group to load different items via broadcast.

[0169] In one example, taking the second compute card group as an example where the checkpoints generated by the four compute cards are identical, continuing to refer to Figure 11(b), the checkpoints generated by the multiple compute cards in data parallelism and optimizer parallelism are identical, where the model parameters are the same but the optimizer states are different; that is, the different items are the optimizer states. For example, the optimizer states generated by the compute cards included in compute node 1111 and compute node 1112 are different, where the first compute card of compute node 1111 sets the optimizer state to O. 00 The copy is stored in the shared memory of the second compute card of compute node 1111, and the optimizer state O generated by the second compute card... 01 The optimizer state shards are stored in the shared memory of the first compute card of compute node 1112, and so on, in a ring configuration across the four compute cards included in compute nodes 1111 and 1112. When compute nodes 1111 and 1112 fail simultaneously, the first compute card of compute node 1111 loads the optimizer state shards from the storage device. 00 The second compute card of compute node 1111 loads the optimizer state O from the storage device. 01 The first compute card of compute node 1112 loads the optimizer state O from the storage device. 02 The second compute card of compute node 1212 loads the optimizer state O from the storage device. 03 .

[0170] It should be noted that after the training task resumes, the large model training system performs the checkpoint saving and fault tolerance check process according to the process shown in Figures 4 and 8 above, which will not be described again.

[0171] In the above method, the checkpoints generated by multiple computing cards under data parallelism and optimizer parallelism can be in two states: the same and different. The multiple computing cards of the large model training system are grouped together, and the checkpoints in the shared memory of the computing nodes where the multiple computing cards are located have the same items and different items. When a failure occurs and training needs to be resumed, if checkpoints generated by multiple computing cards in the first computing card group contain identical items, only one computing card in the first computing card group is selected to load the identical item from the storage device, and then broadcast to the other computing cards in the first computing card group for training resumption. If checkpoints generated by multiple computing cards in the second computing card group contain different items, each computing card in the second computing card group loads its own different item from the storage device for training resumption. This reduces the bandwidth required to load checkpoints from the storage device during training resumption, thereby speeding up the training resumption process and avoiding the bottleneck caused by all computing cards in the group simultaneously loading identical items from the storage device, which would affect the training resumption time. In addition, by evenly distributing identical items within the computing card group, and having each computing card in the computing card group load a fragment of the identical item from the storage device, the loading task of identical items can be evenly distributed to each computing card in the group. This ensures that each computing card in the group loads the fragment of the identical item from the storage device in a balanced manner to obtain the complete identical item, which helps to reduce the bandwidth required to load identical items and increase the concurrency of reading identical items.

[0172] In addition, the above-mentioned large model training method also includes a checkpoint verification process. Both the checkpoint saving process and the checkpoint loading process need to verify the saved checkpoint and the loaded checkpoint. Figure 14 is a flowchart of the checkpoint verification process in a large model training method provided in an embodiment of this application. As shown in Figure 14, it is applied to the above-mentioned large model training system, which includes a storage device and at least one computing card. The at least one computing card is set on at least one computing node. The method includes the following steps 1401 to 1402.

[0173] Step 1401: After the checkpoint is written from the shared memory of the compute node where the compute card is located to the storage device, the checkpoint is verified against the checkpoint generated in the second iteration, or against the default checkpoint.

[0174] In this embodiment, after steps 402, 403, and 404, the integrity of the checkpoint written to the storage device needs to be verified. Specifically, if this write to the storage device is not the first save, the verification is performed based on the checkpoint generated in the second iteration, where the second iteration is the iteration before the target iteration; if this write to the storage device is the first save, the verification is performed based on the default checkpoint, where the default checkpoint can be configured by the aforementioned large model training system or the user.

[0175] It should be noted that the above comparison and verification is based on the file size of the checkpoint. Since the checkpoint generated in each iteration of the training task contains model parameters and optimizer states with the same file size, and only the values ​​of the model parameters and optimizer states are different, the integrity of the checkpoint can be verified based on the file size of the checkpoint.

[0176] Furthermore, during the checkpoint fault tolerance check process, the computing cards in the first computing card group other than the first computing card, such as the second computing card, or the fourth computing card in the second computing card group, check in the storage device after a preset time whether the same or different items included in the checkpoint have been successfully persisted to the storage device. After successfully writing the same or different items included in the checkpoint to the storage device, the checkpoint written to the storage device will be compared and verified with the checkpoint generated in the second iteration or the default checkpoint to ensure that the checkpoint saved each time is consistent with the checkpoints between different computing cards.

[0177] Step 1402: After loading the checkpoint from the storage device, verify the checkpoint against the checkpoint generated in the third iteration.

[0178] After steps 903, 904, and 1103 to 1106, the checkpoint loaded from the storage device needs to be compared and verified. Specifically, the checkpoint loaded from the storage device is compared and verified with the checkpoint generated in the third iteration, which is the iteration before the target iteration, to ensure that the recovery is based on the latest and complete checkpoint loaded, thereby ensuring the correctness of the training recovery.

[0179] Furthermore, during the training of large models, a daemon process is started through a safe exit mechanism. During the lifecycle of the container hosting the training task, if a failure occurs, after steps 402, 403, and 404, the daemon process checks whether the checkpoint has been successfully persisted to the storage device. If the write to the storage device is incomplete, it waits for the checkpoint to be written to the storage device. This daemon process ensures that the checkpoint is successfully persisted to the storage device even after a failure, preventing incomplete writes of the checkpoint to the storage device due to a failure.

[0180] It should be noted that the embodiments of this application do not limit the execution order of steps 1401 and 1042 described above. Furthermore, the checkpoint in the above checkpoint verification process includes identical and different items from the checkpoint saving process, checkpoint fault tolerance check process, and checkpoint loading process.

[0181] In the above method, the checkpoint integrity verification mechanism can ensure the consistency of checkpoints between different iterations, and the daemon process implements a safe exit mechanism to ensure that the checkpoint is complete and successfully persisted to the storage device, thereby ensuring the consistency of checkpoints between shared memory and storage devices.

[0182] It should be noted that the embodiments shown in Figures 6, 7, 11 and 12(a) above are all illustrated with the example of four computing cards storing the same item. In some embodiments, the number of computing cards storing the same item may be less than four or more. The embodiments of this application do not limit the number of computing cards storing the same item. In addition, the embodiments shown in Figures 6, 11(a) and 12(a) above are all illustrated with the example of four computing card groups, each computing card group including four computing cards. In one embodiment, the large model training system may include two or more computing card groups, each of which includes two or more computing cards. This application does not limit the number or size of the computing card groups. Furthermore, the embodiments shown in Figures 6, 11(a), and 12(a) are illustrated using the example of four computing cards in the first computing card group storing checkpoint 0 and four computing cards in the second computing card group storing checkpoint 1. In some embodiments, the large model training system may include other computing card groups besides the first and second computing card groups; in this case, the computing card responsible for storing checkpoint 0 constitutes one computing card group. In summary, this application's embodiments may include scenarios where two or more computing card groups, each including two or more computing cards, are present. The checkpoint saving process, checkpoint fault-tolerant checking process, and checkpoint loading process in this scenario are the same as those shown in Figures 6, 11(a), and 12(a), and will not be repeated here.

[0183] Figure 15 is a schematic diagram of a large model training device 1500 provided in an embodiment of this application. The device is applied to a large model training system, which includes a storage device and at least one computing card. The at least one computing card is disposed on at least one computing node. The at least one computing card includes a first computing card group for executing a first training task. The first training task includes a first sub-training task and a second sub-training task. The first computing card group includes a first computing card and a second computing card. The device includes an allocation module 1501, a determination module 1502, and a sending module 1503.

[0184] The allocation module 1501 allocates the first sub-training task to the first computing card. The first sub-training task is used to instruct the first computing card to execute the first sub-training task and generate a first checkpoint. The first checkpoint includes the model parameters and optimizer state of the target iteration of the first sub-training task.

[0185] The allocation module 1501 allocates the second sub-training task to the second computing card. The second sub-training task is used to instruct the second computing card to execute the first sub-training task and generate a second checkpoint. The second checkpoint includes the model parameters and optimizer state of the target iteration of the second sub-training task.

[0186] The determining module 1502, in response to a first training recovery instruction, determines the same items of the first checkpoint and the second checkpoint, wherein the first training recovery instruction is used to instruct training recovery to be performed according to the target iteration;

[0187] The sending module 1503 sends information about the same items to the first computing card, so that the first computing card loads the same items from the storage device based on the information and sends the same items to the second computing card.

[0188] The allocation module 1501, the determination module 1502, and the sending module 1503 can all be implemented in software or in hardware. For example, the implementation of the determination module 1502 will be described below. Similarly, the implementation of the allocation module 1501 and the sending module 1503 can refer to the implementation of the determination module 1502.

[0189] As an example of a software functional unit, module 1502 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 1502 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0190] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0191] As an example of a hardware functional unit, module 1502 may include at least one computing device, such as a server. Alternatively, module 1502 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0192] The multiple computing devices included in module 1502 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in module 1502 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in module 1502 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0193] It should be noted that, in other embodiments, the determining module 1502 can be used to execute any step in the large model training method, and the allocation module 1501 and the sending module 1503 can be used to execute any step in the large model training method. The steps implemented by the allocation module 1501, the determining module 1502, and the sending module 1503 can be specified as needed. By implementing different steps in the large model training method through the allocation module 1501, the determining module 1502, and the sending module 1503, all the functions of the large model training device shown in FIG15 can be realized.

[0194] This application also provides a computing device 1600. Figure 16 is a schematic diagram of the structure of a computing device provided in an embodiment of this application. As shown in Figure 16, the computing device 1600 includes: a bus 1601, a processor 1602, a memory 1603, and a communication interface 1604. The processor 1602, the memory 1603, and the communication interface 1604 communicate with each other via the bus 1601. Exemplarily, the processor 1603 may be the CPU1 shown in Figure 1 for running a large model training device, and the memory 1603 may be the DRAM of the computing node where the CPU1 is located. The computing device 1600 may be a computing device or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1600.

[0195] Bus 1601 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 16, but this does not imply that there is only one bus or one type of bus. Bus 1601 can include pathways for transmitting information between various components of computing device 1600 (e.g., memory 1603, processor 1602, communication interface 1604).

[0196] Processor 1602 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0197] Memory 1603 may include volatile memory, such as random access memory (RAM). Memory 1603 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0198] The memory 1603 stores executable program code, which the processor 1602 executes to implement the functions of the aforementioned allocation module 1501, determination module 1502, and sending module 1503, thereby implementing the steps in the large model training method. In other words, the memory 1603 stores instructions for executing the large model training method.

[0199] The communication interface 1604 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 1600 and other devices or communication networks.

[0200] This application also provides a computing device cluster. Figure 17 is a schematic diagram of a computing device cluster provided in this application embodiment. As shown in Figure 17, the computing device cluster includes at least one computing device 1600. The memory 1603 of one or more computing devices 1600 in the computing device cluster may store the same instructions for executing large model training methods.

[0201] In some possible implementations, the memory 1603 of one or more computing devices 1600 in the computing device cluster may also store partial instructions for executing large model training methods. In other words, a combination of one or more computing devices 1600 can jointly execute instructions for executing large model training methods.

[0202] It should be noted that the memory 1603 in different computing devices 1600 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the large model training device. That is, the instructions stored in the memory 1603 of different computing devices 1600 can implement the functions of one or more of the aforementioned allocation module 1501, determination module 1502, and sending module 1503.

[0203] It should be understood that the functions of the computing device 1600 shown in Figure 17 can also be performed by multiple computing devices 1600.

[0204] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 18 illustrates one possible implementation. Figure 18 is a schematic diagram of a possible implementation of a computing device cluster provided in an embodiment of this application. As shown in Figure 18, computing devices 1600A and 1600B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1603 in computing device 1600A stores instructions for executing the function of the determination module 1502. Figure 18 uses the example of the memory 1603 in computing device 1600A storing instructions for executing the function of the determination module 1502. Simultaneously, the memory 1603 in computing device 1600B stores instructions for executing the functions of the allocation module 1501 and the sending module 1503. Figure 18 uses the example of the memory 1603 in computing device 1600B storing instructions for executing the functions of the allocation module 1501 and the sending module 1503.

[0205] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method in Figure 17 or Figure 18. The difference is that the memory 1603 in one or more computing devices 1600 in this computing device cluster can store the same instructions for executing large model training methods.

[0206] In some possible implementations, the memory 1603 of one or more computing devices 1600 in the computing device cluster may also store partial instructions for executing large model training methods. In other words, a combination of one or more computing devices 1600 can jointly execute instructions for executing large model training methods.

[0207] It should be noted that the memory 1603 in different computing devices 1600 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the large model training device. That is, the instructions stored in the memory 1603 of different computing devices 1600 can implement the functions of one or more of the aforementioned allocation module 1501, determination module 1502, and sending module 1503.

[0208] This application also provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform a large model training method provided by any possible implementation of the foregoing method embodiments.

[0209] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that direct the computing device to perform a large model training method.

[0210] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the topology information involved in this application was obtained with full authorization.

[0211] Those skilled in the art will recognize that the method steps and units described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0212] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, apparatuses, or units, or they may be electrical, mechanical, or other forms of connection.

[0214] The unit described as a separate component may or may not be physically separate. The component shown as a unit may or may not be a physical unit; that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0215] Furthermore, the units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or software.

[0216] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computing device (which may be a personal computer, server, or computing device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the scope of various examples, a first computing card can be referred to as a second computing card, and similarly, a second computing card can be referred to as a first computing card. Both the first and second computing cards can be node computing cards, and in some cases, they can be separate and different computing cards.

[0218] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. The terms "system" and "network" are often used interchangeably.

[0219] It should also be understood that the term “if” can be interpreted as meaning “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrase “if determination…” or “if detection [the stated condition or event]” can be interpreted as meaning “when determination…”, or “in response to determination…”, or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0220] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0221] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer program instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0222] The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer program instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs, DVDs), or semiconductor media (e.g., solid-state drives).

[0223] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0224] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A large model training system, characterized in that, The large model training system includes a storage device and at least one computing card. The at least one computing card is disposed on at least one computing node. The at least one computing card includes a first computing card group for executing a first training task. The first training task includes a first sub-training task and a second sub-training task. The first computing card group includes a first computing card and a second computing card. The first computing card is used to execute the first sub-training task and generate a first checkpoint, the first checkpoint including the model parameters and optimizer state of the target iteration of the first sub-training task; The second computing card is used to execute the second sub-training task and generate a second checkpoint, the second checkpoint including the model parameters and optimizer state of the target iteration of the second sub-training task; The first computing card is further configured to, in response to a first training recovery instruction, load identical items from the storage device for the first checkpoint and the second checkpoint, recover the first sub-training task based on the identical items, and send the identical items to the second computing card, wherein the first training recovery instruction is used to instruct training recovery to be performed according to the target iteration; The second computing card is also used to receive the identical items and recover the second sub-training task based on the identical items.

2. The system according to claim 1, characterized in that, The at least one computing card also includes a third computing card and a fourth computing card belonging to the second computing card group; The third computing card is used to execute the second training task and generate a third checkpoint. The fourth computing card is used to execute the third training task and generate a fourth checkpoint. The model parameters and optimizer states included in the fourth checkpoint are different from those included in the third checkpoint. The third computing card is also configured to, in response to the second training recovery instruction, load different items of the third checkpoint from the storage device, and recover the second training task based on the different items of the third checkpoint; The fourth computing card is further configured to, in response to the second training recovery instruction, load different items of the fourth checkpoint from the storage device, and recover the third training task based on the different items of the fourth checkpoint.

3. The system according to claim 1, characterized in that, Each compute node comprises multiple compute cards that share memory. The first computing card is further configured to load the same item from the shared memory of the computing node where the first computing card resides before loading the same item from the storage device; The second computing card is further configured to, before loading the identical item from the storage device and in the case that the identical item is not stored in the shared memory of the computing node where the second computing card is located, load the identical item from the shared memory of the computing node where the first computing card is located, and send the identical item to the first computing card.

4. The system according to claim 2, characterized in that, The computing node on the fourth computing card stores copies of different items from the third checkpoint; The third computing card is further configured to load different items of the third checkpoint from the shared memory of the computing node where the third computing card resides before loading the different items of the third checkpoint from the storage device; The fourth computing card is further configured to, before loading the different items of the third checkpoint from the storage device, and in the case that the different items of the third checkpoint are not stored in the shared memory of the computing node where the fourth computing card is located, load a copy of the different items of the third checkpoint from the shared memory of the computing node where the fourth computing card is located, and send the copy of the different items of the third checkpoint to the third computing card.

5. The system according to claim 1, characterized in that, The identical items include the first identical item fragment and the second identical item fragment; The first computing card is further configured to, in response to the first training recovery instruction, load the first identical item fragment from the storage device and send the first identical item fragment to the second computing card; The second computing card is further configured to, in response to the first training recovery instruction, load the second identical item fragment from the storage device and send the second identical item fragment to the first computing card; The first computing card is also used to recover the first sub-training task based on the first identical item segment and the second identical item segment; The second computing card is also used to recover the second sub-training task based on the first identical item segment and the second identical item segment.

6. The system according to any one of claims 1 to 5, characterized in that, Each computing card includes video memory, and the first computing card is further used for: The checkpoint generated by the first computing card is written from the video memory of the first computing card to the shared memory of the computing node where the first computing card is located in the original data format. The checkpoint in the original data format is not serialized.

7. The system according to any one of claims 1 to 6, characterized in that... ； The first computing card and the second computing card are used to determine that the computing node where the first computing card is located writes the same item from the shared memory of the computing node where the first computing card is located to the storage device.

8. The system according to claim 7, characterized in that, Each computing card includes card status information, which is used to indicate the health status of the computing card; The first computing card is also used to send the card status information of the first computing card to the second computing card; The second computing card is also used to receive card status information from the first computing card; The computing node where the second computing card is located is used to write the identical item from the shared memory of the computing node where the second computing card is located to the storage device in the event of a failure of the first computing card, or in the event that the storage device does not store the identical item, wherein the failure of the first computing card is determined based on the card status information of the first computing card.

9. The system according to claim 7, characterized in that... ； The computing node where the first computing card is located is used to write the first identical item fragment from the shared memory of the computing node where the first computing card is located to the storage device; The computing node where the second computing card is located is also used to write the second identical item fragment from the shared memory of the computing node where the first computing card is located to the storage device.

10. The system according to any one of claims 1 to 6, characterized in that... ； The computing node where the third computing card is located is used to write the different items of the third checkpoint from the shared memory of the computing node where the third computing card is located to the storage device; The computing node where the fourth computing card is located is used to write the different items of the fourth checkpoint from the shared memory of the computing node where the fourth computing card is located to the storage device.

11. The system according to claim 9 or 10, characterized in that... ； The first computing card is also used to send the card status information of the first computing card to the second computing card; The second computing card is also used to receive card status information from the first computing card; The computing node where the second computing card is located is also used to write the first identical item fragment from the shared memory of the computing node where the second computing card is located to the storage device in the event of a failure of the first computing card, or in the event that the first identical item fragment is not stored in the storage device. The third computing card is also used to send the card status information of the third computing card to the fourth computing card; The fourth computing card is also used to receive the card status information of the third computing card; The computing node where the fourth computing card is located is also used to write copies of the different items of the third checkpoint from the shared memory of the computing node where the fourth computing card is located to the storage device in the event of a failure of the third computing card or in the event that the different items of the third checkpoint are not stored in the storage device.

12. The system according to any one of claims 1 to 11, characterized in that, The storage device further includes a first-layer storage device and a second-layer storage device, and the computing node where the first computing card is located is also used for: The checkpoint generated by the first computing card is written from the shared memory of the computing node where the first computing card is located to the first layer storage device in the original data format; The checkpoint generated by the first computing card is serialized, and the serialized checkpoint is written from the shared memory of the computing node where the first computing card is located to the first layer storage device. The first-level storage device is used to write the serialized checkpoint from the first-level storage device to the second-level storage device.

13. A large model training apparatus, wherein the large model training apparatus is applied to a large model training system, the large model training system including a storage device and at least one computing card, the at least one computing card being disposed on at least one computing node, the at least one computing card including a first computing card group for executing a first training task, the first training task including a first sub-training task and a second sub-training task, the first computing card group including a first computing card and a second computing card, the apparatus comprising: The allocation module allocates the first sub-training task to the first computing card. The first sub-training task is used to instruct the first computing card to execute the first sub-training task and generate a first checkpoint. The first checkpoint includes the model parameters and optimizer state of the target iteration of the first sub-training task. The allocation module allocates the second sub-training task to the second computing card. The second sub-training task is used to instruct the second computing card to execute the first sub-training task and generate a second checkpoint. The second checkpoint includes the model parameters and optimizer state of the target iteration of the second sub-training task. The determination module, in response to a first training recovery instruction, determines the common items between the first checkpoint and the second checkpoint, wherein the first training recovery instruction is used to instruct training recovery to be performed according to the target iteration; The sending module sends the information recording the identical items to the first computing card, so that the first computing card loads the identical items from the storage device based on the information and sends the identical items to the second computing card.

14. A method for training large models, characterized in that, An application is made in a large model training system, the large model training system including a storage device and at least one computing card, the at least one computing card being disposed on at least one computing node, the at least one computing card including a first computing card group for executing a first training task, the first training task including a first sub-training task and a second sub-training task, the first computing card group including a first computing card and a second computing card, the method including: The first computing card executes the first sub-training task and generates a first checkpoint, which includes the model parameters and optimizer state of the target iteration of the first sub-training task. The second computing card executes the second sub-training task and generates a second checkpoint, the second checkpoint including the model parameters and optimizer state of the target iteration of the second sub-training task; In response to a first training recovery instruction, the first computing card loads identical items from the storage device for the first checkpoint and the second checkpoint, recovers the first sub-training task based on the identical items, and sends the identical items to the second computing card. The first training recovery instruction is used to instruct training recovery to be performed according to the target iteration. The second computing card receives the identical items and recovers the second sub-training task based on the identical items.

15. The method according to claim 14, characterized in that, The at least one computing card further includes a third computing card and a fourth computing card belonging to the second computing card group, and the method further includes: The third computing card performs the second training task and generates the third checkpoint; The fourth computing card executes the third training task and generates the fourth checkpoint. The model parameters and optimizer states included in the fourth checkpoint are different from those included in the third checkpoint. In response to the second training recovery command, the third computing card loads different items of the third checkpoint from the storage device and recovers the second training task based on the different items of the third checkpoint; In response to the second training recovery command, the fourth computing card loads different items of the fourth checkpoint from the storage device and recovers the third training task based on the different items of the fourth checkpoint.

16. The method according to claim 14, characterized in that, Each compute node includes multiple compute cards that share memory, prior to loading the same item from the storage device, including: The first computing card loads the same item from the shared memory of the computing node where the first computing card is located; If the identical item is not stored in the shared memory of the computing node where the first computing card is located, the second computing card loads the identical item from the shared memory of the computing node where the second computing card is located and sends the identical item to the first computing card.

17. The method of claim 15, wherein the compute node on the fourth compute card stores copies of the different items of the third checkpoint, and prior to loading the different items of the third checkpoint from the storage device, includes: The third computing card loads different items of the third checkpoint from the shared memory of the computing node where the third computing card is located; If the shared memory of the computing node where the third computing card is located does not store the different items of the third checkpoint, the fourth computing card loads a copy of the different items of the third checkpoint from the shared memory of the computing node where the fourth computing card is located, and sends the copy of the different items of the third checkpoint to the third computing card.

18. The method according to claim 14, characterized in that, The identical items include a first identical item fragment and a second identical item fragment, and the method further includes: In response to the first training recovery command, the first computing card loads the first identical item fragment from the storage device and sends the first identical item fragment to the second computing card; In response to the first training recovery command, the second computing card loads the second identical item fragment from the storage device and sends the second identical item fragment to the first computing card; The first computing card recovers the first sub-training task based on the first identical item segment and the second identical item segment; The second computing card recovers the second sub-training task based on the first identical item segment and the second identical item segment.

19. The method according to any one of claims 14 to 18, characterized in that, Each computing card includes video memory, and the method further includes: The first computing card writes the checkpoint generated by the first computing card from the video memory of the first computing card to the shared memory of the computing node where the first computing card is located in the original data format. The checkpoint in the original data format is not serialized.

20. The method according to any one of claims 14 to 19, characterized in that, After the first computing card writes the checkpoint generated by the first computing card from its video memory to the shared memory of the computing node where the first computing card resides in its original data format, the process includes: The first computing card and the second computing card determine that the computing node where the first computing card is located writes the same item from the shared memory of the computing node where the first computing card is located to the storage device.

21. The method according to claim 20, characterized in that, Each computing card includes card status information, which indicates the health status of the computing card. The method further includes: The first computing card sends its card status information to the second computing card; The second computing card receives the card status information of the first computing card; In the event of a failure of the first computing card, or in the event that the storage device does not store the same item, the computing node where the second computing card is located writes the same item from the shared memory of the computing node where the second computing card is located to the storage device, wherein the failure of the first computing card is determined based on the card status information of the first computing card.

22. The method according to claim 20, characterized in that, The method further includes: The computing node where the first computing card is located writes the first identical item fragment from the shared memory of the computing node where the first computing card is located to the storage device; The computing node where the second computing card is located writes the second identical item fragment from the shared memory of the computing node where the first computing card is located to the storage device.

23. The method according to any one of claims 14 to 19, characterized in that, The method further includes: The computing node where the third computing card is located writes the different items of the third checkpoint from the shared memory of the computing node where the third computing card is located to the storage device; The computing node where the fourth computing card is located writes the different items of the fourth checkpoint from the shared memory of the computing node where the fourth computing card is located to the storage device.

24. The method according to claim 22 or 23, characterized in that, The method further includes: The first computing card sends its card status information to the second computing card; The second computing card receives the card status information of the first computing card; In the event of a failure of the first computing card, or in the event that the first identical item fragment is not stored in the storage device, the computing node where the second computing card is located writes the first identical item fragment from the shared memory of the computing node where the second computing card is located to the storage device. The third computing card sends its card status information to the fourth computing card; The fourth computing card receives the card status information from the third computing card; In the event of a failure of the third computing card, or in the event that the different items of the third checkpoint are not stored in the storage device, the computing node where the fourth computing card is located writes a copy of the different items of the third checkpoint from the shared memory of the computing node where the fourth computing card is located to the storage device.

25. The method according to any one of claims 14 to 24, characterized in that, The storage device further includes a first-layer storage device and a second-layer storage device. After writing the checkpoint generated by the first computing card from the video memory of the first computing card to the shared memory of the computing node where the first computing card is located in its original data format, the process includes: The computing node where the first computing card is located writes the checkpoint generated by the first computing card from the shared memory of the computing node where the first computing card is located to the first layer storage device in the original data format; The computing node where the first computing card is located serializes the checkpoint generated by the first computing card and writes the serialized checkpoint from the shared memory of the computing node where the first computing card is located to the first layer storage device; The first-level storage device is used to write the serialized checkpoint from the first-level storage device to the second-level storage device.

26. A computing device cluster, characterized in that, The system includes multiple computing devices, each comprising multiple processors and multiple memories, wherein program instructions are stored in the multiple memories, and the multiple processors execute the program instructions, causing the cluster of computing devices to perform the large model training method according to any one of claims 14 to 25.

27. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the large model training method as described in any one of claims 14 to 25.

28. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the large model training method as described in any one of claims 14 to 25.

Images