Checkpoint storage method and device of artificial intelligence model, medium and product

Abstract

The invention relates to the technical field of artificial intelligence, and particularly provides a check point storage method and device of an artificial intelligence model, a medium and a product. The method comprises the following steps: acquiring storage equipment information and PCIe topological information, and constructing a transmission topological graph based on the storage equipment information and the PCIe topological information; based on the transmission topological graph, determining a target parallel transmission path from the GPU to the storage device; dividing the global check point data into a plurality of check point fragments based on the bandwidth capability of the target parallel transmission path; transmitting the plurality of check point fragments from the GPU to corresponding storage equipment in parallel through a plurality of independent transmission streams; and executing a two-stage persistence operation, and transmitting and storing the check points in the storage device to a remote storage system in a fragmented manner. According to the method provided by the invention, a data transmission path can be systematically optimized, heterogeneous storage resources are cooperatively utilized, and high reliability and rapid recovery capability are considered, so that the training efficiency of a large-scale artificial intelligence model is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to methods, devices, media, and products for storing checkpoints of artificial intelligence models. Background Technology

[0002] In the field of artificial intelligence technology, especially in the training process of large-scale models with hundreds of billions of parameters, checkpoint saving is a key mechanism to ensure the fault tolerance and recovery capability of training tasks.

[0003] In related technologies, the synchronous checkpoint saving method requires all model state data to be persisted to disk before the training task can continue. Its saving performance is severely limited by disk I / O bandwidth, which causes the training process to be blocked for a long time, resulting in high-value computing resources such as GPUs being idle and overall resource utilization being low. Summary of the Invention

[0004] In view of this, exemplary embodiments of the present disclosure provide a method, apparatus, medium, and product for saving checkpoints of an artificial intelligence model to address the problems existing in the related art.

[0005] One aspect of an exemplary embodiment of this disclosure provides a method for saving checkpoints in an artificial intelligence model, the method comprising: Acquire storage device information and PCIe topology information, and construct a transmission topology map representing the interconnection relationship and bandwidth constraints between devices based on the storage device information and PCIe topology information; the storage device includes CPU main memory, local solid-state drive SSD and remote network interface card (NIC); Based on the transmission topology, the target parallel transmission path from the GPU to the storage device is determined; Based on the bandwidth capability of the target parallel transmission path, the global checkpoint data is divided into multiple checkpoint fragments; The multiple checkpoint fragments are transmitted in parallel from the GPU to the corresponding storage devices through multiple independent transport streams; A two-phase persistence operation is performed to transmit and save the checkpoint fragments in the storage device to a remote storage system.

[0006] In another aspect of exemplary embodiments of this disclosure, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the methods described in exemplary embodiments of this disclosure.

[0007] In another aspect of exemplary embodiments of this disclosure, a computer-readable storage medium is provided having a computer program / instructions stored thereon that, when executed by a processor, implements the methods described in exemplary embodiments of this disclosure.

[0008] In another aspect of exemplary embodiments of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the methods described in exemplary embodiments of this disclosure.

[0009] As will be described in detail below, a checkpoint saving method for an artificial intelligence model according to an embodiment of this disclosure involves acquiring storage device information and PCIe topology information, and constructing a transmission topology map representing the interconnection relationships and bandwidth constraints between devices based on the storage device information and PCIe topology information; the storage device includes CPU main memory, local solid-state drive (SSD), and remote network interface card (NIC); based on the transmission topology map, a target parallel transmission path from the GPU to the storage device is determined; global checkpoint data is divided into multiple checkpoint fragments based on the bandwidth capability of the target parallel transmission path; multiple checkpoint fragments are transmitted in parallel from the GPU to the corresponding storage device through multiple independent transmission streams; and a two-stage persistence operation is performed to transmit and save the checkpoint fragments in the storage device to a remote storage system.

[0010] Therefore, the checkpoint saving method for the artificial intelligence model disclosed in this publication overcomes the bandwidth limitation of the PCIe root node by constructing a transmission topology graph and applying the maximum flow algorithm, thereby optimizing the transmission path. By using checkpoint fragmentation and parallel transmission, the checkpoint saving time is minimized, significantly reducing GPU resource idleness. Through a two-stage persistence operation, training tasks are avoided from being blocked while ensuring data reliability. Attached Figure Description

[0011] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 This is a schematic diagram of the PCIe topology of the AI ​​training server provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of the breakpoint-based training process provided in an embodiment of the present disclosure; Figure 3a A flowchart illustrating the synchronous checkpointing method provided in this embodiment of the disclosure; Figure 3b A flowchart illustrating the asynchronous checkpointing method provided in this embodiment of the disclosure; Figure 4aThis is a schematic diagram illustrating direct GPU transfer when the GPU direct memory function is not enabled, provided as an embodiment of this disclosure. Figure 4b This is a schematic diagram illustrating GPU direct transfer when GPU direct memory is enabled, provided as an embodiment of this disclosure. Figure 5 This is a schematic diagram of the multi-channel checkpoint architecture and process provided in the embodiments of this disclosure; Figure 6 A flowchart illustrating the checkpoint saving method for an artificial intelligence model provided in this embodiment of the disclosure; Figure 7 This is a schematic diagram of the checkpoint parallel snapshot process provided in an embodiment of the present disclosure; Figure 8 Transmission topology diagram provided for embodiments of this disclosure; Figure 9 A flowchart of the two-stage persistence process for checkpoints provided in this embodiment of the disclosure; Figure 10 A schematic block diagram of the functional modules of the checkpoint storage device for the artificial intelligence model provided in the embodiments of this disclosure; Figure 11 A structural block diagram of an electronic device provided in an embodiment of this disclosure; Figure 12 A schematic diagram of a computer program product provided in an embodiment of this disclosure. Detailed Implementation

[0013] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0014] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0015] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0016] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0017] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0018] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0019] In intelligent computing platforms, servers used for distributed AI model training may include: GPU (Graphics Processing Unit). The core computing device responsible for training AI models. The GPU uses its built-in video memory to store data such as model states and intermediate calculation results.

[0020] The CPU processor is responsible for managing and coordinating multiple GPUs for training, and for inputting and outputting data (including training data and checkpoints) to and from the GPUs.

[0021] Local hard drive or remote network hard drive. Responsible for storing training set data and checkpoint files.

[0022] Network interface card (NIC). Connects to a remote persistent storage system to achieve reliable data preservation.

[0023] For example, Figure 1 This is a schematic diagram of the PCIe topology for an AI training server provided in an embodiment of this disclosure. Figure 1 As shown, the AI ​​training server adopts a dual-CPU architecture, and the two CPUs communicate with each other through a UPI (Ultra Path Interconnect) high-speed interconnect link.

[0024] Each CPU manages an independent PCIe domain, and its root complex (RC) is connected to the corresponding CPU via a PCIe x16 link. Each root complex is connected to a PCIe switch, which further connects multiple GPU devices, network cards, and NVMe SSDs.

[0025] In this topology, devices such as GPUs, network cards, and NVMe SSDs share the uplink bandwidth provided by the same root complex through PCIe switches. When multiple devices access CPU memory or communicate with each other at the same time, their aggregate bandwidth will be limited by the total available bandwidth of the PCIe root node and the switch, which may create a transmission bottleneck in large-scale data exchange scenarios.

[0026] For example, Figure 2 This is a schematic diagram of a checkpoint-based breakpoint resuming training process provided in an embodiment of this disclosure. Figure 2 As shown, during training, the system sequentially saves a series of checkpoints, such as checkpoint 1 and checkpoint 2. When the training process is interrupted due to faults or other reasons, the system can load the previous checkpoint closest to the interruption time (e.g., checkpoint 2) and restore the model state from the point of interruption, thereby continuing the training process. This achieves breakpoint continuation of training and effectively avoids the waste of computing resources.

[0027] Common checkpointing methods use local hard drives or remote network drives to synchronously save checkpoint files, involving two steps: snapshotting and persistence. After the model has trained the Nth batch, training for the N+1th batch needs to be paused, and the snapshot step is initiated to copy the model and optimizer data from the GPU memory to CPU memory. Next, the persistence step is performed, where the data in main memory is serialized and compressed, and then saved to the local or remote hard drive via the file system. Finally, the training task can resume only after the checkpoint files have been written.

[0028] For example, Figure 3a This is a flowchart illustrating the synchronous checkpointing method provided in an embodiment of this disclosure. Figure 3a As shown, the model state in the GPU is first snapshotted to the CPU memory, and then the data in the memory is synchronously written to local or remote storage. The training task will only continue to execute after all data persistence is completed.

[0029] For example, Figure 3b This is a flowchart illustrating the asynchronous checkpointing method provided in an embodiment of this disclosure. Figure 3bAs shown, after the GPU-to-CPU memory snapshot is completed, the training task resumes execution, while the operation of persisting data to local or remote storage is performed asynchronously in the background. Although the asynchronous checkpointing method shortens the training blocking time, it still cannot avoid the overhead caused by the GPU-to-memory transfer bandwidth limitation during the snapshot phase.

[0030] In addition, during checkpoint recovery, regardless of whether the method is synchronous or asynchronous, the checkpoint file must first be loaded from local or remote storage into main memory and then transferred to the GPU. This process may also become a bottleneck for recovery speed.

[0031] For example, Figure 4a This is a schematic diagram illustrating direct GPU transfer when the GPU direct memory function is not enabled, provided in an embodiment of this disclosure. Figure 4b This is a schematic diagram illustrating direct GPU transfer when the GPU direct memory function is enabled, as provided in an embodiment of this disclosure. Figure 4a and Figure 4b As shown, the system includes core components such as CPU, system memory, and multiple GPUs, which are interconnected at high speed through PCIe switches.

[0032] When GPU direct storage is not enabled, data exchange between the GPU and the NVMe SSD needs to be relayed through system memory. The transmission path involves additional data copying operations, which will consume PCIe channel bandwidth and increase transmission latency.

[0033] With GPU Direct Memory enabled, the GPU can establish a direct data transfer channel with the NVMe SSD via a PCIe switch, eliminating the need for system memory intermediaries. This point-to-point transmission method effectively reduces data transfer paths, significantly improves transmission efficiency, reduces transmission latency, and provides higher I / O performance for large-scale data read / write operations such as checkpoints.

[0034] In the field of artificial intelligence, especially in the training of large models with hundreds of billions of parameters, checkpoint saving is a critical operation to ensure the fault tolerance and recovery of training tasks. However, the performance of synchronous checkpointing methods is mainly limited by disk bandwidth, resulting in long disk write-to-disk steps. While asynchronous checkpointing methods reduce disk write-to-disk time overhead, they still cannot shorten snapshot time. Snapshot speed mainly depends on the bandwidth of the GPU-to-CPU memory transfer channel. When data from multiple GPUs needs to be transferred to main memory through the PCIe root node, the bus bandwidth of the PCIe root node becomes a significant bottleneck.

[0035] Furthermore, even if GPU direct transfer technology is used to bypass CPU memory and write data directly to storage devices, the transfer bandwidth is still limited by the I / O capability of a single local or remote storage device, making it difficult to fully realize the potential of multi-device parallel processing.

[0036] In addition, the relevant technologies are insufficient in terms of the reliability of data persistence and the speed of recovery after failure. They either fail to effectively avoid the risk of memory data loss caused by software failure, or only store checkpoints remotely, which requires a long network data transmission process during recovery and prolongs the interruption time of training tasks.

[0037] Therefore, to address the aforementioned issues, this exemplary embodiment provides a checkpoint storage method for an artificial intelligence model. First, addressing the problem of PCIe bus bandwidth becoming a transmission bottleneck, this disclosure constructs a checkpoint storage architecture based on heterogeneous devices, integrating CPU memory, local NVMe SSDs, and RDMA-enabled remote storage into a unified storage system, providing diverse high-speed destinations for parallel transmission. Based on this, to maximize the bandwidth potential of this architecture, this disclosure proposes a multi-channel parallel snapshot method for checkpoints. Specifically, by probing the system's PCIe topology, a device topology map is constructed, and the target parallel transmission path is calculated using the maximum flow algorithm. Subsequently, based on the parallel strategy of the AI ​​model and the bandwidth ratio of the transmission path, the global checkpoint data is intelligently segmented at the neural network layer level. Finally, the checkpoint fragments are transmitted in parallel to memory, SSDs, or network cards in a point-to-point manner through multiple independent CUDA streams, thereby distributing the originally concentrated I / O load across multiple high-speed links, thus overcoming the bandwidth limitation of a single root node.

[0038] Secondly, to ensure data persistence and fault tolerance, and to overcome the risk of data loss due to software or hardware failures, this disclosure designs a two-stage persistence method. Specifically, after a snapshot is completed, the first stage is executed: the checkpoint fragments in memory are quickly written to the local SSD, and checksum data is calculated, thereby defending against software-level failures. The second stage: the local fragments are compressed and uploaded to the remote storage system, thereby defending against server hardware-level failures. This two-stage persistence method ensures data reliability while reducing network transmission volume and improving downlink efficiency due to compression.

[0039] Finally, to achieve rapid resumption of training tasks, this disclosure provides a corresponding parallel loading method. When checkpoints need to be loaded, the system reads the latest checkpoint fragments in parallel from memory, local SSD, and remote storage, and also utilizes multi-CUDA streaming technology to quickly load them into GPU memory, significantly reducing the interruption time of training tasks.

[0040] For example, Figure 5 This is a schematic diagram of the multi-channel checkpoint architecture and process provided in the embodiments of this disclosure. Figure 5As shown, the checkpoint saving method proposed in this disclosure divides the global AI model state into a set of checkpoint fragments. Subsequently, utilizing PCIe point-to-point interconnect functionality, a parallel transmission channel is constructed from the GPU to multiple heterogeneous storage devices. For example... Figure 5 As shown, multiple checkpoint fragments no longer queue through a single PCIe root node and memory channel, but are instead transmitted concurrently from the GPU to the CPU memory, local NVMe SSD, and remote network card supporting RDMA via multiple parallel PCIe links, thereby significantly improving snapshot performance and overcoming the transmission bandwidth bottleneck.

[0041] After completing the parallel snapshot, the process enters the two-phase persistence phase. For example... Figure 5 As shown, the first stage includes asynchronously writing the fragments temporarily stored in CPU memory to local storage 1. The second stage includes compressing the fragments in local storage 1 and then saving them to remote storage 1, thereby building the final line of defense against single-node hardware failures.

[0042] Finally, this architecture forms a distributed checkpoint data redundancy by jointly maintaining a complete set of up-to-date snapshot shards across CPU memory, local storage, and remote storage. When training needs to be resumed, a multi-channel loading process can be immediately initiated to transfer checkpoint shards back to the GPU in parallel from these three storage levels, achieving rapid recovery of the training task and thus completing a closed loop from high-performance storage and reliable persistence to rapid recovery.

[0043] For example, Figure 6 A flowchart illustrating the checkpoint saving method for an artificial intelligence model provided in this embodiment of the disclosure. Figure 6 As shown, the method for saving checkpoints in an artificial intelligence model may include the following steps: Step S610: Obtain storage device information and PCIe topology information, and based on the storage device information and PCIe topology information, construct a transmission topology map representing the interconnection relationships and bandwidth constraints between devices. Storage devices include CPU main memory, local solid-state drives (SSDs), and remote network interface cards (NICs).

[0044] For example, a multi-device storage architecture may include the following components: The processing and caching unit includes the CPU and CPU main memory. The CPU is configured with multiple available processing cores for performing concurrent operations.

[0045] The local persistent storage unit includes at least one solid-state drive (SSD) device, which preferably supports the NVMe protocol and is deployed in the same PCIe root complex or the same PCIe switch as the GPU, thereby enabling point-to-point data transfer.

[0046] The remote storage access unit includes at least one network interface card (NIC) device that connects to the remote storage. The NIC device preferably supports RDMA functionality and is deployed in the same PCIe root complex or the same PCIe switch as the GPU for connecting to the remote storage system.

[0047] Based on this, integrating CPU memory, local SSD, and network-based remote storage into a single checkpoint storage system can avoid the problems of insufficient transmission bandwidth and limited storage capacity caused by storing checkpoint snapshots on a single storage medium.

[0048] For example, Figure 7 This is a schematic diagram of the checkpoint parallel snapshot process provided in an embodiment of this disclosure. The checkpoint segmentation and parallel snapshot method may include: For example, GPU information and storage device information in the system are obtained, and a transmission topology map representing the interconnection relationship and bandwidth constraints between devices is constructed based on PCIe topology information.

[0049] Specifically, GPU information may include: whether the device supports GPU Direct Transfer technology, GPU memory size, and the highest PCIe version supported by the GPU.

[0050] Storage device information may include: available main memory capacity, number of local SSDs, available capacity and read / write bandwidth, and network interface card (NIC) specifications, number, and read / write bandwidth. If the available capacity of a storage device is insufficient, its use for storing checkpoints will be prohibited.

[0051] Allocate paged memory in the CPU's main memory that is the same size as the AI ​​model checkpoint. If memory allocation fails, return a checkpoint operation failure message to the user.

[0052] PCIe topology information can include PCIe topology information for all GPUs, PCIe topology information for user-specified storage devices, and PCIe topology information for all storage devices. For example: the device PCIe number, PCIe bus number, PCIe switch number, and PCIe root node number for all GPUs; and the device PCIe number, PCIe bus number, PCIe switch number, and PCIe root node number for all storage devices.

[0053] Next, a transmission topology graph is constructed. The transmission topology graph is a directed acyclic graph, in which the nodes represent GPUs, PCIe root complexes, and storage devices, while the directed edges accurately represent the physical connections between devices. The actual constraints such as PCIe link bandwidth, SSD read / write speed, and network bandwidth are quantified as edge weights, thereby establishing an accurate system model for subsequent path planning.

[0054] Step S620: Based on the transmission topology map, determine the target parallel transmission path from the GPU to the storage device.

[0055] For example, the maximum flow algorithm can be used to analyze the transmission topology and calculate a set of target parallel transmission paths that maximize the total system transmission bandwidth from all GPUs to all storage devices. Each path in the target parallel transmission path specifies which GPU the checkpoint data originates from, which PCIe switches and root complex it passes through, and which specific storage device it finally reaches, and associates the available bandwidth of that path.

[0056] In one alternative approach, the Ford-Fulkerson algorithm can be used to calculate multiple transmission links TPath that can achieve the maximum transmission bandwidth, and the combination of multiple transmission links TPath is the target parallel transmission path.

[0057] A transmission link (TPath) includes the source device, the bandwidth (band), and the destination device (dest). The source device is a GPU device, and the destination device (dest) can be one of the following: CPU memory, SSD, or NIC device, and can only correspond to one physical device.

[0058] Step S630: Divide the global checkpoint data into multiple checkpoint fragments based on the bandwidth capacity of the target parallel transmission path.

[0059] Based on the total bandwidth capacity of the target parallel transmission paths and combined with the parallel training strategy adopted by the artificial intelligence (AI) model, the global checkpoint data that needs to be saved is intelligently partitioned. By dividing the global data into multiple checkpoint fragments that match the capacity, it is ensured that each fragment can be transmitted through its assigned transmission path, thus achieving load balancing.

[0060] Step S640: Transfer multiple checkpoint fragments from the GPU to their corresponding storage devices in parallel via multiple independent transport streams.

[0061] Create and initialize a separate CUDA stream for each transport link TPath.

[0062] Determine the dest entry for each transmission link TPath. If the dest entry is CPU memory, use the memory copy method to transmit checkpoint fragments; if the dest entry is SSD or NIC, use the PCIe point-to-point method to transmit checkpoint fragments.

[0063] Then, leveraging PCIe point-to-point interconnect capabilities, all CUDA streams are started asynchronously, and the system waits for all CUDA streams to complete their transfers. This allows multiple checkpoint fragments to simultaneously originate from their respective source GPUs, bypassing CPU-memory intermediaries, and be transferred directly or in parallel via memory to their corresponding target storage devices. This aggregates the distributed PCIe link bandwidth, enabling high-throughput export of checkpoint snapshots.

[0064] Once completed, a successful snapshot is returned to the user, and the AI ​​training task continues.

[0065] Step S650: Perform a two-stage persistence operation to transfer and save the checkpoint fragments in the storage device to the remote storage system.

[0066] The first stage of the two-phase persistence operation asynchronously writes the checkpoint fragments temporarily stored in the CPU's main memory to a more stable local SSD to prevent data loss due to software failures. The second stage further saves the checkpoint fragments from the local SSD to a remote storage system, completing the storage of the checkpoint data.

[0067] Based on this, by constructing a transmission topology graph and applying the maximum flow algorithm, all available parallel transmission paths in the system can be discovered, and the I / O traffic that was originally concentrated on a single root node can be dynamically distributed to multiple parallel PCIe links, fundamentally breaking through the bandwidth limitation of a single PCIe root node.

[0068] Furthermore, by performing checkpoint fragmentation based on bandwidth ratios and utilizing PCIe point-to-point interconnects to initiate multiple CUDA streams for parallel transmission, the throughput of checkpoint snapshots is greatly improved, the storage time is significantly shortened, and GPU computing resources are freed from long periods of idle waiting, thereby improving overall training efficiency.

[0069] Finally, the first stage of the two-stage persistence operation provides rapid protection against software failures, while the second stage provides ultimate disaster recovery against hardware failures. Therefore, the two-stage persistence operation effectively addresses the risk of checkpoint data being lost due to software or hardware failures during the persistence process.

[0070] Based on the above embodiments, in another embodiment provided in this disclosure, the construction of a transmission topology map representing the interconnection relationship and bandwidth constraints between devices based on storage device information and PCIe topology information may include: Set the GPU as the source node, the CPU main memory, local SSD and NIC as the aggregation node, and the PCIe root complex as the intermediate node; Add a first auxiliary node to the source node, and add a second auxiliary node to the local SSD and NIC respectively; Add a virtual source node and a virtual aggregation node; The construction of the transmission topology includes: Connect the virtual source node to each first auxiliary node with edges of infinite weight; Based on the PCIe topology information, directed edges with bandwidth weights are established between the first auxiliary node, the intermediate node, the second auxiliary node, and the aggregation node of the CPU main memory. Connect each second auxiliary node to its corresponding local SSD or NIC aggregation node; Connect all aggregation nodes to the virtual aggregation node.

[0071] For example, a directed acyclic transmission topology graph can be constructed based on PCIe topology information. ,in Represents a set of nodes. This represents a set of directed links.

[0072] Set multiple GPUs as source nodes , main memory Solid State Drive and network card devices As a convergence node, the PCIe root complex is used as an intermediate node. .

[0073] For each source node Add a first auxiliary node Add a second auxiliary node to each solid-state drive and network card device. and ).

[0074] Each link Each is mapped to a weight, representing the traffic it carries. Each GPU connects to any other node via PCIe X16, thus enabling... The weights are set to .

[0075] Based on the obtained PCIe topology information, construct the link from the GPU to the corresponding PCIe Root Complex. and RC to MEM link Links from the GPU to other PCIe RCs are prohibited. Since all of the above links are connected via PCIe x16, the bandwidth is limited to [specific value]. .

[0076] Based on the obtained PCIe topology information, construct a link e from the GPU to the SSD and network card located in the same PCIe RC. , Links from the GPU to SSDs or network cards located under other PCIe RCs are prohibited. Network card bandwidth is limited to the actual available bandwidth of the network. SSD bandwidth is set to the actual available bandwidth for storage. .

[0077] Add a virtual source node S and a virtual sink node D. Connect S to each source node with a directed edge of infinite weight. Each aggregation node ( , and Connect to D. The maximum flow from S to D is equal to the maximum sum of feasible flows in the original network.

[0078] For example, Figure 8 This is a transmission topology diagram provided for embodiments of this disclosure. Figure 8 The diagram shows the transmission topology G of a system consisting of 4 GPUs and 1 NIC, where GPU0, GPU1 and NIC are connected to PCIe RC-0, while GPU2 and GPU3 are connected to PCIe RC-1.

[0079] The transmission topology G uses virtual source node S and virtual aggregation node D as the start and end points of the global data flow. GPU0, GPU1, GPU2, and GPU3, acting as data source nodes, and main memory, solid-state drives (SSDs), and network interface cards (NICs), acting as data aggregation nodes, are represented by nodes in the diagram. To accurately model the data transmission path, each GPU is connected to an auxiliary node (GPU0' to GPU3'), and each NIC is also equipped with an auxiliary node NIC'.

[0080] The diagram clearly reflects the system's physical topology: GPU0, GPU1, and the NIC are connected to the PCIe root complex rc0 via a PCIe switch, while GPU2 and GPU3 are connected to another independent PCIe root complex rc1. These two root complexes serve as crucial relay nodes in the diagram. Data transmission links strictly adhere to this physical constraint: the virtual source node S connects to all GPU auxiliary nodes via edges with infinite weights. Each GPU auxiliary node is connected to its respective root complex. Main memory, as a shared resource, is connected to both rc0 and rc1, receiving data from all GPUs. Furthermore, GPU0' and GPU1', located under rc0, also have point-to-point links directly pointing to NIC'. Finally, all aggregation nodes connect to the virtual aggregation node D, thus fully outlining all feasible high-bandwidth, parallel data transmission paths in the system.

[0081] by Figure 8 Taking the system shown as an example, three transmission links can be obtained: TPath_0: source=GPU0, band=Bpcie, dest=memory; TPath_1: source=GPU1, band=Bnic, dest=NIC; TPath_2: source=GPU2, band=Bpcie, dest=memory.

[0082] Based on the above embodiments, in another embodiment provided in this disclosure, step S630 may include: When the artificial intelligence model adopts a data parallel mode, the total bandwidth of the target parallel transmission path and the total amount of checkpoint data of the global checkpoint are obtained; based on the ratio of the total bandwidth and the total amount of checkpoint data, checkpoint fragments to be transmitted are allocated to each transmission link. When the artificial intelligence model adopts a model parallel mode, the target parallel transmission path and GPU are grouped according to the PCIe root complex; the intra-group bandwidth of the target parallel transmission path and the intra-group checkpoint data volume of the GPU are obtained; based on the ratio of intra-group bandwidth and intra-group checkpoint data volume, checkpoint fragments to be transmitted are allocated to each transmission link.

[0083] For example, based on the bandwidth capacity of the parallel data transmission path and the parallel training strategy of the AI ​​model, the global checkpoint data is divided into multiple checkpoint fragments.

[0084] If the server deploys only one GPU, or uses only one GPU for AI training, the checkpoint splitting step is skipped. When the server deploys multiple GPUs and uses multiple GPUs simultaneously for AI training, a parallel strategy for the test tasks needs to be determined.

[0085] Specifically, if AI training uses a data-parallel mode, the current model is recorded as data-parallel (DP) mode, indicating that multiple GPUs on the server contain the same model data. If AI training uses tensor parallel or pipelined parallel methods, the current model is recorded as model-parallel (MP) mode, indicating that multiple GPUs on the server contain different model data.

[0086] In DP mode, the bandwidth terms (band) of all transmission links TPath are summed, denoted as... And obtain the checkpoint size of the model, denoted as . .

[0087] In MP mode, all transmission links (TPaths) are divided into different groups based on their respective PCIe root complexes, and the bandwidth of all TPaths within each group is summed, denoted as . All GPUs are divided into different groups based on their respective PCIe root complexes, and the sum of the checkpoint sizes of all GPUs within each group is denoted as . .

[0088] Obtain global checkpoint data for the deep learning training task, including model parameters, model weights, and optimizer state located on the GPU. Then, calculate the maximum capacity of the corresponding checkpoint slice based on the bandwidth of each transport link (TPath).

[0089] The global checkpoint data is divided into checkpoint fragments according to size.

[0090] For example, the global checkpoint data can be traversed layer by layer, where a layer refers to a layer in a deep learning neural network. The amount of data in the tensors contained in each layer is counted as the data volume of that layer. Based on the data volume of each layer, the global checkpoint data is divided into checkpoint fragments at the layer level, ensuring that the capacity of each checkpoint fragment does not exceed the maximum capacity. And bind the checkpoint fragments to the corresponding transport link TPath.

[0091] Based on this, in data parallel mode, all GPU model data are treated as a whole, and global fragmentation is performed according to the ratio of total system bandwidth to total data volume, ensuring that all transmission links are fully utilized, achieving system-level load balancing and maximizing throughput. In model parallel mode, based on the distribution of model fragments in the physical topology, groups are formed using the PCIe root complex as a domain, and fragmentation is performed based on the ratio of bandwidth to data volume within each group. This avoids redundant data transmission across PCIe root nodes, effectively eliminating latency and bandwidth contention caused by unnecessary cross-domain communication, ensuring extremely high transmission efficiency even in complex multi-GPU topologies.

[0092] Based on the above embodiments, in another embodiment provided in this disclosure, step S650 may include: Write the checkpoint fragments stored in the CPU's main memory to the local SSD; The checkpoint fragments in the local SSD are compressed and saved to the remote storage system via the NIC.

[0093] For example, after the checkpoint parallel snapshot step is completed, a two-phase persistence process can be started in the background. Figure 9 A two-phase persistence flowchart for checkpoints provided in embodiments of this disclosure is shown below. Figure 9 As shown, the two-phase persistence process of a checkpoint can include: First, determine if the available capacity of the local SSD meets the storage requirements. If the capacity is insufficient, skip the first stage; if the capacity is sufficient, proceed to the first stage: calculate the checkpoint shard verification data located in the CPU's main memory, and then call multiple CPU threads to write the checkpoint shard and its verification data to the local SSD in parallel. This achieves fast local data persistence and effectively defends against the risk of data loss due to software failures.

[0094] In one alternative approach, at least four CPU threads can be invoked to maximize write throughput.

[0095] During this process, copies of the slices can be kept in memory for subsequent fast access or for differential computation in the next checkpoint task.

[0096] Once the writing is complete, mark the first phase of the operation as finished and proceed to the second phase.

[0097] After completing the first phase, the process proceeds to the second phase. First, it checks if a connectable remote storage system exists. If not, the second phase is skipped; if it exists, all local checkpoint fragments are compressed into compressed fragments, reducing their data size and saving network bandwidth and remote storage space. Then, the compressed fragments are used to calculate checksum data, and all compressed fragments and checksum data are saved to the remote storage system over the network. Once saved, the second phase is marked as complete, and a persistence success notification can be sent to the user.

[0098] Based on this, by retaining the latest checkpoint shards in CPU memory, local SSD, and network-based remote storage respectively, and loading checkpoints from multiple storage devices in parallel to the GPU during fault recovery, rapid training recovery can be achieved.

[0099] Based on the above embodiments, in another embodiment provided in this disclosure, the checkpoint saving method of the above-mentioned artificial intelligence model may further include: Upon receiving a training task recovery request, the target checkpoint fragment is retrieved from multiple storage devices; The target checkpoint fragments are decompressed and then transmitted concurrently to the GPU via multiple independent transport streams.

[0100] For example, when a training resumption request for an AI model is received, the latest checkpoint fragments are retrieved in parallel from multiple storage devices such as CPU memory, local SSD, and remote storage, and the latest checkpoint fragment is identified as the target checkpoint fragment. Subsequently, the checkpoint fragments retrieved from the remote end are uniformly decompressed, and the processed fragments are concurrently transmitted to the GPU through multiple independent transport streams to resume training.

[0101] Based on this, high-speed data loading is achieved by aggregating the bandwidth of memory, local NVMe SSDs, and the network. Secondly, through a multi-stream concurrent transmission mechanism, the parallel PCIe channels between the GPU and multiple storage devices are fully utilized, transforming the originally serial loading process into a highly efficient parallel pipeline. This overcomes the technical problems of a single loading path and excessively long GPU waiting time for data, thus ensuring that distributed training tasks can be quickly resumed after interruption, significantly improving the overall utilization of the GPU computing cluster and the efficiency of training tasks.

[0102] Based on the above embodiments, in another embodiment provided in this disclosure, obtaining the target checkpoint fragment from multiple storage devices may include: Scan checkpoint fragments across multiple storage devices; If the target checkpoint fragment is stored in the CPU main memory, load the target checkpoint fragment from the CPU main memory. If the target checkpoint fragment is not in the CPU's main memory, load the target checkpoint fragment from the local SSD; If the target checkpoint fragment is not present on the local SSD, load the target checkpoint fragment from the remote storage system. The loaded target checkpoint fragment is verified. If the verification result is inconsistent with the pre-stored verification value, the target checkpoint fragment is retrieved again from the remote storage system.

[0103] For example, the first step is to scan all storage devices and check the integrity of the fragmented files.

[0104] Specifically, priority is given to recovering data from high-speed storage devices. If the latest checkpoint fragment is detected to be stored in both main memory and the local SSD, and this latest checkpoint fragment is identified as the target checkpoint fragment, checkpoint loading can be performed directly.

[0105] If a fragment in main memory is lost, the system automatically searches for and loads the latest checkpoint data that completed the first phase of persistence from local storage, while marking the memory fragment as unavailable to ensure the correctness of subsequent operation logic.

[0106] If a shard in the local SSD is also lost, the system will further search and load the latest checkpoint data that has completed the second phase of persistence from the remote storage, and mark the SSD shard as unavailable. This ensures that even in the extreme case of a single node hardware failure, the training task can still be recovered based on the remote archived data.

[0107] Based on this, the system has the ability to automatically degrade and complete data in scenarios where data is missing. When data in higher-level storage becomes unavailable, the system does not need to interrupt the recovery process and automatically switches to the next level of persistent storage to search for available data.

[0108] Finally, the verification data for all checkpoints to be loaded is calculated and compared with the existing original verification results. If the verification results match, it proves that the data is intact, and the process proceeds to the next parallel loading stage. If the verification results do not match, it indicates that the data may have been corrupted during storage or transmission, and in this case, the corresponding fragments need to be reread from remote storage.

[0109] After completing the shard availability verification and data source scheduling, the parallel loading of checkpoint data begins.

[0110] Specifically, the compressed checkpoint fragments retrieved from storage are decompressed to restore them to their original data format.

[0111] Next, the parallel transfer channel is initialized. An independent CUDA stream is created and initialized for each available checkpoint slice, thus establishing a dedicated data transfer channel from the source storage device to the target GPU memory for each slice.

[0112] Then, asynchronous parallel transmission is performed. All initialized CUDA streams are started asynchronously, enabling multiple checkpoint fragments to be transferred directly from multiple storage devices to their corresponding GPUs simultaneously, achieving high concurrency and high throughput in the data transmission process.

[0113] Finally, after all asynchronous transfer tasks have been completed and all checkpoint data has been successfully loaded into the GPU memory, a status message indicating successful checkpoint loading is returned to the user, allowing the training task of the artificial intelligence model to resume.

[0114] Based on the above embodiments, in another embodiment provided in this disclosure, the checkpoint saving method of the above-mentioned artificial intelligence model may further include: When a new checkpoint save task is triggered, determine whether the previous checkpoint save task has completed the two-phase persistence operation. If the previous checkpoint save task has not completed the two-stage persistence operation, the parallel transmission of the new checkpoint save task is executed first. If the previous checkpoint save task has completed the two-stage persistence operation, the checkpoint fragments retained by the previous checkpoint save task in the CPU main memory are cleaned up. Once the previous checkpoint save task is completed, and the parallel transmission of the new checkpoint save task is finished, the checkpoint fragments retained in the CPU main memory by the previous checkpoint save task are cleared.

[0115] For example, to ensure efficient utilization of system storage resources and prevent data conflicts between consecutive checkpoint tasks, this embodiment also provides a checkpoint resource management strategy. When a new checkpoint save task is triggered, the execution status of the previous checkpoint task is first checked, and different resource management strategies are implemented based on that status: If the previous checkpoint task is found to have failed to complete the checkpoint shard splitting or parallel snapshot operation, the system resources are determined to be in a busy state. To avoid resource contention and ensure data consistency, the newly triggered checkpoint task will be canceled.

[0116] If the previous checkpoint task has completed its parallel snapshot operation but not the subsequent two-stage persistence operation, the data splitting and parallel snapshot operation of the new checkpoint task are executed first, saving the new model state to main memory. This ensures that the latest training progress is captured in a timely manner, avoiding the blocking of the training process due to waiting for persistence. After the persistence operation of the previous checkpoint task is completed, the checkpoint fragments retained in main memory are cleared, thus ensuring a stable data source during the persistence process and preventing accidental data deletion before persistence is complete.

[0117] If it is detected that the previous checkpoint task has been completed, after the slice and parallel transfer of the new checkpoint task are executed, the checkpoint slices retained in main memory by the previous task are cleared, thereby releasing valuable memory resources in a timely manner for the new checkpoint data to be used.

[0118] One or more technical solutions provided in the exemplary embodiments of this disclosure achieve parallel checkpoint saving and loading by utilizing multiple PCIe transmission links from the GPU to the external storage device, effectively reducing the time overhead of checkpoint saving and loading in AI training tasks, reducing training task blocking, and improving the overall computing efficiency of the GPU cluster.

[0119] Therefore, the checkpoint saving method for artificial intelligence models provided in the exemplary embodiments of this disclosure can systematically optimize data transmission paths, collaboratively utilize heterogeneous storage resources, and balance high reliability and fast recovery capabilities, thereby improving the training efficiency of large-scale artificial intelligence models.

[0120] The foregoing primarily describes the solutions provided by exemplary embodiments of this disclosure. It is understood that, in order to achieve the above functions, the electronic device includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0121] The exemplary embodiments of this disclosure can divide the electronic device into functional units according to the above method examples. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into a single processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in the exemplary embodiments of this disclosure is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0122] In the case of dividing each functional module according to its respective function, an exemplary embodiment of this disclosure provides a checkpoint saving device for an artificial intelligence model, which can be a server or a chip applied to a server. Figure 10 A schematic block diagram of the functional modules of the checkpoint saving device for the artificial intelligence model provided in this embodiment of the disclosure. Figure 10 As shown, the checkpoint saving device 1000 of the artificial intelligence model includes: The data acquisition module 1010 is used to acquire storage device information and PCIe topology information, and construct a transmission topology map representing the interconnection relationship and bandwidth constraints between devices based on the storage device information and PCIe topology information; the storage device includes CPU main memory, local solid-state drive SSD and remote network card NIC; Data processing module 1020 is used to determine the target parallel transmission path from the GPU to the storage device based on the transmission topology map; The data processing module 1020 is also used to divide the global checkpoint data into multiple checkpoint fragments based on the bandwidth capability of the target parallel transmission path. The data processing module 1020 is also used to transmit the multiple checkpoint fragments from the GPU to the corresponding storage devices in parallel through multiple independent transmission streams; The data processing module 1020 is also used to perform a two-stage persistence operation, transmitting and saving the checkpoint fragments in the storage device to a remote storage system.

[0123] In another embodiment provided in this disclosure, the data processing module 1020 is further configured to set the GPU as a source node, the CPU main memory, the local SSD, and the NIC as a convergence node, and the PCIe root complex as an intermediate node; add a first auxiliary node to the source node, and add second auxiliary nodes to the local SSD and the NIC respectively; and add a virtual source node and a virtual convergence node. The construction of the transmission topology includes: connecting the virtual source node to each first auxiliary node with edges of infinite weight; establishing directed edges with bandwidth weights between the first auxiliary node, intermediate node, second auxiliary node and CPU main memory aggregation node according to the PCIe topology information; connecting each second auxiliary node to its corresponding local SSD or NIC aggregation node; and connecting all aggregation nodes to the virtual aggregation node.

[0124] In another embodiment provided in this disclosure, the data processing module 1020 is further configured to: obtain the total bandwidth of the target parallel transmission path and the total checkpoint data volume of the global checkpoint when the artificial intelligence model adopts a data parallel mode; allocate checkpoint fragments to be transmitted for each transmission link based on the ratio of the total bandwidth to the total checkpoint data volume; group the target parallel transmission path and GPU according to the PCIe root complex when the artificial intelligence model adopts a model parallel mode; obtain the intra-group bandwidth of the target parallel transmission path and the intra-group checkpoint data volume of the GPU; and allocate checkpoint fragments to be transmitted for each transmission link based on the ratio of the intra-group bandwidth to the intra-group checkpoint data volume.

[0125] In another embodiment provided in this disclosure, the data processing module 1020 is further configured to write checkpoint fragments stored in the CPU main memory to the local SSD; compress the checkpoint fragments in the local SSD; and save them to the remote storage system through the NIC.

[0126] In another embodiment provided in this disclosure, the data processing module 1020 is further configured to, upon receiving a training task recovery request, obtain target checkpoint fragments from multiple storage devices; decompress the target checkpoint fragments; and concurrently transmit the decompressed target checkpoint fragments to the GPU through multiple independent transmission streams.

[0127] In another embodiment provided in this disclosure, the data processing module 1020 is further configured to scan the checkpoint fragments in the plurality of storage devices; if the target checkpoint fragment is stored in the CPU main memory, load the target checkpoint fragment from the CPU main memory; if the target checkpoint fragment is not stored in the CPU main memory, load the target checkpoint fragment from the local SSD; if the target checkpoint fragment is not stored in the local SSD, load the target checkpoint fragment from the remote storage system; verify the loaded target checkpoint fragment, and if the verification result is inconsistent with the pre-stored verification value, re-obtain the target checkpoint fragment from the remote storage system.

[0128] In another embodiment provided in this disclosure, the data processing module 1020 is further configured to, when a new checkpoint save task is triggered, determine whether the previous checkpoint save task has completed the two-stage persistence operation; if the previous checkpoint save task has not completed the two-stage persistence operation, first execute the parallel transmission of the new checkpoint save task, and if the previous checkpoint save task has completed the two-stage persistence operation, clean up the checkpoint fragments retained by the previous checkpoint save task in the CPU main memory; if the previous checkpoint save task is fully completed, after the parallel transmission of the new checkpoint save task is completed, clean up the checkpoint fragments retained by the previous checkpoint save task in the CPU main memory.

[0129] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this disclosure.

[0130] Exemplary embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to embodiments of this disclosure.

[0131] Figure 11The structural block diagram of the electronic device provided in the embodiments of this disclosure will now be described as follows: An electronic device 1100 that can serve as a server or client of this disclosure is an example of a hardware device that can be applied to various aspects of this disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the disclosure described and / or claimed herein.

[0132] like Figure 11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded into a random access memory (RAM) 1103 from a storage unit 1108. The RAM 1103 may also store various programs and data required for the operation of the electronic device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0133] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, output unit 1107, storage unit 1108, and communication unit 1109. Input unit 1106 can be any type of device capable of inputting information to electronic device 1100. Input unit 1106 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 1107 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1108 may include, but is not limited to, disk and optical disk. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0134] The computing unit 1101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above. The various methods described above can all be implemented as computer software programs, which are tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or communication unit 1109.

[0135] Figure 12 The diagram illustrates a computer program product provided in an embodiment of this disclosure. An exemplary embodiment of this disclosure also provides a computer program product 1200, including a computer program 1201, wherein the computer program 1201, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this disclosure.

[0136] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0138] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0140] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0141] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0142] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this disclosure are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center integrating one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0143] Although this disclosure has been described in conjunction with specific features and embodiments, it will be apparent that various modifications and combinations can be made therein without departing from the spirit and scope of this disclosure. Accordingly, this specification and drawings are merely exemplary illustrations of the disclosure as defined by the appended claims and are to be considered as covering any and all modifications, variations, combinations, or equivalents within the scope of this disclosure. It is obvious that those skilled in the art can make various alterations and modifications to this disclosure without departing from its spirit and scope. Thus, this disclosure is also intended to include any such modifications and modifications that fall within the scope of the claims of this disclosure and their equivalents.

Claims

1. A method for saving checkpoints in an artificial intelligence model, characterized in that, The method includes: Acquire storage device information and PCIe topology information, and construct a transmission topology map representing the interconnection relationship and bandwidth constraints between devices based on the storage device information and PCIe topology information; the storage device includes CPU main memory, local solid-state drive SSD and remote network interface card (NIC); Based on the transmission topology, the target parallel transmission path from the GPU to the storage device is determined; Based on the bandwidth capability of the target parallel transmission path, the global checkpoint data is divided into multiple checkpoint fragments; The multiple checkpoint fragments are transmitted in parallel from the GPU to the corresponding storage devices through multiple independent transport streams; A two-phase persistence operation is performed to transmit and save the checkpoint fragments in the storage device to a remote storage system.

2. The method according to claim 1, characterized in that, The step of constructing a transmission topology map representing the interconnection relationships and bandwidth constraints between devices based on the storage device information and PCIe topology information includes: Set the GPU as the source node, the CPU main memory, local SSD and NIC as the aggregation node, and the PCIe root complex as the intermediate node; Add a first auxiliary node to the source node, and add a second auxiliary node to the local SSD and the NIC respectively; Add a virtual source node and a virtual aggregation node; The construction of the transmission topology includes: Connect the virtual source node to each of the first auxiliary nodes with edges of infinite weight; Based on the PCIe topology information, directed edges with bandwidth weights are established between the first auxiliary node, the intermediate node, the second auxiliary node, and the aggregation node of the CPU main memory. Connect each second auxiliary node to its corresponding local SSD or NIC aggregation node; Connect all aggregation nodes to the virtual aggregation node.

3. The method according to claim 1, characterized in that, The bandwidth capability based on the target parallel transmission path divides the global checkpoint data into multiple checkpoint fragments, including: When the artificial intelligence model adopts a data parallel mode, the total bandwidth of the target parallel transmission path and the total checkpoint data volume of the global checkpoint are obtained; based on the ratio of the total bandwidth to the total checkpoint data volume, checkpoint fragments to be transmitted are allocated to each transmission link. When the artificial intelligence model adopts a model parallel mode, the target parallel transmission path and GPU are grouped according to the PCIe root complex; the intra-group bandwidth of the target parallel transmission path and the intra-group checkpoint data volume of the GPU are obtained; based on the ratio of the intra-group bandwidth and the intra-group checkpoint data volume, checkpoint fragments to be transmitted are allocated to each transmission link.

4. The method according to claim 1, characterized in that, The two-phase persistence operation, which transmits and saves checkpoint fragments from the storage device to a remote storage system, includes: Write the checkpoint fragments stored in the CPU's main memory to the local SSD; The checkpoint fragments in the local SSD are compressed and saved to the remote storage system via the NIC.

5. The method according to claim 1, characterized in that, The method further includes: Upon receiving a training task recovery request, the target checkpoint fragment is retrieved from multiple storage devices; The target checkpoint fragments are decompressed and then concurrently transmitted to the GPU via multiple independent transport streams.

6. The method according to claim 5, characterized in that, The step of obtaining the target checkpoint fragment from multiple storage devices includes: Scan the checkpoint segments in the plurality of storage devices; If the target checkpoint fragment is stored in the CPU main memory, the target checkpoint fragment is loaded from the CPU main memory. If the target checkpoint fragment is not in the CPU main memory, load the target checkpoint fragment from the local SSD; If the target checkpoint fragment is not present in the local SSD, load the target checkpoint fragment from the remote storage system. The loaded target checkpoint fragment is verified. If the verification result is inconsistent with the pre-stored verification value, the target checkpoint fragment is retrieved again from the remote storage system.

7. The method according to claim 1, characterized in that, The method further includes: When a new checkpoint save task is triggered, determine whether the previous checkpoint save task has completed the two-phase persistence operation. If the previous checkpoint save task has not completed the two-stage persistence operation, the parallel transmission of the new checkpoint save task is executed first. If the previous checkpoint save task has completed the two-stage persistence operation, the checkpoint fragments retained by the previous checkpoint save task in the CPU main memory are cleared. After the previous checkpoint save task is completed, and after the parallel transmission of the new checkpoint save task is completed, the checkpoint fragments retained in the CPU main memory by the previous checkpoint save task are cleared.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method of claim 1.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the method of claim 1.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the method of claim 1.

Images