Checkpoint updating method and device, electronic equipment and storage medium

Abstract

The invention provides a check point updating method and device, electronic equipment and a storage medium, and relates to the technical field of model training. The method comprises the steps that when a to-be-trained target large language model is trained, target global gradient information corresponding to a current round is acquired through a set communication operation; in an idle period of training, asynchronously and parallelly migrating the target global gradient information to central processing unit (CPU) memories of all training nodes; according to the target global gradient information, iteratively updating the corresponding check point file in the CPU memory of each training node so as to advance the check point state to the next training round; the check point state is isomorphic to the training state of a graphics processing unit (GPU) in the same training node. According to the method, when the check point is updated each time, only the target global gradient information is updated, meanwhile, a snapshot does not need to be created for the GPU training state and a persistent check point file does not need to be constructed, and model training does not need to be paused when the check point is updated, so that the training efficiency is improved.

Description

Technical Field

[0001] This application belongs to the field of model training technology, specifically relating to a checkpoint update method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous advancement of artificial intelligence technology, Large Language Models (LLM), as a subfield of machine learning, has received widespread attention in recent years. LLM models have demonstrated powerful capabilities in natural language processing.

[0003] To mitigate the immediate losses caused by training failures in large language models, checkpointing and restoring are crucial for ensuring the stable operation of GPU (Graphics Processing Unit) tasks. The training framework periodically creates snapshots of the GPU training state and builds persistent checkpoint files after certain training epochs or intervals. When a training failure occurs, the training task can be restarted, the checkpoint files can be read, and training can resume from the most recent checkpoint.

[0004] As the size of the LLM model increases, the size of the checkpoint also increases. At the same time, the construction and updating of checkpoints often require pausing model training, thus introducing significant system overhead and reducing training efficiency. Summary of the Invention

[0005] The technical problem to be solved by this application is to address the above-mentioned shortcomings of the prior art by providing a checkpoint update method, apparatus, electronic device and storage medium. Using the checkpoint update method, the difference between the planning results and the actual deployment can be reduced, and the checkpoint update effect can be improved.

[0006] In a first aspect, embodiments of this application provide a checkpoint update method, including: When training the target large language model, the target global gradient information corresponding to the current round is obtained through ensemble communication operations. During training idle periods, the target global gradient information is asynchronously and in parallel migrated to the CPU memory of all training nodes. The checkpoint files in the CPU memory of each training node are iteratively updated based on the target global gradient information to advance the checkpoint state to the next training round; the checkpoint state is isomorphic to the training state of the GPU in the same training node.

[0007] In some implementations of the first aspect, the target global gradient information corresponding to the current round is obtained through a set communication operation, including: The global gradient information obtained from the current round based on the set communication operation is obtained asynchronously using a pre-set hook function; The target global gradient information for checkpoint file iterative updates is determined from the global gradient information.

[0008] In some implementations of the first aspect, before obtaining the target global gradient information corresponding to the current round through ensemble communication operations when training the target large language model to be trained, the method further includes: The initial state of the checkpoint is constructed in the CPU memory of each training node; the initial state is isomorphic to the training state of the GPU in the same training node. The total number of checkpoint slices in the CPU memory of all training nodes is determined based on the model structure of the target large language model, the hardware resources of all training nodes, and the preset determination algorithm. The number of checkpoint fragments in the CPU memory of each training node is determined based on the total number of checkpoint fragments. The checkpoint file is stored in fragments according to the number of checkpoint fragments, forming multiple checkpoint fragments. At least one checkpoint fragment is stored in the CPU memory of each training node.

[0009] In some implementations of the first aspect, checkpoint fragmentation includes parameter fragmentation data and local metadata corresponding to the target large language model; the parameter fragmentation data and local metadata are stored separately. The checkpoint files in the CPU memory of each training node are iteratively updated based on the target global gradient information, including: The target checkpoint slices are determined based on the target global gradient information and the corresponding local metadata stored in the CPU memory of each training node; the target checkpoint slices store the parameter slice data that needs to be incrementally updated. The target checkpoint segments are iteratively updated based on the target's global gradient information.

[0010] In some implementations of the first aspect, each training node stores a corresponding redundant checkpoint copy; the redundant checkpoint copy corresponds to a checkpoint fragment, and the total number of redundant checkpoint copies stored by all training nodes corresponds to at least all checkpoint fragments corresponding to the checkpoint file. Iteratively updating the checkpoint files in the CPU memory of each training node based on the target global gradient information to advance the checkpoint state to the next training round, also includes: When a fault occurs during training, the iterative update operation of the checkpoint slices of each training node is paused. Retrieve copies of all redundant checkpoints generated in the most recent round from each training node; the most recent round is the training round closest to the time of the failure. Based on a pre-built metadata index, complete and recoverable checkpoint files are generated by parsing and reassembling all redundant checkpoint copies generated in the most recent round; the metadata index corresponds to the parameter data of the target large language model. The training state of the target large language model is restored based on the recoverable checkpoint file.

[0011] In some implementations of the first aspect, before constructing the initial state of the checkpoint in the CPU memory of each training node, the method further includes: A separate checkpoint daemon is derived for each training node; the checkpoint daemon runs on the CPU of the corresponding training node. During initialization, code is injected into the training framework to embed the necessary functional interfaces for building checkpoints; The initial state of the checkpoint is constructed in the CPU memory of each training node, including: The initial state of the checkpoint is built in the CPU memory of each training node based on the checkpoint daemon and functional interface.

[0012] In some embodiments of the first aspect, the method further includes: For target training nodes whose local gradient data is incomplete due to parallel strategies, compensation communication is dynamically initiated to the target training node during the communication idle window so that the target training node can obtain the gradient data missing from the iteratively updated local checkpoint fragments.

[0013] Based on the same inventive concept, in a second aspect, embodiments of this application also provide a checkpoint update apparatus, comprising: The acquisition module is used to obtain the target global gradient information corresponding to the current round through set communication operations when training the target large language model to be trained. The transfer module is used to asynchronously and in parallel transfer the target global gradient information to the CPU memory of all training nodes during the idle period of training. The update module is used to iteratively update the checkpoint files in the CPU memory of each training node based on the target global gradient information, so as to advance the checkpoint state to the next training round; the checkpoint state is isomorphic to the training state of the GPU in the same training node.

[0014] In some implementations of the second aspect, when the acquisition module obtains the target global gradient information corresponding to the current round through set communication operations, it is specifically used for: The global gradient information obtained from the current round based on the set communication operation is obtained asynchronously using a pre-set hook function; the target global gradient information for the checkpoint file iterative update is determined from the global gradient information.

[0015] In some embodiments of the second aspect, the apparatus further includes: The initialization module is used to construct the initial state of checkpoints in the CPU memory of each training node; the initial state is isomorphic to the training state of the GPU in the same training node; the total number of checkpoint fragments in the CPU memory of all training nodes is determined according to the model structure of the target large language model, the hardware resources of all training nodes, and the preset determination algorithm; the number of checkpoint fragments in the CPU memory of each training node is determined according to the total number of checkpoint fragments; wherein, the checkpoint file is stored in fragments according to the number of checkpoint fragments, forming multiple checkpoint fragments, and at least one checkpoint fragment is stored in the CPU memory of each training node.

[0016] In some implementations of the second aspect, checkpoint fragmentation includes parameter fragmentation data and local metadata corresponding to the target large language model; the parameter fragmentation data and local metadata are stored separately. When the update module iteratively updates the checkpoint files in the CPU memory of each training node based on the target global gradient information, it is specifically used for: The target checkpoint slices are determined based on the target global gradient information and the corresponding local metadata stored in the CPU memory of each training node; the target checkpoint slices store the parameter slice data that needs to be incrementally updated; the target checkpoint slices are iteratively updated based on the target global gradient information.

[0017] In some implementations of the second aspect, each training node stores a corresponding redundant checkpoint copy; the redundant checkpoint copy corresponds to the checkpoint fragment, and the total number of redundant checkpoint copies stored by all training nodes corresponds to at least all checkpoint fragments corresponding to the checkpoint file. The device also includes: The fault recovery module is used to pause the iterative update operation of checkpoint shards for each training node when a fault occurs during training; retrieve all redundant checkpoint copies generated in the most recent round from each training node; the most recent round is the training round closest to the time of the fault; based on the pre-built metadata index, parse and reassemble all redundant checkpoint copies generated in the most recent round to generate a complete recoverable checkpoint file; the metadata index corresponds to the parameter data of the target large language model; and restore the training state of the target large language model based on the recoverable checkpoint file.

[0018] In some embodiments of the second aspect, the initialization module is further configured to: A separate checkpoint daemon is derived for each training node; the checkpoint daemon runs on the CPU of the corresponding training node; during initialization, code is injected into the training framework to embed the functional interfaces required for checkpoint construction. When the initialization module constructs the initial state of the checkpoint in the CPU memory of each training node, it is specifically used for: The initial state of the checkpoint is built in the CPU memory of each training node based on the checkpoint daemon and functional interface.

[0019] In some embodiments of the second aspect, the apparatus further includes: The compensation communication module is used to dynamically initiate compensation communication with the target training node during the communication idle window when the local gradient data is incomplete due to the parallel strategy, so that the target training node can obtain the gradient data missing from the iterative update of the local checkpoint fragment.

[0020] Based on the same inventive concept, in a third aspect, embodiments of this application also provide an electronic device, including: a memory and a processor; The memory stores the instructions that the computer executes; The processor executes computer-executable instructions stored in memory to implement a checkpoint update method as described in any of the first aspects.

[0021] Based on the same inventive concept, in a fourth aspect, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the checkpoint update method as described in any of the first aspects.

[0022] According to the checkpoint update method, apparatus, electronic device, and storage medium provided in this application, when training a target large language model, the target global gradient information corresponding to the current round is obtained through ensemble communication operations. During idle periods of training, the target global gradient information is asynchronously and in parallel migrated to the CPU memory of all training nodes. Simultaneously, the checkpoint files corresponding to each training node's CPU memory are iteratively updated based on the target global gradient information to advance the checkpoint state to the next training round. Therefore, each checkpoint update only requires updating the target global gradient information. Furthermore, since the checkpoint files are stored in CPU memory and the target global gradient information is asynchronously and in parallel migrated to the CPU memory of all training nodes, there is no need to create a snapshot of the GPU training state and build a persistent checkpoint file. Checkpoint updates also do not require pausing model training, thereby improving training efficiency. Attached Figure Description

[0023] Figure 1 This illustration shows a flowchart of a checkpoint update method provided in an embodiment of this application; Figure 2 This illustration shows another flowchart of the checkpoint update method provided in an embodiment of this application; Figure 3This diagram illustrates the training location architecture provided in an embodiment of this application. Figure 4 This diagram illustrates the system architecture provided in an embodiment of this application. Figure 5 This illustration shows a schematic diagram of the training operation process provided in an embodiment of this application; Figure 6 This diagram illustrates the checkpoint update process in the training process provided in an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions of this application, the application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0025] The features and exemplary embodiments of various aspects of this application will now be described in detail. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain this application and are not configured to limit this application. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application.

[0026] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0027] As described in the background section, with the continuous expansion of LLM model size, current mainstream LLM models often contain billions or even hundreds of billions of parameters, requiring massive computational resources for training. To improve training efficiency, researchers have proposed various parallel training methods, including data parallelism, pipeline parallelism, tensor parallelism, and export parallelism. Hybrid parallelism distributes computational tasks across multiple GPUs for collaborative processing, effectively improving training efficiency and resource utilization, and has become a core technology for large-scale training. In hybrid parallel training strategies, tensor parallelism partitions the tensor computation of the model within each node and summarizes it through All-Reduce; data parallelism synchronizes gradients across nodes to maintain model copy consistency; forward All-to-All communication is used to pass activations in tensor parallelism; and pipeline parallelism allows different nodes to handle different stages of the model, thereby achieving a cross-dimensional, efficient, and scalable training architecture.

[0028] During the iterative process, the GPUs in the training system need to work collaboratively and synchronize periodically. However, as the scale of LLM model training increases, the stability and fault tolerance of the system face significant challenges. Long-running training jobs are highly susceptible to interference from hardware failures, network jitter, and other factors, causing training interruptions or even failures. In LLM training, which typically lasts for weeks or even months, failures are unavoidable.

[0029] In large-scale distributed training, multiple computing nodes need to frequently synchronize their states. The failure of any GPU or node can cause the entire training process to stall until the node is isolated, repaired, or replaced. Since training often involves hundreds or even tens of thousands of GPUs, the failure of any GPU can cause the entire training machine to stop, resulting in huge economic losses.

[0030] To mitigate the immediate losses caused by training failures, checkpointing and restoring are crucial for ensuring the stable operation of GPU tasks. The training framework periodically creates snapshots of the GPU training state and builds persistent files after certain training epochs or intervals. When a training failure occurs, the training task can be restarted, the checkpoint file can be read, and training can resume from the most recent checkpoint.

[0031] Checkpoint files typically include important data such as model weights, optimizer parameters, and hyperparameters. However, as the size of LLM models increases, the size of checkpoints also increases. For example, the training checkpoints for an 8B model (8 billion parameters model) typically reach 80GB (gigabytes), which brings certain storage and transmission pressures.

[0032] Furthermore, during the training of LLM models, to ensure the consistency of model states, the construction of checkpoints often requires pausing model training to avoid parameter inconsistencies or state mismatches caused by parameter changes during checkpoint creation. This training pause due to checkpoint creation is called a checkpoint pause. As the checkpoint size increases, checkpoint pauses introduce significant system overhead and reduce training efficiency. Therefore, further optimization is needed.

[0033] Example 1

[0034] The checkpoint update method provided in this application is applied to an electronic device. This electronic device can be a computer, or a device within a computer for implementing the checkpoint update method; this application does not specifically limit this. The following description uses the example of the checkpoint update method being executed by an electronic device.

[0035] like Figure 1 As shown, the checkpoint update method provided in this application embodiment may include steps S101 to S103.

[0036] S101. When training the target large language model to be trained, obtain the target global gradient information corresponding to the current round through set communication operation.

[0037] For example, the target large language model can be a commonly used large language model, but this embodiment does not limit it.

[0038] The target global gradient information is used to reflect the globally consistent update of model parameters. Therefore, it can be used to iteratively update the checkpoint file, which includes data such as parameter gradients and optimizer states. The current epoch refers to the current training epoch. Corresponding global gradient information is generated when each epoch completes training.

[0039] Collective communication is a core mechanism in distributed learning and training. In the method of this embodiment, there are multiple training nodes, and each training node includes one or more GPUs and CPUs.

[0040] S102. During the idle period of training, the target global gradient information is asynchronously and in parallel migrated to the CPU memory of all training nodes.

[0041] For example, idle periods include idle periods of collection communication, periods with low resource consumption, etc.

[0042] By asynchronously and in parallel migrating the target global gradient information to the CPU memory of all training nodes, communication interference with the normal training process can be avoided.

[0043] S103. Iteratively update the checkpoint files in the CPU memory of each training node based on the target global gradient information to advance the checkpoint state to the next training round. The checkpoint state is isomorphic to the training state of the GPU in the same training node.

[0044] For example, checkpoint files can be stored in a sharded manner, that is, each training node stores several checkpoint shards, thus eliminating the need for each training node to store a complete checkpoint file.

[0045] By storing checkpoint files on the CPU, serialization overhead can be reduced.

[0046] According to the checkpoint update method provided in this application, when training the target large language model, the target global gradient information corresponding to the current round is obtained through ensemble communication. During the idle period of training, the target global gradient information is asynchronously and in parallel migrated to the CPU memory of all training nodes. At the same time, the checkpoint file corresponding to each training node's CPU memory is iteratively updated according to the target global gradient information to advance the checkpoint state to the next training round. Thus, each checkpoint update only requires updating the target global gradient information. Furthermore, since the checkpoint file is stored in CPU memory and the target global gradient information is asynchronously and in parallel migrated to the CPU memory of all training nodes, there is no need to create a snapshot of the GPU training state and build a persistent checkpoint file. Checkpoint updates also do not require pausing model training, thereby improving training efficiency.

[0047] Example 2

[0048] like Figure 2 As shown, the checkpoint update method provided in this application embodiment is based on the checkpoint update method provided in embodiment 1 of this application, and further describes the method, which may include steps S201 to S204.

[0049] S201. When training the target large language model, a pre-set hook function is used to asynchronously obtain the global gradient information obtained in the current round based on the set communication operation.

[0050] For example, hook functions can directly obtain global gradient information obtained from the ensemble communication operation in the current round when the training framework calls the gradient synchronization operation, improving the efficiency of information acquisition. At the same time, it will not affect the training framework's own process or model training.

[0051] In some implementations, prior to S201, the following steps are also included: The initial state of the checkpoint is constructed in the CPU memory of each training node. The initial state is isomorphic to the training state of the GPU in the same training node.

[0052] The total number of checkpoint slices in the CPU memory of all training nodes is determined based on the model structure of the target large language model, the hardware resources of all training nodes, and the preset determination algorithm.

[0053] The number of checkpoint fragments in the CPU memory of each training node is determined based on the total number of checkpoint fragments. Specifically, the checkpoint file is fragmented according to the number of checkpoint fragments, forming multiple checkpoint fragments, with at least one checkpoint fragment stored in the CPU memory of each training node.

[0054] For example, the process for determining the total number of checkpoint shards and the number of checkpoint shards per training node can be as follows: An optimization model based on given resource constraints (including the number of CPU cores, training iteration time window, and communication idle periods) is proposed. This model aims to maximize the number of checkpoint shards held by each node, thereby achieving a higher recovery rate and shorter rollback time in the event of system failure.

[0055] Objective function: Maximize the number of remote checkpoint fragments for each training machine.

[0056] In the formula, This represents the number of checkpoint fragments in every single machine.

[0057] The above formula This refers to the number of checkpoint segments in each machine, calculated using the following formula:

[0058] In the formula, the number of checkpoints It satisfies update time constraints and idle communication time constraints. Represents the time of a single iteration during training. This represents the update cost of a single checkpoint slice. This constraint ensures that the update cost of checkpoint slices in the checkpoint engine will not be longer than the training iteration time, thus ensuring that training will not be affected by checkpoint updates. This represents the size of the checkpoint sharding gradient partition. These are transmission initiation delay and transmission bandwidth, respectively, following .Model, The default setting is The parameters, For idle time slices, The constraints are as follows: Update time constraints:

[0059] Space communication time constraints:

[0060] The space-time constraints ensure that when the complete checkpoint gradient slices cannot be obtained using the training flow, the introduced external communication will not affect normal training.

[0061] For example, the initial state isomorphic to the training state of the GPU in the same training node means that the initial state of the checkpoint file in the CPU and the training state of the GPU are completely identical in terms of data values, data structure, hierarchical relationship, field names, etc.

[0062] In some implementations, the following process is included before the process of building the initial state of the checkpoint in the CPU memory of each training node: A separate checkpoint daemon is spawned for each training node. The checkpoint daemon runs on the CPU of the corresponding training node.

[0063] During initialization, code is injected into the training framework to embed the necessary functional interfaces for checkpoint construction.

[0064] The initial state of the checkpoint is constructed in the CPU memory of each training node, including: The initial state of the checkpoint is built in the CPU memory of each training node based on the checkpoint daemon and functional interface.

[0065] For example, the checkpoint daemon is a process independent of the training framework. It can construct the initial state of the checkpoint based on the functional interface, and can also perform operations such as updating the checkpoint. That is, the method of this embodiment can perform checkpoint initialization, updating and other operations based on the checkpoint daemon, avoiding interference with model training.

[0066] In some implementations, the method further includes: For target training nodes whose local gradient data is incomplete due to parallel strategies, compensation communication is dynamically initiated to the target training node during the communication idle window so that the target training node can obtain the gradient data missing from the iteratively updated local checkpoint fragments.

[0067] For example, to address the issue that communication primitives cannot directly obtain complete gradient data under some parallel training strategies, low-overhead, fine-grained compensatory gradient communication is dynamically introduced during periods when communication resources are not saturated (i.e., idle windows). This ensures that each target training node can obtain complete gradients while effectively avoiding interference from additional communication to the normal training process.

[0068] S202. Determine the target global gradient information for checkpoint file iterative updates from the global gradient information.

[0069] For example, the target global gradient information to be extracted from the global gradient information can be pre-configured. This includes pre-configuring which information is needed by the GPU and which is needed by the CPU for checkpoint updates. This allows the target global gradient information used for checkpoint file iterative updates to be determined directly from the global gradient information, improving efficiency.

[0070] S203. During the idle period of training, the target global gradient information is asynchronously and in parallel migrated to the CPU memory of all training nodes.

[0071] S204. Based on the target global gradient information, iteratively update the checkpoint files in the CPU memory of each training node to advance the checkpoint state to the next training round.

[0072] In some implementations, checkpoint sharding includes parameter sharding data and local metadata corresponding to the target large language model. The parameter sharding data and local metadata are stored separately. Additionally, global metadata corresponding to the target large language model can be configured during initialization.

[0073] The process in S204 of iteratively updating the checkpoint files in the CPU memory of each training node based on the target global gradient information can be described as follows: The target checkpoint slices are determined based on the target's global gradient information and the corresponding local metadata stored in the CPU memory of each training node. The target checkpoint slices store the parameter slice data that needs incremental updates.

[0074] The target checkpoint segments are iteratively updated based on the target's global gradient information.

[0075] For example, the parameters that need to be updated iteratively can be determined based on the target's global gradient information. At the same time, the corresponding target checkpoint fragments that need to be updated can be further determined based on local metadata.

[0076] In some implementations, each training node stores a corresponding redundant checkpoint copy. The redundant checkpoint copy corresponds to a checkpoint shard, and the total number of redundant checkpoint copies stored across all training nodes corresponds to at least all checkpoint shards corresponding to the checkpoint file.

[0077] Following S204, it also includes: When a fault occurs during training, the iterative update operation of the checkpoint slices of each training node is paused.

[0078] Retrieve copies of all redundant checkpoints generated in the most recent round from each training node. The most recent round is the training round closest to the time of the failure.

[0079] Based on a pre-built metadata index, a complete, recoverable checkpoint file is generated by parsing and reassembling all redundant checkpoint copies generated in the most recent round. The metadata index corresponds to the parameter data of the target large language model.

[0080] The training state of the target large language model is restored based on the recoverable checkpoint file.

[0081] For example, the total number of redundant checkpoint replicas can be several times the number of all checkpoint shards. The most recent rounds that belong to the same round are prioritized, meaning the training round closest to the time of failure is selected based on the same round.

[0082] Metadata indexes refer to global metadata used to index the parameter data of a target large language model. They can be used to index and restore checkpoint files.

[0083] The complete recoverable checkpoint file is generated by parsing and reassembling all redundant checkpoint copies, which allows the training state of the target large language model to be restored based on the recoverable checkpoint file.

[0084] The checkpoint update method in this embodiment implements checkpoint updates through training traffic in the training set communication. By using redundant CPU computing resources, checkpoints are offloaded from the training process and low-overhead iterative-level checkpoints are implemented, thereby reducing checkpoint overhead and minimizing the impact on the training process.

[0085] To better understand the checkpoint update method provided in the embodiments of this application, an exemplary description is given below in conjunction with a specific application implementation.

[0086] In the training of LLM model clusters, problems such as training node failures or task suspensions are unavoidable. The checkpointing mechanism, as a core mechanism for fault recovery, is crucial to the efficiency of training tasks. When a failure occurs, the training task must revert to the previous checkpoint to restart; therefore, high-frequency checkpointing can effectively reduce training task rollbacks. However, high-frequency checkpointing also amplifies the impact of checkpoint pauses on training efficiency. This embodiment aims to redesign the checkpointing mechanism to achieve an iterative checkpointing mechanism through training flow. Its core features are: fast, low-overhead checkpoint construction capability, and the ability to self-update checkpoints during model training, reducing the number of rollback rounds after a failure; efficient read and recovery performance, significantly shortening the recovery time of training tasks after a failure.

[0087] Figure 3 This embodiment demonstrates two key design features: the checkpoint engine and the job executor (based on Kubernetes). Based on user code and the injection of the training framework, this embodiment implements iterative-level self-updating checkpoints that utilize training traffic, offloading checkpoints from training tasks and reducing their impact on training. The job executor schedules training resources to achieve job deployment and recovery.

[0088] Step S301: Design and build the checkpointing engine. This engine initializes the checkpoint shard structure in CPU memory based on the adopted training framework and parallel training strategy. Simultaneously, it extracts key parameters for checkpoint updates from the gradient update stream generated during training, providing data support for the subsequent zero-flow self-update mechanism. To improve loading and storage efficiency, this checkpointing system adopts a structured design that separates parameter data from metadata, avoiding the performance bottleneck caused by the overall serialization operation in traditional checkpointing schemes.

[0089] Step S302: Design and implement a parallel coordinator, which includes two functional sub-modules. Sub-module 1 is a pipelined copy engine. This engine uses an asynchronous pipelined copy mechanism to efficiently copy the global gradient data obtained through cluster communication to CPU memory during idle periods of training and communication, minimizing bandwidth overhead and performance bottlenecks caused by frequent access to the GPU-PCIE (Peripheral Component Interconnect Express, PCIe interface) channel. Sub-module 2 is a gradient synchronization engine. Addressing the issue that communication primitives in some parallel training strategies cannot directly obtain complete gradient data, this engine dynamically introduces low-overhead, fine-grained compensatory gradient communication during periods when communication resources are not saturated. This ensures that each checkpoint engine can obtain complete gradients while effectively avoiding interference from additional communication to the normal training process.

[0090] Step S303: Design and deploy a CPU optimizer. This module utilizes the CPU's idle computing resources to perform synchronous updates on checkpoint partitions. To improve update efficiency, the optimizer is lightweight, featuring low computational overhead and high parallelism, enabling fast partition updates. This mechanism ensures that the checkpoint partition update process can overlap with the training iteration phase, avoiding synchronous blocking time for training due to checkpoint updates and improving overall training and fault tolerance efficiency.

[0091] Step S304: Design and implement a Kubernetes job deployment executor. This module automatically completes the deployment and initialization of training jobs based on user-provided YAML (YAML Ain't Markup Language) format business description files. Specifically, this includes: building the necessary communication environment for the training task, coordinating resources across nodes, and scheduling and submitting distributed training jobs. Simultaneously, this executor also possesses runtime monitoring and stability maintenance functions during the training process, ensuring high availability and continuous execution of training tasks in a large-scale cluster environment.

[0092] The application scenarios of this embodiment are: distributed LLM training, Ring-Reduce architecture (distributed gradient synchronization architecture), data parallel mode, and DeepSpeed ​​(deep learning optimization library).

[0093] (I) System Architecture Overview

[0094] The system architecture of this embodiment is as follows: Figure 4 As shown, the overall system consists of a fault recovery module and a checkpoint creation module. The runtime checkpoint module is responsible for building and updating checkpoints during training. The fault recovery module ensures that training can recover correctly and quickly from faults.

[0095] The checkpoint creation module is designed to efficiently implement fast checkpointing during training, including the training agent, CPU optimizer, and GPU remote storage. Specifically, each training machine runs a built-in checkpoint engine and a training agent. The checkpoint engine executes the specific checkpointing logic, including creating initial states for local and remote checkpoints in CPU memory and updating them throughout the training process.

[0096] The fault recovery module is designed to control the recovery process, including loading the latest checkpoints and even replacing the training machine to resume training. In this embodiment, the fault recovery module consists of a main agent, a coordinator, and a manager. The main agent periodically receives training status updates from agents and instructs them to continue training or wait for fault recovery. The manager holds backup GPU resources and replaces faulty machines as needed. The coordinator instructs the replaced machine to retrieve checkpoints to reload the training status.

[0097] (ii) Memory checkpoints

[0098] The main objective of this embodiment is to implement low-overhead, iterative-level self-updating checkpoints. We note that checkpoint data is essentially a copy of the training state. Therefore, this embodiment introduces a checkpointing process to utilize redundant CPU computing resources for checkpoint self-updating. Based on the number of GPU processes, a checkpointing daemon is configured, and a reasonable number of initial checkpoint states are constructed in CPU memory. These states are isomorphic to the model training states. Simultaneously, key parameters for checkpoint updates are extracted from the gradient update stream generated during training, providing data support for the subsequent zero-traffic self-updating mechanism. To improve loading and storage efficiency, this checkpointing system adopts a structured design that separates parameter data from metadata, avoiding the performance bottleneck caused by the overall serialization operation in traditional checkpointing schemes. This embodiment implements non-intrusive code injection for current mainstream training frameworks. During the training task startup phase, the system automatically spawns an auxiliary process to perform asynchronous checkpoint construction and update operations. To adapt to this system, code injection is required into a portion of the training framework's code during initialization to embed the functional interfaces required for checkpoint construction. Simultaneously, the system adds a checkpoint management interface to the training framework to support the registration, updating, and persistence of training states.

[0099] (ii) Intertwined Flows

[0100] Checkpoint updates are implemented using redundant CPU computing resources. To ensure efficiency, gradient data obtained during training is used for updates. This requires transferring the gradient data obtained from the All-Reduce process during the reverse engineering phase to the CPU for checkpoint updates. Since gradient copying on GPU-to-CPU and training's All-Reduce process use the same PCIe interface, competition between gradient copying and All-Reduce impacts efficiency. Therefore, this embodiment employs a traffic interleaving method, inserting checkpoint copying into the idle time of All-Reduce to interleave copy traffic and training traffic, avoiding disruption to normal training. To this end, we design a parallel coordinator for the fast checkpointing system. Coordinator submodule 1 is a pipelined copy engine. This engine uses an asynchronous pipelined copy mechanism to efficiently copy the global gradient data obtained through cluster communication to CPU memory during idle phases of training and communication.

[0101] During training task initialization, the checkpoint system parses global gradient parameters, determines the number of gradient slices to be updated by the GPU and CPU, and their memory address information. It then migrates the relevant parameters from the communication buffer to the computation buffer. During training, a collection of communication hooks (i.e., the hook function mentioned above) and gradient copy functions are used. After receiving global gradient data, the GPU automatically switches the context of the copy stream, determining the source address, target address, and offset required for copying each global tensor slice from the GPU to the CPU. Subsequently, an independent asynchronous copy stream initiates non-blocking data transfer operations, while simultaneously reclaiming gradient data that has been copied and is no longer involved in computation, thus freeing up GPU memory resources.

[0102] In real-world training scenarios, All-Reduce is not the preferred communication method for all sets of data. Since Reduce-Scatter cannot obtain the complete global gradient, compensatory gradient communication is needed to additionally transmit a portion of the gradient for CPU updates. To address this, a gradient synchronization engine submodule was designed for the parallel coordinator of the fast checkpointing system. This submodule addresses the issue that communication primitives in some parallel training strategies cannot directly obtain complete gradient data. During periods when communication resources are not saturated, this engine dynamically introduces low-overhead, fine-grained compensatory gradient communication, ensuring that each checkpoint engine can obtain the complete gradient while effectively avoiding interference from additional communication to the normal training process.

[0103] (III) CPU Optimizer

[0104] The fast checkpointing system in this embodiment utilizes redundant CPU computing resources to perform synchronous updates on checkpoint fragments. To improve update efficiency, the optimizer employed is lightweight, possessing low computational overhead and high parallelism, enabling rapid fragment updates. This mechanism ensures that the checkpoint fragment update process can overlap with the training iteration phase in parallel, avoiding synchronous blocking time for training caused by checkpoint updates, and improving overall training and fault tolerance efficiency.

[0105] The improved CPU optimizer streamlines computation steps to avoid updating redundant data. Furthermore, it introduces three levels of parallelism to enhance overall computational performance. First, by using the SIMD (Single Instruction, Multiple Data) instruction set, it leverages the hardware-level parallelism supported by the CPU architecture to accelerate vectorized computation. Second, by combining loop unrolling techniques, it improves instruction-level parallelism and optimizes memory access patterns, thereby increasing memory bandwidth utilization. Finally, by utilizing OpenMP (Open Multi-Processing), it executes parameter update tasks in parallel across multiple CPU cores and threads, significantly improving the update speed of checkpoint slices and overall processing throughput.

[0106] (iv) Operation actuator

[0107] To enable user job deployment and implement a Kubernetes job executor, this module automatically completes the deployment and initialization of training jobs based on user-provided YAML format business description files. Specifically, this includes: building the necessary communication environment for the training task, coordinating resources across nodes, and scheduling and submitting distributed training jobs. Simultaneously, the executor also features runtime monitoring and stability maintenance functions during training, ensuring high availability and continuous execution of training tasks in large-scale cluster environments. A Kubernetes ElasticJob (Custom Resource Definition) CRD (Custom Resource Definition) resource is defined, along with corresponding training job scheduling logic and an API (Application Programming Interface). Based on this, a Kubernetes job deployment executor module is built to handle training tasks submitted by users via YAML files. When a user submits a job, the executor requests the necessary computing resources from the cluster scheduler, creates the Pod (container group) corresponding to the training task, configures the environment variables and dependent components required for training, and finally completes the initialization and startup of the training job.

[0108] Figure 5 The training job process of a rapid checkpoint system based on job executor deployment is demonstrated.

[0109] (1) Users initiate a request to create an elastic training task by writing (job configuration) and submitting a YAML configuration file that conforms to the specifications. This configuration file contains the core parameter settings required for training, such as the maximum number of restarts, the version of the elastic training image used, resource requirements, fault tolerance policies, etc. After parsing the configuration, the elastic job executor automatically completes the deployment, scheduling and subsequent management of the job, and supports dynamic recovery and elastic scaling during the training process.

[0110] (2) The elastic job executor first completes the registration and configuration of the custom resource (CRD) type and requests the computing resources required by the training master engine from the Kubernetes cluster. After resource scheduling is completed, the executor automatically initializes the training management node container, starts and configures the various core modules under the training master engine, including the node manager module, the node master agent module, and the coordinator module. Subsequently, according to the user configuration, the executor continues to submit resource requests for training nodes to Kubernetes, creates and starts the worker nodes required for distributed training, and completes the initialization and deployment process of the entire training job.

[0111] (3) During the initialization phase of the training node, the system will automatically configure the training job container and start the training agent module. Subsequently, the creation and binding of the GPU computing process and checkpoint auxiliary process related to the training task will be completed to ensure that training resources and fault tolerance mechanisms are ready. After the above initialization is completed, the training job will officially start and enter the distributed training execution phase.

[0112] (4) During the training process, the training agent continuously monitors the running status of the local GPU process and the overall health of the job, and sends heartbeat signals to the node master agent at fixed intervals to report the status information of the training node, supporting the system to perceive and manage the running status of each node in the cluster in real time.

[0113] (5) When a training node fails, the node master agent module in the training engine immediately sends a resource redeployment request to the node manager to request the rescheduling and deployment of a replacement training node through the Kubernetes platform. At the same time, the node master agent sends a recovery command to the coordinator, which then sends the required configuration information to the replacement node and guides it to complete checkpoint loading and communication environment reconstruction to ensure that the training job can be smoothly restored and continue to execute.

[0114] In this embodiment, the fast checkpointing system is actually a Python (programming language) library that can be embedded into the distributed machine learning framework (DeepSpeed) in code form. Users need to include the Python library in their training code and make minor modifications during training. This invention is compatible with traditional checkpointing mechanisms, providing a reliable and fast checkpointing system for distributed machine learning.

[0115] The core method flow of this embodiment is as follows: Figure 6 As shown: The initial phase includes the user's training task and training configuration, followed by building CPU checkpoint slices, initializing checkpoints, and initializing checkpoint slices. Then, proceed as follows... Figure 6 The training and checkpoint update process is shown.

[0116] (1) During the task training startup phase, the system automatically spawns an auxiliary process to perform asynchronous construction and update operations of checkpoints. To adapt to this system, code injection is required into part of the training framework code during initialization to embed the functional interfaces required for checkpoint construction. At the same time, the system will add a checkpoint management interface to the training framework to support the registration, updating and persistence of training states.

[0117] (2) The traffic-driven self-updating checkpoint system calls the checkpoint engine during the initialization phase to construct the initial state of the checkpoint in CPU memory. This state is generated based on the selected parallel training strategy and memory optimization scheme, and includes key training information such as initial model weights, optimizer state, and training hyperparameters. Simultaneously, the system creates a metadata index based on the model structure to achieve rapid location and efficient access to parameter shards. The checkpoint engine is responsible for holding each shard object and uniformly managing its subsequent updates, synchronization, and persistence. This step relies on the checkpoint engine to reasonably calculate and determine the number of checkpoint shards that each node needs to maintain based on system resource conditions and scheduling strategies.

[0118] (3) During training, the training process obtains global gradient information through aggregate communication operations (such as All-Reduce or Reduce-Scatter). The checkpoint engine listens for and intercepts this gradient data in the background, and initiates an asynchronous non-blocking copy operation through the gradient copy engine to efficiently migrate the global gradients used for checkpoint shard updates from GPU memory to CPU memory. At the same time, in order to release GPU memory resources and reduce GPU memory pressure, the system will promptly clean up redundant gradient data that has been copied and is no longer involved in subsequent calculations.

[0119] (4) During training, the CPU optimizer in the checkpoint assist process immediately starts the checkpoint slice update operation after receiving the gradient data of the corresponding slice. The optimizer iteratively updates the slice content according to the gradient information of the current round, thereby advancing the checkpoint state to the next training round and providing a more accurate and timely parameter snapshot for subsequent recovery.

[0120] (5) When a failure occurs during training, the checkpoint engine immediately triggers the breakpoint protection mechanism, actively suspends the current checkpoint update operation, and retrieves a copy of the redundant checkpoint generated in the same round from the system. Subsequently, the checkpoint engine parses and reassembles the corresponding Byte data content from each checkpoint shard according to the pre-built metadata index, and finally splices it to generate a complete recoverable checkpoint file for restoring the training state.

[0121] (6) After the training task is completed, shut down the system properly, save the checkpoint, and release the relevant memory to ensure the stability of the system.

[0122] The checkpoint engine defines the number of checkpoint shards each training node should hold and is responsible for building and managing the corresponding shard data on the CPU. To fully utilize CPU computing resources and improve the system's fault tolerance and recovery capabilities, an optimization model based on given resource constraints (including the number of CPU cores, training iteration time window, and communication idle periods) is proposed. This model is shown in the aforementioned preset determination algorithm.

[0123] Interleaved Traffic: Without interfering with the main training communication, the gradient synchronization engine initiates compensatory gradient synchronization communication to actively fill in missing gradient fragments at checkpoints, ensuring the consistency and integrity of checkpoint data. Since each training iteration has a high degree of similarity and the training state is stable, the gradient synchronization engine records the timestamps of the previous 20 training iterations to obtain communication idle time during training. :

[0124] Based on the communication idle time, we utilize The model calculates the amount of gradient data that can be transmitted during the communication idle time as follows:

[0125] In the formula, During free time The amount of data that can be transmitted is the same as described above, with other parameters remaining the same. After obtaining the amount of data that can be transmitted in each time slice, the gradient synchronization engine divides the data to be transmitted in each communication interval and moves the data to the communication buffer. Subsequently, the synchronization engine establishes multiple sub-communication groups and uses set communication primitives to simultaneously start gradient synchronization communication to obtain the required communication data.

[0126] The fast checkpoint system implementation and framework layer of this embodiment implements a task-level fast checkpoint framework. Its feature is that it utilizes the redundancy of CPU computing resources to construct additional CPU checkpoint processes, replacing the creation of checkpoints with the self-updating of checkpoints.

[0127] Simultaneously, checkpoint self-updates are achieved using global gradients obtained through ensemble communication. This embodiment directly leverages training traffic without introducing additional checkpoint traffic, achieving low-overhead memory checkpointing.

[0128] This embodiment fully utilizes the data redundancy characteristics of data parallelism in large model training to construct a streamlined CPU optimizer and achieve efficient self-updating of checkpoint shards.

[0129] This embodiment achieves fast deserialized checkpoint storage and retrieval by separating metadata and checkpoint data, effectively reducing the overhead of checkpoint creation and recovery.

[0130] It is understood that the various method embodiments mentioned above in this application can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0131] Example 3

[0132] The checkpoint update device provided in this application embodiment is located in an electronic device, and the checkpoint update device may include: The acquisition module is used to obtain the target global gradient information corresponding to the current round through set communication operations when training the target large language model.

[0133] The migration module is used to asynchronously and in parallel migrate the target global gradient information to the CPU memory of all training nodes during training idle periods.

[0134] The update module iteratively updates the checkpoint files in the CPU memory of each training node based on the target global gradient information, thus advancing the checkpoint state to the next training round. The checkpoint state is isomorphic to the training state of the GPU in the same training node.

[0135] In some implementations, when the acquisition module obtains the target global gradient information corresponding to the current round through set communication operations, it is specifically used for: A pre-defined hook function asynchronously retrieves the global gradient information obtained from the current round's set communication operation. From this global gradient information, the target global gradient information for iterative updates of the checkpoint file is determined.

[0136] In some implementations, the checkpoint update apparatus further includes: The initialization module is used to construct the initial state of checkpoints in the CPU memory of each training node. The initial state is isomorphic to the training state of the GPU in the same training node. Based on the model structure of the target large language model, the hardware resources of all training nodes, and a pre-defined algorithm, the total number of checkpoint fragments in the CPU memory of all training nodes is determined. The number of checkpoint fragments in the CPU memory of each training node is then determined based on the total number of checkpoint fragments. The checkpoint file is stored in fragments according to the number of checkpoint fragments, forming multiple checkpoint fragments, with at least one checkpoint fragment stored in the CPU memory of each training node.

[0137] In some implementations, checkpoint shards include parameter shard data and local metadata corresponding to the target large language model. The parameter shard data and local metadata are stored separately.

[0138] When the update module iteratively updates the checkpoint files in the CPU memory of each training node based on the target global gradient information, it is specifically used for: The target checkpoint slices are determined based on the target's global gradient information and the corresponding local metadata stored in the CPU memory of each training node. Each target checkpoint slice stores parameter slice data that requires incremental updates. The target checkpoint slices are iteratively updated based on the target's global gradient information.

[0139] In some implementations, each training node stores a corresponding redundant checkpoint copy. The redundant checkpoint copy corresponds to a checkpoint shard, and the total number of redundant checkpoint copies stored across all training nodes corresponds to at least all checkpoint shards corresponding to the checkpoint file.

[0140] The checkpoint update device also includes: The fault recovery module pauses iterative updates of checkpoint shards for each training node when a fault occurs during training. It retrieves all redundant checkpoint copies generated in the most recent epoch (the epoch closest to the time of the fault) from each training node. Based on a pre-built metadata index, it parses and reassembles all redundant checkpoint copies from the most recent epoch to generate a complete recoverable checkpoint file. The metadata index corresponds to the parameter data of the target large language model. The training state of the target large language model is then restored based on the recoverable checkpoint file.

[0141] In some implementations, the initialization module is also used for: A separate checkpoint daemon is spawned for each training node. The checkpoint daemon runs on the CPU of the corresponding training node. During initialization, code is injected into the training framework to embed the functional interfaces required for checkpoint building.

[0142] When the initialization module constructs the initial state of the checkpoint in the CPU memory of each training node, it is specifically used for: The initial state of the checkpoint is built in the CPU memory of each training node based on the checkpoint daemon and functional interface.

[0143] In some implementations, the checkpoint update apparatus further includes: The compensation communication module is used to dynamically initiate compensation communication with the target training node during the communication idle window when the local gradient data is incomplete due to the parallel strategy, so that the target training node can obtain the gradient data missing from the iterative update of the local checkpoint fragment.

[0144] The checkpoint update apparatus provided in this application has the beneficial effects and implementation methods of the checkpoint update methods provided in Embodiments 1 and 2 of this application. For details, please refer to the specific descriptions of the checkpoint update methods in Embodiments 1 and 2 above. This embodiment will not repeat them here.

[0145] Example 4

[0146] This application also provides an electronic device, which is intended to be various forms of devices with data processing capabilities, such as servers, checkpoint update devices, and other suitable devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0147] This electronic device includes a processor and memory. The various components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processor processes instructions that execute within the electronic device.

[0148] The memory is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause at least one processor to perform the checkpoint update method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the checkpoint update method provided in this application.

[0149] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the checkpoint update method in the embodiments of this application. The processor executes various functional applications and data processing of the electronic device by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the checkpoint update method in the above method embodiments.

[0150] The electronic device provided in this application has the beneficial effects and implementation methods of the checkpoint update method provided in Embodiments 1 and 2 of this application. For details, please refer to the specific description of the checkpoint update method in Embodiments 1 and 2 above. This embodiment will not repeat the description here.

[0151] Example 5

[0152] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the checkpoint update method in Embodiment 1 or Embodiment 2 above.

[0153] The computer-readable storage medium provided in this application has the beneficial effects and implementation methods of the checkpoint update method in Embodiments 1 and 2 of this application. For details, please refer to the specific description of the checkpoint update method in Embodiments 1 and 2 above. This embodiment will not repeat the description here.

[0154] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0155] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0156] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this application, and this application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this application, and these modifications and improvements are also considered to be within the scope of protection of this application.

Claims

1. A checkpoint update method, characterized in that, include: When training the target large language model, the target global gradient information corresponding to the current round is obtained through ensemble communication operations. During training idle periods, the target global gradient information is asynchronously and in parallel migrated to the CPU memory of all training nodes; The checkpoint files in the CPU memory of each training node are iteratively updated according to the target global gradient information to advance the checkpoint state to the next training round; the checkpoint state is isomorphic to the training state of the graphics processing unit (GPU) in the same training node.

2. The method according to claim 1, characterized in that, The step of obtaining the target global gradient information corresponding to the current round through set communication operations includes: The global gradient information obtained from the current round based on the set communication operation is obtained asynchronously using a pre-set hook function; The target global gradient information for checkpoint file iterative updates is determined from the global gradient information.

3. The method according to claim 1, characterized in that, Before obtaining the target global gradient information corresponding to the current round through ensemble communication operations when training the target large language model to be trained, the process also includes: An initial state for a checkpoint is constructed in the CPU memory of each training node; the initial state is isomorphic to the training state of the GPU in the same training node. The total number of checkpoint fragments in the CPU memory of all training nodes is determined based on the model structure of the target large language model, the hardware resources of all training nodes, and the preset determination algorithm. The number of checkpoint fragments in the CPU memory of each training node is determined based on the total number of checkpoint fragments; wherein, the checkpoint file is stored in fragments according to the number of checkpoint fragments to form multiple checkpoint fragments, and at least one checkpoint fragment is stored in the CPU memory of each training node.

4. The method according to claim 3, characterized in that, The checkpoint segmentation includes parameter segmentation data and local metadata corresponding to the target large language model; the parameter segmentation data and the local metadata are stored separately. The iterative update of the checkpoint file in the CPU memory of each training node based on the target global gradient information includes: The target checkpoint slices are determined based on the target global gradient information and the corresponding local metadata stored in the CPU memory of each training node; the target checkpoint slices store parameter slice data that needs to be incrementally updated. The target checkpoint slices are iteratively updated based on the target global gradient information.

5. The method according to claim 4, characterized in that, Each training node stores a corresponding redundant checkpoint copy; the redundant checkpoint copy corresponds to the checkpoint shard, and the total number of redundant checkpoint copies stored by all training nodes corresponds to at least all checkpoint shards corresponding to the checkpoint file; The step of iteratively updating the checkpoint files in the CPU memory of each training node based on the target global gradient information to advance the checkpoint state to the next training round further includes: When a fault occurs during training, the iterative update operation of the checkpoint slices of each training node is paused. Retrieve all redundant checkpoint copies generated in the most recent round from each of the training nodes; the most recent round is the training round closest to the time of the failure. Based on a pre-built metadata index, a complete recoverable checkpoint file is generated by parsing and reassembling all redundant checkpoint copies generated in the most recent round; the metadata index corresponds to the parameter data of the target large language model. The training state of the target large language model is restored based on the recoverable checkpoint file.

6. The method according to claim 3, characterized in that, Before constructing the initial state of the checkpoint in the CPU memory of each training node, the method further includes: A separate checkpoint daemon process is derived for each training node; the checkpoint daemon runs on the CPU of the corresponding training node. During initialization, code is injected into the training framework to embed the necessary functional interfaces for building checkpoints; The initial state for constructing checkpoints in the CPU memory of each training node includes: The initial state of the checkpoint is constructed in the CPU memory of each training node based on the checkpoint daemon and functional interface.

7. The method according to claim 3, characterized in that, The method further includes: For target training nodes whose local gradient data is incomplete due to the parallel strategy, compensation communication is dynamically initiated to the target training node during the communication idle window so that the target training node can obtain the gradient data missing from the iteratively updated local checkpoint fragments.

8. A checkpoint updating device, characterized in that, include: The acquisition module is used to obtain the target global gradient information corresponding to the current round through set communication operations when training the target large language model to be trained. The migration module is used to asynchronously and in parallel migrate the target global gradient information to the CPU memory of all training nodes during the idle period of training. The update module is used to iteratively update the checkpoint files in the CPU memory of each training node according to the target global gradient information, so as to advance the checkpoint state to the next training round; the checkpoint state is isomorphic to the training state of the graphics processing unit (GPU) in the same training node.

9. An electronic device, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the checkpoint update method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the checkpoint update method as described in any one of claims 1 to 7.

Images