Model training method, distributed training system, device, medium and product

Abstract

The application discloses a model training method, a distributed training system, equipment, a medium and a product. The model training method is applied to a first node in a distributed training system, the first node comprises a first training framework and a first memory management service, and the method comprises the following steps: in response to obtaining that a second node in the distributed training system fails, performing initialization of the first training framework, and acquiring checkpoint data of a training target model from a host memory of the second node through the first memory management service; and in response to the first training framework being initialized, continuing to train the training target model based on the checkpoint data. The application can shorten the time delay of model recovery training.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, a distributed training system, a device, a medium, and a product. Background Technology

[0002] In distributed model training scenarios, as the model and cluster sizes continue to expand, the efficiency of fault recovery for training tasks has become a key factor affecting the utilization of computing resources. In related technologies, when a training node failure leads to training interruption, a breakpoint resumption mechanism is typically used for recovery: the scheduling platform allocates a new replacement node, which then sequentially completes steps such as framework initialization, communication environment establishment, and resource preparation. Finally, it loads the previously saved model checkpoint data from the backend storage system, and training resumes after all steps are completed. However, in the above breakpoint resumption mechanism, the loading process of checkpoint data depends on the read performance of the backend storage system, resulting in a relatively long overall latency from the occurrence of the fault to training recovery. Summary of the Invention

[0003] This application provides a model training method, a distributed training system, a device, a medium, and a product, which can at least shorten the latency of model retraining.

[0004] In a first aspect, embodiments of this application provide a model training method applied to a first node in a distributed training system, the first node including a first training framework and a first memory management service; including: In response to the discovery of a failure in the second node of the distributed training system, the initialization of the first training framework is performed, and the checkpoint data of the training target model is obtained from the host memory of the second node through the first memory management service. In response to the completion of the initialization of the first training framework, the target model continues to be trained based on the checkpoint data.

[0005] Secondly, embodiments of this application provide a model training method applied to a second node of a distributed training system; the second node includes a second training framework and a second memory management service; including: During the training of the target model using the second training framework, the checkpoint data of the target model is stored in the host memory of the second node through the second memory management service. The checkpoint data of the target model is used for: If the first node detects a failure in the second node, it initializes the first training framework of the first node, retrieves checkpoint data from the host memory of the second node through the first memory management service of the first node, and continues to train the target model based on the checkpoint data after the initialization of the first training framework is completed.

[0006] Thirdly, embodiments of this application provide a distributed training system, which includes a first node and a second node: The first node includes a first training framework and a first memory management service; the second node includes a second training framework and a second memory management service. During the training of the target model using the second training framework, the second node stores the checkpoint data of the target model in the host memory of the second node through the second memory management service. In response to the first node's failure, the first node performs the initialization of the first training framework and retrieves the checkpoint data of the training target model from the host memory of the second node through the first memory management service. The first node responds to the completion of the initialization of the first training framework and continues to train the target model based on the checkpoint data.

[0007] Fourthly, embodiments of this application provide an electronic device, including: at least one processor; at least one memory for storing at least one program; and when at least one of the programs is executed by at least one of the processors, implementing the model training method as described in the first aspect, or the model training method as described in the second aspect.

[0008] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for performing the model training method as described in the first aspect, or the model training method as described in the second aspect.

[0009] Sixthly, embodiments of this application provide a machine program product, including a computer program or computer instructions, the computer program or computer instructions being stored in a computer-readable storage medium, a processor of a communication device reading the computer program or computer instructions from the computer-readable storage medium, and the processor executing the computer program or computer instructions to cause the communication device to perform the model training method as described in the first aspect, or the model training method as described in the second aspect.

[0010] In this embodiment, the first node is equipped with a first training framework and a first memory management service. In response to a failure detected by the second node, the first node performs an initialization process via the first training framework, and simultaneously retrieves checkpoint data of the target model from the host memory of the second node via the first memory management service. These two processes overlap in time, meaning that when the first training framework initialization is complete, the checkpoint data has already been pre-retrieved from the host memory of the second node through the first memory management service. The first node can then resume training of the target model based on the checkpoint data without additional waiting. Compared to the serial process in related technologies that requires sequential framework initialization, resource preparation, and then loading checkpoint data from the backend storage system, this application directly retrieves checkpoint data from the host memory of the second node, avoiding or reducing the constraint of read bandwidth from the backend storage system on data loading latency, thereby shortening the latency for resuming model training. Attached Figure Description

[0011] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0012] Figure 1 This is a schematic flowchart of the model training method provided in the first aspect embodiment of this application; Figure 2 This is a schematic diagram of the structure of the distributed training system provided in the embodiments of this application; Figure 3 This is a schematic flowchart of the model training method provided in the second aspect of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0014] It should be understood that in the description of the embodiments of this application, the use of terms such as "first" and "second" is only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order of the technical features indicated. "At least one" refers to one or more, and "more" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any group of these items, including any group of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0015] Model training, especially for large language models, often employs distributed training methods. Artificial intelligence (AI) frameworks utilize tensor parallelism, pipeline parallelism, and data parallelism, simultaneously training a single model using thousands of graphics processing units (GPUs). During training, interruptions often occur due to node, GPU, or network card failures. In such cases, the training system employs a breakpoint-based resume mechanism. This means that during training, model weights and optimizer states are periodically (or triggered by events) saved to a backend storage system (such as a file storage system); this information is called checkpoint data. When a failure occurs, training resumes through resource rescheduling.

[0016] In related technologies, the Mean Time To Recovery (MTTR) after a failure consists of multiple sequentially executed steps, as follows: When a fault occurs, the training operation will be interrupted.

[0017] At this point, the breakpoint resume training system (which can also be called a training fault tolerance system) needs to first isolate the fault and prevent the business from continuing to be scheduled.

[0018] Then, the system allocates new resources (e.g., a new available node) to the training job. After the job resumes training, the AI ​​framework first performs framework initialization, which starts the training job's process and then initializes the distributed process's communication group. This process is usually time-consuming in large language model pre-training and is related to the size of the job.

[0019] Then, the AI ​​framework will perform GPU memory initialization before model loading, that is, pre-plan and allocate GPU memory resources for the checkpoint. This process is related to the model size.

[0020] The AI ​​framework then loads the checkpoint from the backend storage system into the GPU memory of the new node. Since current AI frameworks run independently of the platform, a common solution is to store the checkpoint via a shared file system on the backend storage, thus enabling checkpoint persistence and allowing training to resume from a standby point. Backend storage is a storage system linked to the computing server nodes via a network.

[0021] Finally, after the AI ​​framework initializes and loads the training dataset, training can resume.

[0022] The core issue here is that during the MTTR (Mean Time To Live) from the occurrence of a failure to the resumption of training, the infrastructure resources used for training are idle and waiting, as GPU infrastructure resources are extremely expensive. Therefore, the industry has been trying to shorten the MTTR as much as possible to improve resource utilization efficiency. One pain point is the long time it takes to load the checkpoint from the backend storage system to the GPU memory, which depends on the read bandwidth capacity of the backend storage system. If the performance of the backend storage is poor (but the storage cost is low), it can take tens of seconds or even minutes. How to shorten the checkpoint loading latency is a technical problem that needs to be solved.

[0023] Based on this, embodiments of this application provide a model training method, a distributed training system, a device, a medium, and a product. A first node is configured with a first training framework and a first memory management service. In response to a failure detected by a second node, the first node performs an initialization process via the first training framework, and simultaneously retrieves checkpoint data of the target model from the host memory of the second node via the first memory management service. These two processes overlap in time. Therefore, when the first training framework initialization is complete, the checkpoint data has already been pre-retrieved from the host memory of the second node through the first memory management service. The first node can resume training of the target model based on the checkpoint data without additional waiting. Compared to the serial process in related technologies that requires sequential framework initialization, resource preparation, and then loading checkpoint data from the backend storage system, this application directly retrieves checkpoint data from the host memory of the second node, avoiding or reducing the constraint of read bandwidth from the backend storage system on data loading latency, thereby shortening the latency for resuming model training.

[0024] Meanwhile, this application employs a separate architecture between the first training framework and the first memory management service, enabling the acquisition of checkpoint data and the initialization of the training framework to be executed in parallel. These two operations overlap in time without blocking each other. When the first training framework completes initialization, the checkpoint data is already acquired and ready, allowing the first node to directly resume training the target model without additional waiting for data loading. This parallel execution mechanism further reduces the overall latency from the occurrence of a fault to training recovery, integrating the originally independent serial data loading process into the model initialization process for parallel completion. This reduces the idle waiting time of GPU resources during fault recovery, significantly improving fault recovery efficiency and computing resource utilization in distributed training scenarios.

[0025] The model training method provided in the first aspect of this application will be described in detail below with reference to the accompanying drawings and through some embodiments and application scenarios.

[0026] See Figure 1 , Figure 1 This is a schematic flowchart of the model training method provided in the first aspect of this application.

[0027] like Figure 1 As shown, the model training method is applied to the first node in the distributed training system, the first node including a first training framework and a first memory management service; the method includes at least the following steps: S101, in response to the discovery that the second node in the distributed training system has failed, the initialization of the first training framework is performed, and the checkpoint data of the training target model is obtained from the host memory of the second node through the first memory management service. S102, in response to the completion of the initialization of the first training framework, the target model continues to be trained based on the checkpoint data.

[0028] The following is a detailed description of S101-S102.

[0029] In S101, such as Figure 2 As shown, the distributed training system is used to support parallel training of models, and its architecture includes a scheduling platform and multiple computing server nodes.

[0030] The scheduling platform manages the various computing server nodes, handling resource allocation, task scheduling, and fault handling. Each computing server node is equipped with a training framework and memory management services.

[0031] The training framework is responsible for executing the model training code.

[0032] The memory management service is responsible for storing and retrieving checkpoint data in the host memory. Memory management services on different nodes can transfer checkpoint data to each other via a network (such as a Remote Direct Memory Access (RDMA) network or a high-speed bus network). The memory management service can also interact with the scheduling platform to obtain the node identifiers of different computing server nodes.

[0033] The training framework uses a memory management service to save checkpoint data from the GPU or Neural Processing Unit (NPU) memory to the host memory of the computing server node, or loads checkpoint data from the host memory of the computing server node to the GPU or NPU memory.

[0034] The distributed training system comprises multiple computing server nodes, including a first node and a second node. The second node is the node that fails, causing training to be interrupted; the first node is the node determined and allocated by the scheduling platform through fault rescheduling after detecting the failure of the second node, and is used to replace the second node to continue executing the training task.

[0035] To distinguish the functions of different nodes at different stages, the training framework and memory management service set up on the first node are referred to as the first training framework and the first memory management service, respectively, and the training framework and memory management service set up on the second node are referred to as the second training framework and the second memory management service, respectively. Here, "first" and "second" are used only to identify different nodes and their corresponding functional modules, and not to limit the execution order or importance.

[0036] The first training framework and the second training framework have the same functional architecture, and their specific implementation can be referred to the aforementioned description of the training framework; the first memory management service and the second memory management service have the same functional architecture, and their specific implementation can be referred to the aforementioned description of the memory management service.

[0037] A second node failure refers to an event in a distributed training system where the second node is unable to continue training the target model due to hardware failure, software anomalies, or network problems. This event triggers the scheduling platform to execute a fault isolation and rescheduling process, identifying the first node as a replacement node to resume training.

[0038] The initialization of the first training framework refers to a series of preparatory tasks performed by the training framework on the first node to resume training. Specifically, it can be implemented using conventional initialization methods in the art, such as starting the training process, initializing the distributed process's communication group, pre-allocating video memory resources, and loading training configuration information. The specific procedures and details of the above operations can be implemented with reference to existing technologies, and will not be elaborated upon in this embodiment.

[0039] The target model refers to the model being trained in the distributed training system. This model employs distributed parallelism strategies such as tensor parallelism, pipelined parallelism, and data parallelism, and is trained collaboratively across multiple nodes. It's important to note that before the failure, the training task of the target model was jointly undertaken by multiple nodes, including the second node, with each node responsible for different parts of the model's computation, collaboratively driving the training iterations. The first node, which was not involved in the target model's training before the failure, was selected by the scheduling platform as a replacement node to resume the target model's training upon the failure.

[0040] Checkpoint data refers to a set of information saved during training for resuming training, including but not limited to the model weight parameters of the target model and the optimizer state. In this embodiment, the checkpoint data is saved to the host memory of the second node by the second training framework through the second memory management service before the second node fails, and is retrieved from the host memory of the second node by the first memory management service when the fault is recovered.

[0041] In this embodiment of the application, the first node responds to the discovery that the second node in the distributed training system has failed, and triggers two parallel operation branches: on the one hand, it starts the initialization process of the first training framework, and on the other hand, it obtains the checkpoint data of the target model from the host memory of the second node through the first memory management service.

[0042] The determination of a failure in the second node can be implemented in, but is not limited to, the following ways: 1. After detecting a failure in the second node, the scheduling platform performs fault isolation and designates the first node as the replacement node, then sends a fault rescheduling instruction to the first node. The fault rescheduling instruction may carry at least one of the following information: the node identifier of the second node, the fault type, the identifier of the target model, and the process identifier of the training process to be resumed. Upon receiving this instruction, the first node confirms that the second node has failed and that training must be resumed by this node.

[0043] 2. The first node can also actively detect and obtain fault information. For example, if the first node detects a heartbeat timeout or communication interruption during distributed training communication with the second node, it can determine that the second node has failed.

[0044] The acquisition of checkpoint data can be achieved in various ways, including but not limited to the following: 1. The first memory management service determines the node storing checkpoint data based on the second node identifier carried in the fault rescheduling instruction. Subsequently, the first memory management service establishes a network connection with the second memory management service on the second node, and requests the checkpoint data of the target model through this connection.

[0045] 2. When obtaining checkpoint data, the first memory management service can also request checkpoint data corresponding to the process identifier (such as rank number) of the training process to be restored from the second memory management service.

[0046] 3. If the first memory management service encounters network anomalies, corrupted data in the second node's memory, or the second node is completely offline, causing the retrieval to fail, it can fall back to retrieving checkpoint data from the backend storage system. During normal training, the second node has already persistently stored the checkpoint data through the second memory management service, serving as the ultimate guarantee of data reliability.

[0047] It should be noted that, in this embodiment, the initialization of the first training framework and the acquisition of checkpoint data are executed in parallel on the first node, overlapping in time without blocking each other. The initialization of the first training framework and the acquisition of checkpoint data by the first memory management service are performed independently. The first training framework can start the initialization process without waiting for the acquisition of checkpoint data to be completed; the first memory management service can also start data acquisition without waiting for the initialization of the training framework to be completed.

[0048] In S102, the completion of the first training framework initialization means that the training framework on the first node has completed all preparation work and is ready to resume training. Specific determination methods may include, but are not limited to, the following: 1. The first training framework maintains an internal status flag during initialization. Once all initialization steps are completed, this flag is set to "completed". The first training framework itself or the first memory management service checks this flag to confirm whether initialization is complete.

[0049] 2. After initialization, the first training framework actively sends an initialization completion notification to the first memory management service or other relevant modules.

[0050] Optionally, in one feasible implementation of this application, after initialization, the first training framework calls the loading interface (such as load_checkpoint) provided by the first memory management service, and passes in the identifier of the training process to be resumed and the checkpoint file path. The first memory management service queries the host memory of the local node according to the file path, finds the corresponding checkpoint data, and copies the checkpoint data from the host memory of the first node itself to the GPU or NPU video memory address specified by the training framework. After the copy is completed, the first training framework directly resumes training based on the checkpoint data in the video memory, including loading model weights, restoring the optimizer state, resetting the data loader iteration position, etc., and then continues to execute training iterations.

[0051] After resuming training based on checkpoint data, the first training framework needs to synchronize its state with other nodes in the distributed training cluster to ensure consistent global training progress. Specific implementation methods may include, but are not limited to: 1. The first training framework exchanges training status information such as the current iteration steps and learning rate with other nodes through the initialized set communication group, and adjusts its own progress to match the global state of the cluster.

[0052] 2. The checkpoint data contains information such as the global iteration step count and epoch count at the time of saving. After loading this metadata, the first training framework sets its own training progress to the corresponding number of steps and notifies other nodes (such as through broadcasting or consensus mechanisms) to synchronize, ensuring that all nodes continue training in the same iteration step.

[0053] In one embodiment, obtaining checkpoint data of the training target model from the host memory of the second node through the first memory management service includes: During the initialization of the first training framework, checkpoint data of the training target model is obtained from the host memory of the second node through the first memory management service.

[0054] Optionally, in this embodiment, while the first training framework initiates the initialization process, the first memory management service simultaneously initiates an operation to acquire checkpoint data in the memory of the second node host. The two operations overlap in time, without blocking or waiting for each other. This execution method differs from the process in related technologies that requires waiting for all initialization to be completed before data loading can begin. This embodiment promotes the data acquisition operation and the initialization operation in parallel, transforming their relationship from a "sequential relationship" to a "parallel relationship."

[0055] The parallel execution capability of this embodiment is mainly supported by the following technical features: 1. The first node adopts a separate functional architecture, with a functionally independent first training framework and first memory management service. This functional decoupling design allows the two to run independently without interfering with each other, providing an architectural foundation for parallel execution.

[0056] 2. In the distributed training system, the memory management services between the computing server nodes can interact with each other through interconnected networks such as RDMA networks and high-speed bus networks. This network channel is independent of the communication environment required for the initialization of the training framework. Without waiting for the first training framework to complete the initialization of the distributed process set communication group, the first memory management service can establish a connection with the memory management service of the second node through this network channel to realize the cross-node transmission of checkpoint data.

[0057] 3. During model training, the second node periodically saves the checkpoint data of the target model to the host memory of the second node through its own second memory management service, and maintains the mapping relationship between the file path, memory address and node identifier of the checkpoint data; the first memory management service can directly interact with the scheduling platform to obtain the node identifier of the second node, and the first memory management service can query and obtain the checkpoint data from the host memory of the second node based on the identifier, without the training framework participating in the data query and transmission.

[0058] In these alternative embodiments, the checkpoint data acquisition operation is advanced to be executed in parallel during the initialization process, so that the data loading delay overlaps with the initialization delay in time. When the first training framework is initialized, the checkpoint data has been pre-acquired and stored in the memory of the first node host through the first memory management service. The first node can directly load the data to resume training without waiting for additional time, thereby shortening the time from the occurrence of the fault to the resumption of training.

[0059] In one embodiment, obtaining checkpoint data of the training target model from the host memory of the second node through the first memory management service includes: The first memory management service retrieves checkpoint data for training the target model from the host memory of the second node based on the node identifier of the second node.

[0060] Optionally, in this embodiment, a node identifier refers to the identity information used to uniquely identify a computing server node in the distributed training system. Each computing server node, when accessing the distributed training system, can be assigned or configured with a unique node identifier by the scheduling platform. This node identifier can be the node's IP address, hostname, or a custom number defined by the scheduling platform. The purpose of the node identifier is to enable other components in the distributed training system (such as the scheduling platform, memory management service, training framework, etc.) to accurately identify, locate, and access specific nodes.

[0061] Optionally, in one feasible implementation of this application, when the second node fails, the scheduling platform performs fault isolation and triggers a rescheduling process, determining that the first node will replace the second node to continue training the target model. The scheduling platform sends a fault rescheduling instruction to the first node, which includes at least the node identifier of the second node, and may also include the identifier of the target model, the process identifier of the training process to be resumed, and other information.

[0062] After receiving the instruction, the first memory management service on the first node parses the instruction and obtains the node identifier of the second node. The first memory management service establishes a dedicated data transmission link with the memory management service of the second node through the RDMA network or high-speed bus network in the distributed training system, and reads the checkpoint data of the target model from the host memory of the second node. During the data transmission, the first memory management service can store the acquired checkpoint data into the host memory of the first node it manages, for use by the first training framework to load checkpoint data later.

[0063] In these alternative embodiments, the location of the second node is achieved through node identifiers, allowing the first memory management service to directly obtain checkpoint data from the second node's host memory, thereby improving data acquisition efficiency.

[0064] In one embodiment, the node identifier is determined based on the training process identifier corresponding to the failure of the second node; the checkpoint data corresponds to the training process identifier.

[0065] Optionally, in this embodiment, the training process identifier refers to the identity information used to uniquely identify a training process in a distributed training system. When training a target model using a distributed parallel strategy, the training task is divided into multiple parallel subtasks, each of which is executed by an independent training process. These training processes are distributed across different computing server nodes and work together to complete the training of the model.

[0066] Each training process is assigned a unique identifier in the distributed training cluster. This identifier can be a globally unique rank number or a number defined by the scheduling platform.

[0067] In a distributed training system, each training process is assigned a globally unique process identifier (such as a rank number) when it is created. Therefore, given a process identifier, it is possible to uniquely determine which node the process originally ran on.

[0068] Therefore, when the second node fails, the scheduling platform sends a fault rescheduling instruction to the first node. This instruction contains the process identifier of the training process that needs to be resumed. Upon receiving the instruction, the first memory management service on the first node can determine the failed second node based on this process identifier.

[0069] Subsequently, the first memory management service establishes a network connection with the second memory management service on the second node and requests the corresponding checkpoint data based on the process identifier. Upon receiving the request, the second memory management service queries the checkpoint data corresponding to the process identifier stored in the host memory of its local node and transmits it to the first node via the network.

[0070] Since the location of each training process in the distributed training cluster is determined, its node can be uniquely identified by the process identifier.

[0071] In these alternative embodiments, the node identifier is associated with the training process identifier of the faulty node, and the checkpoint data is matched with the corresponding training process identifier, thereby achieving accurate location and acquisition of checkpoint data, avoiding invalid data interaction, and improving the accuracy and efficiency of fault recovery.

[0072] In one embodiment, the method further includes: In response to the failure to obtain the checkpoint data from the host memory of the second node, the checkpoint data is obtained from the storage device of the distributed training system.

[0073] Optionally, in this embodiment, the storage device of the distributed training system refers to a shared storage resource that is independent of each computing server node and connected to all nodes via a network, used to persistently store checkpoint data generated during training. During normal training, the training framework on each node saves the checkpoint data to the host memory of its local node through the memory management service, and also asynchronously writes the data to the storage device of the distributed training system as a backup copy.

[0074] Optionally, in one specific implementation of this application, firstly, during the normal training phase, the second training framework on the second node periodically calls the save interface (e.g., `save_checkpoint`) provided by the second memory management service to pass the checkpoint data generated during training, the corresponding file path, and the process identifier (e.g., rank number) of the current training process to the second memory management service. After receiving the data, the second memory management service stores it in the host memory of the second node and establishes and maintains a mapping relationship between the file path, memory storage address, process identifier, and the node identifier. Through this mapping, the checkpoint data of each training process is uniquely associated with its node and process identity. Simultaneously, to ensure data reliability, the second memory management service can asynchronously persist the checkpoint data in memory to the storage device of the distributed training system by calling the `update_iteration` interface after all training processes have completed the checkpoint writing, as a data backup in extreme cases.

[0075] When the second node fails, the scheduling platform performs fault isolation and triggers a rescheduling process, determining that the first node will replace the second node to continue training the target model. The scheduling platform sends a fault rescheduling instruction to the first node, which includes at least the process identifier of the training process to be resumed, and may also include the identifier of the target model, the node identifier of the second node, and other information.

[0076] After receiving the instruction, the first memory management service on the first node parses the instruction and determines the faulty second node based on the training process identifier. Subsequently, the first memory management service establishes a network connection with the second memory management service on the second node based on the training process identifier, and sends a checkpoint data retrieval request to the second memory management service through this connection. This request carries the process identifier of the training process to be recovered and the target model identifier, indicating the required data range: only the checkpoint data corresponding to the process to be recovered is requested, not all data stored on the second node.

[0077] Upon receiving the request, the second memory management service queries the mapping relationship maintained in the host memory of its local node based on the process identifier, locating the checkpoint data uniquely corresponding to that process identifier. Subsequently, the second memory management service reads the data from the host memory and sends it to the first node via the established network connection. After receiving the data, the first memory management service stores it in the host memory of the first node, establishes a mapping relationship between the file path and the local storage address, and records information such as the process identifier corresponding to the checkpoint data for subsequent loading.

[0078] If the retrieval of memory from the second node fails due to network anomalies, corrupted memory data on the second node, or the second node being completely offline, the first memory management service can trigger a fault tolerance mechanism to roll back to retrieving the corresponding checkpoint data from the storage device of the distributed training system, thereby ensuring the reliability of the recovery process.

[0079] In these alternative embodiments, when checkpoint data is not obtained in the host memory of the second node, it is obtained from the distributed training system storage device, ensuring the reliability of checkpoint data acquisition and avoiding training failure due to loss of node memory data.

[0080] In one embodiment, the acquired checkpoint data is stored in the host memory of the first node; The step of continuing to train the target model based on the checkpoint data in response to the completion of the first training framework initialization includes: In response to the completion of the initialization of the first training framework, the first training framework is controlled to load the checkpoint data from the host memory of the first node, and the target model is continued to be trained based on the checkpoint data.

[0081] Optionally, in this embodiment, after the first memory management service obtains checkpoint data from the host memory of the second node, it does not directly transfer it to the first training framework. Instead, it first stores the data in the host memory of the first node to form a local cache. After the first training framework completes the initialization process, it directly loads the stored checkpoint data from the host memory of the first node into the GPU or NPU memory and resumes training on the target model based on the data.

[0082] In this way, checkpoint data is stored in the host memory of the first node immediately after acquisition, so that the data loading operation is completed entirely locally on the node, without the need to access across nodes or read from the backend storage. This avoids the network transmission latency and storage system bandwidth limitations caused by remote access, and further improves data loading efficiency and training recovery speed.

[0083] The congestion notification method provided in the second aspect of this application will be described in detail below with reference to the accompanying drawings and through some embodiments and application scenarios.

[0084] See Figure 3 , Figure 3 This is a schematic flowchart of the model training method provided in the second aspect of this application.

[0085] like Figure 3 As shown, the model training method is applied to the second node of the distributed training system; the second node includes a second training framework and a second memory management service, and the method includes at least the following steps: S301, during the training of the target model using the second training framework, the checkpoint data of the target model is stored in the host memory of the second node through the second memory management service; The checkpoint data of the target model is used for: If the first node detects that the second node has failed, it performs the initialization of the first training framework of the first node, and retrieves the checkpoint data from the host memory of the second node through the first memory management service of the first node. After the initialization of the first training framework is completed, it continues to train the target model based on the checkpoint data.

[0086] The explanations of the relevant terms can be found in the explanations of the foregoing embodiments, and will not be repeated here.

[0087] In this embodiment, the first node is equipped with a first training framework and a first memory management service. In response to a failure detected by the second node, the first node performs an initialization process via the first training framework, and simultaneously retrieves checkpoint data of the target model from the host memory of the second node via the first memory management service. These two processes overlap in time, meaning that when the first training framework initialization is complete, the checkpoint data has already been pre-retrieved from the host memory of the second node through the first memory management service. The first node can then resume training of the target model based on the checkpoint data without additional waiting. Compared to the serial process in related technologies that requires sequential framework initialization, resource preparation, and then loading checkpoint data from the backend storage system, this application directly retrieves checkpoint data from the host memory of the second node, avoiding or reducing the constraint of read bandwidth from the backend storage system on data loading latency, thereby shortening the latency for resuming model training.

[0088] In one embodiment, the target model includes multiple iteration cycles, and one iteration cycle corresponds to multiple training processes; The step of storing the checkpoint data of the target model through the second memory management service includes: In response to the second node completing the training of the target model for n iteration cycles, the checkpoint data corresponding to each training process of the most recent iteration cycle is stored in the storage device of the distributed training system through the second memory management service, where n is a positive integer.

[0089] Optionally, in the embodiments of this application, the iteration period refers to the basic progress unit in the training process of the target model, which is used to measure the training steps required for the model to complete a complete parameter update.

[0090] Optionally, during normal training, the second training framework on the second node periodically saves checkpoint data to the host memory of this node through the second memory management service to meet the need for rapid recovery in the event of a failure.

[0091] However, host memory is volatile storage. If the second node experiences a serious failure (such as node crash, power outage, or complete network interruption), the checkpoint data stored in the host memory may be lost and cannot be retrieved by the replacement node.

[0092] To address such extreme and unexpected situations, this embodiment establishes a persistent backup mechanism: The second training framework, according to a preset storage strategy, after completing at least one iteration cycle of training, calls the persistent interface (such as the `update_iteration` interface) provided by the second memory management service, instructing the second memory management service to uniformly write the checkpoint data corresponding to each training process within the current iteration cycle into the storage device of the distributed training system. The trigger frequency of this storage strategy can be configured according to actual needs; for example, it can be configured to trigger a persistent backup once after each iteration cycle, or it can be configured to trigger a persistent backup once after multiple iteration cycles. This application does not impose specific limitations on this.

[0093] In response to the call, the second memory management service reads all checkpoint data related to the current iteration cycle from the host memory of the local node and writes it to the storage device of the distributed training system to form a persistent backup copy.

[0094] Through the above mechanism, the storage device always stores a complete set of checkpoint data corresponding to the training progress of the most recent persistent backup. When the second node fails and the replacement node (such as the first node) cannot obtain the checkpoint data from the second node's host memory, the fault tolerance mechanism can be triggered to load the corresponding checkpoint data from the storage device, thereby ensuring that the breakpoint resume training process can still be reliably executed in extreme cases.

[0095] It should be noted that the various embodiments described in this application can be combined with each other or implemented individually without conflict, and this application does not limit this.

[0096] This application also provides a distributed training system, which includes a first node and a second node: The first node includes a first training framework and a first memory management service; the second node includes a second training framework and a second memory management service. During the training of the target model using the second training framework, the second node stores the checkpoint data of the target model in the host memory of the second node through the second memory management service. In response to the first node's failure, the first node performs the initialization of the first training framework and retrieves the checkpoint data of the training target model from the host memory of the second node through the first memory management service. The first node responds to the completion of the initialization of the first training framework and continues to train the target model based on the checkpoint data.

[0097] This application also provides an electronic device, such as... Figure 4 As shown, the electronic device 400 includes: One or more processors 410; The memory 420 stores one or more programs that, when executed by one or more processors 410, cause the one or more processors 410 to implement the methods described in the above embodiments.

[0098] Memory 420, as a non-transitory network system, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory 420 may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 420 may optionally include remotely located memories 420 relative to processor 410, which can be connected to processor 410 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0099] The memory 420 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 420 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0100] The processor 410 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0101] In some embodiments, the electronic device further includes: Input / output interfaces are used to implement information input and output; The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus transmits information between various components of the device (e.g., processor 410, memory 420, input / output interface, and communication interface); The processor 410, memory 420, input / output interface, and communication interface can communicate with each other within the device via a bus.

[0102] An embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions for performing the methods described in the above embodiments.

[0103] An embodiment of this application also provides a computer program product, including a computer program or computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform the methods described in the above embodiments.

[0104] The system architecture and application scenarios described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. Those skilled in the art will understand that as system architectures evolve and new application scenarios emerge, the technical solutions provided in this application are also applicable to similar technical problems.

[0105] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0106] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0107] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process or execution thread, and components may be located on a single computer or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed training system, or a network, such as the Internet interacting with other systems via signals).

[0108] The above description, with reference to the accompanying drawings, illustrates some embodiments of this application, but does not limit the scope of this application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of this application shall be within the scope of this application.

Claims

1. A model training method, applied to a first node in a distributed training system, the first node comprising a first training framework and a first memory management service; The method includes: In response to the discovery that a second node in the distributed training system has failed, the system performs the initialization of the first training framework and retrieves the checkpoint data of the training target model from the host memory of the second node through the first memory management service. In response to the completion of the initialization of the first training framework, the target model continues to be trained based on the checkpoint data.

2. The method according to claim 1, characterized in that, The step of obtaining checkpoint data for training the target model from the host memory of the second node through the first memory management service includes: The first memory management service retrieves checkpoint data for training the target model from the host memory of the second node based on the node identifier of the second node.

3. The method according to claim 1, characterized in that, The method further includes: In response to the failure to obtain the checkpoint data from the host memory of the second node, the checkpoint data is obtained from the storage device of the distributed training system.

4. The method according to claim 2, characterized in that, The node identifier is determined based on the training process identifier corresponding to the failure of the second node; the checkpoint data corresponds to the training process identifier.

5. The method according to claim 1, characterized in that, The obtained checkpoint data is stored in the host memory of the first node; The step of continuing to train the target model based on the checkpoint data in response to the completion of the first training framework initialization includes: In response to the completion of the initialization of the first training framework, the first training framework is controlled to load the checkpoint data from the host memory of the first node, and the target model is continued to be trained based on the checkpoint data.

6. The method according to claim 1, characterized in that, The step of obtaining checkpoint data for training the target model from the host memory of the second node through the first memory management service includes: During the initialization of the first training framework, checkpoint data of the training target model is obtained from the host memory of the second node through the first memory management service.

7. A model training method applied to a second node of a distributed training system; the second node includes a second training framework and a second memory management service; The method includes: During the training of the target model using the second training framework, the checkpoint data of the target model is stored in the host memory of the second node through the second memory management service; The checkpoint data of the target model is used for: If the first node detects that the second node has failed, it performs the initialization of the first training framework of the first node, and retrieves the checkpoint data from the host memory of the second node through the first memory management service of the first node. After the initialization of the first training framework is completed, it continues to train the target model based on the checkpoint data.

8. The method according to claim 7, characterized in that, The target model includes multiple iteration cycles, and one iteration cycle corresponds to multiple training processes. The step of storing the checkpoint data of the target model through the second memory management service includes: In response to the second node completing the training of the target model for n iteration cycles, the checkpoint data corresponding to each training process of the most recent iteration cycle is stored in the storage device of the distributed training system through the second memory management service, where n is a positive integer.

9. A distributed training system, the distributed training system comprising a first node and a second node; The first node includes a first training framework and a first memory management service; the second node includes a second training framework and a second memory management service. During the training of the target model using the second training framework, the second node stores the checkpoint data of the target model in the host memory of the second node through the second memory management service. In response to the first node's failure, the first node performs the initialization of the first training framework and retrieves the checkpoint data of the training target model from the host memory of the second node through the first memory management service. The first node responds to the completion of the initialization of the first training framework and continues to train the target model based on the checkpoint data.

10. An electronic device, comprising: One or more processors; A memory that stores one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the following: The model training method according to any one of claims 1-6, or the model training method according to any one of claims 7-8.

11. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement as follows: The model training method according to any one of claims 1-6, or the model training method according to any one of claims 7-8.

12. A computer program product comprising a computer program, which, when executed by a processor, implements, as follows: The model training method according to any one of claims 1-6, or the model training method according to any one of claims 7-8.

Images